Blog @ Digital Bazaar

Bitmunk 3.1 Released - Browser-based P2P Commerce

Today marks a significant milestone in the evolution of the Bitmunk peer-to-peer commerce platform. The software release that went live earlier today is the culmination of over 26 months of development, hundreds of thousands of lines of code writes and re-writes and the dream of a small group of us that are trying to fundamentally change the way people buy and sell digital goods on the Internet.

On the surface, Bitmunk looks much like a web-based digital content store specializing in MP3 music sales. People can come to the site and purchase songs and albums for very competitive prices (cheaper than iTunes and Amazon.com).

There is, however, a deeper history and a grander goal for Bitmunk. This blog post outlines why today’s software release is such a significant step towards that goal. We are creating an open, standardized, Internet-scale peer-to-peer commerce infrastructure for the purchase and sale of digital goods. This mechanism, dubbed Collaborate Content Distribution, would allow anything digital to be found, bought and then re-sold via your web browser. This technology shifts the purchase of music, movies, television, books, and any other sort of digital good from being a purely corporation-to-consumer experience to a peer-to-peer experience.

If we’re successful, Bitmunk will help bloggers, artists, writers, tweeple, actors, novelists, and many other people that produce creative and knowledge-industry based content to make a living doing what they do best, without all of the barriers to distribution that have existed to date.

It all started with Bitmunk 1.0… (next page)

Admitting that Javascript was a Mistake

There was an interesting article that was written by Guillaume Marceau recently about visually expressing the usefulness of programming languages. The article uses star-line plots to show how different programming languages compare with one another in speed and expressiveness, as each is used to solve a number of common problems. It’s always nice to check your gut reaction to different programming languages against empirical evidence. Language choice can be as varied as our food preferences, often not based solely on fact. Like our palate, we may find that our preference for our favorite programming languages change over time. As we learn more and use our language of choice to solve real problems, the initial love affair may turn into a nightmare.

At Digital Bazaar, our initial fondness for Javascript is turning into a deep distaste for the way Javascript has evolved in the browsers…

The Looming Cloud Computing Bubble

The number of stories in the online media about Cloud Computing has increased sharply over the last six months. There is a great deal of excitement around this new buzz word, but what is it all about?

The following graph is from Google Trends and shows the average news volume between the term “Cloud Computing” (in blue), and “Web 2.0″ (in red):

As you can see, the Cloud Computing news reference volume overtook Web 2.0 this month, which means that we’re well on our way to another technology bubble. The speed at which this bubble will grow is not known, but one thing is for certain - the media and the IT darlings have latched onto something that they are intent on hyping.

If the IT industry is not careful, we may end up over-promising and under-delivering on the latest tech industry promise of always-on, data-in-the-cloud, always-available computing services. Or worse yet, forgetting about vendor lock-in and forgetting why data portability is such a good thing.

You know those people that can spend an hour saying almost nothing at all? Well, that’s Cloud Computing. It has a good chance of doing just as much harm as good, much like what we went through during the early 90s vendor lock-in and the late 90s dot-com bust.

Read on to find out why Cloud Computing is mostly hot air…

Absorbing Costs Considered Harmful

Bitmunk was founded on a number of principles that we have, unfortunately, not codified on the website yet. One of those principles is the concept that we will always strive to give a detailed break-down of the costs associated with the purchase of any digital good on our network. While some of our customers may not care about where the money goes, others do want to know exactly how much is going to the artist. The fundamental principle at work here is transparency. We believe that transparency regarding how we run our network, manage our costs and reward artists, buyers and sellers is a fundamental operating principle for Digital Bazaar.

Displaying credit card fees have been a part of this transparency. We list all fees that credit card processors charge so that our customers know where their money is going. Typically, this has been about 4.17% per credit card transaction. That is money that goes directly to the credit card agency and we include it as a line item on our website so that our customers know that we aren’t profiting in any way from that charge.

Credit card processors vary widely in the services that they provide as well as their technical sophistication. We have been appalled at how backwards some of the transaction systems are in the banking industry. Credit card processing is no different. Typically, when you use your credit card, some online stores don’t check your address. In other words, they bypass address verification completely because many people enter their addresses incorrectly.

The seedy under-belly of credit card transaction processing…

Fibers are the Future: Scaling Web Services Past 100K Concurrent Requests (Part 2/2)

In a blog post last month, we outlined why a traditional Apache+PHP setup will inevitably fail the growing needs of medium to large AJAX-based websites. The article is continued this month by analyzing different methods of scaling web services past the concurrency barrier inherent in a basic Apache+PHP setup.

While speaking with the technical minds of several companies in our industry, there were several very good questions raised about assumptions we had made when building our system. The feedback from the first blog post about this topic revolved around the following two questions:

• Why would anybody need to scale to 100K (one hundred thousand) concurrent web service requests?
• Isn’t database layout and server bandwidth the primary scalability issue for web services?

We tend to forget that not everybody spends their time thinking about scaling to this level. Scalability requirements that are quite natural to us sometimes come off as exceedingly steep web service requirements to others. The questions above are ones that we have grappled with in the past. Here are the answers that we’ve found over the past several years through vigorous internal debate.

Important questions to ask when scaling past 100K connections…

W3C: RDFa 1.0 is Official

RDFa became an official World Wide Web Consortium Recommendation today. This means that it has undergone an intense amount of design, feedback, development and scrutiny to become a recognized world-wide standard for the expression of web semantics. Manu Sporny, Digital Bazaar’s Founder, has been directly involved with the RDFa Task Force and the standardization work involved with this new Web technology.

We would like to thank Ben Adida (Creative Commons), Chair of the RDFa Task Force, and Mark Birbeck (webBackplane), the primary designer of RDFa, for their vision and tenacity. In addition, we would also like to thank members of the task force - primarily composed of Ralph R. Swick (W3C), Shane McCarron (Applied Testing and Technology), Steven Pemberton (Centrum Wiskunde & Informatica), and Michael Hausenblas (JOANNEUM RESEARCH).

The Web is based on a core dedication to standards. It is individuals, such as those listed above, and their respective organizations that continue to lead the way in standards innovation on the Web. It is a largely thankless job, which is why we would like to extend our deepest appreciation and admiration for an excellent job on RDFa and all that those involved with the W3C continue to do for the World Wide Web and its citizens.

RDFa will have an effect on hundreds of millions of people around the globe. It is a great privilege and honor to be a part of that effort.

POSIX Threads Don’t Scale Past 100K Concurrent Web Service Requests (Part 1/2)

Hard times are upon our financial sector. The US financial markets are in turmoil. Many companies will be cutting spending as a squeeze is placed on operating budgets over the next couple of months, if not years. This is usually good news to the technology sector as most cost cutting measures depend on technology to keep productivity at the same levels as they were before the sky (and stocks) started to fall. These are exciting times as well in the IT sector. We are seeing a shift in the way we compute - from centralized IT to cloud computing, from one core per processor to many cores per processor, from closed data storage to open data portability, from a web of documents to a web of meaning.

At the heart of this transformation is the concept of a Web Service. Web services are used to perform operations on the cloud. They are used to read data from one place on the Web, process and transform that data in another location, and then send the data to yet another location on the Web. It is through this method that we get mash-ups like Google Maps, Facebook apps, Flickr albums and Twitter streams.

These web services are the workhorse of the current Web. They are highly available, highly concurrent, and usually have tens if not hundreds of thousands of people slamming them at a time. This can lead to heartache for software developers. The fine folks at Twitter have had scaling issues over the past two years that required painful changes to their service to avoid continued downtime.

This is a two-part blog post about how traditional software development does not prepare you for the realities of writing scalable web services. Our company focuses a significant portion of our R&D efforts on scalability. One of the lessons that we have learned over the past three years is that pure POSIX threads do not scale for web services.

Check out the graph below and note how the red line (the pure POSIX threads approach) does a very abrupt nose-dive while attempting to reach 400 concurrent web service requests.

You do not want to be in this position, EVER. Most software developers will inevitably choose to use pure POSIX threads for their application servers in order to scale their web services. They do this because most education institutions and websites drill it into our heads that to have concurrency, you must use threads. “I would never make that mistake!”, you exclaim. However, if you use a standard Apache 2 configuration (which uses MPM_prefork) and PHP for your web services (each PHP instance is run in a separate process), you have already made that mistake.

Read on to find out how to scale past 400 concurrent requests…

Thoughts on HTML5, RDFa and Microformats

This article was authored by one of our founders, Manu Sporny. He is an Invited Expert for the RDF in XHTML Task Force at the World Wide Web Consortium and a very active participant in the Microformats community.

We are first and foremost a media services company serving the music, movie, television and electronic book industries. Our mission is to help artists from every walk of life, in every corner of the world, to make a living performing their craft.

Our company does this by creating tools, services and online environments that monetize digital creations for artists by using cutting-edge technology to perform digital distribution. Peer-to-peer networks, swarming distribution, micro-payment-based transaction networks, semantic web technologies, cryptography and steganography are just a few of the tools that we use to deliver on our promise to our artists, publishers and content fans.

Over the past several years, we have been working to help create a standard method of expressing information about music, movies and television on the web with the Microformats and RDFa communities. It has always been our understanding that Microformats are only going to take the web so far, and we will need a more robust, standardized way of expressing semantic information.

Recently, several discussion threads have been started on the WHATWG mailing list (the group of people working on HTML5), that were very discouraging. Not only because they discount the need for such a standardized semantic mark-up technology, but because they are making the decision on factually inaccurate “gut-reactions” about how Microformats work. The exchange expressed a failure to understand the core need of the semantic web by the HTML5 community, mostly because they have never had to create a Microformat through the Microformats Process, nor implement a Microformat parser.

The rest of this article is an attempt to educate the larger web community about the limitations of Microformats, the Microformats Process, and why the web needs both RDFa AND the Microformats community in order to make the semantic web a reality.

We started out where most in the HTML5 community are, we thought Microformats would be the solution to the semantic web…

Blacksburg BarCamp 1.0

The very first BarCamp in Blacksburg is going to be happening on June 14th, 2008. Make sure to tell all your technologist friends and direct them towards the following website:

Blacksburg BarCamp 1.0

Here are the sessions so far for the day, but new ones could be added or these ones could be changed slightly on the day of the event based on camper feedback:

• Introduction to BarCamp (Manu Sporny)
• Multicore Environment Programming (Jonathan Turner)
• Blacksburg Technology Wiki (Cory Donovan)
• Exchange vs. Zimbra/Funambol (Joseph K. Goodman)
• Cloud Computing (Matt Pfeil)
• Ninja Audio Effects (Andy Fabian)
• Ruby on Rails (Josh Eckstein)
• Hacking Parallel Filesystems (Manu Sporny)

The event is starting at 10am this Saturday at Mailtrust:

755 University City Boulevard
Blacksburg, VA 24060

The room that we will be in is on the first floor and is about 1000 feet away from the main Mailtrust offices, towards the Kroger supermarket. If you are facing the Mailtrust logo, turn right and walk down about 1000 feet. There will be signs for Blacksburg BarCamp 1.0.

The event is free to attend and participate.

Dynamic Spectrum Auctions and Digital Marketplaces

Chances are that your cell-phone’s data connection is slow compared to most wired connection speeds. This is especially true in the United States. An emerging field, called Software Defined Radio and Cognitive Radio, is starting to make headway in changing the way we use the airwaves.

Two of the leading universities in the world on Software Defined Radio and Cognitive Radio research are Virginia Tech and Trinity College in Dublin. Digital Bazaar’s founder, Manu Sporny, was asked to join both research organizations along with researchers and officials from Supélec in Rennes, France, the University of Colorado at Boulder, the European Commission, and Vanu to take part in a workshop on the future of SDR/CR research.

Digital Bazaar’s part in all of this was to brainstorm business models and methods of auctioning wireless spectrum in real time. The workshop was hosted by the Centre for Telecommunications and Value-chain Research (CTVR) in Dublin, Ireland. It consisted of an intense 2-day workshop where ideas were exchanged as to the current state of SDR/CR research, what’s going right and what’s going wrong, as well as what the future should hold for this quickly maturing field.

Several ideas were generated at the workshop that dealt with using Digital Bazaar’s technology to buy and then re-sell wireless spectrum in real-time. Effectively, this means that you could buy and sell wireless spectrum from your cellphone or laptop on an as-needed basis.

Here’s how it would work: If you only need 100Kbps in bandwidth on average for sending/receiving e-mail then you can sign up for that wireless plan. If, however, you need 2000Kbps every now and then for the occasional large download to your laptop, you can purchase that wireless bandwidth a la carte from those around you. Not only would you buy the bandwidth, but you would lease the wireless spectrum from those around you as well - effectively increasing your overall download speed. One of the big things that was missing until now was a system that could handle the dynamic auctioning and billing issues. Luckily, Digital Bazaar has spent the last three years building such a system.

Using some of the same technology that Bitmunk employs, it is conceivable that a wireless router or laptop could buy and re-sell spectrum from another system without the need to have a plan setup with a wireless or mobile provider. This means that people could buy and sell bandwidth from each other when they need it and without having a permanent Internet service plan with any provider. Cheap, pay as you go wireless Internet.

We’re very excited about the opportunity to work with these leading research institutions in making this technological dream a reality.