Comments on: Fibers are the Future: Scaling Web Services Past 100K Concurrent Requests (Part 2/2) http://blog.digitalbazaar.com/2008/10/21/scaling-webservices-part-2/ Collaborative Digital Content Distribution Wed, 17 Mar 2010 03:07:06 -0400 http://wordpress.org/?v=2.8.6 hourly 1 By: Jonathan Turner http://blog.digitalbazaar.com/2008/10/21/scaling-webservices-part-2/comment-page-1/#comment-48694 Jonathan Turner Wed, 22 Oct 2008 15:47:12 +0000 http://blog.digitalbazaar.com/2008/10/21/scaling-webservices-part-2/#comment-48694 Another common drawback of fiber-based systems is that you're circumventing the OS and its understanding of your program's parallelism. This means that if you happen to have a multicore or multiple CPUs, a traditional fiber system will not allow you to scale onto those other cpus. This is something a fiber-based system for modern architectures would have to take into consideration, such that there are multiple schedulers instead of a single one. (He says, feverishly trying to finish his own fiber system) Another thing to point out is that there's definitely some pressure to move away from SQL and full-fledged databases towards something that can scale better. The proliferation of cloud datastores is one sign. Another is the idea that a p2p based datastore can actually be quite reliable and allow you to scale easily*. I say "move away", but if it's anything like other technology, they'll pull to one side and meet in the middle somehow. * Chord - http://pdos.csail.mit.edu/chord/ Similar idea, in video: http://skillsmatter.com/podcast/erlang/building-a-transactional-distributed-data-store-with-erlang Another common drawback of fiber-based systems is that you’re circumventing the OS and its understanding of your program’s parallelism. This means that if you happen to have a multicore or multiple CPUs, a traditional fiber system will not allow you to scale onto those other cpus.

This is something a fiber-based system for modern architectures would have to take into consideration, such that there are multiple schedulers instead of a single one. (He says, feverishly trying to finish his own fiber system)

Another thing to point out is that there’s definitely some pressure to move away from SQL and full-fledged databases towards something that can scale better. The proliferation of cloud datastores is one sign. Another is the idea that a p2p based datastore can actually be quite reliable and allow you to scale easily*.

I say “move away”, but if it’s anything like other technology, they’ll pull to one side and meet in the middle somehow.

* Chord – http://pdos.csail.mit.edu/chord/
Similar idea, in video: http://skillsmatter.com/podcast/erlang/building-a-transactional-distributed-data-store-with-erlang

]]>