queuefull blog

My colleague Joe Ammond pointed out a great post on the RightScale blog discussing the Animoto application’s ability to scale dynamically using Amazon cloud services. I’m impressed by what they’ve accomplished, and the article is worth reading. Especially interesting is their use of queues to control how many server instances are required.

What Animoto CTO Stevie Clifton did really well is to connect all the operations using queues, many of them in SQS. One queue contains work items that list photo URLs to fetch from other sites, such as Facebook, Flickr, etc., and that is processed by one array of worker instances. Another queue has the list of render jobs and each work item in there points to the set of photos sitting at the ready in S3 and at the music files also on S3. All of these queues are held in Amazon SQS and the arrays of worker instances are managed by RightScale. This allows the monitoring part of our service to detect when the queue gets too large and more instances need to be launched.

Queues and queue management mechanisms are critical elements of any application that wants to scale using cloud computing services. This is true whether we’re talking about scaling-out with many instances on a platform like Amazon’s EC2 or scaling-up to a larger VM. You have to know what your needs are so that you can ask your cloud provider for it. It will be interesting to see how new applications (especially in the SaaS space) tackle this scaling problem.

As a side-note, Joe and I quickly ran the numbers in our heads and figured that the cost of EC2 for a high-scale app like Animoto is pretty steep compared to other options. The EC2 platform enables a startup to develop and launch new applications quickly with a low entry cost. But it may not be the right platform to sustain a successful business over time. It’s important to figure out how your app can leverage dynamic environments like EC2 for short-term needs (i.e. bursts) while maintaining an established foundation for the day-to-day work.