Posts Tagged ‘outages’

The Amazon Outage: 3 ways to avoid disaster

by Marc Watley on May 9, 2011

Even if your business doesn’t run on Amazon’s infamous on-demand IT services, you’ve no doubt heard about the recent failure in their Virginia datacenter. As I originally began writing this post – 48 hours after the outage occurred – scores of widely-used social media services like Foursquare and Quora were still down in addition to many other businesses. Exactly. No fun. (My music-tinged brain immediately conjured up images of red-faced, smoking-headed CIOs, syncopated to Adam Freeland’s “We Want Your Soul“, “…No simcard. No disco. No photo. Not here.”) Imagine being responsible for IT at one of these companies during the outage? Yeesh. Hats off to you guys for getting back online so quickly.

Now then, nearly two weeks post ze outage, the question remains on many a CIOs mind: what to do to prevent being affected by future outages? Are Amazon’s and other on-demand services going away? Of course not – their services are simply too valuable for today’s business. Spare-room-based startups and established shops alike use (and will continue to use) these services. The Economist reported in a recent article: “…the global market for cloud services could grow from $41 billion last year to $241 billion by 2020.” That said, options do exist to prevent exploding noggins and grey hairs during an outage. Perhaps not necessarily drag-and-drop simple, but not insurmountable either.  A few suggestions to ponder:

Amazon. Now with fewer calories.

One option to consider would be to migrate your core web services from Amazon EBS (their storage service around which the outage occurred) and diversify to other Amazon services – or to alternative services providers, perhaps keeping some services active at Amazon. Michael Krigsman wrote an excellent article for ZDNet about the outage, offering insight from a CIO perspective and sharing how some Amazon customers escaped calamity by employing diversification strategies.

Move. If you wanna.

You may love the low prices, but I’m sure it wasn’t just me who was reminded of the tried-and-true adage “you get what you pay for” when the outage occurred. (Though in fairness to Amazon, and as has been noted in numerous articles regarding the outage, these types of incidents are actually quite rare.) A rather obvious option would be to consider making a move away from Amazon altogether. This may be something you’ve been thinking about anyhow, and if so, be sure to spend the time and investigate your options. (BTW, if your concerns are specifically around storage strategy and exploring alternatives to Amazon EBS, I’d invite you to chat with my good mates over at P1 Technologies.)

Disaster Recovery

Needless to say, this is the option folks know but don’t really want to hear (and I know you knew I’d be going here). Why? Disaster recovery (DR) is neither a quick or simple initiative, as you likely know. It takes many many hours of planning and asking tough questions – principal among which is, naturally, how long can I afford to be down? The answer to this question – understanding your Maximum Tolerable Downtime or MTD – is an important one: if you’re running, say, a social media, gaming, or music service and using on-demand datacenter services, uptime is more than critical – it’s everything. Even a brief outage would mean disaster…hours of downtime might mean irreversible business failure. No users. No ads. No traffic. Not here. If you haven’t yet gone down the DR road, now would be a fantastic time to begin. And by now I mean right now.

The moral of the story here is that datacenter outages – while very infrequent with trusted players like Amazon, Verizon’s Terremark, Rackspace, and others – do and will occur. The key is to be prepared well in advance so that the effect on your business is minimal to none; have a sound strategy and diversify your core datacenter services. Spend time investigating options. Plan, plan, and plan some more, and be sure to have DR initiatives in place. And as I always say, don’t be afraid to ask for help, especially from professionals who can walk you down the road to smart recovery.

What..still here? Daylight’s a-burnin’ my friends!