AWS Outage Root Cause – 1 wrong command took down internet giants!


Update as on 3rd March 2017, 1:31 PM (IST) – AWS outage root cause has been disclosed! The internet giants went in the dark for several hours, because of one wrong command executed by an authorised S3 employee. In an update Amazon said, “At 9:37 a.m. PST, an authorised S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process”. They further added, “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems.” Amazon apologised for the problem caused because of this mistake and stated that they would immediately begin to implement necessary changes to prevent such incident in the future. Summary of the Amazon S3 Service disruption in the Northern Virginia (US-EAST-1) Region can be found here.

You might be thinking that Google owns the internet mostly, but seems like the story is different! On Tuesday, around 9:30 AM Pacific Time, some popular websites like Wix, Quora, Expedia, Netflix etc., and apps have begun to show errors and weren’t responding properly.

This happened because of the outage affecting Cloud King, Amazon Web Services (AWS), as it is currently being used by some of the biggest websites across the globe. A total of 5 hours of outage affected the Amazon’s Simple Storage Service (S3) and took down some of the most popular websites on Tuesday.

Amazon Web Services

Amazon is currently playing as the largest provider of cloud computing platform and holds 40% of the total market alone. This doesn’t only includes clients as private companies or individuals, but also includes U.S. Securities and Exchange Commission and some other massive organisations.

The outage also affected the AWS Service Health Dashboard, where they check the outages and other events. They notified about the same by dropping a tweet.

According to some sources, the information could reach Amazon’s overall network, but they couldn’t establish a connection with the Amazon’s S3 servers.

All those websites and apps which were using Amazon Web Services S3 ‘US-East-1’ servers to host their information, images or any other information were down.

In the beginning, Amazon wasn’t sure that if this happened because of any human error or because of any hardware failure, but as of the now, the issue has been fixed and they have been continuously posting updates on Twitter.

We pulled out the latest message from the AWS Service Health Dashboard: “As of 1:49 p.m. Pacific, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally.”

Some of the well-known websites affected by the AWS outage are Fiverr, Envato, Business Insider, Giphy, Sailthru, Splitwise, Trello, IFTTT etc. Some WordPress businesses also notified about the issues they and their clients were facing because of the S3 outage and notified their clients by tweeting about the same.

Joost de Valk, CEO and founder of Yoast, said, “The outage doesn’t seem to have affected our revenue much, It was slightly annoying and led to some images not working and people not being able to download their plugins for a while, which is always a shame. However, not directly related to this, we’re already looking at ditching S3. That’s because our new hosting setup at SiteGround combined with CDN from MaxCDN actually negates the need for S3 entirely.”

Looks like not just a few but thousands of businesses and websites were affected. However, Amazon handled and fixed the issue with great efforts and to continue as the King of the market, such efforts are always required.

Technologies cannot be 100% efficient and eventually they may cause disruptions at point of time, but the client needs support and technology need updates with time and that’s what makes a company truly worthy of opting for.


Please enter your comment!
Please enter your name here