Amazon Generators Failed During Electric Storms

(Ping! Zine Web Tech Magazine) – This past weekend, many noticed an interruption in some of their favorite services including Netflix, Instagram and Pinterest. That outage was the result of system downtime occurring at Amazon’s North Virginia data center, a facility responsible for key components of the Amazon Web Services cloud platform.

During the crash, the Amazon Web Services health dashboard showed problems with services including Elastic Compute Cloud, CloudWatch, Elastic Beanstalk and more. A brief update from the company cited “electrical storms in the area.”

Amazon was eventually able to restore the data center’s operation. And on Monday, the company elaborated on exactly how the disaster played out.

The storm affected Amazon’s US East-1 Region and when the company went to switch to backup generators, things didn’t necessarily go as planned.

“At 7:24pm PDT, a large voltage spike was experienced by the electrical switching equipment in two of the US East-1 datacenters supporting a single Availability Zone. All utility electrical switches in both datacenters initiated transfer to generator power. In one of the datacenters, the transfer completed without incident. In the other, the generators started successfully, but each generator independently failed to provide stable voltage as they were brought into service,” explained Amazon in a posted message.

The data center was then able to switch to utility power, but the trouble didn’t end there when it failed to stay on. Despite the generator failure, Amazon emphasized that it had thoroughly tested the generators in the past. Amazon’s North Virginia data center has faced problems before.  Last month, a cable fault at the facility resulted in another glitch.