War damage to data centers exceeded what AWS services are designed to withstand.
Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data that was hosted in Bahrain and the United Arab Emirates.
Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers, according to an AWS dashboard update posted on September 15. The development was first reported by Reuters and suggests that the Iranian strikes inflicted catastrophic damage on the data centers.
Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.
Speaking as a member of a company who has just recently realized that we can have products in regions other than us-east-1, this is not surprising. If UE1 went down the economy would collapse overnight
I’ve been working on Disaster Recovery for my company and the amount of sleep I’m losing on this is insane in general.
Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.
It’s incredibly hard to reign that behavior in once it’s entrenched as well.
You basically gotta take away privileges from everyone and it becomes a huge fight if you don’t have organizational support. Usually you have to wait for something to explode spectacularly for that to happen.
Fellow SRE here.
The number of times I’ve heard “we have replication” followed by “we’ve tested it…on paper” make me cry.
Trying to convince a CEO that the only way to know for sure that their backups work is to do an actual disaster test is harder than trying to convince my parents that I don’t need to eat more.
That’d give me nightmares too on top of the DR, hopefully at the minimum they can’t spin up resources in anything but a dev/test environment.
Best of luck to you, governance is super important, if it makes you feel any better you’re definitely not alone.