AWS EC2 North Virginia outage resolves but some issues linger – ZDNet

UPDATE: Signal falls over while Xero and Nest got a bit iffy when the main AWS EC2 region had degraded performance. Amazon Web Service says all is well but some users are still reporting trouble.
By and | September 27, 2021 | Topic: Cloud

Amazon Web Services (AWS) didn’t have a relaxing Sunday night before the work week ahead, with its EC2 instances in its main US-EAST-1 region struggling. And, as of Monday morning PDT, some users are still reporting trouble, although the AWS status page now reports, “The issue has been fully resolved and the service is operating normally.”
It all began at 20:11 PDT, when the AWS status page announced the platform was suffering from degraded performance in its main availability zone.”Existing EC2 instances within the affected availability zone that use EBS volumes may also experience impairment due to stuck IO to the attached EBS volume(s),” a notice said 30 minutes later.
“Newly launched EC2 instances within the affected availability zone may fail to launch due to the degraded volume performance.”
At 21:47 PDT, AWS said the fault was within Amazon Elastic Block Store being overloaded, and customers should “fail out” to another availability zone.
“We continue to make progress in determining the root cause of the issue causing degraded performance for some EBS volumes in a single availability zone (USE1-AZ2) in the US-EAST-1 region. We have made several changes to address the increased resource contention within the subsystem responsible for coordinating storage hosts with the EBS service,” the notice at 22:16 PDT said.
“While these changes have led to some improvement, we have not yet seen full recovery for the affected EBS volumes.”
After a further 25 minutes, AWS said its mitigation had worked, was in process of deploying it fully, and EBS volumes should return to normal in the next hour.
In the final report, at 4:21 AM PDT, AWS reported “the issue was caused by increased resource contention within the EBS subsystem responsible for coordinating EBS storage hosts. Engineering worked to identify the root cause and resolve the issue within the affected subsystem. At 11:20 PM PDT, after deploying an update to the affected subsystem, IO performance for the affected EBS volumes began to return to normal levels. By 12:05 AM on September 27th, IO performance for the vast majority of affected EBS volumes in the USE1-AZ2 Availability Zone were operating normally. However, starting at 12:12 AM PDT, we saw recovery slow down for a smaller set of affected EBS volumes as well as seeing degraded performance for a small number of additional volumes in the USE1-AZ2 Availability Zone.”
AWS continued, “Engineering investigated the root cause and put in place mitigations to restore performance for the smaller set of remaining affected EBS volumes. These mitigations slowly improved the performance for the remaining smaller set of affected EBS volumes, with full operations restored by 3:45 AM PDT. While almost all of EBS volumes have fully recovered, we continue to work on recovering a remaining small set of EBS volumes. We will communicate the recovery status of these volumes via the Personal Health Dashboard. While the majority of affected services have fully recovered, we continue to recover some services, including RDS databases and Elasticache clusters. We will also communicate the recovery status of these services via the Personal Health Dashboard.” 
While AWS was experiencing issues, other sites were also hit with performance issues.
“Hold tight, folks! Signal is currently down, due to a hosting outage affecting parts of our service. We’re working on bringing it back up,” the messaging service tweeted.
Nest said its users had trouble logging in, but the situation was resolved.
At the time of writing, Xero said it was suffering from slowness.
To sum up, as Thaddeus E. Grugq, snarkily tweeted, “The internet was designed to survive nuclear wars, not AWS going down.
Update at 10 AM EDT, 27 September: Added further status update.
By and | September 27, 2021 | Topic: Cloud
Hunker down: The chip shortage and higher prices are set to linger for a while
NBN gives ground on lockdown CVC relief in response to telco CEO letter
Optus parent sells 70% stake in tower business for AU$1.9 billion to AustralianSuper
YouTube expands medical misinformation bans to include all anti-vaxxer content
Please review our terms of service to complete your newsletter subscription.
You agree to receive updates, promotions, and alerts from You may unsubscribe at any time. By joining ZDNet, you agree to our Terms of Use and Privacy Policy.
You agree to receive updates, promotions, and alerts from You may unsubscribe at any time. By signing up, you agree to receive the selected newsletter(s) which you may unsubscribe from at any time. You also agree to the Terms of Use and acknowledge the data collection and usage practices outlined in our Privacy Policy.
Cybersecurity budgets for industrial control systems and operational tech increasing: SANS Institute
The Nozomi Networks survey found 15% of respondents report that they have had a cybersecurity incident in their OT environment over the past 12 months. …
Amazon, Google, Microsoft and other tech giants establish Trusted Cloud Principles
IBM, Salesforce/Slack, Atlassian, SAP, and Cisco have also committed to protecting the rights and privacy of their cloud customers.
Zoom, Five9 call off $14.7 billion deal
Zoom and Five9 have mutually agreed to call off the deal after Five9 failed to get sufficient stockholder support for it.
Microsoft to add new education-focused M365 A1 per-device subscription plan
In addition to adding a new Microsoft 365 A1 per-device option, Microsoft is introducing a new Teams Content from Camera feature aimed at the education market. …
Alexa, Ring, and Astro: Where’s my privacy, Amazon?
It’s not just the cameras; it’s Amazon’s desire to process data created by its customers using its products to further monetize it.
Observe Inc and the great adventure of being one of Snowflake’s best users
Huge SQL table joins at scale, the Snowflake forte, are a machine that Observe intends to ride to success.
How game streaming services are luring the next wave of subscribers
Even after seeing rapid growth when players were forced to stay home, streaming services are preparing to attract the next wave of members.
AWS makes Arm-powered Lambda functions generally available
Serverless functions able to use Graviton2 are said to offer better performance at lower cost.
Best IT certifications for all career stages
Learn how these respected and high-ROI information technology certificates and certifications can lead to better jobs in IT, plus certification costs.
© 2021 ZDNET, A RED VENTURES COMPANY. ALL RIGHTS RESERVED. Privacy Policy | Cookie Settings | Advertise | Terms of Use