Incident Report

Summary

On 8 August 2024, Gmail and Google Drive experienced service degradation globally for a duration of 4 hours and 10 minutes between 12:06 and 16:16 US/Pacific. During the incident, affected users experienced issues with email attachment and delivery functionalities in Gmail, and upload operations in Google Drive.

To our Gmail and Google Drive customers, we apologize for the impact this service disruption had on your organization. We have completed an internal investigation and are taking immediate steps to improve the quality and reliability of our services.

Root Cause

As part of a standard data operation, we restored a large volume of data into an internal Bigtable database. This operation resulted in Bigtable servers being overloaded.

During the restore operation, Bigtable identified certain tablets as ‘unloadable’ and added the corresponding entries into a [1]Chubby file, as expected. As the file size grew, it exceeded the memory limit allocated for Chubby, resulting in Bigtable being unable to write to the file.

The cumulative impact of these events led to a subset of Bigtable instances entering a degraded state, impacting read/write operations. These instances store internal data for Gmail and Google Drive, which consequently affected the performance of those services.

[1] - https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems

Remediation and Prevention

Google engineers were alerted to the issue on Thursday, 8 Aug 2024 at 11:58 US/Pacific via our monitoring system. Once the nature and scope of the issue were understood, our engineers devised and executed a multi-pronged approach to address the root cause and mitigate impact. The Chubby quota was increased to address the memory issue, while traffic from the affected instances was re-routed to avoid further impact. To ensure that any degraded instances were addressed, Bigtable master servers for the affected instances were successfully restarted. Impact was fully mitigated by 16:16 US/Pacific.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

We are committed to preventing a repeat of this issue in the future and are completing the following actions:

• We have completed a deep analysis of our Bigtable instances and ensured that the Chubby space quota limits and configurations are optimal across all instances.
• We are working on enhancing our monitoring for Chubby quota usage in Bigtable to enable early detection and prevention of potential issues.
• We will establish clear guidelines on the recommended volume of data that can be safely restored at any given time, minimizing the risk of service disruptions.
• We have enhanced the logic for diverting traffic from Bigtable clusters to ensure smoother transitions and minimal impact on users.

Detailed Description of Impact

On Thursday, 8 August 2024 from 12:06 to 16:16 US/Pacific, Gmail and Google Drive experienced service degradation for a duration of 4 hours and 10 minutes.

Gmail

Affected users experienced issues with attachment functionality wherein they were unable to send emails or save drafts containing attachments. There was no impact to emails sent without attachments.

Google Drive

Affected users may have observed degraded performance while performing upload operations.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 8 Aug, 2024 12:06

Incident End: 8 Aug, 2024 16:16

Duration: 4 hours, 10 minutes

Affected Services and Features: Gmail and Google Drive

Regions/Zones: Global

Description: Gmail and Google Drive saw service degradation for a duration of 4 hours 10 minutes. From preliminary analysis, the root cause was a latent misconfiguration of our lock-service infrastructure that impacted a critical storage layer during a high-load scenario. Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

• Gmail users experienced issues with attachment functionality. Users may be unable to send emails or save drafts containing attachments. Sending emails without attachments remains functional.
• Google Drive users may have observed degraded experience while performing upload operations.

Summary: Google Drive experienced service degradation

Description: The recent fix implemented by our engineering teams has been successful in fully resolving the issue. We will be publishing a Preliminary Incident Report(IR) shortly which will provide additional details on root cause.

Summary: Google Drive is experiencing service degradation

Description: We continue to see a reduction in observed error rates based on the most recent fix our engineering teams have made. We are continuing to monitor the overall performance and stability of the Workspace impacted products and will provide more information by Thursday, 2024-08-08 18:00 US/Pacific. We appreciate your patience as we continue our efforts to restore full service.

Customer Symptoms: Google Drive users may see degraded experience while performing upload operations.

Workaround: None at this time.

Summary: Google Drive is experiencing service degradation

Description: Our engineers are actively implementing several fixes, and we’re seeing a decrease in error rates. We understand the impact this has had, and we want to assure you that we’re committed to resolving fully, and quickly. We appreciate your patience as we continue our efforts to restore full service.

We do not have an ETA for full mitigation at this point.

We will provide more information by Thursday, 2024-08-08 16:30 US/Pacific.

Customer Symptoms: Google Drive users may see degraded experience while performing upload operations.

Workaround: None at this time.

Summary: Google Drive experiencing service degradation

Description: We are experiencing an issue with Google Drive beginning on Thursday, 2024-08-08 12:15 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2024-08-08 15:00 US/Pacific with current details.

Customer Symptoms: Google Drive users may see degraded experience while performing upload operation.

Workaround: None at this time.