2026-03-09T05:25:43+00:00 Google Cloud https://status.cloud.google.com/ tag:status.cloud.google.com,2026:feed:41E5S3mkTGDfkZuJZH5k.6RrDVD72bLgPwjPGG3ky 2026-03-09T05:25:43+00:00

<p> Incident began at <strong>2026-02-27 04:37</strong> and ended at <strong>2026-02-27 06:35</strong> <span>(all times are <strong>US/Pacific</strong>).</span></p><div class="cBIRi14aVDP__status-update-text"><h1>Incident Report</h1> <h2>Summary</h2> <p>On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes.</p> <p>This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability.</p> <h2>Root Cause</h2> <p>This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions.</p> <h2>Remediation and Prevention</h2> <p>Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation.</p> <p>Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. <br> <br> Google is committed to preventing a repeat of this issue and is taking the following actions:</p> <ul> <li>Reinforcing rollout processes to include mandatory validation checkpoints.</li> <li>Improving alerting systems to monitor critical dependencies more closely.</li> </ul> <h2>## Detailed Description of Impact</h2> <p>On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following:</p> <ul> <li><strong>Affected Models:</strong> All Vertex AI Gemini API models were affected, including gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-pro, gemini-3.0-flash-preview, gemini-3.0-pro-preview, gemini-2.0-flash, gemini-2.0-flash-lite.</li> <li><strong>Error Experience:</strong> <ul> <li><strong>PayGo Customers:</strong> Experienced primarily 429 Resource Exhausted errors.</li> <li><strong>Provisioned Throughput (PT) Customers:</strong> Received 503 Service Unavailable errors.</li> <li>For PT customers, most errors stopped at <strong>06:00</strong>. For PayGo customers, most errors stopped at <strong>06:20</strong>.</li> </ul> </li> <li><strong>Geographic Scope:</strong> Global endpoint, us-central1, us-east4, and other US regions were impacted.</li> </ul> </div><hr><p>Affected products: Agent Assist, Dialogflow CX, Google Cloud Support, Vertex Gemini API</p><p>Affected locations: Global, Montréal (northamerica-northeast1), São Paulo (southamerica-east1), Iowa (us-central1), South Carolina (us-east1), Northern Virginia (us-east4), Columbus (us-east5), Oregon (us-west1)</p>