Prisma Data Platform Status - Incident History

Accelerate partial disruption in us-east-1

Fri, 29 Nov 2024 20:48:47 +0000

Nov 29, 20:48 UTC
Resolved - We're continuing to observe successful provisioning and cleanup of query engine instances in all regions. This incident is resolved.

Nov 29, 19:52 UTC
Monitoring - We've identified an issue in the cleanup flow for inactive query engines that led to resource capacity pressure. We've implemented a fix and confirmed that new query engines are being provisioned successfully.

Nov 29, 19:32 UTC
Investigating - Description: We are investigating an Accelerate disruption in the us-east-1 region affecting the ability to start new query engines.

Affected users: Accelerate users operating in the us-east-1 region

Root cause: Currently unknown

Workarounds: None

Optimize unavailable due to schema migration

Thu, 28 Nov 2024 18:00:00 +0000

Nov 28, 18:00 UTC
Resolved - RESOLVED
A production data migration caused Optimize to remain in maintenance mode for 64 minutes.

Affected Users: All users were unable to access Optimize during the incident.

Root cause: An issue with an upstream provider caused the schema migration process to stall.

Workarounds: None

Accelerate issue provisioning new query engines in eu-central-1

Wed, 02 Oct 2024 16:51:24 +0000

Oct 2, 16:51 UTC
Resolved - This incident has been resolved.

Oct 2, 13:31 UTC
Monitoring - Description: The upstream infrastructure provide issue has resolved and Accelerate is provisioning new query engines successfully. We are continuing to monitor for any further impact.

Affected users: Users provisioning new Accelerate query engines in the eu-central-1 region.

Root cause: Upstream infrastructure provider

Workarounds: None identified

Oct 2, 13:25 UTC
Identified - Description: We identified an upstream infrastructure provider issue affecting the eu-central-1 region. Accelerate experienced issues provisioning resources for new query engine instances.

Affected users: Users provisioning new Accelerate query engines in the eu-central-1 region.

Root cause: Upstream infrastructure provider

Workarounds: None identified

Oct 2, 13:16 UTC
Investigating - Description: We are currently investigating a potential impact on Accelerate identified in our service monitoring.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Accelerate partially unavailable in some regions

Fri, 26 Jul 2024 20:00:00 +0000

Jul 26, 20:00 UTC
Resolved - Description: Between 19:50 and 20:22 UTC some users requesting new query engines took a significantly longer period of time. Majority of users remained unaffected because this incident did not affect existing query engines.

Affected users: Some users in US East, SA East, AP North East

Root cause: Network issue experienced by one of our Cloud providers.

Delay In Usage Data Appearing In Console

Wed, 17 Jul 2024 19:42:47 +0000

Jul 17, 19:42 UTC
Resolved - The remaining data from the affected time range has been recovered and no further impact has been observed.

Jul 17, 17:02 UTC
Monitoring - The incident has been resolved and we are monitoring the issue. We are aware that there is a 57 minute gap in usage data which we're investigating how to recover.

Jul 17, 16:49 UTC
Investigating - Description: We are investigating an incident impacting usage reporting in the Console.
Affected users: All
Root cause: Upstream analytics provider
Workarounds: None identified

Increased 500s from Accelerate

Mon, 08 Jul 2024 15:40:37 +0000

Jul 8, 15:40 UTC
Resolved - We're continuing to observe successful traffic through Accelerate and the upstream incident has been resolved by the hosting provider.

Jul 8, 15:22 UTC
Monitoring - We're observing that Accelerate traffic has recovered. We are continuing to monitor the upstream incident for additional impact.

Jul 8, 15:18 UTC
Identified - Description: We are investigating an increase in 500 responses from Accelerate caused by an incident with one of our hosting providers.

Affected users: Those allocating new query engines due to new Prisma schemas, Prisma versions, or returning from idle activity

Root cause: Upstream hosting provider incident affecting DNS propagation

Workarounds: None

Jul 8, 15:07 UTC
Investigating - We are currently investigating an increase in 500 responses from Accelerate.

Suspected Accelerate Disruption - Investigating

Fri, 14 Jun 2024 06:21:40 +0000

Jun 14, 06:21 UTC
Resolved - Description: Accelerate continues to perform as expected.

Affected users: Because we saw errors in the system that manages Query Engines across regions we treated the incident as all customers affected. After further investigation, ~45% of traffic was actually impacted across the system. The most affected region was EU Central with other regions completely unaffected.

Root cause: We suspect an incident with one of our cloud providers that occurred at the same time which we will validate in our postmortem.

Workarounds: N/A

Jun 14, 05:14 UTC
Monitoring - Description: Query engines are now being started as expected and traffic has returned to normal. We're continuing to investigate the issue and monitor traffic across Accelerate.

Affected users: All

Root cause: Currently unknown

Workarounds: None identified

Jun 14, 04:55 UTC
Investigating - Description: Accelerate is currently experiencing a widespread issue where query engines are not being started. We're are currently investigating the issue.

Affected users: All

Root cause: Currently unknown

Workarounds: None identified

Increased 500s For Accelerate

Thu, 06 Jun 2024 12:59:59 +0000

Jun 6, 12:59 UTC
Resolved - Description: We identified an increase in 500s at 12:30 UTC due a change deployed at the same time. Our monitoring and alerting picked up the issue shortly after this and we immediately rolled back the change.

We are assessing the change and what caused the 500s and will implement the necessary improvements to prevent changes like this from causing any issues in the future.

Affected users: All customers.

Root cause: Deployment

Workarounds: Rollback

Intermittent Request Failures

Wed, 27 Mar 2024 17:55:33 +0000

Mar 27, 17:55 UTC
Resolved - This incident has been resolved.

Mar 27, 10:05 UTC
Update - We are continuing to monitor Accelerate while the underlying incident with our edge provider remains open. Accelerate has had no intermittent failures for the last 16+ hours.

Mar 26, 19:42 UTC
Monitoring - We are continuing to monitor the ongoing incident. The impact on Accelerate remains minor with no intermittent failures for the last 90 minutes.

Mar 26, 15:22 UTC
Identified - Description: We have identified and are actively monitoring intermittent request failures from edge points of presence.

Affected users: All users, intermittently

Root cause: caused by an active incident related to our edge provider which can be monitored here: https://www.cloudflarestatus.com/incidents/b31dxdz60366

Workarounds: None identified

Monitoring Failover Test

Thu, 15 Feb 2024 18:00:56 +0000

Feb 15, 18:00 UTC
Completed - The scheduled maintenance has been completed.

Feb 15, 17:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Feb 15, 16:49 UTC
Scheduled - One of our cloud providers is performing a failover test which we are actively monitoring. The test is expected to last 30 - 60 minutes and we don’t expect any impact to customer operations.

Network Maintenance: IPv6 Network Rollout

Fri, 26 Jan 2024 14:51:12 +0000

Jan 26, 14:51 UTC
Resolved - We are pleased to announce the successful completion of our IPv6 migration!

Throughout this migration process, we have worked diligently to ensure a seamless transition, minimizing any potential impact on your workloads. Our team has closely monitored the migration to ensure stability and performance across all services.

We understand that adapting to new technologies can present challenges. If you are encountering any database-related issues post-migration, these may be due to the broader changes across various service providers as they also transition to IPv6. We are here to support you through these adjustments.

Jan 25, 14:47 UTC
Monitoring - As part of our commitment to staying in step with industry trends, we're implementing IPv6 support across our network. This upgrade aligns with similar shifts by leading cloud and database providers. We anticipate a seamless transition without any impact on customer workloads. However, if you encounter any database-related issues, it may be linked to the broader changes across your various service providers. Should you specifically experience issues with Accelerate, we urge you to contact us for assistance.

Intermittent 500s Caused By Upstream Provider

Fri, 19 Jan 2024 17:57:43 +0000

Jan 19, 17:57 UTC
Resolved - Description: The incident with the upstream provider has been resolved. We are continuing to monitor and anticipate no additional impact.

Affected users: All

Root cause: Upstream provider is experiencing a major incident

Workarounds: None

Jan 19, 16:15 UTC
Monitoring - Description: There are intermittent 500s being caused by an upstream provider. These 500s are within our SLA although we're monitoring the situation closely.

Affected users: All

Root cause: Upstream provider is experiencing a major incident

Workarounds: None

Suspected Accelerate Disruption - Investigating

Fri, 19 Jan 2024 14:39:14 +0000

Jan 19, 14:39 UTC
Resolved - Description: This issue is related to DNS and does not affect any customers.

Affected users: None

Jan 19, 14:34 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Accelerate disruption to new query engine starts

Tue, 09 Jan 2024 20:18:57 +0000

Jan 9, 20:18 UTC
Resolved - Our team has identified the root cause of the issue and applied a fix. No further issues have been identified.

Jan 9, 18:29 UTC
Monitoring - Description: Our team detected an increase in Accelerate 500 responses resulting from failures to allocate new query engines. We were able to associate the issue to a particular deployment and rollback. We're continuing to monitor for any further disruption.

Affected users: users requiring new query engines due to Prisma version updates, Prisma schema changes, or returning from idle activity

Root cause: code deployment affecting request to internal service

Workarounds: None identified

Suspected Accelerate Disruption - Investigating

Mon, 08 Jan 2024 07:35:54 +0000

Jan 8, 07:35 UTC
Resolved - Description: A brief disruption to internal services that did not have customer impact.

Affected users: None

Jan 7, 18:29 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Suspected Accelerate Disruption - Investigating

Sun, 07 Jan 2024 08:15:00 +0000

Jan 7, 08:15 UTC
Resolved - Description: A brief disruption to internal services that did not have customer impact.

Affected users: None

Jan 6, 19:56 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Accelerate ap-northeast-2 region intermittent network issues in internal services

Fri, 05 Jan 2024 20:57:49 +0000

Jan 5, 20:57 UTC
Resolved - Description: The Accelerate ap-northeast-2 region experienced a brief period where requests to replace instances in the region's pool encountered network connectivity issues. This resulted in the workflow succeeding though not acknowledging the event on the queue, causing unnecessary retries and creating an automated incident. No user-facing impact occurred, as the region pool had an excess of instances ready to serve requests.

Affected users: None.

Root cause: Intermittent connectivity between internal services.

Workarounds: Not required.

Jan 5, 20:51 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Accelerate Outage (Investigation)

Thu, 14 Dec 2023 18:12:22 +0000

Dec 14, 18:12 UTC
Resolved - Description: Accelerate raised an automatic incident for a potential warm pool issue due to failed queue messages. This was caused by a brief network disruption causing the requests to the warm pool manager to disconnect, while the operation to replace the instance succeeded. We've confirmed that all regions have a healthy number of warm instances and no impact service impact was identified.

Affected users: none

Root cause: queue messages not acknowledged

Workarounds: not required

Dec 14, 17:07 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Accelerate Outage (Investigation)

Thu, 14 Dec 2023 08:22:11 +0000

Dec 14, 08:22 UTC
Resolved - This incident has been resolved.

Dec 14, 08:02 UTC
Identified - Description: Customers requesting new query engines may experience an increase in start time.

Affected users: US East

Root cause: Currently unknown

Workarounds: None identified

Dec 14, 07:19 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Accelerate Outage (Investigation)

Thu, 30 Nov 2023 18:09:10 +0000

Nov 30, 18:09 UTC
Resolved - Our mitigation was successful and we've observed a full recovery of Accelerate services.

Nov 30, 17:35 UTC
Monitoring - Cloudflare is experiencing an increase in errors in their DNS service, status page below. We have identified a mitigation and are monitoring it.

Nov 30, 17:26 UTC
Update - As we continue to investigate, we recommend that you do not deploy any changes that could affect your database queries or schema.

Nov 30, 17:19 UTC
Investigating - Description: We are investigating the impact of a suspected disruption that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Increased Cold Start Times on Accelerate

Thu, 30 Nov 2023 10:48:17 +0000

Nov 30, 10:48 UTC
Resolved - Description: We are investigating the impact of an internal Queue that backed up. The system is functional. Affected customers will observe increased cold start times of an additional 1 minute. Existing workloads are not affected by this.

Affected users: All customers starting a new Query Engine.

Root cause: One of the internal Accelerate Queues on Cloudflare stopped processing messages.

Workarounds: Do not deploy changes to your Application in order to not trigger a cold start.

Increase In Latency in eu-central-1

Thu, 23 Nov 2023 13:32:57 +0000

Nov 23, 13:32 UTC
Resolved - This incident has been resolved.

Nov 23, 13:11 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Nov 23, 12:08 UTC
Identified - The issue has been identified and a fix is being implemented.

Accelerate Outage (Investigation)

Sat, 18 Nov 2023 05:30:19 +0000

Nov 18, 05:30 UTC
Resolved - Description: Our monitoring detected a reduction in healthy instances in our ap-south-1 pool. One instance failed to be replaced due to a brief network disruption. We have manually recovered the ap-south-1 pool to its desired state. No users were impacted by this incident.

Affected users: None

Root cause: Brief network disruption in ap-south-1

Workarounds: Not required

Nov 18, 05:09 UTC
Investigating - Description: We are investigating the impact of an existing outage that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Partial Accelerate service disruption in eu-central-1

Thu, 16 Nov 2023 21:37:36 +0000

Nov 16, 21:37 UTC
Resolved - This issue has been resolved. All Accelerate regions are reporting healthy statuses.

Nov 16, 21:27 UTC
Monitoring - We've deployed a resolution and are monitoring the results.

Nov 16, 21:20 UTC
Identified - We've identified an issue starting new Query Engines in eu-central-1. A temporary mitigation has been applied and a proper solution is in progress.

Nov 16, 20:42 UTC
Investigating - Description: We are investigating the impact of an existing outage that was automatically detected by our system.

Affected users: Currently unknown

Root cause: Currently unknown

Workarounds: None identified

Cloudflare API Outage Causing Downtime In Some Circumstances

Sat, 04 Nov 2023 04:25:48 +0000

Nov 4, 04:25 UTC
Resolved - All services have been fully restored. More information about the Cloudflare outage itself can be found here: https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/

Nov 3, 17:14 UTC
Update - Accelerate remains mostly functional for customers from what we are able to observe. We are continuing to monitor the situation with Cloudflare.

We continue to recommend against uploading new schemas or deploying changes to your queries.

Nov 3, 09:52 UTC
Monitoring - The majority of Accelerate functionality has returned to service. We are continuing to monitor, however our observability is limited due the ongoing Cloudflare outage. For that reason, we still recommend that you refrain from updating your Prisma schema.

Nov 3, 07:26 UTC
Identified - The issue has been identified and a fix is being implemented.

Some Cloudflare services are still being recovered and creating new Accelerate projects or generating new API keys are still impacted.

Nov 2, 11:54 UTC
Investigating - Description: The Cloudflare API is experiencing an outage causing failures in query engine orchestration.

Affected users: Customers attempting to start a new query engine would be unable to do so.

Root cause: https://www.cloudflarestatus.com/

Workarounds: Investigating

**Recommendation**: Please refrain from updating your Prisma schema. Updates to the Prisma Schema require a start of a new Query Engine which is not working right now. Keep your schema as is and your workloads won't be affected.