Deno Status - Incident history

502 Bad Gateway errors on Deno Deploy

2024-10-28T09:10:00.000+00:00

Oct 28, 09:10:00 GMT+0
Monitoring - We started to see an increase of 502 Bad Gateway errors in the Deno Deploy system at 2024-10-28 09:10 UTC. We already applied a fix to this issue around 2024-10-28 10:00 UTC, now monitoring if the issue is resolved..

Oct 28, 10:28:53 GMT+0
Resolved - This incident has been resolved..

Degraded service on Deploy

2024-06-05T10:59:00.000+00:00

Jun 5, 10:59:00 GMT+0
Monitoring - There was some service degradation on Deploy between 10:59 and 11:23 UTC causing intermittent 502 HTTP errors and longer load times. The issue was caused by some instability on the runtime infrastructure during a production rollout. The root cause is currently under investigation..

Jun 5, 17:51:31 GMT+0
Resolved - The service degradation of Deploy impacted the following two regions: * Southeastern Asia between 10:59 and 11:03 UTC with less than 8% of traffic affected * Eastern South America between 11:05 and 11:21 UTC with less than 17% of traffic affected This occurrence did not impact any of the other global regions. The impact was felt during a service transition period into a new release that was being deployed at the time. This type of impact is not expected and we continue investigating the root cause at this stage. We will be implementing system improvements to ensure that similar situations in the future are prevented..

TLS certificates are not automatically renewed for some custom domains

2024-06-04T08:26:04.290+00:00

Jun 4, 08:26:04 GMT+0
Investigating - We are currently investigating this incident..

Jun 4, 13:56:02 GMT+0
Monitoring - We implemented a fix and all custom domain services are recovering. At peak, 200 domains were temporarily impacted. A full recovery is expected within 10 to 15 minutes..

Jun 4, 14:04:38 GMT+0
Resolved - The service has now been fully restored..

Deno KV Write Operations Unavailable

2024-05-06T05:12:02.086+00:00

May 6, 05:12:02 GMT+0
Investigating - We are seeing degraded performance in Deno KV. We are investigating..

May 6, 05:17:37 GMT+0
Investigating - KV write operations seems to be impacted. Reads are operational. We continue looking into it..

May 6, 05:29:06 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 6, 06:17:51 GMT+0
Resolved - This incident has been resolved..

Deno KV Configuration Update Interruption in Europe

2024-03-25T11:03:00.000+00:00

Mar 25, 11:03:00 GMT+0
Resolved - Our KV cluster that serves requests for Europe regions stopped receiving configuration updates during 2024-03-25 02:03 and 11:28 (UTC). New deployments created during this period were not able to access its attached KV database when accessed from European regions. Existing deployments were not affected, and no data was lost..

dash.deno.com is unavailable

2024-03-13T09:48:30.468+00:00

Mar 13, 09:48:30 GMT+0
Investigating - The dash.deno.com management dashboard is currently unavailable. End user sites hosted on Deno Deploy are not affected. We are investigating..

Mar 13, 10:23:45 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 13, 10:39:05 GMT+0
Resolved - This incident has been resolved..

Degraded service on Deploy

2024-03-12T19:07:57.835+00:00

Mar 12, 19:07:57 GMT+0
Investigating - We're currently seeing a massive spike in traffic that's degrading performance of the Deploy system..

Mar 12, 20:08:02 GMT+0
Resolved - We've monitored and handled the surge of traffic. All systems should be returned to normal..

GitHub Actions Workflow Affected

2024-02-07T18:06:36.149+00:00

Feb 7, 18:06:36 GMT+0
Identified - Deployments made using github actions fail to succeed due to a recent bug. We're currently rolling out a fix and we expect the incident to resolve in 30 minutes..

Feb 7, 19:20:20 GMT+0
Monitoring - We just rolled out a fix and are currently monitoring the result..

Feb 7, 20:22:54 GMT+0
Resolved - We have multiple confirmations that the fix is successful. Resolving incident. .

KV services down in us-east-4

2024-01-25T21:47:00.000+00:00

Jan 25, 22:11:02 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 25, 21:47:00 GMT+0
Identified - We have identified the issue and working on returning KV to service..

Jan 25, 22:08:33 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jan 25, 22:13:10 GMT+0
Resolved - This incident has been resolved..

Degraded performance of Deno KV

2024-01-23T04:41:02.686+00:00

Jan 23, 04:41:02 GMT+0
Investigating - Since 2023-01-23 01:12 UTC, Deno KV write requests started experiencing elevated latencies. We are currently investigating this incident..

Jan 23, 04:45:50 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 23, 05:44:24 GMT+0
Resolved - This incident has been resolved..

Partial outage in us-west2

2024-01-19T06:10:00.000+00:00

Jan 19, 06:10:00 GMT+0
Resolved - We saw increased error rates of `MEMORY_LIMIT` in us-west2 starting around 2024-01-19 06:10 UTC. This issue was addressed at 07:09 UTC..

Deno KV outage

2024-01-05T08:37:00.000+00:00

Jan 5, 08:37:00 GMT+0
Investigating - We are currently investigating this incident..

Jan 5, 09:24:19 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 5, 09:46:36 GMT+0
Resolved - This incident has been resolved..

Intermittent failure when creating deployments

2023-12-21T17:29:02.762+00:00

Dec 21, 17:29:02 GMT+0
Investigating - We're currently experiencing an elevated error rate in our API endpoints that create deployments. This also includes the APIs for modifying environment variables for a project. We're investigating the issue. .

Dec 23, 11:42:36 GMT+0
Resolved - This incident has been resolved..

Dec 22, 16:29:32 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

We're experiencing a DDoS attack

2023-12-09T00:29:43.372+00:00

Dec 9, 00:29:43 GMT+0
Investigating - We're investigating the incident and we'll keep you updated as we make progress..

Dec 9, 01:05:47 GMT+0
Resolved - We've resolved the attack and all systems are returned to normal.

deno.land modules down

2023-12-06T19:26:09.844+00:00

Dec 6, 19:26:09 GMT+0
Investigating - All 3rd party modules are unavailable at this time..

Dec 6, 19:28:32 GMT+0
Resolved - We've rolled back the broken release that caused the issue, back online..

Outage in Deno Deploy

2023-12-01T16:16:21.970+00:00

Dec 1, 16:16:21 GMT+0
Investigating - Investigating. We're seeing increased error rates in US East, South America, and Europe..

Dec 1, 16:25:07 GMT+0
Resolved - The issue was resolved at 16:25 UTC..

Partial outage of Deno KV

2023-11-01T15:29:00.000+00:00

Nov 1, 15:29:00 GMT+0
Investigating - Deployments created before 2023-08-10 are unable to access the KV database. We are currently investigating this incident..

Nov 1, 19:52:13 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Nov 1, 21:07:06 GMT+0
Resolved - This incident has been resolved..

Partial outage of dash.deno.com

2023-09-16T04:21:00.000+00:00

Sep 16, 04:21:00 GMT+0
Investigating - Creation of new deployments is currently broken..

Sep 16, 08:29:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 16, 08:54:16 GMT+0
Resolved - This incident has been resolved..

dash.deno.com down

2023-09-13T18:55:48.758+00:00

Sep 13, 18:55:48 GMT+0
Investigating - We're investigating reports of 502 errors on https://dash.deno.com/projects.

Sep 13, 19:14:14 GMT+0
Monitoring - We believe we were hit with a moderately-sized DOS attack that overloaded a backend database. The attack appears to have cut off, we're monitoring for further issues..

Sep 13, 20:34:32 GMT+0
Resolved - Closing incident as no other active attacks have happened in the last hour or so..

Partial outage of Deno KV

2023-09-09T02:54:00.000+00:00

Sep 9, 02:54:00 GMT+0
Resolved - This incident has been resolved. Due to a protocol change, remote writes to hosted Deno KV databases between 2023-09-07 08:45 and 2023-09-09 02:54 (UTC) were expired immediately. This affects Deno CLI users who used a [new feature](https://deno.com/blog/kv-open-beta#connect-to-kv-from-outside-of-deno-deploy) to connect to hosted KV databases from outside Deno Deploy. Projects hosted on Deno Deploy were not affected. We have continuous backup during the period of the outage. If you experienced data loss after overwriting production data from CLI, please [reach out to us](mailto:[email protected])..

Intermittent API errors

2023-09-06T15:10:00.000+00:00

Sep 6, 15:10:00 GMT+0
Identified - We are experiencing an elevated error rate of 500 Internal Service Error in the Deno Deploy API and dashboard. We have identified the root cause and are working on a fix..

Sep 6, 15:45:39 GMT+0
Resolved - This incident has been resolved..

Degraded performance in some regions

2023-09-05T12:38:03.114+00:00

Sep 5, 13:17:53 GMT+0
Monitoring - We have identified the issue and have implemented a fix. We're monitoring the situation..

Sep 5, 13:52:23 GMT+0
Resolved - This incident has been resolved..

Sep 5, 12:38:03 GMT+0
Investigating - We have received reports of degraded performance in `southamerica-east1`. We are investigating this incident and will provide more information as it becomes available..

Partial outage of deno.com / deno.land

2023-08-22T07:00:00.000+00:00

Aug 22, 07:00:00 GMT+0
Investigating - deno.com and deno.land are currently inaccessible to some users. We are investigating the issue..

Aug 22, 08:30:49 GMT+0
Monitoring - We have implemented a fix, and are currently monitoring the results..

Aug 22, 08:59:50 GMT+0
Resolved - We didn't see any issues for the past 20 minutes since we implemented the fix. We will soon publish an RCA..

Degraded performance of Deno Deploy

2023-08-08T13:40:00.000+00:00

Aug 8, 13:40:00 GMT+0
Investigating - We are currently experiencing degraded performance of the Deno Deploy edge network and management dashboard. We are seeing elevated error rates for some customers. We are investigating the situation..

Aug 8, 14:13:48 GMT+0
Identified - We have identified the root cause, and are working on a fix for the issue..

Aug 8, 17:34:04 GMT+0
Resolved - This incident has been resolved..

Aug 8, 14:30:00 GMT+0
Monitoring - We have implemented a fix for the issue, and the system is operating normally..

Errors for some Deployments

2023-08-01T18:56:06.503+00:00

Aug 1, 20:24:31 GMT+0
Monitoring - Revert rolled out, receiving confirmations that the issue is resolved. We'll continue monitoring to be sure, but recovery looks good..

Aug 1, 18:56:06 GMT+0
Identified - Some Deployments are experiencing "Boot Failure" errors with "Source not found" in the logs. We've identified a recent code change that caused this issue. We're currently working on rolling back the change..

Aug 1, 19:58:13 GMT+0
Identified - Rollback is in-progress now.

Aug 1, 21:31:51 GMT+0
Resolved - Rollback is complete and no further reports of issues have surfaced. Closing incident..