Kontent.ai Status - Incident History

Infrastructure Upgrade

Mon, 14 Oct 2024 14:00:02 +0000

Oct 14, 14:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 14, 06:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 30, 07:05 UTC
Scheduled - We are excited to announce an upcoming infrastructure upgrade designed to further enhance our platform's scalability and performance.

No service outage is planned for this time period. All systems will remain fully operational, though you may experience occasional slowdowns in publishing or API responses.

We appreciate your understanding as we work to provide you with an even better experience. If you have any questions or concerns, please don't hesitate to reach out to our support team.

Issue Accessing Items in Kontent.ai

Tue, 16 Jul 2024 15:10:56 +0000

Jul 16, 15:10 UTC
Resolved - At 3:17 p.m. (UTC) on 7/15/2024, we released a new version of our application that unfortunately contained a misconfiguration. This misconfiguration caused assets and items with assets to be unviewable. As of 5:09 p.m. (UTC) on 7/15/2024, our hotfix to address this issue has been fully deployed.

We apologize for any inconvenience this may have caused and appreciate your patience and understanding as we worked to resolve the matter.

Jul 15, 17:16 UTC
Monitoring - The hotfix has been deployed. We are monitoring the situation for any further disruptions. If you run into anything please let us know at [email protected], thank you!

Jul 15, 16:06 UTC
Identified - Our developers, have identified the problem and are working on a hotfix.

Jul 15, 15:36 UTC
Investigating - After our recent release we are getting reports of users not being able to access Items and being redirected to their content inventory. Our developers are looking into this.

Partial outage of Delivery and GraphQL APIs

Mon, 15 Jul 2024 09:00:00 +0000

Jul 15, 09:00 UTC
Resolved - Due to major infrastructure changes, you might experience an outage of the GraphQL and Delivery APIs between 9:05 AM UTC and 9:30 AM UTC. All the changes were applied, so it shouldn't cause any more issues.

Brief period of UI unresponsiveness

Thu, 27 Jun 2024 18:37:47 +0000

Jun 27, 18:37 UTC
Resolved - At 14:17 UTC we saw a brief spike of high database server latency which caused some requests sent from projects in our US data center to timeout. This resulted in some users experiencing an unresponsive user interface for approximately 15 minutes before latency returned to normal. Since then we have been monitoring our metrics, and don't see any indicators of a prolonged issue or a clear cause of the original spike. At this point we believe it may have been caused by some glitch in one of our service providers, and we are confident that it is resolved and all services are operating without issue.

Jun 27, 16:06 UTC
Update - Since 2:30 pm UTC our service has been fully operational, though we are continuing to monitor the situation.

Jun 27, 14:55 UTC
Monitoring - There was a brief period of failing requests to our service leading to unresponsiveness of the Kontent.ai user interface. This issue affected projects in the US data center between 2:20 and 2:30 pm UTC. We are currently monitoring the situation and working to ensure it does not happen again.

Kontent.ai UI is not responding

Thu, 09 May 2024 13:35:58 +0000

May 9, 13:35 UTC
Resolved - The application became unresponsive in EU data center between 8:47 UTC and 9:15 UTC due to misconfiguration after deploying the new version. We have implemented additional checks to prevent this from happening in the future.

May 9, 09:16 UTC
Monitoring - The fix has been applied, the UI is up again. We are monitoring the service now.

May 9, 09:01 UTC
Update - We are continuing to investigate this issue.

May 9, 08:58 UTC
Investigating - You may experience an infinite loading bar while accessing UI of Kontent.ai. We are working on a fix.

EU Asset Degradation

Thu, 02 May 2024 09:21:07 +0000

May 2, 09:21 UTC
Resolved - The incident has been resolved by the vendor and the asset services are stable.

May 1, 18:27 UTC
Update - Microsoft has marked the issue as resolved we are currently monitoring for any further issues.

May 1, 17:46 UTC
Monitoring - We've been informed by Microsoft that Azure Storage Service in the EU had degraded availability. Kontent.ai asset services are currently operational, but Microsoft is still investigating the issue.

May 1, 16:35 UTC
Investigating - We are currently experiencing degradation in our EU center regarding our asset services. Users may encounter difficulties uploading or requesting assets. Our team is actively working on a solution to resolve this issue.

Learn Portal Outage

Fri, 05 Apr 2024 18:23:00 +0000

Apr 5, 18:23 UTC
Resolved - This incident has been resolved.

Apr 5, 07:58 UTC
Monitoring - There was a brief increase in CPU usage and traffic on our Learn portal. Currently, everything has returned to normal, and we are actively monitoring the service.

Apr 5, 04:15 UTC
Investigating - Our Learn Portal is currently inaccessible - we are investigating this issue. More information will be provided here when available.

UI not responsive

Wed, 27 Mar 2024 03:30:00 +0000

Mar 27, 03:30 UTC
Resolved - On the date of March 27th, 2024, the user interface (UI) of our system experienced unresponsiveness from 3:47 UTC to 4:19 UTC. This occurrence was attributed to connectivity issues encountered between our servers and Content Delivery Network (CDN).

Delivery REST API items feed endpoint - older continuation tokens causing request failures

Mon, 04 Mar 2024 15:38:27 +0000

Mar 4, 15:38 UTC
Resolved - Hotfix has been implemented and continuation tokens will now work regardless of creation time. Please reach out to [email protected] if you have any further issues or questions.

Mar 4, 12:22 UTC
Investigating - Because of the newly released code, users might encounter failed requests for the Delivery REST API items feed endpoint when using older continuation tokens (created before the release, roughly 3 hours ago). The workaround is to stop using those tokens and start consuming the feed from scratch. We are currently investigating further and working on a fix. Delivery REST API is fully operational, only older continuation tokens might be causing failed requests.

Users unable to access Kontent.ai

Tue, 27 Feb 2024 10:35:03 +0000

Feb 27, 10:35 UTC
Resolved - After a 7-day monitoring period, no issues were detected with the implemented solution.

The performance challenges experienced in our US data center were caused by a bug within Microsoft's Azure Cosmos DB SDK, which is currently a top priority for resolution on their side. Kontent.ai has successfully applied a workaround guaranteeing complete functionality until Microsoft addresses the underlying issue.

Feb 20, 13:55 UTC
Monitoring - The fixing process has been finished and all projects in the US data center now seem to be fully operational.

Feb 20, 13:18 UTC
Update - We have applied a fix for the US data center, and projects are continuously returning to normal. It should take a maximum of a few hours to get everything sorted. If you still encounter any issues with your US projects, please let us know.

Feb 20, 11:07 UTC
Update - We are now rolling back changes we made for US datacenter that caused this issue. Hopefully, this should be fixed soon and we'll keep you updated.

Feb 20, 01:05 UTC
Update - Our developers have identified recent infrastructure changes, leading to increased resource consumption, as the probable cause. We are working to mitigate this issue.

Feb 19, 20:27 UTC
Update - Content Production (both via user interface and MAPI) performance for projects in the US data center is still severely degraded. Some number of requests fail, but most are eventually successful after some delay. We believe the issue is due to abnormally high memory usage, and we are continuing to investigate the cause. Thank you for your patience while we continue working toward a solution.

Feb 19, 17:56 UTC
Update - Some users are still unable to access Kontent.ai, and some number of Management API requests are also failing as a result of this issue. Our team is continuing to investigate and work towards a solution. Thank you for your patience.

Feb 19, 17:41 UTC
Update - Some users are still unable to access Kontent.ai, our team is continuing to investigate and work towards a solution. Thank you for your patience.

Feb 19, 17:20 UTC
Identified - Our service should now be back online after about 40 minutes of downtime. Delivery API functionality was not affected. We are still investigating the initial cause of the outage.

Feb 19, 17:19 UTC
Monitoring - Our service should now be back online after about 40 minutes of downtime. Delivery API functionality was not affected. We are still investigating the initial cause of the outage.

Feb 19, 16:55 UTC
Investigating - Several users have reported an inability to log in or access Kontent.ai. Currently we believe the issue is isolated to projects located in the US data center and our development team is actively investigating. Thank you for your patience while we work to resolve this issue.

Asset Domain Blacklisting Issue

Wed, 14 Feb 2024 13:33:11 +0000

Feb 14, 13:33 UTC
Resolved - This incident was resolved and affected security services whitelisted our us asset domain. They also stated our domain was incorrectly blocked and to prevent it again they updated their security rating.

If you are still affected by this asset domain block due to some services not being quick enough in whitelisting us, you can temporary use "assets-us-01-tlsnext.kc-usercontent.com" asset domain instead. For more information please contact our technical support.

Feb 13, 09:12 UTC
Investigating - We have identified that one of our US asset domains has been placed on a blacklist. If you encounter any issues with assets being blocked by security controls or vendors, please contact us immediately.

Our Information Security team is actively working to address this matter. Your reports, including the names of the security vendors involved, are invaluable to our resolution efforts.

We appreciate your understanding and cooperation and will provide updates as the situation progresses.

Kontent.ai Delivery APIs were unavailable for the USA East region.

Fri, 19 Jan 2024 14:13:46 +0000

Jan 19, 14:13 UTC
Resolved - We have resolved the partial outage caused by a vendor's code bug that led to a misconfiguration of our services. We have implemented a fix and all systems are now fully operational. We apologize for any inconvenience and appreciate your understanding.

Jan 19, 09:24 UTC
Monitoring - A fix has been implemented, and we are monitoring the results. Delivery APIs for the USA-East region returned to working at 8.45UTC and are working as expected.

Jan 19, 08:51 UTC
Investigating - We are currently experiencing a disruption in the Kontent.ai APIs specifically affecting the USA East region. Our technical team is doing their best to identify and resolve the root cause. Customers can contact us via chat or email if they need further help. We apologize for any inconvenience.

MFA Outage

Fri, 17 Nov 2023 00:06:11 +0000

Nov 17, 00:06 UTC
Resolved - Between 22:00 UTC and 23:15 UTC, multifactor authentication was not operational due to a bug introduced in a recent deployment. This issue has now been mitigated. Customers using other types of authentication were not affected, and no credentials were compromised.

We are working on a full root cause analysis and investigating options to prevent similar bugs from reaching production.

Nov 16, 23:37 UTC
Update - We are continuing to monitor for any further issues.

Nov 16, 23:32 UTC
Monitoring - Between 22:00 UTC and 23:15 UTC, the MFA was not operational due to a bug in our recent deployment. The issue has been mitigated.

Kontent.ai Login Outage

Tue, 31 Oct 2023 09:58:59 +0000

Oct 31, 09:58 UTC
Resolved - The issue with our authentication provider was solved by the provider. Customers experienced problems with signing in for about 25 minutes between 7:55 PM and 8:20 PM on October 30, 2023.

Oct 30, 21:36 UTC
Update - At this time our services have been functional for some time. Auth0 has reported an incident https://status.auth0.com/incidents/0wnt9fdgq7q3 which caused the outage. That issue is now resolved, and we will continue monitoring to ensure no further impact.

Oct 30, 21:06 UTC
Monitoring - We experienced a brief outage (around 20 minutes) that left users unable to log in to Kontent.ai. We believe the issue may have been related to an issue with our 3rd-party login provider and are continuing to monitor the issue.

Users unable to log in

Tue, 05 Sep 2023 17:54:43 +0000

Sep 5, 17:54 UTC
Resolved - The issue was the result of a small misconfiguration in our infrastructure. We shouldn't happen again and we are concluding our investigation.

Sep 5, 15:40 UTC
Update - Kontent.ai is now back online, we are continuing to investigate the cause of this issue to ensure it does not reappear. If you experience any further issue please report it directly to our support team, thank you!

Sep 5, 15:31 UTC
Investigating - We are currently investigating an issue preventing users from logging into Kontent.ai. We will provide an update as soon as possible. Thank you for your patience while we work to resolve this issue.

Azure issues in AU EAST

Thu, 31 Aug 2023 13:09:26 +0000

Aug 31, 13:09 UTC
Resolved - Our services in the AU data center have been without issue for 18 hours. We will continue to monitor Azure's status but as of now it seems they are nearing full recovery.

Aug 30, 19:25 UTC
Update - Our Australian Data Center is up and running without issue since 17:14 UTC. We are continuing to monitor the situation until Azure lists the disruption as fully resolved. Please report any recurring issues to us so we may continue our investigation. Thank you.

Aug 30, 16:31 UTC
Monitoring - Azure has reported their impacted hardware has been stabilized and restarted. Impacted customers like us should begin to see improvement over time. We are continuing to monitor the situation and will provide an update once our systems are fully restored.

Aug 30, 12:00 UTC
Investigating - We are currently investigating this issue related to: https://azure.status.microsoft/en-us/status (Azure issues in AU EAST)

- propagation of changes might be affected (delay or not coming at all)

- delivery API (or graph) could be affected

Management API not responding correctly

Thu, 31 Aug 2023 09:50:56 +0000

Aug 31, 09:50 UTC
Resolved - The problems that were occurring with the environment endpoint of the Management API have now been fixed.

Aug 31, 09:03 UTC
Investigating - We are currently experiencing issues with the environment endpoint of the Management API, which is responding with 404. We are actively investigating this matter.

Webhook endpoints in MAPI - partial failures

Wed, 30 Aug 2023 10:50:32 +0000

Aug 30, 10:50 UTC
Resolved - The issue has been completely solved.

Aug 30, 09:48 UTC
Investigating - We are currently looking into a partial issue with the webhook endpoints in MAPI which appear to be having difficulty.

Degraded performance of GraphQL API in Australia

Mon, 28 Aug 2023 09:02:24 +0000

Aug 28, 09:02 UTC
Resolved - This incident has been resolved.

Aug 28, 08:40 UTC
Monitoring - There was a short spike (10 min) on AU server. Everything seems to be working as expected now. We are monitoring the server.

Aug 28, 08:37 UTC
Update - We are continuing to investigate this issue.

Aug 28, 08:26 UTC
Update - We are continuing to investigate this issue.

Aug 28, 08:09 UTC
Investigating - There could be failing requests in Australia data center for GraphQL API.
We are investigating the issue.

Secured access for GraphQL is failing

Wed, 16 Aug 2023 13:06:03 +0000

Aug 16, 13:06 UTC
Resolved - This incident has been resolved.

Aug 16, 12:15 UTC
Monitoring - The fix was applied. Now, we are monitoring GraphQL, but the issue should be solved.

Aug 16, 11:58 UTC
Investigating - We have found a bug that prevents GraphQL API calls to finish when Secured access is enabled. We are working on a fix.

Learn Portal Outage

Wed, 09 Aug 2023 08:30:31 +0000

Aug 9, 08:30 UTC
Resolved - This incident has been resolved. The Learn Portal is operational again.

Aug 9, 05:36 UTC
Monitoring - The Learn Portal has been restored. We are monitoring this service while investigating the root cause of the outage.

Aug 9, 00:52 UTC
Update - We are continuing to investigate this issue.

Aug 9, 00:47 UTC
Investigating - Our Learn Portal is currently inacessible - we are investigating this issue. More information will be provided here when available.

Interruption to Scheduled Publishing

Sat, 08 Jul 2023 14:22:17 +0000

Jul 8, 14:22 UTC
Resolved - This incident has been resolved; scheduled publishing is functioning as expected.

Jul 8, 14:09 UTC
Investigating - We are aware of a disruption to scheduled publishing, currently affecting all datacenters. We are currently investigating this issue, and will provide further updates.

Increased latency in EU and AU datacenters.

Wed, 05 Jul 2023 23:31:26 +0000

Jul 5, 23:31 UTC
Resolved - Microsoft has repaired the damaged lines causing unavailable data links. It does not appear that Kontent.ai was affected by the cut lines. For more information you can view Microsoft Azure's status history: https://azure.status.microsoft/en-us/status/history if you have any questions please contact us at [email protected].

Jul 5, 13:16 UTC
Monitoring - After conducting a thorough investigation, our developers have determined that the problem reported by Microsoft only affects internal processes and has not had any impact on our customer projects or API performance.

We are actively monitoring the situation, which is being tracked on the Microsoft Azure status page: https://azure.status.microsoft/en-us/status. Rest assured, we will promptly inform our customers of any relevant developments as they occur.

Jul 5, 09:22 UTC
Investigating - Microsoft Azure is experiencing degradation of performance in EU and AU regions. This means that projects in EU and AU can serve content with increased propagation time.

UI temporarily returning outdated results

Sat, 17 Jun 2023 09:51:43 +0000

Jun 17, 09:51 UTC
Resolved - Between Jun 16, 06:52 UTC, and Jun 17, 06:04 UTC we experienced an issue with our backend service's item indexing, impacting the editing experience in the user interface. Please note that our APIs were not affected.

Due to the nature of the issue, we had to perform item reindexing across all projects to ensure full functionality, which has now been completed in all regions. Thank you for your patience and understanding while we addressed this matter. If you have any lingering concerns or questions, please don't hesitate to reach out to our dedicated support team.

Jun 16, 20:39 UTC
Update - Reindexing tasks are complete for all projects in the European and Australian data centers and this issue should be resolved for customers in those regions. Reindexing of the US data center is still in progress. We will update this status again once that process is complete. Thank you for your patience and understanding while we work to resolve this issue.

Jun 16, 13:05 UTC
Update - The fix is being applied to all data centers. This should resolve the UI issue.

Jun 16, 06:52 UTC
Identified - Our UI is encountering problems because of which it might be temporarily returning outdated results. It may manifest by

- Showing outdated data
- New or updated items not being shown in full-text search results
- New or updated items showing as non-existing when they are used as linked items

We have identified the problem and are working on a fix for the affected projects. Delivery API is not affected by this.

We experienced an outage for Delivery Preview REST API in AU EAST DC and for Delivery Live GraphQL API in AU SCC DC for approximately 45 minutes.

Tue, 18 Apr 2023 10:09:38 +0000

Apr 18, 10:09 UTC
Resolved - For around 45 minutes, we encountered an outage for Delivery Preview REST API in AU EAST DC and for Delivery Live GraphQL in AU SCC DC. The situation is stable now and the outage has been resolved.