Squiz Status - Incident History

Sydney Data Centre Performance Improvement

Tue, 10 Dec 2024 22:00:21 +1100

Dec 10, 22:00 AEDT
Completed - The scheduled maintenance has been completed.

Dec 10, 20:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 27, 10:08 AEDT
Scheduled - Our DC team will be working on network performance improvements in Sydney Data Centre. We do not expect this to affect production traffic. In unlikely event of a disruption we will have rollback plan in place.

Cloudflare Service Maintenance

Tue, 03 Dec 2024 21:30:21 +1100

Dec 3, 21:30 AEDT
Completed - The scheduled maintenance has been completed.

Dec 3, 20:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 19, 10:06 AEDT
Scheduled - Dear Customer,

SQUIZ wishes to notify you that a scheduled maintenance work will be carried out on Squiz Cloudflare CDN.

This includes Squiz DXP Services, & Squiz Cloud customers migrated to Cloudflare.

Any clients on Squiz Edge are not impacted.

Date: 03 Dec 2024

Maintenance Window Start: 09:00 GMT
Maintenance Window End: 12:00 GMT

Expected Duration: 180 min

Impact: During the maintenance window we do not expect there to be any down time or impact to your live websites.

Should you have any questions, please contact us at Squiz Customer Care.

Performance Issues for Squiz UK hosted Clients

Mon, 25 Nov 2024 22:53:50 +1100

Nov 25, 22:53 AEDT
Resolved - We are pleased to confirm that the previously reported issue affecting the performance of our UK hosted systems has been successfully resolved.
Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance.
We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

Nov 25, 21:56 AEDT
Monitoring - Following a thorough investigation from our engineering teams, we have performed mitigation and remediation actions.

These actions have been successful as we are currently experiencing no significant performance impact.

At this time we will continue to monitor the service.

Nov 25, 19:52 AEDT
Identified - We have identified this is as a result of DDoS attack which we have partially mitigated, however we are still experiencing issues at this time and our engineers are working to fully mitigate this.

Further update to be provided as soon as there is more information available.

Nov 25, 18:36 AEDT
Monitoring - We are now monitoring the performance after our engineering have been able to see some results so far.
An update shall be available once we identify no issues.

Nov 25, 18:14 AEDT
Update - Squiz are continuing to look into the potential causes of the performance degredation at hand and will have an update as soon as pertinent news has arisen.

Nov 25, 17:42 AEDT
Update - We are continuing to investigate this issue.

Nov 25, 17:42 AEDT
Investigating - Hey Everyone.
Squiz has identified issues with the performance of our UK servers and wish to let you know that our engineering team is looking into this matter with top priority and will have updates for you as soon as they are available.

Feel free to let us know by our Service Desk if you have any concerns.

UK DC Maintenance

Thu, 14 Nov 2024 10:13:39 +1100

Nov 14, 10:13 AEDT
Completed - These upgrades have been postponed until further notice

Oct 15, 10:12 AEDT
Scheduled - Affects: Internet Transit in Ash, and connectivity between Ash and Newbury DCs will be affected by Network Upgrades being performed by our Provider

When: 20th Nov 00:00 to 04:00 BST up to 30 mins

Impact: Ash DC will be completely Isolated for up to 30 minutes during this Window. Customers behind SquizEdge with DR hosted in Newbury will be served via their DR

UK DC Maintenance

Thu, 14 Nov 2024 10:13:24 +1100

Nov 14, 10:13 AEDT
Completed - These upgrades have been postponed until further notice

Oct 15, 10:10 AEDT
Scheduled - Affects: Internet Transit in Ash, and connectivity between Ash and Newbury DCs will be affected by Network Upgrades being performed by our Provider

When: 16th Nov 00:00 to 04:00 BST up to 30 mins

Impact: Ash DC will be completely Isolated for up to 30 minutes during this Window. Customers behind SquizEdge with DR hosted in Newbury will be served via their DR

DXP / SAAS AUS Maintenance

Wed, 13 Nov 2024 22:00:21 +1100

Nov 13, 22:00 AEDT
Completed - The scheduled maintenance has been completed.

Nov 13, 21:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 13, 10:13 AEDT
Scheduled - Our Engineering team need to perform maintenance on the Matrix instances hosted in DXP / SAAS Australia.
There is a estimated 20 second outage for authoring traffic only during this time, other than that no other issues are expected.
Feel free to let us know if you have any questions by raising a case at https://my.squiz.net/

Melbourne DC maintenance

Mon, 04 Nov 2024 22:30:02 +1100

Nov 4, 22:30 AEDT
Completed - The scheduled maintenance has been completed.

Nov 4, 18:30 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 25, 09:53 AEDT
Scheduled - As part of our mission to bring you the best in system performance we will be performing maintenance on the Melbourne DC
No service impact is expected, but network resiliency will be impacted while we move our Transit Links onto new Hardware.

Melbourne DC maintenance

Wed, 30 Oct 2024 22:30:02 +1100

Oct 30, 22:30 AEDT
Completed - The scheduled maintenance has been completed.

Oct 30, 18:30 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 25, 09:52 AEDT
Scheduled - As part of our mission to bring you the best in system performance we will be performing maintenance on the Melbourne DC
No service impact is expected, but network resiliency will be impacted while we move our Transit Links onto new Hardware.

Squiz Cloudflare CDN Maintenance

Tue, 29 Oct 2024 23:00:02 +1100

Oct 29, 23:00 AEDT
Completed - The scheduled maintenance has been completed.

Oct 29, 22:12 AEDT
Verifying - Roll out of our Cloudflare CDN maintenance is now been complete.
Squiz will continue to monitor and provide updates where necessary.

Oct 29, 20:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 16, 09:46 AEDT
Scheduled - Dear Customer,

SQUIZ wishes to notify you that a scheduled maintenance work will be carried out on Squiz Cloudflare CDN.

This includes Squiz DXP Services, & Squiz Cloud customers migrated to Cloudflare.

Any clients on Squiz Edge are not impacted.

Date: 29 Oct 2024

Maintenance Window Start: 09:00 GMT
Maintenance Window End: 12:00 GMT

Expected Duration: 180 min

Impact: During the maintenance window we do not expect there to be any down time or impact to your live websites.

Should you have any questions, please contact us at Squiz Customer Care.

SSL certificate renewal for Connect AU tenant UI

Mon, 28 Oct 2024 22:44:28 +1100

Oct 28, 22:44 AEDT
Resolved - This issue has now been resolved. We have retested Connect AU and the UI errors are gone and the service is working as expected.

Oct 28, 20:41 AEDT
Monitoring - The SSL certificate has now been renewed and Connect in AU is working as expected. We will continue to monitor for any lasting issues.

Oct 28, 20:21 AEDT
Update - Our network teams are engaged on resolving this issue. We will provide further updates as more information becomes available.

Oct 28, 14:24 AEDT
Identified - Our team continue to work on the SSL certificate renewal

Oct 28, 14:05 AEDT
Investigating - Our team is actively working on the SSL certificate renewal Certificate update process is in progress

Urgent network maintenance in Melbourne DC (No Impact Expected)

Tue, 22 Oct 2024 22:00:02 +1100

Oct 22, 22:00 AEDT
Completed - The scheduled maintenance has been completed.

Oct 22, 20:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 18, 09:30 AEDT
Scheduled - Our network infrastructure team are performing much needed updates within our Melbourne DC which is no expected to have any impact on our services however if you do notice any difference in performance feel free to get in contact with support at [email protected]

UK DC Maintenance

Sat, 19 Oct 2024 14:00:02 +1100

Oct 19, 14:00 AEDT
Completed - The scheduled maintenance has been completed.

Oct 19, 10:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 9, 09:51 AEDT
Scheduled - Affects: Connectivity in the UK between Ash and Newbury DCs will be affected by Network Upgrades being performed by our Provider

When: 19th Oct 00:00 to 04:00 BST up to 30 mins network downtime

Impact: DR (Disaster Recovery) Services hosted in Newbury DC will be affected by up to 30 minutes network downtime during this Window. as long as the main server is still functional during this time then no impact will occur to hosted instances.

Squiz DXP Funnelback degraded performance

Thu, 10 Oct 2024 20:56:46 +1100

Oct 10, 20:56 AEDT
Resolved - We are pleased to confirm that the performance of our UK Funnelback DXP system has been restored.
Our team were able to apply a fix for the issue, which led to significant improvements in performance.

We appreciate your patience and understanding during this time and apologise for any inconvenience caused.
We'll continue to monitor the situation and be quick to provide updates where necessary.

A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

Oct 10, 20:41 AEDT
Monitoring - We are now beginning to see service improvements in our Funnelback search performance after implementing a fix.
We'll continue to monitor the situation and look to confirm resolution when possible.

Oct 10, 20:27 AEDT
Identified - Our engineering team has identified an issue with the DXP Funnelback and is in the process of implementing a resolution.

Oct 10, 20:03 AEDT
Investigating - Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only.

Some customers are experiencing slow response times and/or timeouts.

We are currently working to find the root cause of this issue.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Funnelback DXP Maintenance - No Outage Expected.

Thu, 10 Oct 2024 16:00:04 +1100

Oct 10, 16:00 AEDT
Completed - The scheduled maintenance has been completed.

Oct 10, 15:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 3, 13:09 AEST
Scheduled - Funnelback DXP will soon be undergoing some maintenance in the US and AU regions.
There is no expected outage for this maintenance.
However, traffic may flow differently after the change into the platform, such as IP address pools will change that these services utilise.
The affected hostnames and times listed below:

dxp-us-search.funnelback.squiz.cloud - 10th Oct 0000 EDT / 9th Oct 2100 PDT (10th Oct 1500 AEDT)

dxp-au-search.funnelback.squiz.cloud - 10th Oct 2000 AEDT

UK DR/Backup Datacenter degradation

Tue, 08 Oct 2024 03:20:00 +1100

Oct 8, 03:20 AEDT
Resolved - This incident has been resolved

Oct 8, 02:54 AEDT
Monitoring - Squiz teams have implemented a fix that appears to have resolved this issue for now and we are now monitoring the situation to ensure stability is retained.

Oct 8, 02:51 AEDT
Identified - We are currently experiencing connectivity issues with non production services, hosted in our UK Datacenter. Our Network teams are currently investigating.

Config change by Upstream Provider in Sydney and Melbourne

Fri, 20 Sep 2024 01:00:02 +1000

Sep 20, 01:00 AEST
Completed - The scheduled maintenance has been completed.

Sep 19, 23:00 AEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 6, 08:48 AEST
Scheduled - One of our transit providers in Melbourne and Sydney will be performing routine config changes on our transit links. We have redundant upstream connectivity, so no loss of service is expected, but we will have reduced bandwidth and resiliency during this Hazard Window

04:00 - 06:00 AEST September 18 2024 - Emergency Network Maintenance in Melbourne DC

Wed, 18 Sep 2024 06:00:02 +1000

Sep 18, 06:00 AEST
Completed - The scheduled maintenance has been completed.

Sep 18, 04:00 AEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 17, 05:04 AEST
Scheduled - We will be performing emergency Network Maintenance in our Melbourne DC.

This is not expected to cause any significant service impact, but short network performance degradation might be observed, and resiliency will be partially reduced for the duration of the maintenance

04:00 - 06:00 AEST September 17 2024 - Emergency Network Maintenance in Melbourne DC

Tue, 17 Sep 2024 04:47:40 +1000

Sep 17, 04:47 AEST
Completed - Scheduled maintenance has been completed at this time without any incidents or impact.

Sep 17, 04:00 AEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 17, 00:56 AEST
Scheduled - We will be performing emergency Network Maintenance in our Melbourne DC.

This is not expected to cause any significant service impact, but short network performance degradation might be observed, and resiliency will be partially reduced for the duration of the maintenance

Squiz DXP Funnelback degraded performance

Mon, 16 Sep 2024 22:36:44 +1000

Sep 16, 22:36 AEST
Resolved - We are pleased to confirm that the previously reported issue affecting the performance of our UK Funnelback DXP system has been successfully resolved.
Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance.
We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

Sep 16, 22:23 AEST
Update - We continue to monitor the situation and will confirm resolution as soon as possible.

Sep 16, 21:19 AEST
Update - Customers are now confirming that services are recovering and performing as expected.
Our team continues to monitor the situation closely to ensure everything remains stable.

Sep 16, 20:41 AEST
Update - We are still seeing service improvements in some areas of our Funnelback search performance.
We'll continue to monitor the situation and look to confirm resolution when possible.

Sep 16, 19:52 AEST
Update - We are noting sustained improvements in Funnelback search performance.
We will maintain close monitoring of the situation and will provide confirmation of resolution as soon as it is feasible.

Sep 16, 19:27 AEST
Monitoring - We are now beginning to see service improvements in our Funnelback search performance.
We'll continue to monitor the situation and look to confirm resolution when possible.

Sep 16, 19:11 AEST
Identified - Our engineering team has identified an issue with the DXP Funnelback and is in the process of implementing a resolution.

Sep 16, 19:01 AEST
Update - We are continuing to investigate the situation and our engineers are working to find the root cause of this issue.

Sep 16, 18:42 AEST
Investigating - Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only.

Some customers are experiencing slow response times and/or timeouts.

We are currently working to find the root cause of this issue.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Emergency Maintenance on Sydney DC

Tue, 10 Sep 2024 00:00:02 +1000

Sep 10, 00:00 AEST
Completed - The scheduled maintenance has been completed.

Sep 9, 23:00 AEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 9, 10:01 AEST
Scheduled - To resolve a network performance degradation, we need to perform emergency maintenance on our network infrastructure in the Sydney DC.

This will cause an expected network outage of 10 minutes during this window, but assume the network is 'at risk' throughout.

Support Telephone Line Issues

Fri, 06 Sep 2024 21:47:55 +1000

Sep 6, 21:47 AEST
Resolved - We are pleased to inform you that our support phone line has been successfully repaired and is now fully operational.

Sep 6, 19:24 AEST
Investigating - Please be advised we are currently experiencing issues with our Support Telephone lines.

This issue is not affecting other services so please continue to contact us via email at [email protected] and through MySquiz.

This is only affecting the below phone lines as far as we are aware:

Squiz 24x7 Helpdesk:
(+61) 1300 077 849 from Australia
(+64) 4 805 0098 from New Zealand
(+44) 808 267 7849 from UK / Europe
(+1) 877 394 0640 from the US

Sydney border router maintenance.

Thu, 05 Sep 2024 23:00:02 +1000

Sep 5, 23:00 AEST
Completed - The scheduled maintenance has been completed.

Sep 5, 22:00 AEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 21, 14:24 AEST
Scheduled - On 5th September at 10 PM Sydney (1PM GMT) DC Team will be doing proactive maintenance on one of the two border routers in Sydney DC. It will take up to 1 hour. Expected impact is minimal due to the redundancy setup but in worst case scenario there will be up to 10 minute outage.

DXP - Component content changes not showing in Admin

Thu, 22 Aug 2024 17:41:22 +1000

Aug 22, 17:41 AEST
Resolved - This update should be out as the latest patch.
Feel free to get in touch at [email protected] if you see any further issues.

Aug 22, 13:29 AEST
Update - The proposed patch has now been pushed into the release cycle.
Next update here will be posted once this is complete.

Aug 22, 13:23 AEST
Update - We have further tested the patch in question and are looking to release to more test sites.

Aug 22, 13:08 AEST
Update - We have gone through a new stage of testing with no issues to the solution so far.

Aug 22, 12:51 AEST
Update - Further testing of a proposed solution are underway and we are awaiting more positive results before pushing.

Aug 22, 12:31 AEST
Update - We are looking at a possible solution and are being sure to test all facets to ensure a safe fix.

Aug 22, 12:14 AEST
Update - We are continuing to work on implementing a fix to compile previews into the component directories.

Aug 22, 11:54 AEST
Update - We are continuing to implement a solution that we believe will fix the issue.

Aug 22, 11:36 AEST
Update - We are continuing to implement a solution that we believe will fix the issue.

Aug 22, 11:10 AEST
Update - We have identified the issue and have deployed a fix to customers who have reported an impact. We will be deploying the fix to remaining customers soon. If you are affected and wish to be prioritised to have the fix deployed, please contact Squiz.

Aug 22, 11:08 AEST
Monitoring - We identified an issue and applied patches to all known instances. We are currently monitoring the outcome of the patch.

Aug 22, 10:51 AEST
Investigating - We are aware of an incident affecting DXP customers when making changes when editing components, causing the updates not to appear in the admin interface. We are currently investigating the issue.

Squiz DXP funnelback degraded performance

Thu, 01 Aug 2024 20:18:08 +1000

Aug 1, 20:18 AEST
Resolved - We are pleased to confirm that the previously reported issue affecting the performance of our Funnelback DXP system has been successfully resolved.
Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance.
We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

Aug 1, 19:56 AEST
Monitoring - We are continuing to see service improvements across the board.
We'll continue to monitor the situation and look to confirm resolution when possible.

Aug 1, 19:50 AEST
Update - We are beginning to see slight improvements in Funnelback search performance.
Squiz will continue to investigate the root cause and provide regular updates via this status page.

Aug 1, 19:27 AEST
Update - We are continuing to investigate the situation and our engineers are working to find the root cause of this issue.

Aug 1, 19:10 AEST
Update - Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only.

Some customers are experiencing slow response times and/or timeouts.

We are currently working to find the root cause of this issue.

A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Aug 1, 18:55 AEST
Investigating - Our monitoring systems found slowness on some DXP Funnelback instances. Squiz is actively working on finding the root cause and fixing the issue.

Squiz DXP - Rollback of settings from previous incident.

Fri, 28 Jun 2024 14:31:19 +1000

Jun 28, 14:31 AEST
Completed - We have received word from our team that something has come up in our planning and the operations planned in this maintenance window are postponed until further notice.
Keep an eye out for a future notice.

Jun 21, 09:38 AEST
Scheduled - We are scheduling a 30min maintenance window during which a short outage is anticipated, to implement a domain migration required to revert changes made during a prior incident response.

Time for this change will be:

AEST 06:30
NZST 08:30
BST 21:30

The following services will impacted partially or fully during this time:

DXP Console (https://dxp.squiz.cloud/)

DXP Services

- Component Service

- Metrics Collection Service

- Logging Service

- Event Service

- API Key Service

- Job Runner

- AB Testing

- CDP

Authentication to these services:

- Advanced Forms

- Funnelback DXP

- Squiz Connect