This article is more than 1 year old
Canadian ISP Rogers falls over for hours, takes out broadband, cable, cellphones
ATMs, 911 calls, and more hit by networking snafu
Updated Canadian ISP Rogers today suffered a major outage that has left subscribers unable to use cellular, broadband internet, and cable services.
ATMs and debit card payments were also affected, leaving folks unable to take out cash and buy stuff in stores, and some were unable to make 911 emergency calls. Rogers has more than 10 million wireless subscribers in Canada. The snafu cut the nation's connectivity by a quarter, according to network watchers at NetBlocks, and it may have knock-on effects for other ISPs and service providers.
The outage has been going on for several hours, seemingly nearly 11 hours at time of writing. The communications giant acknowledged the broken connectivity at about 1300 UTC (0900 ET) but it appears from online complaints and the Cloudflare internet radar, the trouble began much earlier, around 0830 UTC (0430 ET).
According to that radar, there was a surge in BGP routing announcements at around 0845 UTC for Rogers' AS812 block of the internet.
Cloudflare Radar's graph showing a spike in BGP announcements at 0845 UTC for Rogers' AS812 around the same time connectivity died
A few hours ago, Rogers tweeted: "On behalf of all of us at Rogers, we sincerely apologize to our customers, and we will continue to keep you updated as we have more information to share, including when we expect service to be back up."
And on its support message board, it confirmed: "We are currently experiencing an outage across our wireline and wireless networks and our technical teams are working hard to restore services as quickly as possible."
We are seeing some reports by netizens that their Rogers services are returning, and plenty others saying it's still down, so your mileage may vary. The comms giant told CBC it is making progress in repairing its networking.
According to the broadcaster, Rogers-owned brands, such as Fido and Chatr, were also knocked offline, and access to travel and financial networks as well as emergency services, was derailed, too. CBC's radio station in Kitchener, Ontario, was even taken off the air. Police stressed that 911 call centers were still working, even if Rogers' subscribers couldn't get through, and rival communications provider Bell said it was unaffected.
As this article was published, Rogers tweeted to say it is working with its techies and suppliers to bring back connectivity:
Our technical teams are working to restore our services alongside our global technology partners, and are making progress. pic.twitter.com/BsufX2Q92Q
— RogersHelps (@RogersHelps) July 8, 2022
Suffice to say, Rogers' customers are furious. "Welcome to Rogers-ocalypse, where a third of Canada's populace becomes a hostage to their service provider," as one put it.
We'll update this story as more information arrives. ®
Updated to add at 2200 UTC
This outage is still ongoing. In the past hour, Rogers emitted this statement:
We acknowledge the impact our outage is having on your life. We have every technical resource and partner fully deployed to solve the problem. As soon as we know the specific time the networks will be fully operational, we will share that with you. Right now, we are focused on the solution. Some of our customers have raised the question of credits and of course we will be proactively crediting all customers and will share more information soon.
It sounds like it's been a long day, not just for the ISP but also its subscribers.
Final update at 1900 UTC, July 11
At about 0700 ET on Saturday, July 9, Rogers declared it had mostly fixed its service. On Sunday, its CEO Tony Staffieri emailed subscribers to say maintenance work on its systems likely caused the entire thing to collapse.
We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
"We know how much you rely on our networks and I sincerely apologize," he added. "We’re particularly troubled that some customers could not reach emergency services and we are addressing the issue as an urgent priority.
"We let you down on Friday. You have my personal commitment that we will do better."
Staffieri also said customers would be refunded for the loss of connectivity, promised a root cause analysis will be carried out, redundancy may be added, and steps may be taken to prevent this from happening again.
Biting the hand that feeds IT
