Spamhaus DDoS grows to Internet-threatening size

Last week, anti-spam organization Spamhaus became the victim of a large denial of service attack, intended to knock it offline and put an end to its spam-blocking service. By using the services of CloudFlare, a company that provides protection and acceleration of any website, Spamhaus was able to weather the storm and stay online with a minimum of service disruptions.

Since then, the attacks have grown to more than 300 Gb/s of flood traffic: a scale that's threatening to clog up the Internet's core infrastructure and make access to the rest of the Internet slow or impossible.

It now seems that the attack is being orchestrated by a Dutch hosting company called CyberBunker. CyberBunker specializes in "anything goes" hosting, using servers in a former nuclear bunker (hence the name). As long as it's not "child porn and anything related to terrorism," CyberBunker will host it. This includes sending spam.

Spamhaus blacklisted CyberBunker earlier in the month. A CyberBunker spokesman, Sven Olaf Kamphuis, told the New York Times that CyberBunker was fighting back against Spamhaus because the anti-spam organization was "abusing [its] influence."

Update: Kamphuis has written on his Facebook page that the NYT has gone for "sensational reporting" and that CyberBunker is not, in fact, responsible for the attacks.

When the attack started, on March 18, it measured around 10 Gb/s. On March 19, it hit 90 Gb/s, on March 22 it reached 120 Gb/s. This still wasn't enough to knock CloudFlare or Spamhaus offline. So the attackers escalated.

Today, CloudFlare wrote that one of the Internet's big bandwidth providers is seeing 300 gigabits per second of traffic related to this attack, making it one of the largest ever reported.

This is bad news for the Internet. 300 Gb/s is the kind of scale that threatens the core routers that join the Internet's disparate networks.

As Ars wrote last week, CloudFlare uses a technique called anycast to distribute traffic to nearby servers. This greatly diffuses the potency of DDoS attacks, by preventing the attackers from focusing their traffic on a single system on the Internet. Instead, the attack traffic all gets directed to a nearby machine—one of CloudFlare's geographically distributed mirrors. A sufficient flood of traffic could still knock one of those local mirrors offline, but the impact of that should be relatively restricted, with users throughout the rest of the world unaffected.

Once an attack has been detected, the companies that CloudFlare buys bandwidth from—known as "Tier 2" providers—can then block the traffic to prevent it from entering their networks. That doesn't stop the problem, however; it just moves it upstream.

Tier 2 providers buy their bandwidth from the small number of Tier 1 providers. Tier 1 providers work a bit differently than Tier 2. They don't buy bandwidth from anyone. Instead, they just connect to other Tier 1 providers for free. These Tier 1 providers are the high-speed backbone that joins all the Tier 2 providers together, and hence makes the Internet a single global network, rather than a bunch of separate networks.

If a Tier 1 provider fails, that risks breaking the entire Internet.

Though the Tier 2 providers are blocking the flood traffic, the Tier 1 providers are still carrying it. As the DDoS attack has grown, so too has this load. The 300 Gb/s figure came from one of these Tier 1 providers. CloudFlare says that several of the Tier 1 networks have started to become congested, particularly in Europe. This congestion can make the entire Internet slower for everyone.

This has been particularly significant in London. Dotted around the globe are a number of "Internet Exchanges" (IXs). These are places where multiple networks from different service providers connect to each other. The London Internet Exchange (LINX), through which an average of about a terabit of traffic passes each second, suffered a substantial outage on March 23. At peak time, its traffic dropped from about 1.5 Tb to around half that.

The LINX team has subsequently changed some aspects of their network configuration to make their systems more robust against this kind of large scale attack, and normal service was resumed a little over an hour after the first attack.

The fundamental problem, however, remains. The traffic is being generated primarily from DNS amplification attacks. Small requests are sent to DNS servers, generating responses from those servers that are about 50-100 times larger. The sending address of these requests are spoofed, so the DNS servers think that they originated not from the attacker's machine but from the victim's machine; accordingly, the large responses are sent to that victim, overwhelming it with traffic.

To perform these attacks, the attackers need servers that are open to anyone (and arguably misconfigured). The Open DNS Resolver Project reports that there are about 25 million of these open DNS servers, and hence 25 million servers that can be used to generate enormous quantities of traffic. Making this worse is the fact that, unlike DDoS attacks using home PCs, these DNS servers typically have fast Internet connections.

The number of open DNS resolvers is dropping—CloudFlare reported that it was down by about 30 percent in February—but they're still abundant, and as the current attacks on SpamHaus make clear, still enough to be tremendously problematic.

To guard against these attacks in future, the open DNS servers need to be reconfigured in some way (to either restrict the IP addresses that can use them, or limit the number of queries they'll respond to, or both), and networks need to be reconfigured so that they won't send traffic with spoofed sender addresses.

Both of these fixes are well-known, and the problems have long been acknowledged. However, they require coordinated action from many parties: every DNS server operator and every ISP needs to do the reconfiguration work.

As for CyberBunker, the company boasts that although "Dutch authorities and the police have made several attempts to enter the bunker by force, none of these attempts were successful." Even a Dutch SWAT team allegedly failed to get in. CyberBunker argues that it is currently engaged in a blackmail war with Spamhaus. As Internet wars go, this one is using the nuclear option, and everyone is at risk of being caught in the blast.