Yesterday I posted a post mortem on anoutage we had Saturday. The
outage was caused when we applied an overly aggressive rate limit to
traffic on our network while battling a determined DDoS attacker. In
the process of writing it I mentioned that we'd seen a 65Gbps DDoS
earlier on Saturday. I've received several questions since that all go
something like: "65Gbps DDoS!? Who launches such an attack and how do
youdefend yourself against it?!" So I thought I'd give a bit more
detail.
What Constitutes a Big DDoS?
A 65Gbps DDoS is a big attack, easily in the top 5% of the biggest
attacks we see. The graph below shows the volumeof the attack hitting
our EU data centers(the green line represents inbound traffic). When
an attack is 65Gbps that means every second 65 Gigabits of datais sent
to our network. That's the equivalent data volume of watching 3,400 HD
TV channels all at the same time. It's a ton of data. Most network
connections are measured in 100Mbps, 1Gbps or 10Gbps so attacks like
this would quickly saturate even a large Internet connection.
At CloudFlare, an attack needs to get over about 5Gbps to set off
alarms with our ops team. Even then, our automated network defenses
usually stop attacks without the need of any manual intervention. When
an attack gets up in the tens of Gigabits of data per second, our ops
team starts monitoring the attack: applying filters and shifting
traffic to ensure the attacked customer's site stays online and none
of the rest of our network is affected.
So You Want to Launch a DDoS
So how does an attacker generate 65Gbps of traffic? It is highly
unlikely that the attacker has a single machine with a big enough
Internet connection to generate that much traffic on its own. One way
to generate that much traffic is through a botnet. A botnet is a
collection of PCs that have been compromised with a virus and can be
controlled by what is known as a botnetherder.
Botnet herders will often rent out access to their botnets, often
billing in 15 minute increments (just like lawyers). Rental prices
depend on the size of the botnets. Traditionally, email spammers
purchased time on botnets in order to send their messages to appear to
come from a large number of sources. As email spam has become
lessprofitable with the rise of better spam filters, botnet herders
have increasinglyturned to renting out their networks of compromised
machines to attackers wanting to launch a DDoS attack.
To launch a 65Gbps attack, you'd need a botnet with at least 65,000
compromised machines each capable ofsending 1Mbps of upstream data.
Given that many of these compromised computers are in the developing
world where connections are slower, and many of the machines that make
up part of a botnet may not be online at any given time, the actual
size of the botnet necessary to launch that attack would likely need
to be at least 10x thatsize. While by no means unheard of, that's a
large botnet and using all its resources to launch a DDoS risks ISPs
detecting many of the compromised machines and taking them offline.
Amplifying the Attacks
Since renting a large botnet can be expensive and unwieldy, attackers
typically look for additional ways to amplify the size of their
attacks. The attack on Saturday used one such amplification technique
called DNS reflection. To understand how these work, you need to
understand a bit about how DNS works.
When you first sign up for an Internet connection, your ISP will
provide you with a recursive DNS server, also knownas a DNS resolver.
When you click on a link, your computer sends a lookup to your ISP's
DNS resolver. The lookup is asking a question, like: what is the IP
address of the server for cloudflare.com? If the DNS resolver you
query knows the answer, because someone has already asked it recently
and the answer is cached, it responds. If it doesn't, it passes the
request on to the authoritative DNS for the domain.
Typically, an ISP's DNS resolvers are setup to only answer requests
from the ISP's clients. Unfortunately, there are a large number of
misconfigured DNS resolversthat will accept queries from anyone on
the Internet. These are known as "open resolvers" and they area sort
of latent landmine on the Internet just waiting to explode when
misused.
DNS queries are usually sent via the UDPprotocol. UDP is a
fire-and-forget protocol, meaning that there is no handshake to
establish that where a packet says it is coming from actually is where
it is coming from. This means, if you're an attacker, you can forge
the header of a UDP packet to say it is coming from a particular IP
you want toattack and send that forged packet to an open DNS resolver.
The DNS resolver will reply back with a response to the forged IP
address with an answer to whatever question was asked.
To amplify an attack, the attacker asks a question that will result in
a very large response. For example, the attacker may request all the
DNS records for a particular zone. Or they may request the DNSSEC
records which, often, are extremely large. Since resolvers typically
have relatively high bandwidth connections to the Internet, they have
no problem pumping out tonsof bytes. In other words, the attacker can
send a relatively small UDP request and use open resolvers to fire
back at an intended target with a crippling amount of traffic.
Mitigating DNS Reflection Attacks
One of the great ironies when we deal with these attacks is we'll
often get an email from the owner of the network where an open
resolver is running asking us to shut down the attack our network is
launching against them. They're seeing a large number of UDP packets
with one of our IPs as the source coming in to their network and
assume we're the ones launching it. In fact, it is actually their
network which isbeing used to launch an attack against us. What's
great is that we can safely respond and ask them to block all DNS
requests originating from our network since our IPs should never
originate a DNS request to a resolver. Not only doesthat solve their
problem, but it means there's a smaller pool of open resolvers that
can be used to target sites on CloudFlare's network.
There have been a number of efforts to clean up open resolversthat
are currently active. Unfortunately, it is slow going and the default
installation of many DNS clients still has them open by default. While
we actively reach out to the worst offenders to protect our network,
to protect the Internet generally there will need to be a concerted
effort to clean up open DNS resolvers.
In terms of stopping these attacks, CloudFlare uses a number of
techniques.It starts with our network architecture. We use
Anycastwhich means the response from a resolver, while targeting one
particular IP address, will hit whatever data center is closest.
Thisinherently dilutes the impact of an attack, distributing its
effects across all 23 of our data centers. Given the hundreds of gigs
of capacity we have across our network, even a big attack rarely
saturates a connection.
At each of our facilities we take additional steps to protect
ourselves. We know, for example, that we haven't sent any DNS
inquiries out from our network. We can therefore safely filter the
responses from DNS resolvers: dropping the response packets from
theopen resolvers at our routers or, in some cases, even upstream at
one of our bandwidth providers. The result is that these types of
attacks are relatively easily mitigated.
What was fun to watch was that while the customer under attack was
being targeted by 65Gbps of traffic, not a single packet from that
attack made it to their network or affected their operations. In fact,
CloudFlare stopped the entire attack without the customer even knowing
there was a problem. From the network graph you can see after about 30
minutes the attacker gave up. We think that's pretty cool and,as we
continue to expand our network, we'll get even more resilient to
attacks like this one.
No comments:
Post a Comment