GitHub Survived the Biggest DDoS Attack Ever Recorded
On Wednesday, at about 12:15 pm ET, 1.35 terabits per second of traffic hit the developer platform GitHub all at once. It was the most powerful distributed denial of service attack recorded to date—and it used an increasingly popular DDoS method, no botnet required.
GitHub briefly struggled with intermittent outages as a digital system assessed the situation. Within 10 minutes it had automatically called for help from its DDoS mitigation service, Akamai Prolexic. Prolexic took over as an intermediary, routing all the traffic coming into and out of GitHub, and sent the data through its scrubbing centers to weed out and block malicious packets. After eight minutes, attackers relented and the assault dropped off.
The scale of the attack has few parallels, but a massive DDoS that struck the internet infrastructure company Dyn
in late 2016 comes close. That barrage peaked at 1.2 Tbps and caused
connectivity issues across the US as Dyn fought to get the situation
under control.
“We modeled our capacity based on
fives times the biggest attack that the internet has ever seen,” Josh
Shaul, vice president of web security at Akamai told WIRED hours after
the GitHub attack ended. “So I would have been certain that we could
handle 1.3 Tbps, but at the same time we never had a terabit and a half
come in all at once. It’s one thing to have the confidence. It’s another
thing to see it actually play out how you’d hope."
Real-time traffic from the DDoS attack.
Akamai
Akamai
defended against the attack in a number of ways. In addition to
Prolexic's general DDoS defense infrastructure, the firm had also
recently implemented specific mitigations for a type of DDoS attack
stemming from so-called memcached servers.
These database caching systems work to speed networks and websites, but
they aren't meant to be exposed on the public internet; anyone can
query them, and they'll likewise respond to anyone. About 100,000
memcached servers, mostly owned by businesses and other institutions,
currently sit exposed online with no authentication protection, meaning
an attacker can access them, and send them a special command packet that
the server will respond to with a much larger reply.
Unlike
the formal botnet attacks used in large DDoS efforts, like against Dyn
and the French telecom OVH, memcached DDoS attacks don't require a
malware-driven botnet. Attackers simply spoof the IP address of their
victim, send small queries to multiple memcached servers—about 10 per
second per server—that are designed to elicit a much larger response.
The memcached systems then return 50 times the data of the requests back
to the victim.
Known as an amplification attack, this type of DDoS has shown up
before. But as internet service and infrastructure providers have seen
memcached DDoS attacks ramp up over the last week or so, they've moved
swiftly to implement defenses to block traffic coming from memcached
servers.
"Large DDoS attacks such as those made
possible by abusing memcached are of concern to network operators," says
Roland Dobbins, a principal engineer at the DDoS and network-security
firm Arbor Networks who has been tracking
the memcached attack trend. "Their sheer volume can have a negative
impact on the ability of networks to handle customer internet traffic."
The
infrastructure community has also started attempting to address the
underlying problem, by asking the owners of exposed memcached servers to
take them off the internet, keeping them safely behind firewalls on
internal networks. Groups like Prolexic that defend against active DDoS
attacks have already added or are scrambling to add filters that
immediately start blocking memcached traffic if they detect a suspicious
amount of it. And if internet backbone companies can ascertain the
attack command used in a memcached DDoS, they can get ahead of malicious
traffic by blocking any memcached packets of that length.
"We
are going to filter that actual command out so no one can even launch
the attack," says Dale Drew, chief security strategist at the internet
service provider CenturyLink. And companies need to work quickly to
establish these defenses. "We’ve seen about 300 individual scanners that
are searching for memcached boxes, so there are at least 300 bad guys
looking for exposed servers," Drew adds.
'It’s one thing to have the confidence. It’s another thing to see it actually play out how you’d hope.'
Josh Shaul, Akamai
Most
of the memcached DDoS attacks CenturyLink has seen top out at about 40
to 50 gigabits per second, but the industry had been increasingly
noticing bigger attacks up to 500 gbps and beyond. On Monday, Prolexic
defended against a 200 gbps memcached DDoS attack launched against a
target in Munich.
Wednesday's onslaught wasn't the
first time a major DDoS attack targeted GitHub. The platform faced a
six-day barrage in March 2015, possibly perpetrated by Chinese
state-sponsored hackers. The attack was impressive for 2015, but DDoS
techniques and platforms—particularly Internet of Things–powered
botnets—have evolved and grown increasingly powerful when they’re at
their peak. To attackers, though, the beauty of memcached DDoS attacks
is there's no malware to distribute, and no botnet to maintain.
The web monitoring and network intelligence firm ThousandEyes observed
the GitHub attack on Wednesday. "This was a successful mitigation.
Everything transpired in 15 to 20 minutes," says Alex Henthorne-Iwane,
vice president of product marketing at ThousandEyes. "If you look at the
stats you’ll find that globally speaking DDoS attack detection alone
generally takes about an hour plus, which usually means there’s a human
involved looking and kind of scratching their head. When it all happens
within 20 minutes you know that this is driven primarily by software.
It’s nice to see a picture of success."
GitHub
continued routing its traffic through Prolexic for a few hours to ensure
that the situation was resolved. Akamai's Shaul says he suspects that
attackers targeted GitHub simply because it is a high-profile service
that would be impressive to take down. The attackers also may have been
hoping to extract a ransom. "The duration of this attack was fairly
short," he says. "I think it didn’t have any impact so they just said
that’s not worth our time anymore."
Until
memcached servers get off the public internet, though, it seems likely
that attackers will give a DDoS of this scale another shot.
Sign up here with your email
ConversionConversion EmoticonEmoticon