(Ping! Zine Issue 50) – On a busy afternoon at HostingCon 2012 in Boston, I stood at a podium in a room of no fewer than 50 web hosting industry professionals, fumbling with my audio equipment as my eyes darted around the room searching for Matthew Prince of CloudFlare. I had already greeted Carlos Morales of Arbor Networks, Dr. Richard Zhao of the Asian security firm NSFOCUS, and my last minute selection, Matt Mahvi of Staminus Communications. Over the next 45 minutes the five of us began scratching the surface of the most pervasive threat facing hosting companies, threats to availability or distributed denial of service (DDoS) attacks as they’re better known.
I was rather naive to believe that my hasty brain trust could completely tackle this problem in such a short period of time. The hour was marked by brief, enlightening, and sometimes entertaining comments about the trials and tribulations of the handful of engineers who have stepped up to the challenge of keeping the internet online and responsive. This mini-symposium was the culmination of nearly 13 years of research and ventures which ultimately yielded what is colloquially referred to as the “DDoS industry.” Months prior, Arbor, Black Lotus, and CloudFlare each submitted a proposal for a DDoS mitigation talk. Empirically speaking, these types of presentations are little more than FUD cannons which leave a crowd of attendees frightened, confused, or somewhat annoyed. I wanted to do something different and teach the hosting industry how to defend itself against the avalanche which has been approaching for the past decade. What will happen when the hosting industry is completely eliminated?
The Threat Landscape
More than 10 years ago, the term DDoS mitigation was extremely uncommon. No one was searching for “DDoS protection,” and services like Prolexic, Verisign, and NeuStar did not yet exist. The art of defending web sites was guerilla warfare, a few guys from California (early pioneers, Black Lotus and Staminus) scooping up the customers no one wanted to deal with, the targets of DDoS attacks. So-called DDoS mitigation appliances did not exist, if you wanted to stop a DDoS attack at the turn of the century, you had to build something from scratch.
Fast forward to 2012, the problem is so pervasive that literally anyone can, and likely will become the target of a DDoS attack. The business of defending companies against DDoS attacks is growing at an alarming rate. From the time I returned from lunch today, I had orchestrated the turn-up of two GRE tunnel services (commonly known as “Clean Pipe” services) for hosts under active attack, fielded an emergency call from a large DNS provider, and educated a customer on the benefits of using a Layer 7 proxy to scrub extremely small, “low and slow” attacks. A Google News search for “DDoS” currently yields 11,400 results. WebHostingTalk is buzzing with discussions about DDoS attacks and many hosts are now advertising some form of DDoS protection as a standard feature, yet very few appear to have a comprehensive plan for dealing with this problem.
DDoS attacks are no longer a problem isolated to the target; they’re a threat to our entire industry. Customers are leaving hosts every day because of DDoS attacks, yet hosting industry entrepreneurs have not come to terms with this reality. An anecdote that I like to use to illustrate this point is the case of a pre-IPO venture hosted at one of the big brand server farms, the CEO of which assuring me just last year that the DDoS problem is not substantially pervasive, further indicating that the problem is easily solved with off the shelf hardware. A week later that company’s entire cloud infrastructure is hard down and the fledgling venture is asked to leave that same evening. This company was well funded and was in a position to call up the services of a DDoS mitigation company late in the evening, but others are not as fortunate.
DDoS attacks are growing in size and complexity at an alarming rate, yet the standard convention in the hosting industry continues to involve removing the customer, resulting in damage to the target, the hosting company, and the industry at large. As DDoS attacks scale, smaller hosts will be unable to cope and will begin losing their businesses to larger, more capable enterprise hosts. If today’s hosts are not prepared for this shake-up, our entire industry will regress. Competition will be limited as small proprietors will go out of business and mega-hosts begin scaling pricing to cope with the costs of hiring security engineers and over-provisioning systems and networks. DDoS mitigation services and mega-hosts will consolidate into unified solutions providers, perhaps out of necessity vice desire.
Education, Preparation, and Contingency
Don’t worry, it’s not all FUD. Hosts can and will survive this problem if properly equipped. It all starts with education. The answer is not to refer your customers to DDoS mitigation providers or suggest that they sit behind a cloud service. This strategy completely outsources the problem and indicates to the customer that their host is unable to cope with modern threats. Instead, hosts should become familiar with the types of DDoS attacks, how to recognize them, and basic techniques that can be used to mitigate these threats in-house:
Understand the OSI model and the layers at which each attack operates. Layer 3 and 4 attacks can be defeated at the network and service layers using ACL’s, policies, and commercially available DDoS mitigation appliances, whereas Layer 7 attacks require inspection by proxy. This is critical to identifying an attack and understanding the proper course of action.
Don’t feed the animals
Many attackers will contact their victims and make demands or simply taunt them for entertainment value. My experience indicates that ignoring these attempts often lessens the likelihood of the attack commencing or continuing.
Firewalls are stateful, meaning that they track the state of each connection passed through the device. This is actually a threat to availability as even the largest firewalls, such as high-end Cisco ASA or Juniper SRX systems, will buckle under the pressure of relatively small attacks.
Build systems with availability in mind
Hosts can manage and secure servers on behalf of customers, or build proxies using load balancers like nginx or haproxy, allowing the host to lessen the impact of low and slow Layer 7 attacks against the backend server. This is especially imperative when defending Windows Server customers which are substantially more susceptible to Layer 7 attacks than Linux counterparts.
Build secure networks
While this may seem like common sense, I often see hosting provider networks that are haphazardly constructed in a manner that cannot sustain DDoS attacks at the edge or core. Defending your customers is impossible when your own infrastructure buckles during an attack. Purchase modern equipment under valid service contracts, use models sufficient to withstand heavy attacks, and follow best practices to secure the devices themselves.
Have a contingency plan
Not all DDoS attacks will be simple, some may require professional help. Do you have security experts in-house? Have you contracted the services of a DDoS mitigation company for use in a worst case scenario? Can you activate this service in a moment’s notice?
The DDoS Mitigation Ecosystem
The commercialization of DDoS mitigation solutions has added a new layer of complexity to the problem. Today, everyone is a specialist, every piece of hardware is a standalone solution, and every service is a magic fix. Who should you trust with the future of your hosting business? Will you outsource the problem or champion it in-house? The correct answer depends on your understanding of your own company and the due diligence you have performed on your solutions partners. Bigger does not necessarily indicate better and a single vendor is rarely a holistic solution. Ideally, hosts should have a plan that involves education, in-house resources, basic self-defense, threat mitigation systems, and a robust partnership with a DDoS mitigation service provider.
Writer’s Bio: Jeffrey A. Lyon, CISSP is the President of Black Lotus Communications and an early pioneer in the mitigation of distributed denial of service (DDoS) attacks. He led a panel at HostingCon 2012 titled “Technology Strategies and Practices for Defending Against DDoS Attacks” and is a co-inventor of the Human Behavior Analysis (HBA) technology, a Layer 7 DDoS defense technology used to identify and validate real human users of an information system.