Sonic → Re: Poor handling by sonic.net support - DDoS attack

(Note that this information is general, and is not specific to this customer's case)

The question I posed was, "To what extent should we inconvenience many customers when one customer is the target of attack?"

It's a bit of a set up question, and I'm posing it to address the point that Adam made - that the NOC was in no hurry to put your link back online. That is the most inflammatory issue, I believe.

If a customer is attacked over and over again, should we terminate their access for the greater good? Also, should we suspend it temporarily in hopes of avoiding a recurrence of the attack from another direction which would again affect others?

In response, I'll say that the situation hasn't yet arisen where we've had enough repeated attacks directed at a single customer, in my opinion partly because of the choices that our NOC makes about response and restoration.

When a Sonic.net end-user is attacked by someone who has a large enough bot network under their control to require that we react, something is behind it. Absent a typo of IP address, systems are attacked for a reason. If only to keep the bot net viable (the most it's used, the more bots are likely to get fixed), they are not used without reason.

In almost all cases, the customer falls into one of two categories. Either they run a Unix like host on their network, or they participate in IRC.

In the first case, we find that often customers are running compromised systems, which are being used by third parties to source spam, to hack or source DOS, or to participate in the IRC. In the second case, we get the impression that someone pissed off someone in the IRC and is engaging in a power struggle.

I believe that the network ops team has found that by defending customers TOO well, the customers do not address the issue which triggered the attack. In other words, it wasn't much inconvenience to the target because the NOC was able to wake up, drag themselves to a terminal, coordinate with upstream providers, etc, and got the customer back online just as soon as the attack could be stopped.

Then, the customer (even many businesses!) simply doesn't make it a priority to resolve the root issue (generally a compromised host).

The other benefit of keeping an attacked customer offline for some time is that the attacker goes away having won, rather than repeatedly attacking from different vectors, each time taking down thousands of other customers. They move on with their day/night, go to sleep, forget about the insult or slight or attack or whatever.

So - for the good of all customers, sometimes those who are attacked may not be restored as quickly as might be possible. This prevents recurrences which would affect other customers, and also causes enough inconvenience that the customer is far more likely to resolve the root cause.

So that's the rest of the story. Time allowing, I always try to be as blunt as I can about the realities of the business, rather than simply doing spin control. That said, running a network is a bit like making sausage - you might not want to see what goes into it in all cases, and you may or may not agree with these steps.

-Dane