how-to block ads
Santa Rosa, CA
|reply to Almighty1 |
Re: Poor handling by sonic.net support - DDoS attack
The lack of access to dialup is a side effect. We only have two locked statuses: "accounting", which means a bill is more than 30 days overdue, and "security", which is everything else. In most cases, security is used when customer's systems or accounts are compromised, and the lockout must be complete. Because "target of DOS" isn't a common occurrence, we don't have a separate category.
Regarding filtering, in a DDOS, which is most common, this isn't viable. The customer is simply offline, either because they're being flooded with traffic, or because we coordinate with upstreams to black-hole the target IP (the customer). It's a moot point - you're down.
More in another followup.
Santa Rosa, CA
(Note that this information is general, and is not specific to this customer's case)
The question I posed was, "To what extent should we inconvenience many customers when one customer is the target of attack?"
It's a bit of a set up question, and I'm posing it to address the point that Adam made - that the NOC was in no hurry to put your link back online. That is the most inflammatory issue, I believe.
If a customer is attacked over and over again, should we terminate their access for the greater good? Also, should we suspend it temporarily in hopes of avoiding a recurrence of the attack from another direction which would again affect others?
In response, I'll say that the situation hasn't yet arisen where we've had enough repeated attacks directed at a single customer, in my opinion partly because of the choices that our NOC makes about response and restoration.
When a Sonic.net end-user is attacked by someone who has a large enough bot network under their control to require that we react, something is behind it. Absent a typo of IP address, systems are attacked for a reason. If only to keep the bot net viable (the most it's used, the more bots are likely to get fixed), they are not used without reason.
In almost all cases, the customer falls into one of two categories. Either they run a Unix like host on their network, or they participate in IRC.
In the first case, we find that often customers are running compromised systems, which are being used by third parties to source spam, to hack or source DOS, or to participate in the IRC. In the second case, we get the impression that someone pissed off someone in the IRC and is engaging in a power struggle.
I believe that the network ops team has found that by defending customers TOO well, the customers do not address the issue which triggered the attack. In other words, it wasn't much inconvenience to the target because the NOC was able to wake up, drag themselves to a terminal, coordinate with upstream providers, etc, and got the customer back online just as soon as the attack could be stopped.
Then, the customer (even many businesses!) simply doesn't make it a priority to resolve the root issue (generally a compromised host).
The other benefit of keeping an attacked customer offline for some time is that the attacker goes away having won, rather than repeatedly attacking from different vectors, each time taking down thousands of other customers. They move on with their day/night, go to sleep, forget about the insult or slight or attack or whatever.
So - for the good of all customers, sometimes those who are attacked may not be restored as quickly as might be possible. This prevents recurrences which would affect other customers, and also causes enough inconvenience that the customer is far more likely to resolve the root cause.
So that's the rest of the story. Time allowing, I always try to be as blunt as I can about the realities of the business, rather than simply doing spin control. That said, running a network is a bit like making sausage - you might not want to see what goes into it in all cases, and you may or may not agree with these steps.
San Francisco, CA
|reply to DaneJasper |
Perhaps there should be two clases of security, one for inbound and one for outbound. Obviously, the outbound means the something bad came from the customers computer.
As for target of DDoS, it seems to be more common as if you looked at the MOTD, it seems like there is almost some type of DDoS attack every few months 3 to 4 months:
Atleast for this week, there would have been 2 DDoS attacks if you included my case.
I was just looking at a older thread »hmm... I seem to be offline from JohnInSJ and it seems like he got attacked too and what is it about all these attacks coming from France these days?
As far as filtering, if you don't filter, what happens to the traffic that was targetted at the customer that now doesn't exist as the customer is already offline but I'm sure it would still flood the rest of the sonic.net internal network and/or the other customers who share the same Redback SMS so in effect, it will still affect others unless you do something at the gateway.
So in response to the other post, what's the maximum length of time the customers link will be offline since it's better for the customer to know the worst case scenario instead of being in a panic situation because there is no ETA so assuming you said X amount of time which is the maximum possibility, there is always a chance it'll come back sooner.
Adam never made the point tht the NOC was in no hurry to put the link back online since all he did was said the customer was at fault to cause the issue and also the Network Admin is the one that decides on if the account itself should be locked up or not as it's on a case by case basis.
Now, if a customer is attacked over and over again, this has to mean the customer has done something to upset the attacker. Since that's what I meant that putting the customer offline will not do any good if that was the case because let's imagine that the customer gets attacked again as soon as you put the customer back online since if the attacker was pissed, they would have something that automatically attacks as soon as they can ping the host in question. It's no different that I had a ping to the sonic.net DNS server so that I'll know when the connection is back up.
As for the question of if you should terminate the access for the greater good, it depends since what would happen if the customer was under the 1 year term or something, they would have the early termination fee so even if you tell the customer to go find a new ISP, is sonic.net going to eat the early termination fee because that would end up costing sonic or are you still going to past the early termination fee to the customer?
As far as the two categories, I know I'm on the Unix like host on the network but not participate in IRC since I have not used IRC for longer than I can remember. Actually, there can be more than two categories like posting on a forum, newsgroups or sending e-mails that the attacker found offensive. While forums don't display IP addresses, what happens if it was the admin of the forums who was doing the attacking?
As for the first case, I know a few months ago that I was getting all these bounces for e-mail that was supposedly from me except it was firstname.lastname@example.org instead of email@example.com and basically the headers were forged so the original e-mail never originated from my network except they just forged the headers so that it would seem like it came from my network.
I always make it a priority to resolve the root issue because #1, I don't like my system compromise because unlike others who simply load a OS and can reformat, etc. All my work is on the system so I have to be crazy to not make it a priority as I don't want all my hard work to go down the drain especially when it's 15+ years of things that while it gets backed up at 4AM each morning to another HD on the machine, if the system is compromise, it's just too big of a risk. That was the reason why even when I talked to Kory initially and when put on hold, I looked at the trafshow output and also checked to see if there were any weird processes running since the later would tell me if the system is compromised or not.
I do have a question that never gotten answered, in my attack, was there a reason that the NOC didn't do anything after 7:43AM when the attacks started occuring and didn't even seem to have known about it until I called at 9:15AM since Kory even asked me to unplug the system and see if it made a difference or I can have the NOC shut down the circuit for a few hours. The former probably won't work because if it really was a targetted attack, I'll probably see the attack as soon as I plug it back in and in my mind, as long as I get th connection back by early afternoon, then it's fine. And if you could, can you provide more information on the attack in question other than the 125,000 packets per second since DDoS is really too generic as a smurf attack is still a DoS attack as curiousity kills.
DNA Logic Corporation