how-to block ads
San Francisco, CA
|reply to DaneJasper |
Re: Poor handling by sonic.net support - DDoS attack
Thanks Dane for your response, it is always appreciated. Now before we go further, remember in one way or another,the case was considered resolved when
we e-mailed each other. Even though I said in the email@example.com e-mail that I CC'ed you originally that I was going to post on DSLReports.com, I changed my mind because of how you handled the matter. The only reason that I even posted about the incident was because Adam who is the Manager of Technical Support basically called the next morning with remarks that were uncalled for and basically it added more injuries to the wounds that was moving into the forgotten category, sooner or later and we wouldn't even have this thread if it wasn't for Adam.
Communications is always the key to resolving problems but when there is a lack of it, that is where the chaos starts. Everyone should atleast be up to date with the situation.
So speaking about the NOC, where was the NOC for the 1:30 minutes the attack was occuring since like what I said before, it seems like until I called twice and the second time talking to Kory, they were totally unaware of what was happening other than I have a huge download on my circuit and what was more weird was that I had to provide the IP's of the attacking source before they can even do anything. I'm sure this would have affected other clients since 90 minutes of attacks and the NOC didn't know is really dropping the ball in my book. As far as the response from the NOC, were the monitoring tools just not working for that 90 minutes since I'm sure if others were affected, someone would have called in and claimed their connection was slow or something and there was no MOTD concerning this even though there was one for earlier today of another DDoS attack on a customer here:
hopefully it's not someone we know. How big of a attack was this as the MOTD entry did say it was handled promptly so my attack must be small compared to this and not as important as it was unnoticed for atleast 90 minutes and it never made it to the MOTD.
So while I realize that they can do it from a small level of filtering the IP being attacked to the severe solution of shutting off the entire circuit, both of which I did accept but there was really no reason to turn off other sonic.net services like basically anything with the login/password such as webmail and dial-up access as that is supposed to be the backup solution just in case the DSL is not working due to whatever reason. It's not like the customer will try to retaliate using a dialup since it will never work as your target will have more bandwidth that will put you in your place like there is no tomorrow.
As you can see in my posts in this thread, I do think from both sides. Otherwise, I would not be talking about the liability issue and the upsetting customers issue which is for the ISP side of things and then ofcourse from the customer side, the anxiety of the customers and those who are innocent which the victim may or may not be.
As far as my opinion, just like veloslave, I would gladly put up with some bandwidth problems while you guys put up the fight for another fellow sonic.net customer as sonic.net is just like a big family, we all hope things go smoothly just like in life but there are always people out there who want to cause trouble and basically portscans and tries to take down targets by launching DDoS attacks as well as hack into the targets.
Playing some cat and mouse might be needed but just don't try to really push it too hard. I'll give you a example. At one point, we ran a IRC Server for DALNet and we all know how people like to hack IRC Servers so one day, we saw a hacker on the system. Normally, what people would do is get into a nice little war and kick the hackers butt on the spot. Instead, our strategy was to have my business partners talk to the hacker in a friendly way just to stall time and keeping him busy while I figure out how he got in and patch the system at the same time and then booted his butt to the point of no return since I even blackholed his route so anything from his IP will never be able to connect and then also contacting both his ISP and the backbone provider of his ISP by e-mail so they deal with him.
If you just turn off the connection of the victim, the attacker will just get more excited because you just created a challenge so they now will try to attack
something else on the sonic.net sooner or later as it seems only the victim has been punished since technically speaking, you never gotten rid of the root cause yet which is the person launching the attacks so what do they do, they just continue doing their thing since no one went after them or taught them a lesson they will never forget.
Now let's replace a DSL customer with a co-location customer whose server happens to be under DDoS attack and assuming this was a big customer such as a bank or something, would you cut the connection of this customer because you know for one thing, they can easily sue you for liability and with the amount of money they have, they can hire an army of lawyers just to put you out of business. What would you do in that situation?
That was also the reason why I mentioned shell.sonic.net being DDoS in 2000/2001 as it's in the MOTD archive. How would sonic.net feel if UUNet did cut sonic.net's connection assuming that was the only connection for sonic.net, not sure if sonic.net had the Cable & Wireless USA/Internet MCI circuits back then.
I found the URL for the MOTD in question:
So in my case, I would be sonic.net and the actual sonic.net is now UUNet. sonic.net I doubt will just stay quiet since I'm sure sonic.net will go after UUNet for damages, disruption of service and whatever else the lawyers can think of.
Even here - »corp.sonic.net/status/2000/05/24···eir-598/
It did say that sonic.net was upset and will be pursuing the matter and that's spoken as the perspective as a victim as sonic.net was the one being attacked.
Using a real world example, let's say some foreign nation or organization attacked the United States such as 911 for example. Does this mean we should just shut whatever object that is being attacked down or should we go retaliate against the attacker?
When all the sonic.net customers unite, the power shown to the attacker is infinite. If you just shut down the victim's circuit, then all that tells the attacker is that they will not get in trouble since all sonic.net will do is kill the victim and not go after them so in their minds, they'll just attack sonic.net again and again since no one will go after them anyways.
As far as if a customer gets attacked like a numerous amount of times continuously, it really depends on how
much you value the customer and if the customer really did anything. Let's just say the customer is once again a big customer such as a bank, are you willing to face the consequences if you did pull the plug on the customer or would you rather work out some sort of solution for the customer and basically eliminate the hacker so the customer is a happy camper.
I should add that being part of the sonic.net family of customers who are supposed to be labeled technically knowledgeable, we would have more understanding when there are attacks, hacking, etc of any kind that there are times, when performance can be impacted but atleast we still have connectivity. It's only the people who are like unknowledgeable who would not be understanding and really jump all over your case when there is a issue with performance because of a attack.
DNA Logic Corporation
Santa Rosa, CA
For the purposes of this exercise, consider that during the attack, other customers are offline, not just suffering poor performance. And, the hypothetical is a home DSL user, not a large enterprise connection.
Any additional comments? I will follow up with more when I am at a real keyboard, likely late today.
San Francisco, CA
My response would be the same because let's say that the victim was one of the DSLR members and someone we know even though sonic.net doesn't release that information due to privacy issues but if there was a attack, other customers are offline, it still wouldn't feel right that the victim should get punished since there is always a chance someday that you're the one being attacked so you have to put yourself in their shoes too and think from both yurself as a affected customer and also fom the viewpoint of the customer who happens to be the victim.
It's better to just do whatever it takes to block the traffic at the edge because as long as the traffic can't make it into your network, it wouldn't really impact anyone. In other words, as long as the source IP is identified meaning the attacker, it's just easier blocking traffic all traffic from them because if you just take out the connection of the customer being attacked, the attacker can aways find another customer to attack, so what then are you going to do, keep taking down customer connections as each one is attacked which will probably end up with many upset customers. And besides, if the traffic is already blocked at the edge, how will it be able to still attack the target customer since from my point of view, it's basically better to just filter the root cause of all evil at the edge where you peer and have transit with the rest of the world because it would seem that if you have to take a customers conection down which means there is no traffic even making it to the Redback SMS to/from the customer and if the attacker used a new soure IP for the attack, the
targetted customer is now not part of the equation but you will still have a flood of traffic from the attacker to the Redback SMS, not to mention that they can still attack other customers even though the original target is down for the time being. It's like hackers, they hack one of your servers, you get rid of them. They come back on another server so it's better to just make sure they can't get into the network instead of a computer. Assuming you have a attacker who thinks like this, he will attack each DSL IP one at a time on sonic.net until it can't be attacked (i.e. connection shut off), I don't think anyone at Operations wants to go and shut the thousands of customer circuits down right after each one is attacked assuming this was happening 24/7 but instead, i's easier to just kill all traffic to your network from the attacker at the edge routers and if the attacker finds a new source for attacking, block that as well.
Let's use a real example, let's say that a person physically goes into attack one of the tenants in a building with let's say 10 tenants. Assuming that you can't get the tenant criminally or legally, would you lock basically block access at the tenant level or would it be better to block access of the attacker to all tenants at the entrance of the building assuming his objective is to attack the other 9 tenants but will continute to the next one only after he's been blocked access to the first one since unless you had 10 persons to guard each tenant, it's easier to just guard the main entrance since you don't have to worry that the attacker can be all over the building hiding somewhere.
So even assuming that the customer (victim) connection gets shutoff, which brings me back to the question, what's the real purpose of turning off the dial-up backup internet access since that will really make customers mad because I remember one of the purposes that every sonic.net DSL connection included a dial-up account just incase the DSL connection is not available as I recall you always mention that when perspective clients were worried about the "I'll be without internet access" for some time if I they switch from another ISP or if they relocate, etc.
Santa Rosa, CA
The lack of access to dialup is a side effect. We only have two locked statuses: "accounting", which means a bill is more than 30 days overdue, and "security", which is everything else. In most cases, security is used when customer's systems or accounts are compromised, and the lockout must be complete. Because "target of DOS" isn't a common occurrence, we don't have a separate category.
Regarding filtering, in a DDOS, which is most common, this isn't viable. The customer is simply offline, either because they're being flooded with traffic, or because we coordinate with upstreams to black-hole the target IP (the customer). It's a moot point - you're down.
More in another followup.
Santa Rosa, CA
(Note that this information is general, and is not specific to this customer's case)
The question I posed was, "To what extent should we inconvenience many customers when one customer is the target of attack?"
It's a bit of a set up question, and I'm posing it to address the point that Adam made - that the NOC was in no hurry to put your link back online. That is the most inflammatory issue, I believe.
If a customer is attacked over and over again, should we terminate their access for the greater good? Also, should we suspend it temporarily in hopes of avoiding a recurrence of the attack from another direction which would again affect others?
In response, I'll say that the situation hasn't yet arisen where we've had enough repeated attacks directed at a single customer, in my opinion partly because of the choices that our NOC makes about response and restoration.
When a Sonic.net end-user is attacked by someone who has a large enough bot network under their control to require that we react, something is behind it. Absent a typo of IP address, systems are attacked for a reason. If only to keep the bot net viable (the most it's used, the more bots are likely to get fixed), they are not used without reason.
In almost all cases, the customer falls into one of two categories. Either they run a Unix like host on their network, or they participate in IRC.
In the first case, we find that often customers are running compromised systems, which are being used by third parties to source spam, to hack or source DOS, or to participate in the IRC. In the second case, we get the impression that someone pissed off someone in the IRC and is engaging in a power struggle.
I believe that the network ops team has found that by defending customers TOO well, the customers do not address the issue which triggered the attack. In other words, it wasn't much inconvenience to the target because the NOC was able to wake up, drag themselves to a terminal, coordinate with upstream providers, etc, and got the customer back online just as soon as the attack could be stopped.
Then, the customer (even many businesses!) simply doesn't make it a priority to resolve the root issue (generally a compromised host).
The other benefit of keeping an attacked customer offline for some time is that the attacker goes away having won, rather than repeatedly attacking from different vectors, each time taking down thousands of other customers. They move on with their day/night, go to sleep, forget about the insult or slight or attack or whatever.
So - for the good of all customers, sometimes those who are attacked may not be restored as quickly as might be possible. This prevents recurrences which would affect other customers, and also causes enough inconvenience that the customer is far more likely to resolve the root cause.
So that's the rest of the story. Time allowing, I always try to be as blunt as I can about the realities of the business, rather than simply doing spin control. That said, running a network is a bit like making sausage - you might not want to see what goes into it in all cases, and you may or may not agree with these steps.
San Francisco, CA
|reply to DaneJasper |
Perhaps there should be two clases of security, one for inbound and one for outbound. Obviously, the outbound means the something bad came from the customers computer.
As for target of DDoS, it seems to be more common as if you looked at the MOTD, it seems like there is almost some type of DDoS attack every few months 3 to 4 months:
Atleast for this week, there would have been 2 DDoS attacks if you included my case.
I was just looking at a older thread »hmm... I seem to be offline from JohnInSJ and it seems like he got attacked too and what is it about all these attacks coming from France these days?
As far as filtering, if you don't filter, what happens to the traffic that was targetted at the customer that now doesn't exist as the customer is already offline but I'm sure it would still flood the rest of the sonic.net internal network and/or the other customers who share the same Redback SMS so in effect, it will still affect others unless you do something at the gateway.
So in response to the other post, what's the maximum length of time the customers link will be offline since it's better for the customer to know the worst case scenario instead of being in a panic situation because there is no ETA so assuming you said X amount of time which is the maximum possibility, there is always a chance it'll come back sooner.
Adam never made the point tht the NOC was in no hurry to put the link back online since all he did was said the customer was at fault to cause the issue and also the Network Admin is the one that decides on if the account itself should be locked up or not as it's on a case by case basis.
Now, if a customer is attacked over and over again, this has to mean the customer has done something to upset the attacker. Since that's what I meant that putting the customer offline will not do any good if that was the case because let's imagine that the customer gets attacked again as soon as you put the customer back online since if the attacker was pissed, they would have something that automatically attacks as soon as they can ping the host in question. It's no different that I had a ping to the sonic.net DNS server so that I'll know when the connection is back up.
As for the question of if you should terminate the access for the greater good, it depends since what would happen if the customer was under the 1 year term or something, they would have the early termination fee so even if you tell the customer to go find a new ISP, is sonic.net going to eat the early termination fee because that would end up costing sonic or are you still going to past the early termination fee to the customer?
As far as the two categories, I know I'm on the Unix like host on the network but not participate in IRC since I have not used IRC for longer than I can remember. Actually, there can be more than two categories like posting on a forum, newsgroups or sending e-mails that the attacker found offensive. While forums don't display IP addresses, what happens if it was the admin of the forums who was doing the attacking?
As for the first case, I know a few months ago that I was getting all these bounces for e-mail that was supposedly from me except it was firstname.lastname@example.org instead of email@example.com and basically the headers were forged so the original e-mail never originated from my network except they just forged the headers so that it would seem like it came from my network.
I always make it a priority to resolve the root issue because #1, I don't like my system compromise because unlike others who simply load a OS and can reformat, etc. All my work is on the system so I have to be crazy to not make it a priority as I don't want all my hard work to go down the drain especially when it's 15+ years of things that while it gets backed up at 4AM each morning to another HD on the machine, if the system is compromise, it's just too big of a risk. That was the reason why even when I talked to Kory initially and when put on hold, I looked at the trafshow output and also checked to see if there were any weird processes running since the later would tell me if the system is compromised or not.
I do have a question that never gotten answered, in my attack, was there a reason that the NOC didn't do anything after 7:43AM when the attacks started occuring and didn't even seem to have known about it until I called at 9:15AM since Kory even asked me to unplug the system and see if it made a difference or I can have the NOC shut down the circuit for a few hours. The former probably won't work because if it really was a targetted attack, I'll probably see the attack as soon as I plug it back in and in my mind, as long as I get th connection back by early afternoon, then it's fine. And if you could, can you provide more information on the attack in question other than the 125,000 packets per second since DDoS is really too generic as a smurf attack is still a DoS attack as curiousity kills.
DNA Logic Corporation