dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
4848
share rss forum feed


Almighty1
Premium
join:2003-05-14
San Francisco, CA

4 edits

Poor handling by sonic.net support - DDoS attack

Greetings everyone:

I normally praise sonic.net for all the years I've been with them but this time, they really dropped the ball and it's not Dane's fault. First of all, I hardly have time to do anything these days due to having to watch my grandmother 24 hours a day and researching on my investments.

On June 29, 2008 - at about 7:45AM, on my Windows box, .225, I saw my AOL Instant Manager (AIM) connection drop so I thought it was just the occassional AIM routing issue so I went to do some other things in the house and then came back at 8:50AM or so when I discovered that I couldn't even reach the net using traceroute or ping as I couldn't even see the other side of the DSL connection .1 so I called sonic.net support who told me I had a very large download going on and that is weird because I am the only person using 2 windows machines (.225 and .226) with the .231 being a wireless router and .224 is my outoing traffic shaping gateway. So I was looking at my trafshow on my FreeBSD box and I saw there was a bunch of UDP from two IP's that was flooding the link so I called sonic.net to ask them to stop it. Now before we go further, the only machines I use are .225 and .226 and I only use that for browsing. .224 is a smtp, dns server but even for that, for the last 2 years, the only thing I had time for is e-mail and even with that, I'm a few days behind since you know investment news by e-mail is a pain in the butt to read. So I called sonic.net to ask them to block the two source IP's at approximately 9:15AM, they told me they have to shut the entire connection down which seems to be drastic as only the .224 IP was being attacked which even at this minute, I still have no idea what happen because I did not do anything to anyone else as I don't even have the time for it. I was told the connection will be up by afternoon. But I had credit card bills to pay since I always pay on the due date so at 12:00PM, I decided to try to use the sonic.net dialup and basically, sonic.net locks the entire account including dial-up access and webmail even when I was able to use the tethering hack on my verizon wireless cellphone to connect to the net which was unreliable as I would get disconnected every 1-2 minutes while it works fine on the PDA. So I called sonic.net afterwards and talked to Jonathan who said he would call me back with more info which he never did.

So at 6:00PM, I called sonic.net and talked to Tristan who finally got my DSL up and running again.

I later e-mailed Dane who was actually competent. Today, I received a call from Adam who is the Manager for Techical Support and basically he said that it's sonic.net's policy to shut down the entire connection and remove the users account from sonic.net's databases for 24 hours when there is a attack as from experience it's always the user who done something to get attacked. That is almost like saying someone is guilty before proven innocent which is not how this country works. All sonic.net had to do was block the IP's from hitting the network since it's my circuit that can't handle the flood and not sonic's other customers who were also getting affected as there was no MOTD for this and it was obvious that NOC didn't even know about the issue until I called some 1 hour and 30 minutes later so if it did affect everyone else, we all know that there would have been a MOTD on it already. I was being attacked and thetreatment I got was that I was guilty and gotten all my services turned off as a result when there was no due dilligence performed at all since if I did anything, don't you think the attacker would have said what I did especially when I don't even know the attacker. So if this daily routine of visiting:
www.tigerfinancial.com, finance.yahoo.com, www.garfied.com, www.facebook.com, mail.live.com, mail.google.com, www.fidelity.com, www.schwab.com, www.scottrade.com and sending e-mail to only a few friends and also communicating with people at the above websites would generate a DDoS, I don't know what the world is coming to. Not to mention that there are always people who port scan and just decide to take down machines running some type of server such as smtp, dns. I have ran a ISP before and customers have been under DDoS attacks but that doesn't mean it's always the customer who is the guilty party. I can think of atleast a ton of different ways to remdy the situation without any downtime.

1) block the source ip's at sonic's border routers
2) shut down only the .224 routing but this will still have a effect of downtime since I have DNS and SMTP servers running and the SMTP is the big issue as I don't have a MX backup yet.

But shutting down the entire circuit would be drastic enough already but then shutting down even webmail and
dial-up access is bad. Apparently Adam and sonic.net support thinks I have too much time when it's exactly the opposite,I have more things to do than time allows and I am awake 23 hours per day and the internet connection is something I depend on to get my work done as I couldn't do any research on investments without going online. Atleast JohnInSJ has access to my FreeBSD box and can see all logs and stuff so it's not like I'm making up things. Correction, JohnInSJ doesn't have a account on my FreeBSD box, other people on DSLR do.

Looking at the logs in /var/log/messages on the FreeBSD box, it shows:

Jul 29 07:41:04 bigbang kernel: Connection attempt to UDP 208.201.x.224:12316
from 208.201.224.11:53
Jul 29 07:43:27 bigbang kernel: ttempt to UDP 208.201.x.224:5087 from 88.191.3
9.101:51517
Jul 29 07:43:27 bigbang kernel: Connection attempt to UDP 208.201.x.224:25972
from 88.191.39.101:51517
Jul 29 07:43:27 bigbang kernel: Connection attempt to UDP 208.201.x.224:42819
from 88.191.39.101:51517
Jul 29 07:43:27 bigbang kernel: Connection attempt to UDP 208.201.x.224:64824
from 88.191.39.101:51517
Jul 29 07:43:27 bigbang kernel: Connection attempt to UDP 208.201.x.224:56624
from 88.191.39.101:51517

So there are no events I can see before this that can trigger this other than my machine is a DNS server for my own domain and I am running BIND 9.5 with the latest patches.

The attack ended at 9:23:54 when sonic.net operations shut down my connection I think.

Jul 29 09:23:54 bigbang kernel: Connection attempt to UDP 208.201.x.224:46589 from 88.191.39.101:51517
Jul 29 09:23:54 bigbang kernel: Connection attempt to UDP 208.201.x.224:41600 from 88.191.23.112:64194

The only thing in the logs that happened the day before was a bunch of:

Jul 28 21:00:01 bigbang sshd: warning: /etc/hosts.allow, line 39: can't verify hostname: getaddrinfo(23.hosting-5.xtream.co.il, AF_INET) failed

and then

Jul 29 07:44:08 bigbang inetd[53957]: warning: /etc/hosts.allow, line 39: can't verify hostname: getaddrinfo(sd-5892.dedibox.fr, AF_INET) failed

I just wished sonic.net would handle things better in the future since the customer is not always the one to blame as there are always exceptions to every rule especially when a accusation cannot be proven which normally can be a case of libel/slander.

Now,let's reverse the situation and imagine that I was sonic.net's upstream and sonic.net is under a DDoS attack which has happened as can be seen in the MOTD's, UUNet Security didn't go out and turned off sonic.net's circuits since there is always a liability issue.

And the experience downtime did cost me money which I will not go into as I do online investing with free trades so calling the broker as Dane implied will be costly as every penny saved is a penny earned and then there is the lost of time.

And I haven't been to DSLReports for a few years due to the lack of time. People that know me will know that I will always get to the bottom of things regardless how long it takes and I will not sleep until that happens. Not to mention, no one in their right mind except for newbies would want to leave a clear path for any retaliation to be done which is why this can just be a random attack or because someone randomly attacked someone else and used IP spoofing which happens to be my FreeBSD box.

All I know at this point was that I was a victim of a attack which basically killed my WAN resources and then sonic.net operations made it worst by taking even backup resources away so hopefully other sonic.net users don't somehow get a random attack as it is understandable if the victim was the root cause of the issue to disable their connection but when it's nothing more than a hypothesis, then something is really wrong with the way the policies.
And actions speak louder than words, the only one who is guilty of any act of wrongdoing is the attacker, not the victim.

So I would like to give a thumbs up to Dane as the CEO and Tristan in Tech Support but thumbs down goes to Kory, Jonathan and Adam in Technical Support.

--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET


Veloslave
Geek For God
Premium
join:2003-07-11
Martinez, CA
kudos:1
Reviews:
·Comcast
·PHONE POWER

1 edit
Wow....

That was one of the Lee sisters for sure.

Sorry to hear about all of that Almighty1... hopefully Dane can make some changes so the innocent are not victimized by the company as well as the perpetrator; at the end of the day this all rests on his shoulders. I would expect that this experience will make them stronger & better... the whole "you did something to cause this" punishment of a day without IP is way out of bounds. Doesn't sound like the Sonic we all know, hopefully it was just an ex-employee that was being a jerk.
--
Mom was right.... I NEED fiber!


Almighty1
Premium
join:2003-05-14
San Francisco, CA

1 edit
Dane wasn't the issue since he has always done a great job... It's just the way they handle things and claim this can't be done and it's standard procedure, etc. That excuse will work for customers who had never been at the service provider level before. You can run a IRC server and everyone will try to hack it or DDoS it as we have experienced before. And then there are people who didn't know better and try to do something to others but they left a clear trail and there are always those who are innocent. I'm not a stranger to the internet and dnalogic.net is in all my signatures on the over forums on over 150+ websites I had visited in some point or another so it was not hard to find my IP. Technically, I could be dead and someone would still DDoS me at random. I just think that as a customer which is still a business relationship, sonic.net should not have gone killing the dial-up portion of the account and webmail since that really serves no purpose. If I was a corporate business and this happened, you can bet on it that sonic.net will have some form of legal action filed as it would be a disruption of business and there are always monetary losses when there is downtime.
Even for the ISP that I own and run, I always think about liability issues for each action I take since unless there was a documented complaint, shutting off a customers connection can really get us in trouble and under the water.
Unlike a business who isn't living, a person such as myself for whatever time loss, the time flows away like a river and will never return. Everything was fine when Dane and I were e-mailing each other since even Dane said I was supposed to have sonic.net dial-up access as a backup and while sonic.net live support was available, it kept saying no one is available so I left the message and ofcourse the response went to webmail which I can't access because my account is supposedly suspended for all sonic.net services.
And what I find amusing is that for a NOC that is supposed to monitor their networks stats, if I didn't call them, the DDoS would probably have gone indefinitely since I knew exactly how to fix the problem as the attack never got past the FreeBSD kernel but the circuit was being filled to capacity by the UDP packets. I still like to see what their research so far shows. Since it seems that if I didn't call, I could have just let the DDoS go and atleast I'll be able to use my dial-up sonic.net connection even though it might be slow but atleast I will be able to get
work done but what I ended up with is a permanent DDoS attack with the attacker being sonic.net.

From wikipedia, under the Surving an attack section:

»en.wikipedia.org/wiki/Denial-of-···e_attack

It's supposed to be for sonic.net to work upstream until the attacker or their provider is contacted instead of downstream and killing the customer instead.

--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET


Veloslave
Geek For God
Premium
join:2003-07-11
Martinez, CA
kudos:1
Reviews:
·Comcast
·PHONE POWER
reply to Almighty1
I just meant that since Dane is the Big Kahuna, that is where the buck stops. That is also where change will start.

In my EXCELLENT experiences with Sonic and Dane, I would be VERY surprised if something either wasn't done as is normally directed or policies will be changing.

I am the first to defend Sonic, maybe to a fault but I must admit that I am a little surprised that the 1.5M connection is running $50.00. I too have clients using Sonic on my recommendation and while that will not change, I might be more cautious about warning them on pricing. It sure does not help Sonic that pachell runs their $15.00 special all the time.
--
Mom was right.... I NEED fiber!

devedander

join:2002-12-15
said by Veloslave:

I am the first to defend Sonic, maybe to a fault but I must admit that I am a little surprised that the 1.5M connection is running $50.00. I too have clients using Sonic on my recommendation and while that will not change, I might be more cautious about warning them on pricing. It sure does not help Sonic that pachell runs their $15.00 special all the time.
Glad I am not the only one who thinks it's a bit much...


Almighty1
Premium
join:2003-05-14
San Francisco, CA

2 edits
reply to Veloslave
Yep, that's my point. Dane already resolved the problem when he is always competent like usual on Tuesday night, the day of the incident. So I don't know why Adam who is only a sonic.net employee even though he's Manager of Technical Support bothered to call me with more criticism when there really are no facts and basically threw what Dane basically said back in the water as Dane said I could have used my sonic.net dial-up as a backup, apparently people under Dane is trying to go against what Dane had said. I'll take Dane's word over Adam's anyday of the week since it's in writing like the old U.S. Sprint (This was the period when Sprint was previously known as GTE Sprint and got purchased by U.S. Telecom and before they were known as Sprint) commercial slogan says to "Get It In Writing". Customers are what built sonic.net to what it is today and being the loyal customer that I am, I will continue to use sonic.net even though I have changed my review here on DSL Reports because of the handling of the matter the other day.

I remember when you had the 6Mbps offering at the really high price ($100+/month) with PacHell/SBC and all you had to go through but that was expected since we all know how PacHell is as far as being a ISP. It's just shocking when it's performed by sonic.net. I know DSLExtreme has gone down the tubes when my friend ordered DSL from them after their acquisition by someone larger. My friend was getting 128kbps on a 3.0Mbps circuit and it didn't make a difference when they lowered him to 1.5Mbps or 768kbps, it was still 128kbps and they never resolved it even after sending AT&T out who replaced the DSL splitter with filters and charged him for it, that sounds so backwards. He was the first to get PacHell DSL back in 1990 when it was first available in the Sunset neighborhood of San Francisco before it was available in other neighborhoods and he had a static IP with reliable 1.5Mbps service. I tried to convince him to sign up with sonic.net and he was worried he would have the same issue so he ended up with Comcast and he's happy so far until probably those new charge for bandwidth policies hit him.

So sonic.net & Dane should really take customer complaints as constructive criticism as I am a Toastmaster now so I do really evaluate to the extreme and hopefully customer feedback will help sonic.net improve in their policies as well as other things.

Perhaps, because DSLExtreme's standards has gone down, sonic.net can go down in excellence too as far as standards are concerned since they just need to be better than the best competitor out there which can mean 0.0000000000000000000000001% better. I'm a Astrophysicist so I do measure things at a level below what most people would.

As for pricing, it really depends on what the costs are for sonic.net since there is no such thing as a free lunch and what the benefits are compared to the competition and then you vote with your wallet since there are always trade-off's when going with the competition, including DSLExtreme and it depends on what kind of user you are, etc. It's a lot of factors just like UUNet is one of the most expensive backbones but they are still alive and have a ton of customers. Dane has probably explain it before that the promo pricing is a loss leader product which is no different than how Pacific Telesis (parent company of PacHell) about a decade ago under the CalREN (California Research and Education Network) project gave free T1 internet circuits without the transit (no ISP) for 18 months and then slap you with a $2500/month charge after the 18 months is over, this is excluding ISP charges so you had to find someone to connect to at the other end.

So sonic.net has to factor in the lowest price they are able to provide the services for without losing money, otherwise, we wouldn't have seen sonic.net celebrate it's most recent 14th birthday!

devedander

join:2002-12-15
said by Almighty1:

So I don't know why Adam who is only a sonic.net employee even though he's Manager of Technical Support bothered to call me with more criticism when there really are no facts and basically threw what Dane basically said back in the water as Dane said I could have used my sonic.net dial-up as a backup, apparently people under Dane is trying to go against what Dane had said.
I think we have all seen small, wondeful businesses undergo a negative support trend as they grow in the past, I have to wonder if Sonic is experiencing the same.

I have noticed over the years, that while support is generally quick to pick up and personable, the level of their answers has gone down a lot. I get a lot of "Well I would think" and "I am pretty sure" when asking questions and they aren't the kind that sounds confidend. I have had to resort to following up most of my questions with "can we verify that somehow?" which really shouldn't have to be asked... if the support person doesn't know, they should just say please wait while I verify.

I think Dane is probably still driving sonic in the right direction, but maybe his staff is getting big enough or his attention is being placed away from verifying his staffs perforamance (so easy as a business grows) that they aren't necessarily going in the same direction.

I heard a saying when I was growing up in China that seems to ring true more and more often, but never so much so as in a business that is growing:

The Mountains Are High and the Emperor is Far Away


Almighty1
Premium
join:2003-05-14
San Francisco, CA

3 edits
My earlier reply never made it so I'm doing this from memory.

I think with the issue with growth we've seen here is the left hand does not know what the right hand is doing which I didn't think before Tuesday, July 29, 2008 that I would ever say about sonic.net.

A good example is what happened in my situation on Tuesday, July 29, 2008. The DDoS (Distributed Denial of Service) attacks started at 7:43AM and basically went indefinitely until I called sonic.net support at 9:15AM. If I didn't call at all, the DDoS attack would probably have went indefinitely and probably had resulted in a real big event that would have affected atleast other sonic.net DSL customers if not their co-location customers, business customers with dedicated circuits or even sonic.net's servers that would have really caused chaos because it's obvious that the NOC dropped the ball in monitoring the network and the attacker would have probably just attacked everything inside sonic.net's network one by one as obviously, the entire NOC is out of commission and that is the best time to attack when people are not watching... Not to mention that despite that Dane saying that he heard it was a 125,000 packet/sec attack which has not been verified so far and Adam claiming that this also affected other customers, the NOC's monitoring of the network has really gone south and dropped the ball big time. And fast forward and it's two days after the incident, unless other sonic.net users know something I don't, there is still no MOTD's on the webpage or by e-mail concerning this event which is something one would expect from sonic.net given their excellent track record.

So let's look at the persons I talked to. We have Kory in support who told me that the downtime would be a few hours so I assume noon would be when my connection would be working and then I have talked to Jonathan who said the operations guy is out to lunch after putting me twice on hold for 15+ minutes each and said he will call me back with more information as there were no notes which he failed to even after 6 hours and using Dane's words, he basically dropped the ball. So just using what Jonathan said about the operations guy, one would think that sonic.net's NOC is a one man operation with no one else should he be unavailable for whatever reason such as this case. The mission of the NOC is to always be on alert, escalate issues and be prompt in resolving issues as well as being a effective communicator with their associates at sonic.net if not the customer. I'll give a example here which is a real story.

I own/run a ISP in Hawaii and also run a ISP in Beverly Hills (network of the founders of Concentric Network Corporation now known as XO Communications) all from my keyboard remotely in the San Francisco Bay Area or wherever in the world I happen to be if I'm on a trip and whenever there is a issue with the circuit, DDoS/DoS attack, hackers, I am always there to resolve the issue and don't sleep until the problem is resolved. I am the NOC and the only one running the NOC so I am always on alert and put all issues on priority in real-time as they happen because not only are there liability concerns from customers and whoever else but there is a reputation I have to protect. I was physically in Hawaii from May 3, 1997 until July 19, 1997 and this is to briefly mention the only issue ever that has gone unresolved. It is a GTE Internetworking (BBNPlanet) Frame Relay T1 circuit which was outsourced to UUNet as GTEI did not have a Point of Presence in Hawaii.
The circuit basically was doing 5kbps 80% of the time so we had to escalate the issue on day 1 which was May 3, 1997 until July 19, 1997 which involved not only the GTEI/BBNPlanet and UUNet NOC's but also the Director of Operations of both companies as well as GTE Hawaiian Telephone Company. Ofcourse after working 24 hours a day for that entire period and having a trouble ticket that when printed is over 250+ pages, we basically gave up as it was not going anywhere. Over the years, we had our shares of hackers, attackers and never once had I dropped the ball on our over 5,000+ clients with co-location, DSL, Wireless, dial-up connections. Unlike the users of sonic.net, if their connection is even down for 1 minute, the phone will ring off the hook. Remember, I am a one person NOC and I have never failed, it's surpising how sonic.net really failed this time as far as my latest experience goes.

I like to call Kory in technical support, Jonathan in technical support, and Adam who is Manager of technical support, as the AM (daytime) people as the AM people were the ones who dropped the ball. The PM (night) people such as Dane the CEO and Tristan in technical support are the ones who happily resolved the problem and what's funny is that the AM gang and the PM gang have exact opposite answers of the sonic.net policy. The AM gang claims that all sonic.net services is supposed to be turned off which includes webmail and dial-up access while the PM gang believes that I am supposed to have webmail and a dial-up access which is provided for backup.

So after waiting until 6PM, I called Tristan who said that my webmail and dial-up access was not supposed to be restricted and he apologized for that which was fine. He the talked to operations and came back and said it should work now. Before I called him, I can ping/traceroute as far as the gateway and got the sonic.net security lockdown page no matter where I tried to surf but now, I can't even ping/traceroute the gateway and surfing would result in a timeout so I told him that my profile probably has to be rebuilt on the Redback SMS so then he put me on hold again and everything worked again so when he came back, I asked him if it was before the Redback SMS needed a reboot and he said that was exactly it so I thanked him and all was well until I saw Kory's e-mail on webmail saying they can't turn my connection as they are still researching the reason for the attack, that was when I responded and CC'ed Dane and Dane was tops as far as response goes! sonic.net support should probably note accounts of users who either participates on DSLReports or sonic.net newsgroups because these are the customers who are tech savvy and really knows the internals of sonic.net and how things work. Let's roll the clock back to 2004 when the 6Mbps/608Kbps connection was first offered. If it wasn't for me, JohnInSJ, and other sonic.net users as well as a DSLExtreme user, sonic.net would have never known about the profiling issue needed on the Redback SMS for both CO and Remote Terminal connections so that the speeds are stable instead of being spotty and this was all thanks to the DSLExtreme user who had similar issues at DSLExtreme, I can't remember his name right now. Oh, his name is deblin.

But in any case, Dane and sonic.net should remember that while sonic.net is big today over the past 14 years and counting, the bread and butter of the company is it's customers and the last thing you want to do is to get customers upset since customer retention is more important than new customers as a big majority of sonic.net's customers are due to word of mouth and whenever a customer gets upset due to the action of someone at sonic.net, the word spreads faster than you might think which results in customers leaving and also new customers probably going elsewhere instead due to the reviews and feedback from those ex-customers or current customers with bad experience.

As far as the quality of support levels go. I think the reason is because if I'm correct, DSLExtreme is their largest competitor with a excellent level of service so when DSLExtreme's level of service went down, sonic.net's crew probably has the thinking that as long as they are even a atom better than DSLExtreme is then they are doing good.

As I am Chinese as well, there is the other saying which is also used in the english language. There is always mountains higher. Basically, what it means is that even though you might already be at the highest level, you can always exceed your previous record and even be better than you already are previously since when it comes to going up,there are no limits as the limit upward is infinite or an umpty amount. However, we all know that 0 is the worst you can go downwards. I mean it doesn't take doing a MRTG type chart to monitor the performance of sonic.net's staff or even a stochastics graph like used in the financial world with minute, hourly, daily, monthly, 5 year, 10 year, 14 year charts to monitor so that it shows it going up and then a flat line and then up instead of going up and then rolling over which is bad as when it rolls over, this means that sonic.net is in serious trouble unless and they better get that chart moving up again or else the company will be in deep water type of trouble financially and you can look at either a merger and acquisition and should there be no suitors, it's time for the chapter 7 or chapter 11 bankruptcy.

So Dane needs to make sure his upper management is competent and then work the way down the ladder until the entire company is in sync with policies and being competent and make sure that everyone communicates with each other effectively. Speaking about communication, that was probably the source of the problem as there was no ETA of when the circuit would be back up and it seems in reality, it can be back up probably anytime sonic.net feels like it, probably a hour afterwards since if the attack was continuing while I was offline, it would probably have cause more damage already. If it was not for my 6PM call and talking to Tristan, I would probably have been offline until who knows when.

As far as communications go, I prefer written over telephone for several reasons. The most important is that it is documented and that what you say in written form can always be referenced later so there is no argument of you said this and the other person saying that they did not.

Adam for being the Technical Support Manager goes is probably more clueless than we give him credit for since
#1, sonic.net users who posts on sonic.net newsgroups and here on DSLR are tech savvy as I mentioned earlier and if we were going to excercise a act on someone, we are smart
enough to not use our own connections for the attack and for the other side to be able to find a clear trail back, not to mention, no one would be dumb enough to have the source on their DSL connection since attacks take bandwidth. If I had that much bandwidth to waste, both I and JohnInSJ and other users would not need to have Fair Queue Traffic Shaping on our circuits that we all spend so much time on to bespoke solutions. Not only that, I would launch the attack or whatever act from a source with lots of bandwidth instead since no one will notice anything and it will not affect others except those with smaller pipes.

--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET

devedander

join:2002-12-15

4 edits
quote:
I think with the issue with growth we've seen here is the left hand does not know what the right hand is doing which I didn't think before Tuesday, July 29, 2008 that I would ever say about sonic.net.
I have to agree. I look at Danes actions, the direction the company has taken and can only conclude that overall Dane is trying to run a good company in the right direction. But like a militia that grows into a military, the good guys at the top start to become undermined by the numerous and sometimes weaker troops at the bottom.

My guess is that the marjor metric Dane uses to measure his staffs performance are reports from managers and customer satisfaction surveys (which managers will generally try to clean up before presenting in any business).

I bet if Dane would do some secret shopper type activity, he might be shocked what is really happening compared to what he has told people to do and what he is being told people are doing.

I remember a friend of mine recounting to me a story of a senior staff member at a hospital who was always sure they were doing everything right and anyone who complained was doing so wrongly. The numbers looked great and reports all said everything was top notch.

Then his step daughter came in for some care. She had a different last name and no one recognized her.

Let's just say there were some serious shake ups and changes in the weeks following.

quote:
Remember, I am a one person NOC and I have never failed, it's surpising how sonic.net really failed this time as far as my latest experience goes.
And that is why you have probably never let your customers down, because there is no one to blame but you if it happens.

When you can blame a manager, fire someone or cut a budget in response to a mistake, that's when many start to find details slipping.

Likewise, when you don'town the company and your drivinng force is your paycheck and not necessarily the big picture or achieving your dream by shaping and directing the company, it gets easier to rationalize actions that aren't necessarily in the best interest of the customer and the business.

In my onsite technology work I am usually overloaded and have to turn customers away at times because I can't give them quality service with the current workload (they usually wait and make do until I can get to them which says something I would think) but I won't hire anyone under me because I would feel 100% responsible for their level of service and I am just not comfortable slapping my name on someone elses work...

Sonic is getting bigger, and every company has an "expected" number of errors or unhappy customers per capita. As sonic grows it may have enough happy customers to unhappy ones to meet their expectations, but looking at the hospital story, it's not hard to see the logic behind "1 unhappy customer is 1 too many if you are that customer".

I hope sonic doesn't get too ahead of itself thinking that since sonic is the premium company, not the cheapest, it's acceptable to have unhappy customers because those who are unhappy must have only been here under the misguided belief sonic was the cheapest.

As long as we are tossing sayings around, someone once recounted to me that if you think 1% failure rate is acceptable, that would mean that you would accept 30 crashes a day at Chicago O'Hare Airport.


Almighty1
Premium
join:2003-05-14
San Francisco, CA
You have good points and I'm sure Dane actually gets good feedback when he is contacted by e-mail from customers or
via the DSLReports forums or even the sonic.net newsgroups, it's just a lot of the other sonic.net staff who you only
hear on the telephone and support@sonic.net e-mails that
really need to be more aware of things.

I don't let my customers down for other reasons too since basically I built everything from scratch including routers which are bespoke FreeBSD boxes with DS1-OC3 interface cards which works wonders around Cisco provided solutions so I don't want to see what I built fail.

Even when I own the company, I haven't been paid either other than maybe $300 for the entire 12 years so it was more of helping people than thinking of it as a job as I am a Astrophysicist working for NASA where my income actually comes from except for the past few years when my daily income from investing beats what I make monthly at work.

I believe in doing things myself since I can let someone else read all the investment news daily for me but the bottom line is I will not learn anything at the end of the day, I rather have a text to speech program read it all to me instead which is a good idea as soon as I find one that's good.

That Chicago O'Hare Airport is a good one.... Do we have to bring the how each OS is different and let's not talk about Microsoft making cars... Just for laughs...

DOS: everybody pushes it till it glides, jumps on, and lets it coast till it skids... then jumps off, pushes, jumps back on, etc.

DOS w/QEMM: same as DOS but with more leg room to push.

MAC: all the stewards, stewardesses, captains, baggage handlers, etc., look the same, act the same, and talk the same. Every time you ask questions about details you are told you don't need to know, don't want to know, and everything will be done for you without knowing, so just shut up.

OS/2: to get on board you have to have your ticket stamped 10 different times by standing in 10 different lines; then you have to fill out a form that states how you want your seating arrangement to be--whether it should have the look and feel of an ocean liner, a passenger train, or a bus. If you are successful in getting on board and getting off the ground you have a wonderful, enjoyable trip... except for times when the rudder and flaps freeze stuck, in which case you have time to say your prayers and get your personal things in order before you crash.

Windows: nice colorful airport terminal, friendly stewards/stewardesses, easy access to a plane, uneventful takeoff.... then BOOM!

NT: everyone sits on the runway and forms the outline of a plane, then they just sit there and go "PHHLLZZZSST" like they're flying.

Unix: everyone brings one piece of the plane with them when they come to the airport. Then they go out on the runway and piece it together, all the time arguing about what kind of plane they are building.

Don't have one for Mac OSX yet as it didn't exist when the above was written but MacOSX is basically FreeBSD/Unix underneath with the Aqua GUI.
--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET

devedander

join:2002-12-15

1 edit
I haven't visited the sonic newsgroups so I can't say but I would guess you are probably right about the level of info Dane recieves. At this point I can only speculate on where issues may be comming from, but I think we both have empirical data showing that the employees are probably not doing what Dane expects of them at least some times. Humans are imperfect, I just hope that when these imperfections are brought to attention Dane see's it as an opportunity to improve his staff, not as an opportunity to figure out what is wrong with the customer complaining that got him or her into that situation.

I like the airplane thing... I can just see a bunch of bearded suspender wearing guys screwing a hellicopter rotor onto a jet plane frame and swearing that if they just had access to the root it would work.

I just realized... you are actually a rocket scientist...


Almighty1
Premium
join:2003-05-14
San Francisco, CA
Yeah so I think even for this thread, it would atleast make other fellow sonic.net users aware what's the worst scenario they will be in if they get attacked for whatever reason so they can't say no one told them ahead of time and atleast they won't get a heart attack!


DaneJasper
Sonic.Net
Premium,VIP
join:2001-08-20
Santa Rosa, CA
kudos:9
reply to Almighty1
I've been talking to my team about this situation, and it seems like the basic troubles involve mis-communication. When we're doing "normal" things, our processes work well, but when it comes to an attack like this, network ops and support didn't communicate well. One support rep also didn't follow up.

NOC's priority is stability for the majority of customers, so in the face of 125,000 packets per second, they shut down your circuit as part of the response. While in theory they could attempt to filter the traffic by specific IP, I think the concern is that begins a cat and mouse game with the attacker, and for the good of others, it's better to take the attacked IP out of service.

The situation is a quandary, particularly if there are repeated attacks.

When a customer is hit with a huge DDOS, they're offline, one way or the other. You can't squeeze Niagara falls down a garden hose.

Also, other customers are often affected. Sometimes it's customers on the same Redback aggregation device (a few thousand users), other times it's huge chunks of the network, affecting larger numbers. It takes some time to respond, coordinating with upstreams to basically drop incoming traffic to our customer's IP. The other customers are impacted during this response interval.

So what choice should a service provider make if a customer is repeatedly attacked. Say for example that once a week, thousands of other innocent customers are impacted for five to thirty minutes. The victim becomes part of the equation, and it's not a simple one.

Thankfully, we haven't had this sort of persistent attack, but we do have customers who have been attacked two or three times in the past.

It's an interesting dilemma. What's your opinion? Think of yourself as a customer suffering outages because someone else is being attacked. Then, think of it from the position of someone who is the target. Is your opinion the same? How do you anticipate other customers might feel?

After some reactions, I've got more to say on the topic. Please follow up!

-Dane


Veloslave
Geek For God
Premium
join:2003-07-11
Martinez, CA
kudos:1
Reviews:
·Comcast
·PHONE POWER
reply to Almighty1
IMO for the sake of polling,

I would gladly put up with some bandwidth problems while you guys were putting up a fight for another Sonic brother (or sister)

I say play some cat and mouse, if that is what it turns out to be, and tell the attacker that THEY lose and not the customer that would have had his connection cut otherwise. Seems too passive to me for you guys to just roll over (please forgive the saying, you know I respect you) and cut the customer... the net is filled with jerk-offs that would love for that to be the outcome... even if only for a few hours or a day.

I would be proud to be part of the battle, even if losing some pipe, with you guys are telling the scum of the net to go bother someone with a different ISP because you have got our backs.
--
Mom was right.... I NEED fiber!


Almighty1
Premium
join:2003-05-14
San Francisco, CA

4 edits
reply to DaneJasper
Thanks Dane for your response, it is always appreciated. Now before we go further, remember in one way or another,the case was considered resolved when
we e-mailed each other. Even though I said in the support@sonic.net e-mail that I CC'ed you originally that I was going to post on DSLReports.com, I changed my mind because of how you handled the matter. The only reason that I even posted about the incident was because Adam who is the Manager of Technical Support basically called the next morning with remarks that were uncalled for and basically it added more injuries to the wounds that was moving into the forgotten category, sooner or later and we wouldn't even have this thread if it wasn't for Adam.

Communications is always the key to resolving problems but when there is a lack of it, that is where the chaos starts. Everyone should atleast be up to date with the situation.

So speaking about the NOC, where was the NOC for the 1:30 minutes the attack was occuring since like what I said before, it seems like until I called twice and the second time talking to Kory, they were totally unaware of what was happening other than I have a huge download on my circuit and what was more weird was that I had to provide the IP's of the attacking source before they can even do anything. I'm sure this would have affected other clients since 90 minutes of attacks and the NOC didn't know is really dropping the ball in my book. As far as the response from the NOC, were the monitoring tools just not working for that 90 minutes since I'm sure if others were affected, someone would have called in and claimed their connection was slow or something and there was no MOTD concerning this even though there was one for earlier today of another DDoS attack on a customer here:
»corp.sonic.net/status/2008/08/01···nd-ddos/

hopefully it's not someone we know. How big of a attack was this as the MOTD entry did say it was handled promptly so my attack must be small compared to this and not as important as it was unnoticed for atleast 90 minutes and it never made it to the MOTD.

So while I realize that they can do it from a small level of filtering the IP being attacked to the severe solution of shutting off the entire circuit, both of which I did accept but there was really no reason to turn off other sonic.net services like basically anything with the login/password such as webmail and dial-up access as that is supposed to be the backup solution just in case the DSL is not working due to whatever reason. It's not like the customer will try to retaliate using a dialup since it will never work as your target will have more bandwidth that will put you in your place like there is no tomorrow.

As you can see in my posts in this thread, I do think from both sides. Otherwise, I would not be talking about the liability issue and the upsetting customers issue which is for the ISP side of things and then ofcourse from the customer side, the anxiety of the customers and those who are innocent which the victim may or may not be.

As far as my opinion, just like veloslave, I would gladly put up with some bandwidth problems while you guys put up the fight for another fellow sonic.net customer as sonic.net is just like a big family, we all hope things go smoothly just like in life but there are always people out there who want to cause trouble and basically portscans and tries to take down targets by launching DDoS attacks as well as hack into the targets.

Playing some cat and mouse might be needed but just don't try to really push it too hard. I'll give you a example. At one point, we ran a IRC Server for DALNet and we all know how people like to hack IRC Servers so one day, we saw a hacker on the system. Normally, what people would do is get into a nice little war and kick the hackers butt on the spot. Instead, our strategy was to have my business partners talk to the hacker in a friendly way just to stall time and keeping him busy while I figure out how he got in and patch the system at the same time and then booted his butt to the point of no return since I even blackholed his route so anything from his IP will never be able to connect and then also contacting both his ISP and the backbone provider of his ISP by e-mail so they deal with him.

If you just turn off the connection of the victim, the attacker will just get more excited because you just created a challenge so they now will try to attack
something else on the sonic.net sooner or later as it seems only the victim has been punished since technically speaking, you never gotten rid of the root cause yet which is the person launching the attacks so what do they do, they just continue doing their thing since no one went after them or taught them a lesson they will never forget.

Now let's replace a DSL customer with a co-location customer whose server happens to be under DDoS attack and assuming this was a big customer such as a bank or something, would you cut the connection of this customer because you know for one thing, they can easily sue you for liability and with the amount of money they have, they can hire an army of lawyers just to put you out of business. What would you do in that situation?

That was also the reason why I mentioned shell.sonic.net being DDoS in 2000/2001 as it's in the MOTD archive. How would sonic.net feel if UUNet did cut sonic.net's connection assuming that was the only connection for sonic.net, not sure if sonic.net had the Cable & Wireless USA/Internet MCI circuits back then.

I found the URL for the MOTD in question:

»corp.sonic.net/status/2000/05/

So in my case, I would be sonic.net and the actual sonic.net is now UUNet. sonic.net I doubt will just stay quiet since I'm sure sonic.net will go after UUNet for damages, disruption of service and whatever else the lawyers can think of.

Even here - »corp.sonic.net/status/2000/05/24···eir-598/

It did say that sonic.net was upset and will be pursuing the matter and that's spoken as the perspective as a victim as sonic.net was the one being attacked.

Using a real world example, let's say some foreign nation or organization attacked the United States such as 911 for example. Does this mean we should just shut whatever object that is being attacked down or should we go retaliate against the attacker?

When all the sonic.net customers unite, the power shown to the attacker is infinite. If you just shut down the victim's circuit, then all that tells the attacker is that they will not get in trouble since all sonic.net will do is kill the victim and not go after them so in their minds, they'll just attack sonic.net again and again since no one will go after them anyways.

As far as if a customer gets attacked like a numerous amount of times continuously, it really depends on how
much you value the customer and if the customer really did anything. Let's just say the customer is once again a big customer such as a bank, are you willing to face the consequences if you did pull the plug on the customer or would you rather work out some sort of solution for the customer and basically eliminate the hacker so the customer is a happy camper.

I should add that being part of the sonic.net family of customers who are supposed to be labeled technically knowledgeable, we would have more understanding when there are attacks, hacking, etc of any kind that there are times, when performance can be impacted but atleast we still have connectivity. It's only the people who are like unknowledgeable who would not be understanding and really jump all over your case when there is a issue with performance because of a attack.

--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET


DaneJasper
Sonic.Net
Premium,VIP
join:2001-08-20
Santa Rosa, CA
kudos:9
For the purposes of this exercise, consider that during the attack, other customers are offline, not just suffering poor performance. And, the hypothetical is a home DSL user, not a large enterprise connection.

Any additional comments? I will follow up with more when I am at a real keyboard, likely late today.

-Dane


Almighty1
Premium
join:2003-05-14
San Francisco, CA

3 edits
My response would be the same because let's say that the victim was one of the DSLR members and someone we know even though sonic.net doesn't release that information due to privacy issues but if there was a attack, other customers are offline, it still wouldn't feel right that the victim should get punished since there is always a chance someday that you're the one being attacked so you have to put yourself in their shoes too and think from both yurself as a affected customer and also fom the viewpoint of the customer who happens to be the victim.

It's better to just do whatever it takes to block the traffic at the edge because as long as the traffic can't make it into your network, it wouldn't really impact anyone. In other words, as long as the source IP is identified meaning the attacker, it's just easier blocking traffic all traffic from them because if you just take out the connection of the customer being attacked, the attacker can aways find another customer to attack, so what then are you going to do, keep taking down customer connections as each one is attacked which will probably end up with many upset customers. And besides, if the traffic is already blocked at the edge, how will it be able to still attack the target customer since from my point of view, it's basically better to just filter the root cause of all evil at the edge where you peer and have transit with the rest of the world because it would seem that if you have to take a customers conection down which means there is no traffic even making it to the Redback SMS to/from the customer and if the attacker used a new soure IP for the attack, the
targetted customer is now not part of the equation but you will still have a flood of traffic from the attacker to the Redback SMS, not to mention that they can still attack other customers even though the original target is down for the time being. It's like hackers, they hack one of your servers, you get rid of them. They come back on another server so it's better to just make sure they can't get into the network instead of a computer. Assuming you have a attacker who thinks like this, he will attack each DSL IP one at a time on sonic.net until it can't be attacked (i.e. connection shut off), I don't think anyone at Operations wants to go and shut the thousands of customer circuits down right after each one is attacked assuming this was happening 24/7 but instead, i's easier to just kill all traffic to your network from the attacker at the edge routers and if the attacker finds a new source for attacking, block that as well.

Let's use a real example, let's say that a person physically goes into attack one of the tenants in a building with let's say 10 tenants. Assuming that you can't get the tenant criminally or legally, would you lock basically block access at the tenant level or would it be better to block access of the attacker to all tenants at the entrance of the building assuming his objective is to attack the other 9 tenants but will continute to the next one only after he's been blocked access to the first one since unless you had 10 persons to guard each tenant, it's easier to just guard the main entrance since you don't have to worry that the attacker can be all over the building hiding somewhere.

So even assuming that the customer (victim) connection gets shutoff, which brings me back to the question, what's the real purpose of turning off the dial-up backup internet access since that will really make customers mad because I remember one of the purposes that every sonic.net DSL connection included a dial-up account just incase the DSL connection is not available as I recall you always mention that when perspective clients were worried about the "I'll be without internet access" for some time if I they switch from another ISP or if they relocate, etc.


DaneJasper
Sonic.Net
Premium,VIP
join:2001-08-20
Santa Rosa, CA
kudos:9
The lack of access to dialup is a side effect. We only have two locked statuses: "accounting", which means a bill is more than 30 days overdue, and "security", which is everything else. In most cases, security is used when customer's systems or accounts are compromised, and the lockout must be complete. Because "target of DOS" isn't a common occurrence, we don't have a separate category.

Regarding filtering, in a DDOS, which is most common, this isn't viable. The customer is simply offline, either because they're being flooded with traffic, or because we coordinate with upstreams to black-hole the target IP (the customer). It's a moot point - you're down.

More in another followup.

-Dane


DaneJasper
Sonic.Net
Premium,VIP
join:2001-08-20
Santa Rosa, CA
kudos:9
(Note that this information is general, and is not specific to this customer's case)

The question I posed was, "To what extent should we inconvenience many customers when one customer is the target of attack?"

It's a bit of a set up question, and I'm posing it to address the point that Adam made - that the NOC was in no hurry to put your link back online. That is the most inflammatory issue, I believe.

If a customer is attacked over and over again, should we terminate their access for the greater good? Also, should we suspend it temporarily in hopes of avoiding a recurrence of the attack from another direction which would again affect others?

In response, I'll say that the situation hasn't yet arisen where we've had enough repeated attacks directed at a single customer, in my opinion partly because of the choices that our NOC makes about response and restoration.

When a Sonic.net end-user is attacked by someone who has a large enough bot network under their control to require that we react, something is behind it. Absent a typo of IP address, systems are attacked for a reason. If only to keep the bot net viable (the most it's used, the more bots are likely to get fixed), they are not used without reason.

In almost all cases, the customer falls into one of two categories. Either they run a Unix like host on their network, or they participate in IRC.

In the first case, we find that often customers are running compromised systems, which are being used by third parties to source spam, to hack or source DOS, or to participate in the IRC. In the second case, we get the impression that someone pissed off someone in the IRC and is engaging in a power struggle.

I believe that the network ops team has found that by defending customers TOO well, the customers do not address the issue which triggered the attack. In other words, it wasn't much inconvenience to the target because the NOC was able to wake up, drag themselves to a terminal, coordinate with upstream providers, etc, and got the customer back online just as soon as the attack could be stopped.

Then, the customer (even many businesses!) simply doesn't make it a priority to resolve the root issue (generally a compromised host).

The other benefit of keeping an attacked customer offline for some time is that the attacker goes away having won, rather than repeatedly attacking from different vectors, each time taking down thousands of other customers. They move on with their day/night, go to sleep, forget about the insult or slight or attack or whatever.

So - for the good of all customers, sometimes those who are attacked may not be restored as quickly as might be possible. This prevents recurrences which would affect other customers, and also causes enough inconvenience that the customer is far more likely to resolve the root cause.

So that's the rest of the story. Time allowing, I always try to be as blunt as I can about the realities of the business, rather than simply doing spin control. That said, running a network is a bit like making sausage - you might not want to see what goes into it in all cases, and you may or may not agree with these steps.

-Dane


Almighty1
Premium
join:2003-05-14
San Francisco, CA

3 edits
reply to DaneJasper
Perhaps there should be two clases of security, one for inbound and one for outbound. Obviously, the outbound means the something bad came from the customers computer.
As for target of DDoS, it seems to be more common as if you looked at the MOTD, it seems like there is almost some type of DDoS attack every few months 3 to 4 months:

»corp.sonic.net/status/?s=DDoS

Atleast for this week, there would have been 2 DDoS attacks if you included my case.

I was just looking at a older thread »hmm... I seem to be offline from JohnInSJ See Profile and it seems like he got attacked too and what is it about all these attacks coming from France these days?

As far as filtering, if you don't filter, what happens to the traffic that was targetted at the customer that now doesn't exist as the customer is already offline but I'm sure it would still flood the rest of the sonic.net internal network and/or the other customers who share the same Redback SMS so in effect, it will still affect others unless you do something at the gateway.

So in response to the other post, what's the maximum length of time the customers link will be offline since it's better for the customer to know the worst case scenario instead of being in a panic situation because there is no ETA so assuming you said X amount of time which is the maximum possibility, there is always a chance it'll come back sooner.

Adam never made the point tht the NOC was in no hurry to put the link back online since all he did was said the customer was at fault to cause the issue and also the Network Admin is the one that decides on if the account itself should be locked up or not as it's on a case by case basis.

Now, if a customer is attacked over and over again, this has to mean the customer has done something to upset the attacker. Since that's what I meant that putting the customer offline will not do any good if that was the case because let's imagine that the customer gets attacked again as soon as you put the customer back online since if the attacker was pissed, they would have something that automatically attacks as soon as they can ping the host in question. It's no different that I had a ping to the sonic.net DNS server so that I'll know when the connection is back up.

As for the question of if you should terminate the access for the greater good, it depends since what would happen if the customer was under the 1 year term or something, they would have the early termination fee so even if you tell the customer to go find a new ISP, is sonic.net going to eat the early termination fee because that would end up costing sonic or are you still going to past the early termination fee to the customer?

As far as the two categories, I know I'm on the Unix like host on the network but not participate in IRC since I have not used IRC for longer than I can remember. Actually, there can be more than two categories like posting on a forum, newsgroups or sending e-mails that the attacker found offensive. While forums don't display IP addresses, what happens if it was the admin of the forums who was doing the attacking?

As for the first case, I know a few months ago that I was getting all these bounces for e-mail that was supposedly from me except it was vince@dnalogic.net instead of vince@bigbang.dnalogic.net and basically the headers were forged so the original e-mail never originated from my network except they just forged the headers so that it would seem like it came from my network.

I always make it a priority to resolve the root issue because #1, I don't like my system compromise because unlike others who simply load a OS and can reformat, etc. All my work is on the system so I have to be crazy to not make it a priority as I don't want all my hard work to go down the drain especially when it's 15+ years of things that while it gets backed up at 4AM each morning to another HD on the machine, if the system is compromise, it's just too big of a risk. That was the reason why even when I talked to Kory initially and when put on hold, I looked at the trafshow output and also checked to see if there were any weird processes running since the later would tell me if the system is compromised or not.

I do have a question that never gotten answered, in my attack, was there a reason that the NOC didn't do anything after 7:43AM when the attacks started occuring and didn't even seem to have known about it until I called at 9:15AM since Kory even asked me to unplug the system and see if it made a difference or I can have the NOC shut down the circuit for a few hours. The former probably won't work because if it really was a targetted attack, I'll probably see the attack as soon as I plug it back in and in my mind, as long as I get th connection back by early afternoon, then it's fine. And if you could, can you provide more information on the attack in question other than the 125,000 packets per second since DDoS is really too generic as a smurf attack is still a DoS attack as curiousity kills.

--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET


Almighty1
Premium
join:2003-05-14
San Francisco, CA

1 edit
reply to Almighty1
I just had a thought that as strange as it may sound. Let's say that a sonic.net customer was being DDoS so that sonic.net either shuts their connection off or terminated their relationship with the customer. What happens if we had a disgruntled customer, we've heard about disgruntled employees and this disgruntled customer happens to be a hacker and basically he/she launches a attack on sonic.net's network and/or servers or even sonic.net brothers/sisters networks and/or servers as a act of retaliation and then assuming that if sonic.net blocks him, he has has a unlimited amount of sources to launch the attack. What would sonic.net do then?

--
Cheers,
Vince
DNA Logic Corporation
»www.DNALOGIC.NET


guhuna
5149.5
Premium
join:2001-03-31
Port Costa, CA
Shut the whole network down?

kmcmurtrie

join:2006-04-18
Sunnyvale, CA
kudos:1
Reviews:
·SONIC.NET
reply to Almighty1
The "shut down the victim" policy is troubling. I've been DDoSed a few times and none of them were personal or even intentional. It was a Windows virus with broken load distribution that was popular around 2006. My computer was targeted only because it had ports open that it was interested in hacking.