My earlier reply never made it so I'm doing this from memory.
I think with the issue with growth we've seen here is the left hand does not know what the right hand is doing which I didn't think before Tuesday, July 29, 2008 that I would ever say about sonic.net.
A good example is what happened in my situation on Tuesday, July 29, 2008. The DDoS (Distributed Denial of Service) attacks started at 7:43AM and basically went indefinitely until I called sonic.net support at 9:15AM. If I didn't call at all, the DDoS attack would probably have went indefinitely and probably had resulted in a real big event that would have affected atleast other sonic.net DSL customers if not their co-location customers, business customers with dedicated circuits or even sonic.net's servers that would have really caused chaos because it's obvious that the NOC dropped the ball in monitoring the network and the attacker would have probably just attacked everything inside sonic.net's network one by one as obviously, the entire NOC is out of commission and that is the best time to attack when people are not watching... Not to mention that despite that Dane saying that he heard it was a 125,000 packet/sec attack which has not been verified so far and Adam claiming that this also affected other customers, the NOC's monitoring of the network has really gone south and dropped the ball big time. And fast forward and it's two days after the incident, unless other sonic.net users know something I don't, there is still no MOTD's on the webpage or by e-mail concerning this event which is something one would expect from sonic.net given their excellent track record.
So let's look at the persons I talked to. We have Kory in support who told me that the downtime would be a few hours so I assume noon would be when my connection would be working and then I have talked to Jonathan who said the operations guy is out to lunch after putting me twice on hold for 15+ minutes each and said he will call me back with more information as there were no notes which he failed to even after 6 hours and using Dane's words, he basically dropped the ball. So just using what Jonathan said about the operations guy, one would think that sonic.net's NOC is a one man operation with no one else should he be unavailable for whatever reason such as this case. The mission of the NOC is to always be on alert, escalate issues and be prompt in resolving issues as well as being a effective communicator with their associates at sonic.net if not the customer. I'll give a example here which is a real story.
I own/run a ISP in Hawaii and also run a ISP in Beverly Hills (network of the founders of Concentric Network Corporation now known as XO Communications) all from my keyboard remotely in the San Francisco Bay Area or wherever in the world I happen to be if I'm on a trip and whenever there is a issue with the circuit, DDoS/DoS attack, hackers, I am always there to resolve the issue and don't sleep until the problem is resolved. I am the NOC and the only one running the NOC so I am always on alert and put all issues on priority in real-time as they happen because not only are there liability concerns from customers and whoever else but there is a reputation I have to protect. I was physically in Hawaii from May 3, 1997 until July 19, 1997 and this is to briefly mention the only issue ever that has gone unresolved. It is a GTE Internetworking (BBNPlanet) Frame Relay T1 circuit which was outsourced to UUNet as GTEI did not have a Point of Presence in Hawaii.
The circuit basically was doing 5kbps 80% of the time so we had to escalate the issue on day 1 which was May 3, 1997 until July 19, 1997 which involved not only the GTEI/BBNPlanet and UUNet NOC's but also the Director of Operations of both companies as well as GTE Hawaiian Telephone Company. Ofcourse after working 24 hours a day for that entire period and having a trouble ticket that when printed is over 250+ pages, we basically gave up as it was not going anywhere. Over the years, we had our shares of hackers, attackers and never once had I dropped the ball on our over 5,000+ clients with co-location, DSL, Wireless, dial-up connections. Unlike the users of sonic.net, if their connection is even down for 1 minute, the phone will ring off the hook. Remember, I am a one person NOC and I have never failed, it's surpising how sonic.net really failed this time as far as my latest experience goes.
I like to call Kory in technical support, Jonathan in technical support, and Adam who is Manager of technical support, as the AM (daytime) people as the AM people were the ones who dropped the ball. The PM (night) people such as Dane the CEO and Tristan in technical support are the ones who happily resolved the problem and what's funny is that the AM gang and the PM gang have exact opposite answers of the sonic.net policy. The AM gang claims that all sonic.net services is supposed to be turned off which includes webmail and dial-up access while the PM gang believes that I am supposed to have webmail and a dial-up access which is provided for backup.
So after waiting until 6PM, I called Tristan who said that my webmail and dial-up access was not supposed to be restricted and he apologized for that which was fine. He the talked to operations and came back and said it should work now. Before I called him, I can ping/traceroute as far as the gateway and got the sonic.net security lockdown page no matter where I tried to surf but now, I can't even ping/traceroute the gateway and surfing would result in a timeout so I told him that my profile probably has to be rebuilt on the Redback SMS so then he put me on hold again and everything worked again so when he came back, I asked him if it was before the Redback SMS needed a reboot and he said that was exactly it so I thanked him and all was well until I saw Kory's e-mail on webmail saying they can't turn my connection as they are still researching the reason for the attack, that was when I responded and CC'ed Dane and Dane was tops as far as response goes! sonic.net support should probably note accounts of users who either participates on DSLReports or sonic.net newsgroups because these are the customers who are tech savvy and really knows the internals of sonic.net and how things work. Let's roll the clock back to 2004 when the 6Mbps/608Kbps connection was first offered. If it wasn't for me, JohnInSJ, and other sonic.net users as well as a DSLExtreme user, sonic.net would have never known about the profiling issue needed on the Redback SMS for both CO and Remote Terminal connections so that the speeds are stable instead of being spotty and this was all thanks to the DSLExtreme user who had similar issues at DSLExtreme, I can't remember his name right now.
Oh, his name is deblin.
But in any case, Dane and sonic.net should remember that while sonic.net is big today over the past 14 years and counting, the bread and butter of the company is it's customers and the last thing you want to do is to get customers upset since customer retention is more important than new customers as a big majority of sonic.net's customers are due to word of mouth and whenever a customer gets upset due to the action of someone at sonic.net, the word spreads faster than you might think which results in customers leaving and also new customers probably going elsewhere instead due to the reviews and feedback from those ex-customers or current customers with bad experience.
As far as the quality of support levels go. I think the reason is because if I'm correct, DSLExtreme is their largest competitor with a excellent level of service so when DSLExtreme's level of service went down, sonic.net's crew probably has the thinking that as long as they are even a atom better than DSLExtreme is then they are doing good.
As I am Chinese as well, there is the other saying which is also used in the english language. There is always mountains higher. Basically, what it means is that even though you might already be at the highest level, you can always exceed your previous record and even be better than you already are previously since when it comes to going up,there are no limits as the limit upward is infinite or an umpty amount. However, we all know that 0 is the worst you can go downwards. I mean it doesn't take doing a MRTG type chart to monitor the performance of sonic.net's staff or even a stochastics graph like used in the financial world with minute, hourly, daily, monthly, 5 year, 10 year, 14 year charts to monitor so that it shows it going up and then a flat line and then up instead of going up and then rolling over which is bad as when it rolls over, this means that sonic.net is in serious trouble unless and they better get that chart moving up again or else the company will be in deep water type of trouble financially and you can look at either a merger and acquisition and should there be no suitors, it's time for the chapter 7 or chapter 11 bankruptcy.
So Dane needs to make sure his upper management is competent and then work the way down the ladder until the entire company is in sync with policies and being competent and make sure that everyone communicates with each other effectively. Speaking about communication, that was probably the source of the problem as there was no ETA of when the circuit would be back up and it seems in reality, it can be back up probably anytime sonic.net feels like it, probably a hour afterwards since if the attack was continuing while I was offline, it would probably have cause more damage already. If it was not for my 6PM call and talking to Tristan, I would probably have been offline until who knows when.
As far as communications go, I prefer written over telephone for several reasons. The most important is that it is documented and that what you say in written form can always be referenced later so there is no argument of you said this and the other person saying that they did not.
Adam for being the Technical Support Manager goes is probably more clueless than we give him credit for since
#1, sonic.net users who posts on sonic.net newsgroups and here on DSLR are tech savvy as I mentioned earlier and if we were going to excercise a act on someone, we are smart
enough to not use our own connections for the attack and for the other side to be able to find a clear trail back, not to mention, no one would be dumb enough to have the source on their DSL connection since attacks take bandwidth. If I had that much bandwidth to waste, both I and JohnInSJ and other users would not need to have Fair Queue Traffic Shaping on our circuits that we all spend so much time on to bespoke solutions. Not only that, I would launch the attack or whatever act from a source with lots of bandwidth instead since no one will notice anything and it will not affect others except those with smaller pipes.