dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
37

Veloslave
Geek For God
Premium Member
join:2003-07-11
Martinez, CA

Veloslave to Almighty1

Premium Member

to Almighty1

Re: Poor handling by sonic.net support - DDoS attack

I just meant that since Dane is the Big Kahuna, that is where the buck stops. That is also where change will start.

In my EXCELLENT experiences with Sonic and Dane, I would be VERY surprised if something either wasn't done as is normally directed or policies will be changing.

I am the first to defend Sonic, maybe to a fault but I must admit that I am a little surprised that the 1.5M connection is running $50.00. I too have clients using Sonic on my recommendation and while that will not change, I might be more cautious about warning them on pricing. It sure does not help Sonic that pachell runs their $15.00 special all the time.
devedander
join:2002-12-15

devedander

Member

said by Veloslave:

I am the first to defend Sonic, maybe to a fault but I must admit that I am a little surprised that the 1.5M connection is running $50.00. I too have clients using Sonic on my recommendation and while that will not change, I might be more cautious about warning them on pricing. It sure does not help Sonic that pachell runs their $15.00 special all the time.
Glad I am not the only one who thinks it's a bit much...

Almighty1
Premium Member
join:2003-05-14
San Francisco, CA

2 edits

Almighty1 to Veloslave

Premium Member

to Veloslave
Yep, that's my point. Dane already resolved the problem when he is always competent like usual on Tuesday night, the day of the incident. So I don't know why Adam who is only a sonic.net employee even though he's Manager of Technical Support bothered to call me with more criticism when there really are no facts and basically threw what Dane basically said back in the water as Dane said I could have used my sonic.net dial-up as a backup, apparently people under Dane is trying to go against what Dane had said. I'll take Dane's word over Adam's anyday of the week since it's in writing like the old U.S. Sprint (This was the period when Sprint was previously known as GTE Sprint and got purchased by U.S. Telecom and before they were known as Sprint) commercial slogan says to "Get It In Writing". Customers are what built sonic.net to what it is today and being the loyal customer that I am, I will continue to use sonic.net even though I have changed my review here on DSL Reports because of the handling of the matter the other day.

I remember when you had the 6Mbps offering at the really high price ($100+/month) with PacHell/SBC and all you had to go through but that was expected since we all know how PacHell is as far as being a ISP. It's just shocking when it's performed by sonic.net. I know DSLExtreme has gone down the tubes when my friend ordered DSL from them after their acquisition by someone larger. My friend was getting 128kbps on a 3.0Mbps circuit and it didn't make a difference when they lowered him to 1.5Mbps or 768kbps, it was still 128kbps and they never resolved it even after sending AT&T out who replaced the DSL splitter with filters and charged him for it, that sounds so backwards. He was the first to get PacHell DSL back in 1990 when it was first available in the Sunset neighborhood of San Francisco before it was available in other neighborhoods and he had a static IP with reliable 1.5Mbps service. I tried to convince him to sign up with sonic.net and he was worried he would have the same issue so he ended up with Comcast and he's happy so far until probably those new charge for bandwidth policies hit him.

So sonic.net & Dane should really take customer complaints as constructive criticism as I am a Toastmaster now so I do really evaluate to the extreme and hopefully customer feedback will help sonic.net improve in their policies as well as other things.

Perhaps, because DSLExtreme's standards has gone down, sonic.net can go down in excellence too as far as standards are concerned since they just need to be better than the best competitor out there which can mean 0.0000000000000000000000001% better. I'm a Astrophysicist so I do measure things at a level below what most people would.

As for pricing, it really depends on what the costs are for sonic.net since there is no such thing as a free lunch and what the benefits are compared to the competition and then you vote with your wallet since there are always trade-off's when going with the competition, including DSLExtreme and it depends on what kind of user you are, etc. It's a lot of factors just like UUNet is one of the most expensive backbones but they are still alive and have a ton of customers. Dane has probably explain it before that the promo pricing is a loss leader product which is no different than how Pacific Telesis (parent company of PacHell) about a decade ago under the CalREN (California Research and Education Network) project gave free T1 internet circuits without the transit (no ISP) for 18 months and then slap you with a $2500/month charge after the 18 months is over, this is excluding ISP charges so you had to find someone to connect to at the other end.

So sonic.net has to factor in the lowest price they are able to provide the services for without losing money, otherwise, we wouldn't have seen sonic.net celebrate it's most recent 14th birthday!
devedander
join:2002-12-15

devedander

Member

said by Almighty1:

So I don't know why Adam who is only a sonic.net employee even though he's Manager of Technical Support bothered to call me with more criticism when there really are no facts and basically threw what Dane basically said back in the water as Dane said I could have used my sonic.net dial-up as a backup, apparently people under Dane is trying to go against what Dane had said.
I think we have all seen small, wondeful businesses undergo a negative support trend as they grow in the past, I have to wonder if Sonic is experiencing the same.

I have noticed over the years, that while support is generally quick to pick up and personable, the level of their answers has gone down a lot. I get a lot of "Well I would think" and "I am pretty sure" when asking questions and they aren't the kind that sounds confidend. I have had to resort to following up most of my questions with "can we verify that somehow?" which really shouldn't have to be asked... if the support person doesn't know, they should just say please wait while I verify.

I think Dane is probably still driving sonic in the right direction, but maybe his staff is getting big enough or his attention is being placed away from verifying his staffs perforamance (so easy as a business grows) that they aren't necessarily going in the same direction.

I heard a saying when I was growing up in China that seems to ring true more and more often, but never so much so as in a business that is growing:

The Mountains Are High and the Emperor is Far Away

Almighty1
Premium Member
join:2003-05-14
San Francisco, CA

3 edits

Almighty1

Premium Member

My earlier reply never made it so I'm doing this from memory.

I think with the issue with growth we've seen here is the left hand does not know what the right hand is doing which I didn't think before Tuesday, July 29, 2008 that I would ever say about sonic.net.

A good example is what happened in my situation on Tuesday, July 29, 2008. The DDoS (Distributed Denial of Service) attacks started at 7:43AM and basically went indefinitely until I called sonic.net support at 9:15AM. If I didn't call at all, the DDoS attack would probably have went indefinitely and probably had resulted in a real big event that would have affected atleast other sonic.net DSL customers if not their co-location customers, business customers with dedicated circuits or even sonic.net's servers that would have really caused chaos because it's obvious that the NOC dropped the ball in monitoring the network and the attacker would have probably just attacked everything inside sonic.net's network one by one as obviously, the entire NOC is out of commission and that is the best time to attack when people are not watching... Not to mention that despite that Dane saying that he heard it was a 125,000 packet/sec attack which has not been verified so far and Adam claiming that this also affected other customers, the NOC's monitoring of the network has really gone south and dropped the ball big time. And fast forward and it's two days after the incident, unless other sonic.net users know something I don't, there is still no MOTD's on the webpage or by e-mail concerning this event which is something one would expect from sonic.net given their excellent track record.

So let's look at the persons I talked to. We have Kory in support who told me that the downtime would be a few hours so I assume noon would be when my connection would be working and then I have talked to Jonathan who said the operations guy is out to lunch after putting me twice on hold for 15+ minutes each and said he will call me back with more information as there were no notes which he failed to even after 6 hours and using Dane's words, he basically dropped the ball. So just using what Jonathan said about the operations guy, one would think that sonic.net's NOC is a one man operation with no one else should he be unavailable for whatever reason such as this case. The mission of the NOC is to always be on alert, escalate issues and be prompt in resolving issues as well as being a effective communicator with their associates at sonic.net if not the customer. I'll give a example here which is a real story.

I own/run a ISP in Hawaii and also run a ISP in Beverly Hills (network of the founders of Concentric Network Corporation now known as XO Communications) all from my keyboard remotely in the San Francisco Bay Area or wherever in the world I happen to be if I'm on a trip and whenever there is a issue with the circuit, DDoS/DoS attack, hackers, I am always there to resolve the issue and don't sleep until the problem is resolved. I am the NOC and the only one running the NOC so I am always on alert and put all issues on priority in real-time as they happen because not only are there liability concerns from customers and whoever else but there is a reputation I have to protect. I was physically in Hawaii from May 3, 1997 until July 19, 1997 and this is to briefly mention the only issue ever that has gone unresolved. It is a GTE Internetworking (BBNPlanet) Frame Relay T1 circuit which was outsourced to UUNet as GTEI did not have a Point of Presence in Hawaii.
The circuit basically was doing 5kbps 80% of the time so we had to escalate the issue on day 1 which was May 3, 1997 until July 19, 1997 which involved not only the GTEI/BBNPlanet and UUNet NOC's but also the Director of Operations of both companies as well as GTE Hawaiian Telephone Company. Ofcourse after working 24 hours a day for that entire period and having a trouble ticket that when printed is over 250+ pages, we basically gave up as it was not going anywhere. Over the years, we had our shares of hackers, attackers and never once had I dropped the ball on our over 5,000+ clients with co-location, DSL, Wireless, dial-up connections. Unlike the users of sonic.net, if their connection is even down for 1 minute, the phone will ring off the hook. Remember, I am a one person NOC and I have never failed, it's surpising how sonic.net really failed this time as far as my latest experience goes.

I like to call Kory in technical support, Jonathan in technical support, and Adam who is Manager of technical support, as the AM (daytime) people as the AM people were the ones who dropped the ball. The PM (night) people such as Dane the CEO and Tristan in technical support are the ones who happily resolved the problem and what's funny is that the AM gang and the PM gang have exact opposite answers of the sonic.net policy. The AM gang claims that all sonic.net services is supposed to be turned off which includes webmail and dial-up access while the PM gang believes that I am supposed to have webmail and a dial-up access which is provided for backup.

So after waiting until 6PM, I called Tristan who said that my webmail and dial-up access was not supposed to be restricted and he apologized for that which was fine. He the talked to operations and came back and said it should work now. Before I called him, I can ping/traceroute as far as the gateway and got the sonic.net security lockdown page no matter where I tried to surf but now, I can't even ping/traceroute the gateway and surfing would result in a timeout so I told him that my profile probably has to be rebuilt on the Redback SMS so then he put me on hold again and everything worked again so when he came back, I asked him if it was before the Redback SMS needed a reboot and he said that was exactly it so I thanked him and all was well until I saw Kory's e-mail on webmail saying they can't turn my connection as they are still researching the reason for the attack, that was when I responded and CC'ed Dane and Dane was tops as far as response goes! sonic.net support should probably note accounts of users who either participates on DSLReports or sonic.net newsgroups because these are the customers who are tech savvy and really knows the internals of sonic.net and how things work. Let's roll the clock back to 2004 when the 6Mbps/608Kbps connection was first offered. If it wasn't for me, JohnInSJ, and other sonic.net users as well as a DSLExtreme user, sonic.net would have never known about the profiling issue needed on the Redback SMS for both CO and Remote Terminal connections so that the speeds are stable instead of being spotty and this was all thanks to the DSLExtreme user who had similar issues at DSLExtreme, I can't remember his name right now. Oh, his name is deblin.

But in any case, Dane and sonic.net should remember that while sonic.net is big today over the past 14 years and counting, the bread and butter of the company is it's customers and the last thing you want to do is to get customers upset since customer retention is more important than new customers as a big majority of sonic.net's customers are due to word of mouth and whenever a customer gets upset due to the action of someone at sonic.net, the word spreads faster than you might think which results in customers leaving and also new customers probably going elsewhere instead due to the reviews and feedback from those ex-customers or current customers with bad experience.

As far as the quality of support levels go. I think the reason is because if I'm correct, DSLExtreme is their largest competitor with a excellent level of service so when DSLExtreme's level of service went down, sonic.net's crew probably has the thinking that as long as they are even a atom better than DSLExtreme is then they are doing good.

As I am Chinese as well, there is the other saying which is also used in the english language. There is always mountains higher. Basically, what it means is that even though you might already be at the highest level, you can always exceed your previous record and even be better than you already are previously since when it comes to going up,there are no limits as the limit upward is infinite or an umpty amount. However, we all know that 0 is the worst you can go downwards. I mean it doesn't take doing a MRTG type chart to monitor the performance of sonic.net's staff or even a stochastics graph like used in the financial world with minute, hourly, daily, monthly, 5 year, 10 year, 14 year charts to monitor so that it shows it going up and then a flat line and then up instead of going up and then rolling over which is bad as when it rolls over, this means that sonic.net is in serious trouble unless and they better get that chart moving up again or else the company will be in deep water type of trouble financially and you can look at either a merger and acquisition and should there be no suitors, it's time for the chapter 7 or chapter 11 bankruptcy.

So Dane needs to make sure his upper management is competent and then work the way down the ladder until the entire company is in sync with policies and being competent and make sure that everyone communicates with each other effectively. Speaking about communication, that was probably the source of the problem as there was no ETA of when the circuit would be back up and it seems in reality, it can be back up probably anytime sonic.net feels like it, probably a hour afterwards since if the attack was continuing while I was offline, it would probably have cause more damage already. If it was not for my 6PM call and talking to Tristan, I would probably have been offline until who knows when.

As far as communications go, I prefer written over telephone for several reasons. The most important is that it is documented and that what you say in written form can always be referenced later so there is no argument of you said this and the other person saying that they did not.

Adam for being the Technical Support Manager goes is probably more clueless than we give him credit for since
#1, sonic.net users who posts on sonic.net newsgroups and here on DSLR are tech savvy as I mentioned earlier and if we were going to excercise a act on someone, we are smart
enough to not use our own connections for the attack and for the other side to be able to find a clear trail back, not to mention, no one would be dumb enough to have the source on their DSL connection since attacks take bandwidth. If I had that much bandwidth to waste, both I and JohnInSJ and other users would not need to have Fair Queue Traffic Shaping on our circuits that we all spend so much time on to bespoke solutions. Not only that, I would launch the attack or whatever act from a source with lots of bandwidth instead since no one will notice anything and it will not affect others except those with smaller pipes.
devedander
join:2002-12-15

4 edits

devedander

Member

quote:
I think with the issue with growth we've seen here is the left hand does not know what the right hand is doing which I didn't think before Tuesday, July 29, 2008 that I would ever say about sonic.net.
I have to agree. I look at Danes actions, the direction the company has taken and can only conclude that overall Dane is trying to run a good company in the right direction. But like a militia that grows into a military, the good guys at the top start to become undermined by the numerous and sometimes weaker troops at the bottom.

My guess is that the marjor metric Dane uses to measure his staffs performance are reports from managers and customer satisfaction surveys (which managers will generally try to clean up before presenting in any business).

I bet if Dane would do some secret shopper type activity, he might be shocked what is really happening compared to what he has told people to do and what he is being told people are doing.

I remember a friend of mine recounting to me a story of a senior staff member at a hospital who was always sure they were doing everything right and anyone who complained was doing so wrongly. The numbers looked great and reports all said everything was top notch.

Then his step daughter came in for some care. She had a different last name and no one recognized her.

Let's just say there were some serious shake ups and changes in the weeks following.
quote:
Remember, I am a one person NOC and I have never failed, it's surpising how sonic.net really failed this time as far as my latest experience goes.
And that is why you have probably never let your customers down, because there is no one to blame but you if it happens.

When you can blame a manager, fire someone or cut a budget in response to a mistake, that's when many start to find details slipping.

Likewise, when you don'town the company and your drivinng force is your paycheck and not necessarily the big picture or achieving your dream by shaping and directing the company, it gets easier to rationalize actions that aren't necessarily in the best interest of the customer and the business.

In my onsite technology work I am usually overloaded and have to turn customers away at times because I can't give them quality service with the current workload (they usually wait and make do until I can get to them which says something I would think) but I won't hire anyone under me because I would feel 100% responsible for their level of service and I am just not comfortable slapping my name on someone elses work...

Sonic is getting bigger, and every company has an "expected" number of errors or unhappy customers per capita. As sonic grows it may have enough happy customers to unhappy ones to meet their expectations, but looking at the hospital story, it's not hard to see the logic behind "1 unhappy customer is 1 too many if you are that customer".

I hope sonic doesn't get too ahead of itself thinking that since sonic is the premium company, not the cheapest, it's acceptable to have unhappy customers because those who are unhappy must have only been here under the misguided belief sonic was the cheapest.

As long as we are tossing sayings around, someone once recounted to me that if you think 1% failure rate is acceptable, that would mean that you would accept 30 crashes a day at Chicago O'Hare Airport.

Almighty1
Premium Member
join:2003-05-14
San Francisco, CA

Almighty1

Premium Member

You have good points and I'm sure Dane actually gets good feedback when he is contacted by e-mail from customers or
via the DSLReports forums or even the sonic.net newsgroups, it's just a lot of the other sonic.net staff who you only
hear on the telephone and support@sonic.net e-mails that
really need to be more aware of things.

I don't let my customers down for other reasons too since basically I built everything from scratch including routers which are bespoke FreeBSD boxes with DS1-OC3 interface cards which works wonders around Cisco provided solutions so I don't want to see what I built fail.

Even when I own the company, I haven't been paid either other than maybe $300 for the entire 12 years so it was more of helping people than thinking of it as a job as I am a Astrophysicist working for NASA where my income actually comes from except for the past few years when my daily income from investing beats what I make monthly at work.

I believe in doing things myself since I can let someone else read all the investment news daily for me but the bottom line is I will not learn anything at the end of the day, I rather have a text to speech program read it all to me instead which is a good idea as soon as I find one that's good.

That Chicago O'Hare Airport is a good one.... Do we have to bring the how each OS is different and let's not talk about Microsoft making cars... Just for laughs...

DOS: everybody pushes it till it glides, jumps on, and lets it coast till it skids... then jumps off, pushes, jumps back on, etc.

DOS w/QEMM: same as DOS but with more leg room to push.

MAC: all the stewards, stewardesses, captains, baggage handlers, etc., look the same, act the same, and talk the same. Every time you ask questions about details you are told you don't need to know, don't want to know, and everything will be done for you without knowing, so just shut up.

OS/2: to get on board you have to have your ticket stamped 10 different times by standing in 10 different lines; then you have to fill out a form that states how you want your seating arrangement to be--whether it should have the look and feel of an ocean liner, a passenger train, or a bus. If you are successful in getting on board and getting off the ground you have a wonderful, enjoyable trip... except for times when the rudder and flaps freeze stuck, in which case you have time to say your prayers and get your personal things in order before you crash.

Windows: nice colorful airport terminal, friendly stewards/stewardesses, easy access to a plane, uneventful takeoff.... then BOOM!

NT: everyone sits on the runway and forms the outline of a plane, then they just sit there and go "PHHLLZZZSST" like they're flying.

Unix: everyone brings one piece of the plane with them when they come to the airport. Then they go out on the runway and piece it together, all the time arguing about what kind of plane they are building.

Don't have one for Mac OSX yet as it didn't exist when the above was written but MacOSX is basically FreeBSD/Unix underneath with the Aqua GUI.
devedander
join:2002-12-15

1 edit

devedander

Member

I haven't visited the sonic newsgroups so I can't say but I would guess you are probably right about the level of info Dane recieves. At this point I can only speculate on where issues may be comming from, but I think we both have empirical data showing that the employees are probably not doing what Dane expects of them at least some times. Humans are imperfect, I just hope that when these imperfections are brought to attention Dane see's it as an opportunity to improve his staff, not as an opportunity to figure out what is wrong with the customer complaining that got him or her into that situation.

I like the airplane thing... I can just see a bunch of bearded suspender wearing guys screwing a hellicopter rotor onto a jet plane frame and swearing that if they just had access to the root it would work.

I just realized... you are actually a rocket scientist...

Almighty1
Premium Member
join:2003-05-14
San Francisco, CA

Almighty1

Premium Member

Yeah so I think even for this thread, it would atleast make other fellow sonic.net users aware what's the worst scenario they will be in if they get attacked for whatever reason so they can't say no one told them ahead of time and atleast they won't get a heart attack!