Company Forums » Companies beginning with S » SnappyDSL.net (COFS.NET) > Outage?

reply to pcmike

Re: Outage?

reply to pcmike
I guess everyone has seen this by now but thought I would post it anyway..

Update: Sat. 22nd Jan 6:00pm

We are pleased to inform everyone that source of the problem has been identified and corrected. The final culprit was a server system which had been compromised and was causing havoc with the core router. All system are back to normal. Extended monitoring has shown that system performance is back to normal levels.

We would like to thank you all for your patience and cooperation.

I’d sure like to know what kind of setup they have, as something like that shouldn’t happen. I have some computers that can control my routers, but there is almost 0 chance of them being compromised, as they are not connected to the internet, and for good reason.

Also I am surprised to took them that long to figure this out, as the first thing I would have done in a situation where everything was knocked out, and I couldn’t see why, is bring everything down, and start bringing it up one piece at a time until I found what was killing it. If they had done that, when they switched on those servers they would have know real fast where the culprit was.

reply to pcmike
"...server system which had been compromised and was causing havoc with the core router."

In my opinion, I interpret that to mean that the compromised server was being using in a DDOS and was completely using all the available bandwidth. Though, it could also mean something else. At any rate I still have two questions for COFS:

(1) What was stored on the 'compromised server' (anything relating to user details?)
(2) How was this 'compromised server' causing 'havoc' with the 'core router'?

This whole situation wreaks of someone screwing up on the job and I just want to know that such an 'incident' will not take place in the future and that all my credit card information and the like is safe and secure.

I'd just like to also put in a good word for COFS. I've been with them for 2-3 years now and their service up until this point has always far surpassed everyone else I've been with. If past history is any judge of the future I don't think such an incident like this will be happening any time soon. Don't misjudge the tone of my postings and assume I'm somehow unhappy with COFS. I'm very much a happy customer, I just wish such things would not ever occur, but atlast every once in awhile 'shit breaks.'

I have collocated servers where I work, and it’s not uncommon to have several of them a year become compromised. I have had more than a few used for DDOS attacks, and although on a few occasions groups of compromised servers have managed to overload my circuits to the internet, I’ve never had any come remotely close to overloading the router it’s self. This is because the busses and backplane of the router has many times the throughput of any server I have ever seen. So it’s almost imposable for a DOS attack to overload it, and I would expect Snappy to have even a larger router than I have making theirs even harder to overload. Which makes it hard for me to believe that is was just a DOS attack that shut them down so completely.

My guess is either the server managed and controlled the core router, or the core router had firmware vulnerable to certain malformed packets that should have been updated. I had to update all my routers not long ago because of that, because the right packets sent them would have locked them up solid, but with the info so incomplete about what happened, we can only guess what happened for now.

Hello Everyone.

Sorry for the 'Lack of timely updates' unfortunately the series of events which happened and the sequence of these did create a very interesting experience.

To put the conspiracy theories to rest. Yes it was some sort a DOS attack, however it was coming from the inside going out. Unfortunately it was enough to cause a very strange type of overload on the router, (Not passing traffic over two independent 100mbs links). Making this look very much like a hardware / router issue. It took some time to go through the process of checking everything out, replacing router with spare unit, checking out the cables, changing the ports with the upstream provider, etc etc etc. ( Looks like some of you are very familiar with NOTA, boy I did not know that it takes them about 3 hrs to test out an Ethernet Cable !).

The source of the problem was not identified until Sat. afternoon, after a closer review of the system looking for a compromised host.

Sorry for the inconvenience. Certainly hope not to have issues as such again, I don't know about you, but working 12hrs straight through the night in the icebox of NOTA is not my idea of fun.

The compromised host has not been fully dissected yet, could not tell you what kind of compromise it was at the moment.

P.S. The routers we are using are Riverstone's, their behaviour was very strange, but seeing this type of behaviour for the first time, will make it easier to identify if this ever happens again. (Cisco might be easier to identify with DOS Attacks, they simply keep rebooting !)

Anyway. Thank you all for your patience, advice, and offers of assistance. I would not mind knowing who you are, just drop me a note via email.

Thanks.
Faisal

reply to pcmike
I'll assume that last part was for Dr-IP. I'm just a user!

pcmike, if you login is the same with us, then I know you.

-:)

Thanks for everything.

reply to pcmike
Yea, it is. Thanks for the service over the years. The only thing that would make it better would be 6mbit/768kbps in the coming years.

PCMike,
You might want to qualify that statement... How about a 6Meg/768k for about the same price(sub $100/month) range ?

Lets see what 2005 bring. you never know what surprise is looming inside the 'box'.

If $ was not an issuse, 6M/768k is available today, and should also be offering 10meg Ethernet shortly, but pricing is more inline for commercial ventures / business.

Faisal

Yeah, now if we could only have it offered in Daytona..:(