KatOak
VIP
join:2001-09-10
Seattle, WA
| Packet Loss/Latency Reports
Just a quick update here:
Many of you have provided excellent information regarding the packet loss and/or latency that you are either monitoring or directly experiencing in your Point of Presence.
I have passed all account information to our Engineering team and they are thoroughly investigating this situation at this time. I'm hopeful that we'll have more information to provide within the next couple of days.
We really appreciate the time that many of you have spent tracking and reporting this problem as it has been extremely helpful.
--
Kat Oak
Speakeasy
kat@speakeasy.net |
|
jaykaykay
4 Ever Young
Premium,MVM
join:2000-04-13
Scottsdale, AZ
 ·Speakeasy
| Great and glad to hear it. While I have not posted to this forum about my packet loss and latency problems, I have done so on the site. I am glad to note that it is a wider problem than just my own, and since I haven't been able to spend much time here of late, I hadn't even checked this forum.
--
JKK Age is a very high price to pay for my maturity. If I can't stay young, I can at least stay immature! |
|
JonW
join:2002-08-13
Herndon, VA | reply to KatOak
There are lots of problems all around the internet lately, I assume the multiple worms going around haven't quite been defeated. |
|
Endorphine
Endorphine
Premium
join:2002-09-30
Bremerton, WA | reply to KatOak
Thanks for the update Kat!
Hope they figure out what is going on with the border routers. |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| reply to KatOak
Comcast, Bay Area |
You -- and this post of yours -- are literally the reason I am still a customer. I had just logged in to MySpeakeasy to file a cancellation (yes, after all the hard work you, myself, and lots of other reps have put into dealing with the issues I've brought forth), but remembered I hadn't checked the SE forum in a week or so. I'm thankful I did.
The problem I have with all of this is that it's confusing SE's engineering crew. I am an engineer (system and networking), and have been since 1991. I guess I'm having trouble accepting that an engineering group simply can't pinpoint what/where the problem is.
I know you've probably heard all of this before, but it's just that, well... we're going on, what, almost 5 weeks of this now (in regards to SFO)? It's becoming hard to stomach. I shake my head every day when I stare at my graphs and see the same repetitious behaviour and over between the hours of 09:00 and midnight. I keep waiting to see some change in it, even something as trivial as the Window-of-Doom being audited to something smaller (or heck, even LARGER!).
To make matters worse, as of this evening, things have seemingly become downright foul. It's nearly 2 in the morning and I'm witnessing packet loss (huge margins might I add, suddenly appearing out of no where in clumps of literally 35-40%!), as well as the flippant latency all of us have been reporting. It's the first time I've ever seen it happen at this late of an hour. I even tried power-cycling my DSL bridge: no love.
I'm presently on my Comcast connection, but good lord -- you can tell from the graph I've posted just how absolutely HORRID Comcast is right now here in the Bay. This has been a long-standing problem with AT&T and Comcast for over 3 years now, and it's getting worse. It's been confirmed as well: it's over-saturation due to the populus here.
Despite my comments, I'm going to continue sticking with SE for as long as I can. It's just that... *sigh*... SBC doesn't seem to be having problems like this, or rather, at least not to this degree. I'm sure you can understand my frustration.
Please keep us updated, even with small things like "there was an engineering meeting today about this." I just like knowing someone is at least acknowledging what we're seeing as legitimate, since we (customers) fought so hard to get SE Support to accept this issue in the first place.
I haven't forgotten about your orchids either, by the way. I'm still keeping my word on getting you some. I just happen to be swamped due to work, and a lazy person to bat.
--
Making life hard for others since 1977. |
|
warcorp
join:2000-09-19
Novi, MI
clubs:
| Koitsu,
We seem to be having the same problem out of the Chicago POP as well so I feel your pain. Not nearly to the degree you are however and seems to be a problem starting at about 7:30 and going all the way till 4:00am ish the next day and it's regular as clockwork (for the past 4 or 5 weeks that I know of. This is the last 12 hours: |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Can you beat this?
This has been going on since about 23:30 or so. Prior to that, it was what I'm used to seeing (flippant latency, occasional packet loss (3-4%)).
You know we're sad individuals when we're comparing results of broken connectivity!
--
Making life hard for others since 1977. |
|
warcorp
join:2000-09-19
Novi, MI
clubs: |  YIKES!
Nope can't beat that. We are broken, but not THAT broken.... |
|
bhan261
join:2001-02-12
New York, NY
| reply to koitsu
Koitsu,
I don't know whether you are a saint or an idiot to endure this kind of poor service without some sort of offer for reduction in fee from SE. I (and others) are having problems in NYC but nothing this extreme.
But you are right that a continuing flow of information about what's being done to investigate and resolve the problem will go a long way to maintaining the peace. Let's hope SE follows through accordingly. |
|
Chicago_DSL6
join:2003-08-04
Palatine, IL
| reply to KatOak
Koitsu, I hope they at least waive the cancellation fee. I am glad that there is an end user in these forums that DOES actually know what they are talking about. The fact that you have presented this much information about your problem and still not gotten your issue resolved in unreal. I never wanted to go to the dark side (SBC), but I called them yesterday for status on install (moved over the weekend) and they mentioned how ALL of their DSL customers are getting doubled downstream. I do not know what you need your DSL connection for, but if it isn't for hosting, SBC might be your most "reliable" and inexpensive choice.
Again, nice work koitsu and best of luck! |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| 
09/04 @ 02:00 to 15:30 |
I don't go the hosting route; I have actual co-location for that. All I want here in the Bay is 1.5mbit down and 384kbit up (or higher -- but those are my minimum requirements). The download speed is pretty much a standard by now, and the upstream I mainly require for a) a fast Remote Desktop connection while at work, and b) being able to stream mp3s from my home box while at work (I use 96-320kbit VBR, so 256kbit upstream induces stuttering on occasion)).
So far SE has gone to great lengths to provide me with pretty much everything I need, including renegotiating my line speed (I was originally 1.5/768 syncing at 1.5/540 and had occasional sync problems, now I'm 1.5/384 -- but all of these network problems were still apparent throughout that fiasco too).
--
Making life hard for others since 1977. |
|
sh0V3L
join:2002-02-16
San Carlos, CA
|  reply to KatOak
Im really fed up i have regulars complainig about lag ( i run a ut2003 game server btw ) i dont blame the my connection is getting worse by the min when will they take this serious ive had and have tickets open nothing gets done same thing everytime from our side things are fine lol what an answer so check this out »/quality/nil/1152730
that really suxorz the sour cho cho man FIX IT GUYS!!!!!!!!!!!!!!!!!! THIS is an sdsl 768/768 circuit here $$$$
GET IT RIGHT please
FED UP!!!!!!! |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| 
SE, evening 09/04 |
Graph shows what's happening; large gray portion is due to me simply not running PP during that time (I was doing some hardware work on my workstation).
I also began witnessing the "stalling socket" problem I've documented here on the forum a few times (which leads me to believe that the problems I had with it back when I was 1.5/768 were the beginning stages of this whole thing!). It also took me a good 10-15 full minutes to download a battle.net update for Warcraft 3 (something that should take about 2 minutes max), due to all the packet loss.
Sadly, if it's not fixed by the end / near the end of this month, I'm going to have to cancel. Two months of this is just unacceptable, and I'm sure Kat can understand our frustrations.
I'll power up my DSL bridge a couple times a week to see how things are doing. I may end up sticking a spare gateway I have (Netgear RP614) on my DSL line just so I can run mtr against it from my co-lo to see if there's ever any improvement (otherwise a tr just times out due to nothing listening on the IP).
--
Making life hard for others since 1977.
[text was edited by author 2003-09-05 02:27:51] |
|
KatOak
VIP
join:2001-09-10
Seattle, WA
| reply to KatOak
UPDATE
We met last night and this morning about the packet loss and latency reports in a few of our POPs. We have a separate, unique concern and overriding issue in our Seattle POP that is still under investigation, so I cannot speak to that at this time – but I can say that the issue under investigation is only being exacerbated by the root cause of the issue across our network.
The crux of it is this: The MSBlaster worm and variants have caused an increase in unnecessary traffic by 30% - 50%, depending on the POP. This is impacting Speakeasy’s network and the Internet as a whole – resulting in both monitored packet loss and increased latency Internet-wide.
The original report, by brikholl on 8/7, was due to the circuit transitions that we performed in SFO on 8/6 and 8/7. From the information he provided, it appeared that the connection returned to normal and then the packet loss and latency increase returned on 8/13, two days after the MS Blaster worm was identified and began propagation.
Due to the worm activity, you may see degraded service during peak hours. This is not due to bandwidth constraints but rather to maxing out how many packets the router can forward per second.
The monitored packet loss graphs displayed in this forum recently tell part of the story, but can be misleading. Routers and servers treat “ping” (ICMP traffic) with a very low “handle” priority. In practice, this means that if a given router is busy handling other traffic, it can delay the response or even decide not to respond (which appears as packet loss). While some of our POPs have high usage during a few hours in the evening, San Francisco is one of our largest markets and has a high concentration of heavy usage throughout the day. Add the unnecessary worm traffic to this usage pattern and the dropped packets will increase accordingly. The general rule is that as long as you are reaching your destination host with no packet loss, or only a small jump in latency, you are not being affected by the current state of things. Latency will probably be higher than usual until these worms die down.
Regarding the SBC connection posed by koitsu & brikholl awhile back – We read through these threads and it appears that SBC is also seeing a significant hit due to these worms. At one point, a rep states that they isolated this into one of the routers in one of their CO's and that it was resolved, but we have no further information on the state of this.
If SBC was having problems with equipment related specifically to their DSL services, our customers shouldn't have been impacted at all. However, if the problems were related to their backbone/Internet services, we may have seen some impact, assuming Internap was routing through SBC at the time – which is highly unlikely.
So, that’s the diagnosis – this is what we’re doing to fix it:
Our abuse team is notifying customers reportedly infected with this worm and will work with them to secure their network. We’re going to aggressively work this until we can reduce the number of infected computers on our network by at least 75%. Industry experts have stated that this worm has peaked and is on the decline, although we haven’t seen any real world evidence of that quite yet.
As you know, we are in the process of a major network upgrade and will be deploying equipment that allows us to be more resilient during situations like this in the future.
We have informed our support staff of this outstanding, global network issue and requested that while they are troubleshooting specific connections, they keep this in mind. We want to avoid any more of the “it’s your LAN, nothing we can do” and any unnecessary frustration on the part of the customer if at all possible, but still continue to troubleshoot connections to determine if there is another issue, unrelated to the worm, at hand.
We are continuing to monitor and work on this situation and I’ll post any additional updates here.
--
Kat Oak
Speakeasy
kat@speakeasy.net |
|
Yourself
join:2000-08-10
Lynnwood, WA
| Thanks for the update Kat!! Given the fact that I am on the Seattle POP, I am actually intrigued by the first paragraph of the update and look forward to the next bit of info(it's like a soap opera )
Self |
|
yoonix
Floating Taco Of Doom
Premium
join:2001-03-27
Teaneck, NJ | reply to KatOak
Thanks for a very useful and detailed post, Kat, good to see that Speakeasy is looking into this issue. And not so good to see that worm writers are responsible for this.  |
|
MrCoal
join:2001-08-21
Crystal Lake, IL
| reply to KatOak
I think it is a cop out to blame these problems on the Blaster worm.
I have had intermittent severe packet loss for WEEKS, long before Blaster started appearing.
In my case, my connection will just suddenly start experiencing packet loss throughout the internet. This can occur anytime and lasts for several minutes, then just as mysteriously, the packet loss goes away and everything works great again.
Sometimes I can not even reach many sites. Lowering my MTU to 1400 helps but I still get intermittent packet loss.
I am just living with it, but there must be some underlying routing problem.
--
MrCoal
Crystal Lake, IL |
|
bhan261
join:2001-02-12
New York, NY
| I will admit that this isn't my area of expertise, but if it IS a result of the Blaster worm, wouldn't I be seeing similar problems on my work connection? (NYConnect T1) I'm not. Wouldn't my wife see it on her work connection? (DSL.net T1) She's not.
As I said, I am not an engineer so Blaster could very well be the problem. It just seems odd. |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| reply to KatOak
Excellent update, Kat.
I did take router load and pipe saturation into mind when considering the situation. 30-50% sounds a little high, to be honest (I would guess more along the lines of 20-25%), but I won't argue because I simply don't have access to the networking equipment: period.
I realise SE doesn't filter, but I strongly urge reconsideration -- filtering TCP src port 135, or TCP dst port 135 at the border routers. Border routers are specifically deployed for this purpose, and it takes less processing time to drop a packet on a Cisco or Juniper than it does to route it. Don't respond with ICMP_UNREACH_PORT or ICMP_UNREACH_ADMIN_PROHIBIT: just silently drop the things, both directions. If there are customers who are ACTUALLY RELYING on RPC, unfiltered, across their connections, they need to get a grip on security and start using a VPN with IPsec.
If this is indeed the W32.Blaster.Worm, then I can conclude the following from my graphs:
If you notice, most of the heavy (80-100%, "large red blocks") packet loss appears in sections. If you examine the latency (black line), you'll see it takes a sharp ramp or dip simultaneously.
This is a route change, which is EXTREMELY prominent with InterNAP (I've seen them change routing tables every 30 minutes -- I have no idea what criteria they use to decide when to drop BGP with someone and switch, but it seems almost random). From examining my data files for all of this, I've found that ,ost of the heavy packet loss sections are caused when routing through Verio. Being as I worked for Verio for over 3 years, this comes as no surprise -- they refuse to filter ANYTHING ANYWHERE, regardless if their entire network is being hit (that includes DoS/DDoS), and they don't play very well with end-users, ISPs, or other peering providers.
I don't know what kind-of agreement you have with InterNAP, but it might be wise to request that Verio be taken out of the BGP table for SE netblocks until this can be resolved. The "lingering" packet loss you see in the other portions of my graphs I can accept being caused by the Blaster variants.
What I want to know is why the packet loss is induced upon my DSL gateway depending upon which route packets are taking at InterNAP. The only explanation I can come up with is that the DSL gateways have multiple interfaces, and while one interface may be saturated, the other is seemingly fine (only noticed on a BGP change). If this is the case, alright, but still pretty odd.
P.S. -- Don't forget W32/SoBig.F, which I guarantee is giving routers a work-out as well. Luckily the thing terminates itself on 09/09. W32.Blaster.Worm stops working between January 1st and August 15th (but good luck getting people to put up with this for another 4 months).
--
Making life hard for others since 1977. |
|
brikholl
join:2003-01-23
Sunnyvale, CA
| reply to KatOak
I'll 2nd those that don't think that the blaster (or variants) is the root cause of these issues. Partially, maybe, but not entirely.
I had been seeing problems before my initial call to support way back last month. I just waited a few days to call in, hoping it'd get better.
I've since left SE (was in the 30 day trial at the time) and am really surprised that not only is the issue still going on, but that it's worse now than it was when I had SE.
I've moved to another provider, and see NONE of these issues anywhere. What I find interesting is that at times I'd be routed with SE over just about the same path I take to places at my new ISP, and while I saw massive PL when at SE, I see none now.
Lastly, Kat does a great job, but the folks on the phone need to stop spouting the "it's your line, not us" stuff. Covad came out and checked my line, stating it's fine. My current ISP is running flawlessly on it. I'm glad to see that in Kat's post they seem to be heading in that direction, so that's a good thing.
Overall, I thought that for the premium price I was paying for SE, that the techs on the phone should have followed through with things better. Rather than always playing the "blame game" with Covad or my line. I mean, I didn't know Covad was coming out until I called SE to request a status update and was told "yeah, Covad is coming out tomorrow." No one called me to tell me that. I wouldn't have been home if I hadn't called in.
But, for the week or so the service worked "right" it was great.  |
|
