how-to block ads
|
KatOak
VIP
join:2001-09-10
Seattle, WA
| Packet Loss/Latency Reports Just a quick update here:
Many of you have provided excellent information regarding the packet loss and/or latency that you are either monitoring or directly experiencing in your Point of Presence.
I have passed all account information to our Engineering team and they are thoroughly investigating this situation at this time. I'm hopeful that we'll have more information to provide within the next couple of days.
We really appreciate the time that many of you have spent tracking and reporting this problem as it has been extremely helpful.
--
Kat Oak
Speakeasy
kat@speakeasy.net | |
|  
jaykaykay
4 Ever Young
Premium,MVM
join:2000-04-13
Scottsdale, AZ
 ·Speakeasy
| Re: Packet Loss/Latency Reports Great and glad to hear it. While I have not posted to this forum about my packet loss and latency problems, I have done so on the site. I am glad to note that it is a wider problem than just my own, and since I haven't been able to spend much time here of late, I hadn't even checked this forum.
--
JKK Age is a very high price to pay for my maturity. If I can't stay young, I can at least stay immature! | |
| JonW
join:2002-08-13
Herndon, VA | There are lots of problems all around the internet lately, I assume the multiple worms going around haven't quite been defeated. | |
| 
Endorphine
Endorphine
Premium
join:2002-09-30
Bremerton, WA | Thanks for the update Kat!
Hope they figure out what is going on with the border routers. | |
| 
koitsu
Premium
join:2002-07-16
Mountain View, CA
| 
Comcast, Bay Area |
You -- and this post of yours -- are literally the reason I am still a customer. I had just logged in to MySpeakeasy to file a cancellation (yes, after all the hard work you, myself, and lots of other reps have put into dealing with the issues I've brought forth), but remembered I hadn't checked the SE forum in a week or so. I'm thankful I did.
The problem I have with all of this is that it's confusing SE's engineering crew. I am an engineer (system and networking), and have been since 1991. I guess I'm having trouble accepting that an engineering group simply can't pinpoint what/where the problem is.
I know you've probably heard all of this before, but it's just that, well... we're going on, what, almost 5 weeks of this now (in regards to SFO)? It's becoming hard to stomach. I shake my head every day when I stare at my graphs and see the same repetitious behaviour and over between the hours of 09:00 and midnight. I keep waiting to see some change in it, even something as trivial as the Window-of-Doom being audited to something smaller (or heck, even LARGER!).
To make matters worse, as of this evening, things have seemingly become downright foul. It's nearly 2 in the morning and I'm witnessing packet loss (huge margins might I add, suddenly appearing out of no where in clumps of literally 35-40%!), as well as the flippant latency all of us have been reporting. It's the first time I've ever seen it happen at this late of an hour. I even tried power-cycling my DSL bridge: no love.
I'm presently on my Comcast connection, but good lord -- you can tell from the graph I've posted just how absolutely HORRID Comcast is right now here in the Bay. This has been a long-standing problem with AT&T and Comcast for over 3 years now, and it's getting worse. It's been confirmed as well: it's over-saturation due to the populus here.
Despite my comments, I'm going to continue sticking with SE for as long as I can. It's just that... *sigh*... SBC doesn't seem to be having problems like this, or rather, at least not to this degree. I'm sure you can understand my frustration.
Please keep us updated, even with small things like "there was an engineering meeting today about this." I just like knowing someone is at least acknowledging what we're seeing as legitimate, since we (customers) fought so hard to get SE Support to accept this issue in the first place.
I haven't forgotten about your orchids either, by the way. I'm still keeping my word on getting you some. I just happen to be swamped due to work, and a lazy person to bat.
--
Making life hard for others since 1977. | |
| | 
warcorp
join:2000-09-19
Novi, MI
clubs:
| Re: Packet Loss/Latency Reports Koitsu,
We seem to be having the same problem out of the Chicago POP as well so I feel your pain. Not nearly to the degree you are however and seems to be a problem starting at about 7:30 and going all the way till 4:00am ish the next day and it's regular as clockwork (for the past 4 or 5 weeks that I know of. This is the last 12 hours: | |
| | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports Can you beat this?
This has been going on since about 23:30 or so. Prior to that, it was what I'm used to seeing (flippant latency, occasional packet loss (3-4%)).
You know we're sad individuals when we're comparing results of broken connectivity!
--
Making life hard for others since 1977. | |
| | | | | | | | 
bhan261
join:2001-02-12
New York, NY
| Koitsu,
I don't know whether you are a saint or an idiot to endure this kind of poor service without some sort of offer for reduction in fee from SE. I (and others) are having problems in NYC but nothing this extreme.
But you are right that a continuing flow of information about what's being done to investigate and resolve the problem will go a long way to maintaining the peace. Let's hope SE follows through accordingly. | |
| 
Chicago_DSL6
join:2003-08-04
Palatine, IL
| Koitsu, I hope they at least waive the cancellation fee. I am glad that there is an end user in these forums that DOES actually know what they are talking about. The fact that you have presented this much information about your problem and still not gotten your issue resolved in unreal. I never wanted to go to the dark side (SBC), but I called them yesterday for status on install (moved over the weekend) and they mentioned how ALL of their DSL customers are getting doubled downstream. I do not know what you need your DSL connection for, but if it isn't for hosting, SBC might be your most "reliable" and inexpensive choice.
Again, nice work koitsu and best of luck! | |
| |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports
09/04 @ 02:00 to 15:30 |
I don't go the hosting route; I have actual co-location for that. All I want here in the Bay is 1.5mbit down and 384kbit up (or higher -- but those are my minimum requirements). The download speed is pretty much a standard by now, and the upstream I mainly require for a) a fast Remote Desktop connection while at work, and b) being able to stream mp3s from my home box while at work (I use 96-320kbit VBR, so 256kbit upstream induces stuttering on occasion)).
So far SE has gone to great lengths to provide me with pretty much everything I need, including renegotiating my line speed (I was originally 1.5/768 syncing at 1.5/540 and had occasional sync problems, now I'm 1.5/384 -- but all of these network problems were still apparent throughout that fiasco too).
--
Making life hard for others since 1977. | |
| 
sh0V3L
join:2002-02-16
San Carlos, CA
| Im really fed up i have regulars complainig about lag ( i run a ut2003 game server btw ) i dont blame the my connection is getting worse by the min when will they take this serious ive had and have tickets open nothing gets done same thing everytime from our side things are fine lol what an answer so check this out »/quality/nil/1152730
that really suxorz the sour cho cho man FIX IT GUYS!!!!!!!!!!!!!!!!!! THIS is an sdsl 768/768 circuit here $$$$
GET IT RIGHT please
FED UP!!!!!!! | |
| |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports
SE, evening 09/04 |
Graph shows what's happening; large gray portion is due to me simply not running PP during that time (I was doing some hardware work on my workstation).
I also began witnessing the "stalling socket" problem I've documented here on the forum a few times (which leads me to believe that the problems I had with it back when I was 1.5/768 were the beginning stages of this whole thing!). It also took me a good 10-15 full minutes to download a battle.net update for Warcraft 3 (something that should take about 2 minutes max), due to all the packet loss.
Sadly, if it's not fixed by the end / near the end of this month, I'm going to have to cancel. Two months of this is just unacceptable, and I'm sure Kat can understand our frustrations.
I'll power up my DSL bridge a couple times a week to see how things are doing. I may end up sticking a spare gateway I have (Netgear RP614) on my DSL line just so I can run mtr against it from my co-lo to see if there's ever any improvement (otherwise a tr just times out due to nothing listening on the IP).
--
Making life hard for others since 1977.
[text was edited by author 2003-09-05 02:27:51] | |
| KatOak
VIP
join:2001-09-10
Seattle, WA
| UPDATE
We met last night and this morning about the packet loss and latency reports in a few of our POPs. We have a separate, unique concern and overriding issue in our Seattle POP that is still under investigation, so I cannot speak to that at this time – but I can say that the issue under investigation is only being exacerbated by the root cause of the issue across our network.
The crux of it is this: The MSBlaster worm and variants have caused an increase in unnecessary traffic by 30% - 50%, depending on the POP. This is impacting Speakeasy’s network and the Internet as a whole – resulting in both monitored packet loss and increased latency Internet-wide.
The original report, by brikholl on 8/7, was due to the circuit transitions that we performed in SFO on 8/6 and 8/7. From the information he provided, it appeared that the connection returned to normal and then the packet loss and latency increase returned on 8/13, two days after the MS Blaster worm was identified and began propagation.
Due to the worm activity, you may see degraded service during peak hours. This is not due to bandwidth constraints but rather to maxing out how many packets the router can forward per second.
The monitored packet loss graphs displayed in this forum recently tell part of the story, but can be misleading. Routers and servers treat “ping” (ICMP traffic) with a very low “handle” priority. In practice, this means that if a given router is busy handling other traffic, it can delay the response or even decide not to respond (which appears as packet loss). While some of our POPs have high usage during a few hours in the evening, San Francisco is one of our largest markets and has a high concentration of heavy usage throughout the day. Add the unnecessary worm traffic to this usage pattern and the dropped packets will increase accordingly. The general rule is that as long as you are reaching your destination host with no packet loss, or only a small jump in latency, you are not being affected by the current state of things. Latency will probably be higher than usual until these worms die down.
Regarding the SBC connection posed by koitsu & brikholl awhile back – We read through these threads and it appears that SBC is also seeing a significant hit due to these worms. At one point, a rep states that they isolated this into one of the routers in one of their CO's and that it was resolved, but we have no further information on the state of this.
If SBC was having problems with equipment related specifically to their DSL services, our customers shouldn't have been impacted at all. However, if the problems were related to their backbone/Internet services, we may have seen some impact, assuming Internap was routing through SBC at the time – which is highly unlikely.
So, that’s the diagnosis – this is what we’re doing to fix it:
Our abuse team is notifying customers reportedly infected with this worm and will work with them to secure their network. We’re going to aggressively work this until we can reduce the number of infected computers on our network by at least 75%. Industry experts have stated that this worm has peaked and is on the decline, although we haven’t seen any real world evidence of that quite yet.
As you know, we are in the process of a major network upgrade and will be deploying equipment that allows us to be more resilient during situations like this in the future.
We have informed our support staff of this outstanding, global network issue and requested that while they are troubleshooting specific connections, they keep this in mind. We want to avoid any more of the “it’s your LAN, nothing we can do” and any unnecessary frustration on the part of the customer if at all possible, but still continue to troubleshoot connections to determine if there is another issue, unrelated to the worm, at hand.
We are continuing to monitor and work on this situation and I’ll post any additional updates here.
--
Kat Oak
Speakeasy
kat@speakeasy.net | |
| | Yourself
join:2000-08-10
Lynnwood, WA
| Re: Packet Loss/Latency Reports Thanks for the update Kat!! Given the fact that I am on the Seattle POP, I am actually intrigued by the first paragraph of the update and look forward to the next bit of info(it's like a soap opera )
Self | |
| | 
yoonix
Floating Taco Of Doom
Premium
join:2001-03-27
Teaneck, NJ | Thanks for a very useful and detailed post, Kat, good to see that Speakeasy is looking into this issue. And not so good to see that worm writers are responsible for this.  | |
| 
MrCoal
join:2001-08-21
Crystal Lake, IL
| I think it is a cop out to blame these problems on the Blaster worm.
I have had intermittent severe packet loss for WEEKS, long before Blaster started appearing.
In my case, my connection will just suddenly start experiencing packet loss throughout the internet. This can occur anytime and lasts for several minutes, then just as mysteriously, the packet loss goes away and everything works great again.
Sometimes I can not even reach many sites. Lowering my MTU to 1400 helps but I still get intermittent packet loss.
I am just living with it, but there must be some underlying routing problem.
--
MrCoal
Crystal Lake, IL | |
| |
bhan261
join:2001-02-12
New York, NY
| Re: Packet Loss/Latency Reports I will admit that this isn't my area of expertise, but if it IS a result of the Blaster worm, wouldn't I be seeing similar problems on my work connection? (NYConnect T1) I'm not. Wouldn't my wife see it on her work connection? (DSL.net T1) She's not.
As I said, I am not an engineer so Blaster could very well be the problem. It just seems odd. | |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Excellent update, Kat.
I did take router load and pipe saturation into mind when considering the situation. 30-50% sounds a little high, to be honest (I would guess more along the lines of 20-25%), but I won't argue because I simply don't have access to the networking equipment: period.
I realise SE doesn't filter, but I strongly urge reconsideration -- filtering TCP src port 135, or TCP dst port 135 at the border routers. Border routers are specifically deployed for this purpose, and it takes less processing time to drop a packet on a Cisco or Juniper than it does to route it. Don't respond with ICMP_UNREACH_PORT or ICMP_UNREACH_ADMIN_PROHIBIT: just silently drop the things, both directions. If there are customers who are ACTUALLY RELYING on RPC, unfiltered, across their connections, they need to get a grip on security and start using a VPN with IPsec.
If this is indeed the W32.Blaster.Worm, then I can conclude the following from my graphs:
If you notice, most of the heavy (80-100%, "large red blocks") packet loss appears in sections. If you examine the latency (black line), you'll see it takes a sharp ramp or dip simultaneously.
This is a route change, which is EXTREMELY prominent with InterNAP (I've seen them change routing tables every 30 minutes -- I have no idea what criteria they use to decide when to drop BGP with someone and switch, but it seems almost random). From examining my data files for all of this, I've found that ,ost of the heavy packet loss sections are caused when routing through Verio. Being as I worked for Verio for over 3 years, this comes as no surprise -- they refuse to filter ANYTHING ANYWHERE, regardless if their entire network is being hit (that includes DoS/DDoS), and they don't play very well with end-users, ISPs, or other peering providers.
I don't know what kind-of agreement you have with InterNAP, but it might be wise to request that Verio be taken out of the BGP table for SE netblocks until this can be resolved. The "lingering" packet loss you see in the other portions of my graphs I can accept being caused by the Blaster variants.
What I want to know is why the packet loss is induced upon my DSL gateway depending upon which route packets are taking at InterNAP. The only explanation I can come up with is that the DSL gateways have multiple interfaces, and while one interface may be saturated, the other is seemingly fine (only noticed on a BGP change). If this is the case, alright, but still pretty odd.
P.S. -- Don't forget W32/SoBig.F, which I guarantee is giving routers a work-out as well. Luckily the thing terminates itself on 09/09. W32.Blaster.Worm stops working between January 1st and August 15th (but good luck getting people to put up with this for another 4 months).
--
Making life hard for others since 1977. | |
| |
Endorphine
Endorphine
Premium
join:2002-09-30
Bremerton, WA
| Re: Packet Loss/Latency Reports Is the Seattle POP outage/problems related to the blaster worm, or the other reason you didn't tell us yet? | |
| | 
sporkme
drop the crantini and move it, sister
Premium,MVM
join:2000-07-01
Morristown, NJ
 ·Optimum Online
| said by koitsu  :
I realise SE doesn't filter, but I strongly urge reconsideration -- filtering TCP src port 135, or TCP dst port 135 at the border routers. Border routers are specifically deployed for this purpose, and it takes less processing time to drop a packet on a Cisco or Juniper than it does to route it.
A few problems with that... Blaster (or the next worm, or NACHI) doesn't really care if it's being filtered upstream. It will still blast away out your pc, out your modem, out the dslam, out Covad's ATM network, out SE's backhaul. SE does not have any real breathing room for this, so they get slammed. Unlike the cable guys, they can't filter back at your CPE to save their network.
And depending on IOS, filtering is much more CPU intensive. On 12.2T(mumble), adding 135 filters gobbled up 20% more CPU, and that is with CEF enabled.
But I don't think SE has Cisco or Juniper boxes, just a Redback and an FE or GigE handoff to InterNAP. And from what I've seen, the Redback doesn't get too fancy with filtering...
Another thing with Redback is that they prioritize ICMP replies *really low*, so pointing pingplotter at the router gives a less accurate picture than say pointing it to a DNS or game server at your pop. That also gives you more leverage if their network guy knows that pinging routers gives flawed results.
said by koitsu :
This is a route change, which is EXTREMELY prominent with InterNAP (I've seen them change routing tables every 30 minutes -- I have no idea what criteria they use to decide when to drop BGP with someone and switch, but it seems almost random).
While their PR stance is that they have all these transit connections for "robust connectivity", I think a big part of their decisions are based on closely meeting minimums with each provider to meet certain price points. Since they went in the crapper, you see almost 0% of traffic going out "expensive" paths. I think they put the rest of their money into automating route selection based on economic constraints rather than "optimum routing". No one likes to dis InterNap, but they really aren't that "premium" anymore. Sorry.
said by koitsu :
What I want to know is why the packet loss is induced upon my DSL gateway depending upon which route packets are taking at InterNAP.
Coincidence? Or the bgp scanner process is sucking up cpu after a change? Another reason not to ping a router, but a host at the POP instead.
--
just a minute | |
| | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports Where did you conclude I was "pinging a router and not a host at a POP?" Not a single graph of mine that I've posted here in the past 2 months has done that -- I test from endpoint to endpoint.
--
Making life hard for others since 1977. | |
| brikholl
join:2003-01-23
Sunnyvale, CA
| I'll 2nd those that don't think that the blaster (or variants) is the root cause of these issues. Partially, maybe, but not entirely.
I had been seeing problems before my initial call to support way back last month. I just waited a few days to call in, hoping it'd get better.
I've since left SE (was in the 30 day trial at the time) and am really surprised that not only is the issue still going on, but that it's worse now than it was when I had SE.
I've moved to another provider, and see NONE of these issues anywhere. What I find interesting is that at times I'd be routed with SE over just about the same path I take to places at my new ISP, and while I saw massive PL when at SE, I see none now.
Lastly, Kat does a great job, but the folks on the phone need to stop spouting the "it's your line, not us" stuff. Covad came out and checked my line, stating it's fine. My current ISP is running flawlessly on it. I'm glad to see that in Kat's post they seem to be heading in that direction, so that's a good thing.
Overall, I thought that for the premium price I was paying for SE, that the techs on the phone should have followed through with things better. Rather than always playing the "blame game" with Covad or my line. I mean, I didn't know Covad was coming out until I called SE to request a status update and was told "yeah, Covad is coming out tomorrow." No one called me to tell me that. I wouldn't have been home if I hadn't called in.
But, for the week or so the service worked "right" it was great.  | |
| Yourself
join:2000-08-10
Lynnwood, WA | Kat-
Any update on the Seattle POP?
Self | |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| 
SE SFO, 09/08 |
The "huge red blotch" you see from about 16:12 until 17:00 or so is Verio. I verified this since PP logs route changes. Why are BGP route changes at InterNAP affecting my DSL gateway? This makes absolutely no sense to me what-so-ever.
The "smaller packet loss" lines shown are probably caused by MSBlast and SoBig.F. That I can justify.
But I still don't understand why latency shoots through the roof from 09:00 until midnight. That has absolutely nothing to do with MSBlast or SoBig.F, that's for sure. Plus, you can see a point in the graph above (a little after 11:00) where the latency dips. Unless all of these people (and there would have to be a LOT meeting this condition) using the 'net through SE don't leave their PCs from 00:01 until 08:59, and simultaneously suffer from MSBlast.
Tomorrow is 09/09. MSBlast stops working after that point in time. If what's being shown here (from everyone) continues past that, my vote is that it has absolutely nothing to do with MSBlast.
EDIT: Typos.
--
Making life hard for others since 1977.
[text was edited by author 2003-09-08 21:00:26] | |
| |
bhan261
join:2001-02-12
New York, NY | Re: Packet Loss/Latency Reports Yes...today is the day that we'll find out whether SE's engineers were correct or simply living in hope. Anyone want to make any bets? | |
| | |
Chicago_DSL6
join:2003-08-04
Palatine, IL |  Re: Packet Loss/Latency Reports
| |
| | | | | | | | | 
GamingGirl
@speakeasy.n
| Re: Packet Loss/Latency Reports I am out of the LAX pop and I am seeing continuous (throughout the day) packet loss... anywhere from 1% to 28%. I have the Gamer package and have never seen packet loss on my line before last week... obviously this is affecting my gaming =( and although I've contacted support, I'm still waiting for contact from advanced support (it's been three days now). If this truly was related to the worms going around, why is it that nobody else I game with is experiencing any problems?
And where do you get that ping plotter? | |
| | | | | |
bhan261
join:2001-02-12
New York, NY
| Re: Packet Loss/Latency Reports www.pingplotter.com
Also, yesterday was the first day in a month that I didn't have several spikes of P/L. So maybe it was MSBlaster...I'll keep my fingers crossed. | |
| Yourself
join:2000-08-10
Lynnwood, WA
| I have a feeling that my issue is not worm related, as I am on the Seattle POP. This morning(5:45 am)I am still showing up to 20-30% PL on the 2nd hop of my connection. In my opinion it sounds like a NOC(Internap) issue for us Seattle POP folks...I just wish we weren't still in the dark.
Self | |
| 
paulhaskew
Unoffical Dominos Spokesman
join:2002-01-10
Vancouver, WA
clubs: | heh, my modem just lost sync for the third time in two days... hmmm | |
|
sh0V3L
join:2002-02-16
San Carlos, CA
| Thats all fine and dandy I think speakeasy could care less about our ping plotter charts they know thier system is below par internap sucks routing is just rediculius i can see how spiky the network is i have a 768/768 sdsl circuit i run a game server ok im in sancarlos ca guys in sanfrancisco with a speakeasy dsl connection going through the same pop have a ping of 50 to 80 thats just way to high its 25 miles away yet a tracert reveals 11 or 12 hops internap just flat sux i trace a game server in sanjose again just 25 miles away 12 friggin hops abd a ping of 40 hell it shoould be 6 hops a ping of 15 to 20 max you know that i know that and SE Damn well knows it too i think the only reason they say send in ping plotter logs is to give you something to do so your not constantly calling sorry if this offends anyone its not ment as a flame its reality dont let them fool ya we must demand a cure and this is not a worm issue as i have been trying to rectify this for months as koitsu has to .
Speakeasy u know the cure apply it now please | |
| | 
ykrad
join:2001-08-23
Petaluma, CA
| Re: Packet Loss/Latency Reports off topic but; hooooly crap. that is one HELL of a run on!
on topic; yeah, pl sucks, especially for gamers. they're by far the most critical of broadband analyzers. catskinner, no offense but I suggest you post traceroutes with your comments so they're at least semi-understandable.
I also go through the Seattle POP and have experienced the pl but haven't been significantly impacted by it. My roomate plays SW:Galaxies and hasn't expressed any grief with it - except when SE was dropping altogether the other day(s)...
while I agree it is a pain, I still think SE is doing the best they can. They wouldn't be the #1 independant bandwidth reseller if they had a bunch of no-nothing network engineer monkeys on staff. Lets just have a little more faith. | |
| | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports He and I go through the same Speakeasy POP in SFO. See my PP graphs throughout all my Speakeasy threads (click on my username and look at my recent posts); I can guarantee you what he's seeing is identical to what I am.
--
Making life hard for others since 1977. | |
| | | |
GamingGirl
@speakeasy.n
 from:
koitsu
GamingGirl
| Re: Packet Loss/Latency Reports I'm not as technically savvy as most of you are but here are some traceroutes I did last night... I'm too tired to wait for them to complete tonight but this will show you that something wacky is going on as I've seen this consistantly for over a week now...
Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:
1 11 ms 11 ms 11 ms dsl081-084-001.lax1.dsl.speakeasy.net [64.81.84.
1]
2 11 ms 13 ms 11 ms border12.fe4-0.speakeasy-32.lax.pnap.net [63.251
.223.91]
3 12 ms 13 ms 13 ms core1.ge2-0-bbnet1.lax.pnap.net [216.52.255.1]
4 14 ms 13 ms 13 ms sl-st20-la-10-0.sprintlink.net [144.232.154.205]
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
Trace complete.
Here's a trace to a website I visit based on the east coast...
Tracing route to www.iglnet.com [66.151.180.190]
over a maximum of 30 hops:
1 10 ms 11 ms 11 ms dsl081-084-001.lax1.dsl.speakeasy.net [64.81.84.
1]
2 * 11 ms 11 ms border12.fe4-0.speakeasy-32.lax.pnap.net [63.251
.223.91]
3 12 ms 15 ms 11 ms core1.ge2-0-bbnet1.lax.pnap.net [216.52.255.1]
4 10 ms 13 ms 11 ms gigabitethernet5-1-180.ipcolo2.LosAngeles1.Level
3.net [63.208.234.133]
5 13 ms 11 ms 11 ms unknown.Level3.net [209.244.10.245]
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
Trace complete.
Here is to Speakeasy...
Tracing route to speakeasy.net [216.254.0.95]
over a maximum of 30 hops:
1 11 ms 11 ms 11 ms dsl081-084-001.lax1.dsl.speakeasy.net [64.81.84.
1]
2 390 ms 105 ms 337 ms border12.fe4-0.speakeasy-32.lax.pnap.net [63.251
.223.91]
3 11 ms 11 ms 11 ms core1.ge3-0-bbnet2.lax.pnap.net [216.52.255.65]
4 11 ms 11 ms 11 ms gigabitethernet5-1-180.ipcolo2.LosAngeles1.Level
3.net [63.208.234.133]
5 13 ms 13 ms 13 ms gigabitethernet5-0.core2.LosAngeles1.Level3.net
[209.244.10.45]
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 *
Obviously I stopped it before I had to go all the way to 30 hops but I think you get the picture.
I get spikes of up to 28% packet loss in game (at least that I've seen with my own eyes) no matter what time of day that I play. The Blaster worm is expired but yet I saw no difference. It's been five days since I opened my service ticket and I STILL haven't heard from advanced support. Is everyone off this week? Do any of you know what could cause this?
Oh well, I'm going to bed, I bet in the morning nothing will have changed =(. | |
|
GamingGirl
@speakeasy.n | I just got that ping plotter program and I'm wondering... what size packets should I use to accurately show the way my line reacts when playing online games (Unreal Tournament to be exact)? | |
| |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports The default (56 bytes) probably isn't going to be good, but don't pick something absurdly large either (most routers will ignore large windowed ICMP packets). Pick something like 1020 bytes, or maybe 516.
--
Making life hard for others since 1977. | |
| |
sh0V3L
join:2002-02-16
San Carlos, CA
| i finaly got them to aknowledge the fact the sfo pops are borked fubar and then some lol.i recieved an email stating a temp fix in the next few days then a permanent one in 2 weeks as they do some upgrades i realized se was violating the tos for sdsl premium routing as it states 2hops to backbone.for 2 months or whatever it is since they did the last duct tape and tie wire fix at sfo its been minimum 3 hops so i had them by the...... heh you get the picture lol so we will se if they do as said.
said by GamingGirl:
I just got that ping plotter program and I'm wondering... what size packets should I use to accurately show the way my line reacts when playing online games (Unreal Tournament to be exact)?
hi gaming girl im an ut nut too well now ut2003 i ran ut servers for 3 yrs and now ut2003 for 1 yr or more.
i dont think the packet really matters, why? because ut is all udp as far as actual game play server to client. the ping is icmp and the initial handshake is tcp so the routers will handle udp different than the icmp ping.
my feelings are this se knows its got problems ok but if they can buy a lil time by having you do the ping ploter thing they will play that game with you as they did me till i called their bluff. just tell them you cannot reproduce the symptoms and document them because its udp connectionless service when i told them thats when things started going my way so give that a shot
frag ya l8r gg | |
| | |
GamingGirl
@speakeasy.n
| Re: Packet Loss/Latency Reports "my feelings are this se knows its got problems ok but if they can buy a lil time by having you do the ping ploter thing they will play that game with you as they did me till i called their bluff. just tell them you cannot reproduce the symptoms and document them because its udp connectionless service when i told them thats when things started going my way so give that a shot"
Hiya Catskinner... well, that didn't work for me. I called and spoke to the first person, told him what was going on and he proceeded to tell me that there is nothing they could do for UDP traffic...something about it not being able to be corrected en route or something. He told me that it was my game that was causing it (huh?!). He wanted me to believe that it was the netcode or something... two years of never experiencing packet loss and then all of a sudden the game socks me in the back? I don't think so =(.
Of course I began to ask him why there is no way it could have to do with the network if the packets still use the network to reach me. He decided to transfer me to advanced support (finally!). The advanced support guy said that he sent 100 packets to me and saw no packet loss. When I explained about my game and the UDP traffic he said "we can't even count what happens in your game"... Ok, my big problem with this is, I pay for the so called Gamer Package. Like duh, I use my connection for gaming. When my connection becomes useless to me for that purpose, of course I want things fixed. For them to tell me that the problems I am experiencing in game (and to a lesser extent outside of it) are not relevant and deserve no time for discussion is ludicrous. Why sell me a package geared for gaming (knowing that UDP traffic is what consists of gaming) if they refuse to support me when I am having a problem with the very marketing tool they sold me? That's like me buying shampoo, finding out there's dirt inside and being told "I'm sorry, it may be called shampoo but we didn't say it worked like it!" =(
I'm not normally someone who gets upset easily but I feel ignored and I'm just very frustrated. I'm sure that if all I did was surf the net I wouldn't have much to complain about, but then again I wouldn't be paying 100 bucks a month to do it... sigh =(.
He did say that he is going to have Covad reprovision my line to see if that helps my latency (a whole other issue) so who knows, maybe that will work. | |
| | | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports *sigh* Shady technical support reps. Thumbs down.
First off, what the rep. was referring to was the fact that UDP traffic by nature is lossy -- there is absolutely no guarantee that the packet will reach it's destination. The protocol is basically a "send the packet and pray" medium. TCP, on the other hand, guarantees that your packet makes it to it's destination due to handshaking (TCP SYN/ACK). Most games use UDP for their actual transmission of movement, shots fired, blah blah blah, because of two reasons: 1) UDP is by far less intensive on a server (as far as processing goes), and 2) if some bloke running along happens to miss a delivery packet, big whoop, he'll send another one half a second later anyways. Now you know why gamers who have slow or saturated net connections "jump around" a lot, and why latency plays a big role.
The issue I have with what the rep. told you is that the problem we're all experience has _NOTHING_ to do with the kind-of traffic that's being propagated -- TCP, UDP, ICMP. It doesn't matter. The results are the same no matter which protocol you use: the POPs are a mess. The rep. shouldn't be giving you some schmooze about how they "can't do anything about UDP traffic." I get stalls and lag via ssh to my co-lo, which is _entirely_ TCP-based, and I still don't see them bending over backwards about it (because the problem has NOTHING to do with protocols! ).
The fact that SE calls their "gamer package" a "package for gamers" is misleading. All of the packages are the same, just with different speeds or different account options (i.e. shell accounts, 10 Email boxes, and so on). There's absolutely no difference, DSL or network wise, between the SysAdmin package and the Gamer package. It's just marketing schmooze.
Your latency problem will probably get solved when they deploy what Kat said above in her most recent post. I can attest to seeing a _very_ marginal increase in stability since 9/9 (re: MSBlast terminating itself), but the problem (not to sound like a braggart) is definitely looking like oversaturation, which is what I claimed in the first place. *shakes head*
Ah well. We'll see how it goes. I'm glad the Seattle folks are getting the upgrade first since their POP is absolutely horrid. I just hope SFO and LAX are next.
--
Making life hard for others since 1977. | |
| Yourself
join:2000-08-10
Lynnwood, WA | FYI....
I got a call last night from SE(yes they called me) to inform me that there would be a hardware upgrade/changeout taking place next Wednesday that should take care of my issues on the Seattle POP. I will update....
Self | |
| KatOak
VIP
join:2001-09-10
Seattle, WA
| Status:
Although we've been aggressively pursuing customers that have been reported as infected with the MSBlaster worm or its variants, the packet loss that is occurring as a result of this has decreased only marginally.
The Blaster worm & its variants basically scan everything within its class B subnet relentlessly, and we've a couple of these shared with other organizations with whom we've also been in contact with in order to cut down the traffic we're seeing as a result of this worm.
Packet loss is as a result of a maxing of the packets per second that can be processed by our routers. We will be upgrading all of our equipment so in the future we are able to filter/block these kinds of attacks based on more individualized rule sets, instead of the global filters we'd need to put in place right now.
We did attempt a filter on port 135 about 3 weeks ago, but had to pull it due to complaints from Exchange and Windows Filesharing users.
All recorded complaints of packet loss and latency in the SFO area have been customers on one of our routers - we will be transitioning 40% of the customers off of this router and circuit and onto a new router and circuit to alleviate the issue in the short term.
In Seattle, we will begin our POP upgrade next week and will have all new equipment in place within the next few weeks.
We have just a handful of issues in a couple of our other POPs, and will continue to monitor those and work with customers infected with this worm in order to alleviate any load that we're seeing.
--
Kat Oak
Speakeasy
kat@speakeasy.net | |
| | 
GeminiCub4U
Premium
join:2003-04-07
Livermore, CA |  Re: Packet Loss/Latency Reports
--
Reality TV has taken over me. | |
| 
TrueAudio
192khz
Premium
join:2002-02-24
Richmond, CA
| Nobody wants to upgrade there systems till its absolutely necessary. Those routers and servers are super expensive.
Its just gets to a point where customers know what there talking about and experiencing, and start calling the company on it, and then they are forced to take some kind of action. Since they do a great deal of advertising from DSL reports I would think it would be in there best interest to keep the customers that post here happy, and with service they are expecting to receive.
I ordered this service because of the ads on this website. I saw allot of people praising the service and that really turned me on. Im 3 or 4 days into this service, and im already experiencing complete loss of connection for about 15-20 seconds, then it pops back, and there seems to be allot of route changes going on. I haven't been keep much track of the packet loss, but I do notice it.
Well... I hope they are upgrading around here(SFO). Its seems like they bay area has allot of packet loss on some of the major ISP's. Tons of data flowing out of Cali & Seattle.. that's for sure.
I guess we wait and see. Good luck to those who are having problems.
--
»www.mtndewbuzz.com | |
| | GamingGirl
join:2003-09-13
Folsom, CA
| Re: Packet Loss/Latency Reports said by TrueAudio :
...and there seems to be allot of route changes going on. I haven't been keep much track of the packet loss, but I do notice it...
I've noticed a lot of routing changes as well. I asked the rep. that I spoke with earlier why I was seeing all these different routes to one of the websites I typically trace (out of curiosity of course =) ) and he told me that "sometimes when a router goes down the route will get changed but Speakeasy hasn't made any changes to the way things get routed". However, I am seeing a lot of sprintlink and att routes that I never saw before. I used to see mainly ALTER for my trips across the US. However, as I said in a previous post, I'm not technically inclined in the least so it could all still be the same but with a different name... what do I know =) | |
| | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports InterNAP is who Speakeasy peers with. InterNAP peers with all sorts-of different providers; it's synonymous with the PAIX in Palo Alto (i.e. a central point for many backbone providers to connect to one another, hence making the Internet work).
InterNAP is a double-edged sword, and I'm sure Kat can talk a bit about why this is as well. It's fantastic -- when everything is working how it should be. Otherwise it's an absolute nightmare to debunk, since they seem to change routes almost at the drop of a coin.
InterNAP has some seriously magic BGP voodoo going on -- they drop/swap routes practically on an hourly basis. Even when the interface you're going through is absolutely 100% clean (e.g. prior to all of this packet loss crap), they will _still_ drop the route and switch you from, for sake of example, GBLX to ALTER for absolutely no reason. That's why I say it's "magic BGP voodoo." This is NOT how BGP is normally supposed to be used, and it looks to me like InterNAP simply uses some excessively anal settings, or simply plays spin-a-roo with their routes (I've heard of peering providers doing this as well, but it's not very common).
The rep. you spoke to was correct in his description of what's supposed to happen -- when there's a network outage or major anomaly (i.e. packet loss, CRC errors, line errors, or anything else that generally affects POS (Packet Over Sonet) links), BGP will automatically say "Link is screwed, dropping route automatically, moving to another peer." In about 90% of the cases out there, this works great. InterNAP simply deploys this along side a bunch of other stuff -- it's the other stuff which is causing your traceroutes to show different routes every hour, or sometimes as often as every 15 minutes. Oh, BGP stands for Border Gateway Protocol, by the way -- it's just a monitoring protocol that allow routers to make automatic configuration changes based upon rulesets to determine if a network is functioning "up to par" or not.
The best way to understand all of this is to get the utility called Ping Plotter and let it run for a few hours again a website you commonly visit (use www.news.com if you need a default). If you know how to use traceroute, then PP will make perfect sense to you -- except that it's a graphical representation of a continual traceroute. One of the coolest features of PP is that it detects and logs route changes, so you can see exactly where, when, and why a route was changed. The other thing it's useful for is showing actual graphs of packet loss and latency. Try it out, you might like it -- for *IX I prefer mtr (matt's traceroute), while for Windows I prefer Ping Plotter.
Hope this helps.
--
Making life hard for others since 1977. | |
| | | |
sporkme
drop the crantini and move it, sister
Premium,MVM
join:2000-07-01
Morristown, NJ
·Optimum Online
| Re: Packet Loss/Latency Reports said by koitsu :
InterNAP is who Speakeasy peers with. InterNAP peers with all sorts-of different providers; it's synonymous with the PAIX in Palo Alto (i.e. a central point for many backbone providers to connect to one another, hence making the Internet work).
It's not really synonymous. InterNAP buys *transit* from mutliple providers, likely at some fixed price point if they guarantee "X" number of megabits/sec average. At a peering point like PAIX, providers go there to exhange *peering*, which is *not* the same as paying for transit. They will agree on a pipe of some size, who will pay what portion of the interconnect, and agree that this is of mutual benefit to them as ISP X's customer's can now directly reach ISP Y's customers without either side paying each other for transit. These days it's a bit more complicated than that, but that's the theory at least.
InterNAP is simply a paying customer of a number of backbones. They have no network interconnecting their PoPs, each one is an "island". The original plan was to sell "premium connectivity", but ever since they've had financial difficulties, it's become a bit different. You now see more and more traffic going out the cheapest providers. One day we'll see Cogent in the mix.
said by koitsu :
InterNAP has some seriously magic BGP voodoo going on -- they drop/swap routes practically on an hourly basis. Even when the interface you're going through is absolutely 100% clean (e.g. prior to all of this packet loss crap), they will _still_ drop the route and switch you from, for sake of example, GBLX to ALTER for absolutely no reason. That's why I say it's "magic BGP voodoo." This is NOT how BGP is normally supposed to be used, and it looks to me like InterNAP simply uses some excessively anal settings, or simply plays spin-a-roo with their routes (I've heard of peering providers doing this as well, but it's not very common).
This is certainly on purpose, and it's so that they meet their contractual agreements with each provider without going over/under what they've agreed to buy from each provider. I would not be at all surprised to find a RouteScience box at each PoP twiddling all these paths for them. RouteScience, »www.routescience.com/, is a really nifty thing, but it's just a tool. It's generally used to *improve* performance, but it also has many options that let you say "we don't want to average more than 10Mb/s this month, we don't want to average more than 40Mb/x this month to Level3, but we'll go up to 100Mb/s to Verio, and Cogent is unlimited".
Hopefully one day SE will get more neteng people on board and think about simply multihoming themselves to 5 or 6 providers at each PoP and managing this themselves. It requires more time and talent, but in today's fluctuating bandwidth market, it makes sense to add and drop providers on a regular basis. If they are in good facilities with low x-connect charges, it's a win-win.
--
just a minute | |
| | | | |
koitsu
Premium
join:2002-07-16
Mountain View, CA
| Re: Packet Loss/Latency Reports So to sum everything up, literally InterNAP is just a customer who happens to have service with multiple backbone providers (Global Crossing, Verio, Level 3, Alter, and a few other who's names are slipping my mind since I just woke up), then proceed to resell portions of their own POP, with their mysterious BGP setup deployed, to customers (like SE) for network connectivity.
Likewise, the reason for their entertaining merry-go-round routing procedures is a purely financial one due to the costs of aggregate bandwidth total per peering point (hence balancing out the traffic between all providers rather than shoving it all through one which happens to have the lowest hop count and more reliable link (at that time)).
What a bummer.
--
Making life hard for others since 1977. | |
|
paulhaskew
Unoffical Dominos Spokesman
join:2002-01-10
Vancouver, WA
clubs:
| agreed, i am out of the PDX pop, so far no issues as your guys are talking about...
only the two major outages that lasted long than 4 hours, once was the who dang pop went down, then AT&T screwed up everything...
Did that to the company i work for Charter, numerous times...
I hate seeing posts like these anywhere especially for people who use the same service as me, or the company I work for...
Oh, did I mention that tracerts no longer work correctly???
Have the time every other hop times out now... hmm...
My fingers are crossed for resolution of the issue you are all having...
As for my just look at my thread, Transfer Nightmare (still is by the way, being ignored still )
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.
[text was edited by author 2003-09-13 00:53:11] | |
| | | |
|
