Re: Packet Loss/Latency Reports in Speakeasy http://www.dslreports.com/forum/r8085406 en Tue, 01 Dec 2009 04:53:34 EDT Tue, 01 Dec 2009 04:53:34 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8114304 sh0V3L : Ok now 5 mins later it clears up so i stress test a lil by pinging epic with 1472 byte packets :

Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\>ping www.epicgames.com -t -l 1472

Pinging www.epicgames.com [207.135.145.4] with 1472 bytes of data:

Reply from 207.135.145.4: bytes=1472 time=80ms TTL=117

Reply from 207.135.145.4: bytes=1472 time=70ms TTL=117

Ping statistics for 207.135.145.4:
Packets: Sent = 199, Received = 199, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 70ms, Maximum = 90ms, Average = 71ms
Control-C

This Instability should not be a charicteristic of an sdsl circuit and they need to address this issue as it has not gone away by itself ive been backhauled depro repro diff ips nothing changes the internitant instabilty and of course it wont do it when im on the phone with them and pipj can kiss my ass my personal account mngr wtf it was so much better service /tech support before they started the personal account mngr BS. IMHO speakeasy has changed they have become the attbi/comcast of the dsl market i feel for kat being stuck in this she used to be able to get things done for ppl but now it appears she just cant do the things as before , must be the new ceo of course this just my opinion

the above jpeg shows they can do sdsl right but why cant they do stability also ? i have had an open ticket for over a month i refuse to close it until this is stable and will continue to request credit for services not recieved
Click for full size
]]> http://www.dslreports.com/forum/remark,8114304 Wed, 01 Oct 2003 13:10:44 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8114237 Yourself : Lol...they closed my trouble ticket saying it was resolved. I get the same PL and I also have serious circuit issue thats brought my speedtest down to 352/33. Yet, my ticket shows "resolved"...

Self]]> http://www.dslreports.com/forum/remark,8114237 Wed, 01 Oct 2003 13:00:51 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8114023 sh0V3L : i dont know what to say i geuss a picture is worth a 1000 words

the battle for a decent connection continues of course they say my connections fine @##%&$!!!!!!!!
[text was edited by author 2003-10-01 12:41:53]
Click for full size
]]> http://www.dslreports.com/forum/remark,8114023 Wed, 01 Oct 2003 12:28:52 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8091625 Yourself :
said by macmouse See Profile:
For whatever its worth on speakeasy's site (system status) it looks like they now officially saying there is an problem. (Or at least, its on their page instead of just in this forum.) Its for SEA,NYC and CHI. Message identical except for city names. Adjust as necessary ;)


09/26/03 02:18:07 PM  Seattle POP Packet Loss

Region : Seattle
E.T.A. : (none)
Services Affected : Some broadband services


We are presently seeing packet loss on one of our Seattle POP’s backhaul circuits caused by an unexpected increase in traffic caused by Internet worms. We will be fully upgrading this POP within the next few months and are presently investigating interim solutions to these packet loss issues.


I'm going on 6 weeks and counting with a worthless connection(gaming). I wonder how long it's going to take to "investigate" and find an interim fix before the POP upgrade. sigh...:(]]> http://www.dslreports.com/forum/remark,8091625 Sun, 28 Sep 2003 21:27:09 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8091435 koitsu : Switched back over to the ADSL line at 08:45 this morning. By 13:00 it was unusable due to the infamous "socket stalling" problem. It's now 18:01 and the problem is quite apparent.

Back to the cable modem.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8091435 Sun, 28 Sep 2003 21:02:22 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8085406 koitsu : Interesting post, scooby. I've taken up the details with you in private, but the Juniper in question was an M40 (I provided pictures so you can see that I'm not talking out my rear ;-) ).

It's likely that the unit was misconfigured, and it's highly likely that they're _still_ misconfigured. I no longer work there (thanks for laying off 85% of your work force, Verio!), but I still occasionally see "Verio networking madness," especially since InterNAP has a peering link with Verio.

Thanks for clearing up the software vs. hardware issue as well. The individuals who told me they were software-based was one of the Verio network administrators I knew, which may explain how and why the units were misconfigured.

Either way, thanks for stepping in and saying something. *thumbs up*
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8085406 Sun, 28 Sep 2003 01:26:41 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8085181 paulhaskew : sorry folks, looks like its wait till it gets done
[text was edited by author 2003-09-28 00:55:53]
]]> http://www.dslreports.com/forum/remark,8085181 Sun, 28 Sep 2003 00:51:12 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8084689 macmouse : For whatever its worth on speakeasy's site (system status) it looks like they now officially saying there is an problem. (Or at least, its on their page instead of just in this forum.) Its for SEA,NYC and CHI. Message identical except for city names. Adjust as necessary ;)


09/26/03 02:18:07 PM  Seattle POP Packet Loss

Region : Seattle
E.T.A. : (none)
Services Affected : Some broadband services


We are presently seeing packet loss on one of our Seattle POP’s backhaul circuits caused by an unexpected increase in traffic caused by Internet worms. We will be fully upgrading this POP within the next few months and are presently investigating interim solutions to these packet loss issues. ]]> http://www.dslreports.com/forum/remark,8084689 Sat, 27 Sep 2003 23:46:16 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8084672 scooby :
said by koitsu See Profile:
IMPORTANT: The problem at hand with SE might not have ANYTHING to do with Juniper equipment! It might be something else; I have no idea. But when Kat says the equipment is oversaturated in regards to how many pps (packets per second) the equipment can push out, the first thing I think of is their Junipers. Software routing, ugh.


Either you were using a beta product or one of their 'lollypops'. Juniper used to let you have the OS to install on a pc to play with so you could teach people how they work without using a router that was in production.

Junipers are complete hardware routers. Whoever told you they were software routers was incorrect. There is no way you could push the following amount of data through a software router.

M5 - 3.2-Gbps full duplex (6.4 Gbps total)
M10 - 6.4-Gbps full duplex (12.8 Gbps total)
M20 - 12.8-Gbps full duplex (25.6 Gbps total)
M40 - 25.6-Gbps full duplex (51.2 Gbps total)

and the processor in each can handle 40 million packets per second.

Oh and btw, a very common router used at ISPs the Cisco 7513 spec'd to the max only handles .5 million packets per second.
[text was edited by author 2003-09-27 23:47:56]
]]> http://www.dslreports.com/forum/remark,8084672 Sat, 27 Sep 2003 23:44:22 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8084374 koitsu : Dave,

Your problem definitely sounds unrelated to this thread. This thread is discussing problems with Speakeasy IP network reliability and not DSL-related issues. Your issue definitely sounds DSL-related, especially if you were losing sync. I'd ask for noise margins to be provided (you can ask for this on the Covad forum if you have your Covad Circuit ID #), as well as ask Speakeasy to have Covad run an MLT test (which will report back the physical electrical distance between you and your Central Office (Verizon)).

Hope this helps, but please remember this thread is for IP networking issues (packet loss, primarily). :-)
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8084374 Sat, 27 Sep 2003 23:12:46 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8083520 davec22 : Hi:

I have been a customer of Speakeasy’s since January 2000. My legacy SDSL 384/128 service typically runs in the 300’s in both directions. I am connected to the Boston POP, and my setup is a SpeedStream 5250 bridge attached to a Linksys WAP Router configured with a static IP. I have 3 PC’s networked, 2 Win98 SE’s using Cat 5 cable and 1 XP Pro laptop using WiFi. For the first 3 ½ years, I was as happy as any SE customer could be…maybe a few hours of downtime in several years, just a handful of trouble tix, and plenty of help from tech support when needed. I touted my Speakeasy DSL service as awesome to anyone who would listen.

In early August 2003, I started to experience intermittent outages. First, my connection was more up than down, then it was more down then up. The bridge lights would go from 4 green to the retraining cycles, typically not successfully retraining. I opened a trouble ticket, a few days later the problem disappeared, so I closed the ticket.

A few days after that, the problem reappeared (intermittently more down than up), so I opened a second trouble ticket and began working with advance tech Jesse x248. We went through the litany of tests…swapping out cables, doing the on the phone testing with Covad, etc. We had Verizon come out and check the box on the side of my house as well as the connection at the pole 600 ft. off of the house. All was deemed fine, and Jesse fed me the “it’s either your inside wiring or your bridge is failing” line. I believed him and was either going to a) upgrade to ADSL with a Covad Pro Install to get new equipment, a NID filter, and my wires checked, b) buy a new bridge and stick with the service I have, or c) (heaven forbid) switch to Verizon DSL or Comcast.

Before I could decide which way to go (leaning towards a), of course), the problem disappeared. I closed the ticket and life was grand again thru much of September. Then, late this week (Th 9/25), the problems began again, this time with a new symptom. Not only did the bridge keep retraining unsuccessfully, but for most of Thursday and part of Friday, the 4 lights were green, but I seemingly had no connection at all.

I stumbled upon this thread on Friday, and after reading it, it seems that my problems aren’t really my own. If I read this stuff correctly, all of you seem to be having similar ones. Is it the hurricane, the blackout, the worms, or what? I believed Jesse until I saw that the same “it’s your problem” line has been fed to others.

Of course, by Friday night and all of today (Sat), my connection is up and humming as it should be. So, my questions are:

1) Do I have the same problem as all of you, or am I really missing the boat here and I have a different problem? Could it truly be my wiring or my bridge?
2) Can someone point me to a FAQ re: ping plotter, etc.? I run the bos.speakeasy.net speed test, but haven’t tested any of the other stuff you guys are throwing around on this thread. What else should I be looking at, if anything?
3) Should I upgrade or switch ISP’s? Based on what I read in this forum and the VOL forum, the intermittent problems are not going to disappear if I upgrade or switch. You guys all refer to packet loss, ping times, etc…is that the same thing as my intermittent problems?
4) I am a little leery of upgrading to ADSL and the relating “line sharing”. I like the fact that my SDSL is on a separate pair of copper wires than my dial tone. Other than giving me more download speed for the same $$, what’s the benefit to me of upgrading? My 384 is fast enough for what I do (no gaming, little to no FTP, etc. – just surfing, streaming music and emailing mostly) when it is humming.
5) How else can I make this problem go away so that I can go back to being the happily satisfied, always connected SE customer that I once was?

Any thoughts or input any of you (including Kat!!) can provide would be much appreciated.

Thanks,

Dave C.]]> http://www.dslreports.com/forum/remark,8083520 Sat, 27 Sep 2003 21:28:03 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8083442 koitsu : Junipers are essentially FreeBSD-based routers, to put it lightly. Their interface cards do some CPU offloading (duh), but not as much as the Cisco cards do. For example, at the time of my experiences with Junipers (2001 or so), their top-of-the-line advertised "backbone router" (model number I forget, sorry) was running off a Pentium II 450MHz CPU. This was what they felt was suiting for a backbone router -- and not a border router, mind you.

Story time!

We experienced "Juniper stupidity" the instant a host on our network (just for documentation purposes, an EFnet-based IRC hub) got DDoS'd (the uplink was an OC48, the pps count was essentially that of an OC3). That top-of-the-line Juniper turned into hunk of junk in a matter of seconds -- dropping packets, and doing weird things like packet forwards not getting executed (?!) and things of that nature. It affected the ENTIRE facility -- which sported over 70 racks of co-location customers as well as technical support and sales departments. Our Cisco would get slow and start to chunk a little bit, but it wouldn't die/malfunction.

Some of us SAs started poking fun at the Juniper with some tools we had, just to see what could effectively kill it. We found that a simple 10mbit half-duplex Ethernet connection, running smurf (an extremely ancient DoS method), could literally bring that Juniper to a stand-still. Yes, you read that right; 8-9mbit/s of broadcast traffic could totally bring down Juniper's top-of-the-line badboy.

We discussed this our immediate networking department (since Verio's NOC refused to discuss any networking details with us for reasons unknown to my manager as well as my manager's manager), who basically said "the Ciscos held up because they do hardware-level routing. Most of the packet switching and filters are done via dedicated ICs; the Junipers offload most of this onto the CPU. They can't handle heavy loads as well."

The advantages to the Juniper were that the interface for administrators was a lot more streamlined (the command-line interface is essentially UNIX), and navigation is easier. Things are lot more dynamic (which I realise sounds pretty "buzzword-ish" coming from me, but it's hard to explain); sure, IOS is that way as well, but IOS relies extensively on the applicable hardware working how it should (i.e. any sort-of fault and IOS says "GAH!" and reloads the entire config). The Junipers permitted the NOC folk to deploy filters and access lists a lot more quickly and reliably, plus be able to get more detailed information from the unit as to what was going on during packet storms and things of that nature.

Each company has their reasons for going to Juniper. I think the three big ones are the following

1) Lower cost of ownership / maintenance
2) Easier to maintain / administrate
3) Tired of dealing with Cisco hardware and/or IOS bugs

Most of the time, these are the reasons I see people moving to Junipers. You'd have to point out all of this over in the Advanced Networking forum -- although, I think the guys there would probably jump all over me for some of my statements (I'm not a network administrator! Go easy on me, guys!).

I have absolutely *NO* problem with SE using Juniper equipment (other than the fact that I hate Junipers ;-) ). If they want to go down that path, all power to them. However, when it comes to deploying something like that, I URGE them to do what's called TESTING. Send out an Email to some customers in each region deployment is planned for, and ask them if they'd like to be guinea pigs for new routing equipment/new network topologies. Don't deploy it and then be like "BUT IT'S GREAT!". TEST things first. Users who would become guinea pigs could provide feedback.

IMPORTANT: The problem at hand with SE might not have ANYTHING to do with Juniper equipment! It might be something else; I have no idea. But when Kat says the equipment is oversaturated in regards to how many pps (packets per second) the equipment can push out, the first thing I think of is their Junipers. Software routing, ugh.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8083442 Sat, 27 Sep 2003 21:17:56 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8082975 paulhaskew : :(

this issue/thread has been going on how long????

service rating keeps dropping...]]> http://www.dslreports.com/forum/remark,8082975 Sat, 27 Sep 2003 20:11:31 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8082052 anon :
Hi koitsu -

Is the motive to move to Juniper just cost? Or does software controlled routing mean that Juniper boxes offer ISPs many more options for traffic management policies?

Flexibility is usually expensive to implement in hardware, but cheap and easy in software, especially if the software architecture is partitioned into appropriate abstractions. The knobs will be cool and will all function as advertised, as long as the performance demands don't exceed operating limits, BEYOND WHICH the exception detection, propagation and handling elements of the software's architecture had better be pretty robust, or else there will be a constant refrain of, "I've fallen down and I can't get up." ;-)

Maybe the problem isn't just that Juniper equipment offers less head room because of what it does in software rather than in hardware. Maybe the problem is also that SpeakEasy is experimenting with new capabilities in the Juniper system, or worse, sold premium bandwidth on the assumption that they could implement new policies in the new equipment, either without onerous side-effects, or that they could do adequate spin-control on whatever side-effects they anticipated... Maybe SpeakEasy bought a lemon, and is now stuck convincing the world that there must be something wrong with us if we're not happy being served lemonade. ;-)


Am I just blowing smoke here? Does anyone know enough about the flavors of traffic management policy that Juniper equipment offers on paper, even if some of them don't work as advertised?

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8082052 Sat, 27 Sep 2003 17:52:02 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8081865 koitsu : Carl,

Keep one thing in mind: Juniper routers are NOT hardware-based. FE or ATM cards are used in the boxes, sure, but the actual processing is NOT done via hardware (re: ASICs). Junipers are pure software routers, running a BSD derivative. Ciscos, on the other hand, do a lot of hardware-based routing, and IOS simply interfaces with the appropriate chip (while some other features are handled purely by software).

I'm not going to say if this is the cause for what's going on, but in my eyes, software routers are absolute garbage for the sole reason you see here. Junipers are presently a strong contender in the router market that's presently dominated by Cisco, for one sole reason: price. They're much more affordable, which is why so many ISPs and backbone providers are steering away from Cisco.

Sadly, cutting on costs will simply bite them in the ass later when their customers cancel due to performance impact of the newly-deployed equipment (oops, did I say that outloud?). We went through this exact situation at Verio back in 2001 or so; the Junipers made router administration much easier, and the unit itself was a lot easier to use. However, the performance... ugh ugh ugh...
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8081865 Sat, 27 Sep 2003 17:28:45 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8081860 anon : Carl,

I was just razzin ya on the choice of words (re: segregation)

:)

"In this case by contrast, we'd be imposing a temporary segregation in order to get a behavior change, that is, installation of a firewall and antiviral software. That makes my proposal more like giving a child a time-out, until they're ready to behave in a responsible, well-socialized way. "


That sounds like a good plan. I think instead of just speakeasy-wide, this should be internet wide. Whoever is still blasting my home apache testing server with code red should be sent to their room for 6 months :)]]> http://www.dslreports.com/forum/remark,8081860 Sat, 27 Sep 2003 17:28:14 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8080906 anon :
Hi SSIO -

Thanks for your detailed, informative post.

I now have some understanding as to why SpeakEasy is in the 90+th percentile in terms of Blaster impact.

Perhaps someone with relevant expertise can chime in and model how the degradation comes to be so uniform, if there are no restraining traffic management policies in place. (As Bateson said, if he walked into a room and saw a group of monkeys banging out Shakespeare on typewriters, he'd look inside the typewriters. ;-) )

As to your statement that no switching products provide the information to do analyses of traffic for behavioral signatures:

While it would make no sense to build any specific heuristics into hardware, ASICs are these wonderfully parallelized processing engines, where you could capture and buffer attributes of the stream, packet by packet, as it flows through, without impeding the stream. You just fan-out copies of the attributes into parallel pipeline stages that write to dual-port buffers, that yet other processes read out.

You then write the software that uses heuristics to scan the stream, in this case, looking for IPs exhibiting problematic behavior. Once a worm was captured and reverse-engineered, heuristics could be very tailored. Since I would guess that processing would need to make multiple passes over the data, the software itself could also be pipelined, to run across multiple machines in the kind of cheapo, multi Linux-on-Intel server farm that is used for ASIC development.

The hardware buffers would likely overflow doing a 24/7 data capture. But intermittent sampling of an hour or so would seem feasible, and could be triggered during meltdown traffic conditions. Even if the offline heuristic analysis took a little while, I'm guessing it would be faster, more sensitive and more reliable than whatever ISPs are doing now to identify infected machines.

I have to say that it would astonish me to think that nothing like this exists, and that a total newbie like me could invent something like this in a newsgroup, especially since various flavors of DOS attack have been a problem for years now. If the ability to capture this flow-through data does not exist in the current generation of switching products, perhaps this is the beginning of a lucrative business plan... ;-)

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8080906 Sat, 27 Sep 2003 15:31:32 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8080807 Bondman : It looks like the Detroit NAP to Chicago has gone to ***** again. I guess I spoke too soon in my message earlier in this topic. I am getting packet loss and double ping times since this morning.....
:[ sad
Ping statistics for 64.81.159.2:
Packets: Sent = 76, Received = 73, Lost = 3 (3% loss),
Approximate round trip times in milli-seconds:
Minimum = 14ms, Maximum = 44ms, Average = 32ms
:( frown]]> http://www.dslreports.com/forum/remark,8080807 Sat, 27 Sep 2003 15:19:02 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8080609 anon :
Hi Grocers Pet -

:-)

I didn't mean that computers become asymptomatic by themselves, as though their immune system eventually overcomes an infection.

I did mean that after infected users do a virus clean-up and install a firewall, their computers will observably stop generating lots of recognizably bogus traffic--using particular protocols and ports--and thereby be observably asymptomatic.

Also, I think I'd like to narrow your statement about segregation just a little. Segregation based upon arbitrary, uncorrelated, superstitious, etc. criteria have never solved anything, and almost always makes things worse.

In this case by contrast, we'd be imposing a temporary segregation in order to get a behavior change, that is, installation of a firewall and antiviral software. That makes my proposal more like giving a child a time-out, until they're ready to behave in a responsible, well-socialized way.

Perhaps 'segregation' was not the best choice of words, given the many injustices with which it's associated.

Cheers,

Carl F.


]]> http://www.dslreports.com/forum/remark,8080609 Sat, 27 Sep 2003 14:49:49 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8080597 borborpa :
said by Carl F:
1. What's the relationship between "incidents" and "infected computers"?
The incidents are abuse E-Mails sent to rr.com for infected computers, after they receive a certain number of complaints about a specific IP address. There are only 1400 people that use MyNetWatchman, so the 650+ reports to rr is a lot.
quote:
2. Are you saying that SpeakEasy has more infected computers within its customer base, or that SpeakEasy IP addresses are more frequently targeted from infected computers outside of SpeakEasy's customer base?
No, SE is more frequently targeted by rr.com addresses. See more below.
quote:
3. Could you elaborate a little more how the worm chooses its target IP addresses, and why that makes SpeakEasy more vulnerable?

The full explaination is at »www.f-secure.com/v-descs/msblast.shtml , but for basic terms, it takes your IP, and goes out from there. SE IP's start with 66.92 and 66.93. rr's major IP space sits at 66.91, so since SE is so close, it becomes an instant target.?
quote:
4. What is the relationship of rr.com to SpeakEasy? Why would rr.com's incidents take a toll on SpeakEasy, more than any other ISP? (Also, what's the home URL for rr.com?)?
See above. RoadRunner.com should be their main page.
quote:
5. My understanding is that there are decent, basic firewall programs available for free. What's the disincentive to SpeakEasy to do exactly what you're saying, that is, give folks a reasonable time to resolve the problem, under penalty of suspension of service? I should think that losing income from a small percentage of customers would not begin to approach the costs of the equipment required to carry the extra load. Not to mention the costs of all the calls to Tech Support, and the clear damage to their reputation.
Aside from bitchy customer's, there's no disincentive at all. That's why I suggested it. :)
quote:
6. Better yet, can SpeakEasy, as part of its "load-balancing", write a Perl-script or something that will dynamically segregate all symptomatic computers to their own router, until a computer becomes asymptomatic? I think that would definitely qualify to match each customer's embodiment of "best effort", segregating the bad citizens to their own little community, where they can degrade each other's service, in isolation from the rest of us. ;-)
That is actually a little too hard to try and do. Since nothing exists to do that, they would have to put time, money, and effort into creating that...which would be a bit too much.
quote:
7. Why, under random stress, does SpeakEasy's service degrade and then recover in such a well-structured way, if, as SpeakEasy claims, no traffic throttling or other traffic management policies are in force? Apart from everything you've said, these data still need explanation from SpeakEasy. To me, at least, the Blaster worm was the stressor that clearly exposed this well-structured behavior. But whatever is imposing this structure on SpeakEasy's traffic flow is, I think, the root cause of the degraded service. Traffic generators like worms amplify the damaging impact, but the root cause looks to me like some kind of traffic management policy that assures average bandwidth at the expense of increased variability.
Well, once you begin maxing out a connection, it does become very unstable, and variable. Routers are "trained" to drop certain types of packet when they become overloaded. I'm not the pro on this end of things, so my knowledge is limited to what I have seen and heard.
--
There are no stupid questions, but there are a LOT of inquisitive idiots.[AIM - BoyBandsMakeUGay]]]> http://www.dslreports.com/forum/remark,8080597 Sat, 27 Sep 2003 14:47:57 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8080340 anon : 5. My understanding is that there are decent, basic firewall programs available for free. What's the disincentive to SpeakEasy to do exactly what you're saying, that is, give folks a reasonable time to resolve the problem, under penalty of suspension of service? I should think that losing income from a small percentage of customers would not begin to approach the costs of the equipment required to carry the extra load. Not to mention the costs of all the calls to Tech Support, and the clear damage to their reputation.

What's a reasonable amount of time? I suggest 3 days after notification. The reason is: The brain numbing amount of apathy I experience telling people that having a 24/7 always on net connection means a bit of responsibility. The response I hear often is: I am not into security, it wont happen to me. Until its too late.

Its not "if" its "when"

6. Better yet, can SpeakEasy, as part of its "load-balancing", write a Perl-script or something that will dynamically segregate all symptomatic computers to their own router, until a computer becomes asymptomatic? I think that would definitely qualify to match each customer's embodiment of "best effort", segregating the bad citizens to their own little community, where they can degrade each other's service, in isolation from the rest of us.

Computers just dont become 'asymptomatic' like getting over a cold.

Segregation, as we have seen through history has never really solved a thing.]]> http://www.dslreports.com/forum/remark,8080340 Sat, 27 Sep 2003 14:12:04 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8079496 anon :
Hi SSIO -

For those of us who are relative newbies, your post needs a little unpacking. :-)

1. What's the relationship between "incidents" and "infected computers"?

2. Are you saying that SpeakEasy has more infected computers within its customer base, or that SpeakEasy IP addresses are more frequently targeted from infected computers outside of SpeakEasy's customer base?

3. Could you elaborate a little more how the worm chooses its target IP addresses, and why that makes SpeakEasy more vulnerable?

4. What is the relationship of rr.com to SpeakEasy? Why would rr.com's incidents take a toll on SpeakEasy, more than any other ISP? (Also, what's the home URL for rr.com?)

5. My understanding is that there are decent, basic firewall programs available for free. What's the disincentive to SpeakEasy to do exactly what you're saying, that is, give folks a reasonable time to resolve the problem, under penalty of suspension of service? I should think that losing income from a small percentage of customers would not begin to approach the costs of the equipment required to carry the extra load. Not to mention the costs of all the calls to Tech Support, and the clear damage to their reputation.

6. Better yet, can SpeakEasy, as part of its "load-balancing", write a Perl-script or something that will dynamically segregate all symptomatic computers to their own router, until a computer becomes asymptomatic? I think that would definitely qualify to match each customer's embodiment of "best effort", segregating the bad citizens to their own little community, where they can degrade each other's service, in isolation from the rest of us. ;-)

7. Why, under random stress, does SpeakEasy's service degrade and then recover in such a well-structured way, if, as SpeakEasy claims, no traffic throttling or other traffic management policies are in force? Apart from everything you've said, these data still need explanation from SpeakEasy. To me, at least, the Blaster worm was the stressor that clearly exposed this well-structured behavior. But whatever is imposing this structure on SpeakEasy's traffic flow is, I think, the root cause of the degraded service. Traffic generators like worms amplify the damaging impact, but the root cause looks to me like some kind of traffic management policy that assures average bandwidth at the expense of increased variability.

Thanks ahead of time for your elaborations.

Cheers,

Carl F.


]]> http://www.dslreports.com/forum/remark,8079496 Sat, 27 Sep 2003 11:57:35 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8078511 borborpa :
said by Carl F:
I have no data that says that the SpeakEasy customer base had significantly more infections than average. Personally, I think stories about bizarrely high infection rates are just red herrings, and that SpeakEasy was not infected at a significantly higher rate than other ISPs.
Actually, SE is still at the high end of it all, along with rr.com. I use MyNetWatchmen to filter my Zone Alarm logs and send them to the proper ISP's. I would say 75% of the reports go to RR. Because of the way the worms choose their IP address, SE is at considerable risk from RR infected computers. Accoring to »www.mynetwatchman.com/ListIncide···ider.asp , RR has 3.5 TIMES more incidents than any other ISP...so that definitely takes it's toll on SE. I'm still seeing over 3000 hits per day, per IP (I have 8). That's a LOT of extra overhead just tramping it's way around the internet all day pointlessly.

Of course, SE definitely needs to address the issue by sending out information to infected subscribers (which I'm sure they are doing). They need to start getting a little strong-arm though, and if they see after 2-3 days the infected computer is STILL going at it, suspend service until that person corrects the problem. RR REALLY needs to get on the ball, and get their systems fixed as well...
--
There are no stupid questions, but there are a LOT of inquisitive idiots.[AIM - BoyBandsMakeUGay]]]> http://www.dslreports.com/forum/remark,8078511 Sat, 27 Sep 2003 08:58:25 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8074098 anon :
Hi Whiteshp -

This does go round and round. One after another engineer "connected" to SpeakEasy offering similar scenarios that do not make contact with critical data available on these threads.

SpeakEasy's own "propeller heads" need to be telling us the story, directly. Leaking off the record to you is not nearly so helpful as would be talking on the record to all of us, the user base.

1. I'm sorry to hear that business class SDSL is giving you grief as well. But that is a far cry from a definitive statement that there are NO premium classes of the kind I describe.

Mind you, I have no data that such a class exists. It just occurred to me that such a premium class would be consistent with observation, and that someone with much more experience than I have might be able to tell me if such an animal exists, and what kind of customer/application it might be. If no one makes any concrete suggestions over the next few days, I'll take it that this is just pointless speculation on my part.

2. Interesting story about CPU time-slicing... Are routers really implemented with general purpose processors in this way? I would have thought this would have been done primarily in ASICs, to be able to keep up. In either case, whether the algorithm is in software or silicon, it is not rocket science to build an arbiter that assigns priority on any of a number of criteria.

Again, I don't know nearly enough about network infrastructure equipment to know what traffic management policies are available to the ISPs who own and operate the equipment. I'm asking questions to which I'm looking for answers. Your time-slicing story doesn't fit, because I would expect much greater packet loss as opposed to just stalls, if queues were thrashing in the way you suggest. If I think about it, it doesn't make sense to me. :-) But maybe I'm just another asshole with an opinion...

3. The infection rate for Blaster was about 1%. If the propeller heads you're connected to indicate that SpeakEasy's infection rate was much higher, then SpeakEasy owes us some explanation as to how the virus targeted SpeakEasy's customers with such greater precision than other ISP's customers, for example, a security breach of some broadly construed kind.

I have no data that says that the SpeakEasy customer base had significantly more infections than average. Personally, I think stories about bizarrely high infection rates are just red herrings, and that SpeakEasy was not infected at a significantly higher rate than other ISPs. At the same time, SpeakEasy does seem in the 90th percentile or higher in terms of impact incurred, and that does need explaining...but in terms of network architecture, not in terms of infection rates that imply a security or some other egregious issue. On this one, I trust SpeakEasy, and reject conjectures about high infection rates at SpeakEasy.

4. Again, here are the data that need explaining (round numbers, from memory, as my ability to get to these forums has been all wonky for the last few days).

6Sep03, Midday:
================
Download 350Kbps +/- 180Kbps, Upload 290Kbps +/- 1700Kbps (not a typo!!)

19Sep03, Midday:
================
Download 635Kbps +/- 26Kbps, Upload 640Kbps +/- 300Kbps

This behavior shows a remarkable amount of structure, especially for a system under stress. Degraded performance is usually progressively more chaotic, less predictable performance. Why not in this case, with SpeakEasy?

A. Why do download and upload bandwidths track each other so closely, both improving by about 1X?

B. How do upload and download bandwidths track each other so closely, when the upload variability is about 10X the download variability? (This is really the most striking data point, given the randomness of traffic on the internet.)

C. Why do variabilities (the +/-) track each other so closely, both improving by about 6X?

D. Why is upload variability consistently so much greater than download variability?

To me these look, not like the random failures of a system under stress, but rather like traffic management policies that remained remarkably stable despite the high stress of the Blaster worm. But what do I know, being fairly far away from my areas of expertise?

To be very clear here, I'm not saying I know the answers to any of these questions. I'm not a networking engineer. I do believe these are all good questions, and that SpeakEasy would do best to give us the answers to these questions, in the voice of their own propeller heads. Those propeller heads built what was a great ISP until just very recently, so I am sure that they are a really competent bunch of folks with some substantial grip on what is really happening, whether or not they are allowed to say.

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8074098 Fri, 26 Sep 2003 18:05:31 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8073268 whiteshp : I've done a little digging. I won't condone the actions of the techs to some of you. Though I doubt they were given a choice as I'm sure someone in upper management thought delay would help the hemorrhage of customers. I don’t think it is the best decision but it sounds like someone in upper management hopes for the general user not known is not there. Thing is Speakeasy has many propeller heads who are well connected to each other.

If you log your pings like I do you can see that ICMP pings are flooding into all IP’s. With routers having to check packets for where to go millions of pings multiplied by thousands of infected computers is a nightmare for any router CPU. Add in Microsoft Exchange servers must have ICMP and you can’t just block it or you lose your business.

For those wondering if “premium” customers are protected while the others are left to the wolves. I was told by tech that my SDSL priority would help shield me. But my SDSL line is affected exactly and at the same as my ADSL line. I can watch pings on both lines and watch them both grow at the same time and even hit packet loss. But if you think it makes sense. Packet priority only works when your CPU has enough time slices to check all the packets (decide which one should go first). The CPU’s in this case (routers connecting Speakeasy and some other ISP’s) to Internap can’t keep up. So I can vouch as a SDSL customer with my business class routing my packets are getting missed too.

I have been told a (hoped) ETA that some new network equipment will be going into CHI at the end of this month to the early part of October. This should help a lot as everyone goes through CHI one way or another.

To stay or go? A lot of other providers are having problems with these worms as well. This is not isolated to just Speakeasy so if I left I’d need to be careful. The other side of the coin is I LIKED my service quite well before all this garbage happened and I hate to lose it. I’ve been with quite a few providers who really did SUCKK . I can’t speak for the rest of you but the ETA is fairly close. It would take me longer to sign up for new DSL service. So I’ll wait and watch the date. If it gets fixed good life goes on as expected. If not I may be forced to check my alternatives.]]> http://www.dslreports.com/forum/remark,8073268 Fri, 26 Sep 2003 16:27:09 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8072824 bhan261 : No doubt many of us will start (or are already) doing this math. And no doubt some of us will come to the same conclusion that even with the $300 cancellation fee, it will be cheaper AND you get a more reliable circuit by going elsewhere. As I said a couple of months ago, the myth and the reality of Speakeasy being "worth more" are starting to diverge in the minds of many users.

And let me anticipate the response from certain SE fans....

Yes, we know your circuit is fine but do you "get" that ours isn't?

Yes, we know that RADSL is a "best efforts" service but do you "get" that there's good reason to suspect that "best efforts" aren't being given?

Yes, we like Kat and appreciate the work she does but it does not compensate for the fact that we're paying a premium price for what is a sub-premium service right now. (Again, yes, we KNOW your circuit is perfect.)]]> http://www.dslreports.com/forum/remark,8072824 Fri, 26 Sep 2003 15:27:35 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8072084 LowJack :
said by Yourself See Profile:
said by borborpa See Profile:
Kat has said that Seattle has not yet been corrected, but will be in a week (or something). I'm sure she will let everyone know as soon as it's fixed, so we can see results.


The problem with that is that the tech support rep I spoke with 2 days ago said the interim fix had already been completed, which supposedly included my circuit. Obviously that is not the case.


Exactly. Tech support keeps lying to me.

I'm somewhat considering simply paying the $300 fee to cancel, and then going to cable, which is $20 a month for the next 6 months (a deal in my neighborhood) and then goes up to around $40 a month.

$300 + ($20*6) + ($40*6) = $660 /f year

or I could continue with speakeasy

$105 X 12 = $1,260 /f year]]> http://www.dslreports.com/forum/remark,8072084 Fri, 26 Sep 2003 13:52:57 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8070840 anon :
I have a question.

What kind of customer/application would fit the following profile?

1. Traffic is generated outbound, that is, for upload, in large bursts, but with substantial intervals in between.

2. The traffic is time sensitive, such that the customer would be willing to pay a premium to have the traffic delivered onto the network/internet as quickly as possible.

I'm asking this question because SpeakEasy seems to have made solid progress on both download bandwidth and download variability, but on the upload side, the progress has been adequate only on bandwidth.

One plausible reason is that they have a premium class of customer whose overall upload output allows SpeakEasy to maintain its promised upload bandwidth to all customers. That is, the customers in this premium class do not need disproportionate bandwidth. They just need to have other customers' traffic throttled when they are uploading, to assure high priority for their traffic, whenever they are uploading.

Customers like us, who are not in this premium upload class, would get the leftovers. We'd get our advertised upload bandwidth, maybe even more. However, whenever the premium customers were online, our upload variability would be unsuitable for many real-time interactive purposes, including those that SpeakEasy has advertised like Gaming.

Do such applications / customers exist? If so, what flavor of animal or vegetable are they?

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8070840 Fri, 26 Sep 2003 11:10:18 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8069718 bhan261 : In the past day or two, the horrible spikes in P/L in NYC have been accompanied by huge ping spikes. In the words of Gilda Radnor, "if it's not one thing, it's another."]]> http://www.dslreports.com/forum/remark,8069718 Fri, 26 Sep 2003 07:29:20 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8069275 koitsu : Socket stalling problem has re-appeared tonight at about 23:30-23:45. Line has been working "flawlessly" (i.e. no socket stalls) since my last post. Ping Plotter shows happiness regardless (my ping packets are 56 byte, however).

I'm hoping this is being caused by some unmentioned maintenance somewhere, otherwise I'm going to get quite irate.

Back to the cable modem for the time being...
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8069275 Fri, 26 Sep 2003 03:14:18 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8067212 Yourself : I, for one, have no intention of being an apologist for SE. My contract is up with one more payment and if this isn't resolved soon, I will move on to another ISP. This is the 4th trouble ticket on my circuit in 10 months. When I had a Verizon circuit through an independent ISP, I had a total of 4 service incidents in just under 4 years. So this is getting a bit tiresome....

Self]]> http://www.dslreports.com/forum/remark,8067212 Thu, 25 Sep 2003 21:58:18 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8066418 anon :
Hi SSIO -

Please don't slip into being an apologist for SpeakEasy.

From every report I've read lately, the mandate given
to Tech Support seems to be to avoid mentioning the
systemic problems, and to make each caller feel as though
they are the only one having the problem, or that the
problem is normal and not a problem or what-have-you.

I agree with what you said on another thread about
SpeakEasy having been great in the past. But companies
change, especially when the board decides to bring in a
new CEO.

SpeakEasy is working on the Quality of Service problems,
but the Quality of Support has clearly degraded into
a circling of the wagons, rather than the honest and open
exchange that I felt I could always count on.

Please note that Kat is working hard on our Quality of
Service issues, but is remarkably silent on our Quality of
Support issues. Wouldn't you expect Quality of Support
to be part of her charter, as Customer Experience Manager?
With Quality of Service, she seems to be managing the
SpeakEasy side, but with Quality of Support, she seems to
be putting more energy into managing *us* and how we
interpret and feel about SpeakEasy's support behavior.

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8066418 Thu, 25 Sep 2003 20:33:57 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8066158 Endorphine : I also have seen a huge degrade in service this last week on the Seattle POP. :(
I have been keeping ping plotter logs and my first hop used to be 12-13ms all day long. As you can see from the picture below you get past 9am and it goes to around 30-34ms.
I understood also that they were going to being doing a major upgrade "this fall", but when is that going to be?
I am also in the Speakeasy Gaming Beta test, but that will be a joke if I keep getting so much packet loss/drops in most of the games I play.
I will try to be as patient as possible, but I hope Kat can give us an update soon.

Marc
Click for full size
]]> http://www.dslreports.com/forum/remark,8066158 Thu, 25 Sep 2003 20:04:00 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8065040 whiteshp : Anyone seeing ping spikes on the Dayton NAP or is it just me? I'm wondering if it's just me. Things look near normal then they climb to 400+ for 10-15 seconds then drop back down. I have asked tech support twice just curious and they act suprised and say there should be no problem here on our NAP.]]> http://www.dslreports.com/forum/remark,8065040 Thu, 25 Sep 2003 17:54:31 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8064432 Yourself :
said by borborpa See Profile:
Kat has said that Seattle has not yet been corrected, but will be in a week (or something). I'm sure she will let everyone know as soon as it's fixed, so we can see results.


The problem with that is that the tech support rep I spoke with 2 days ago said the interim fix had already been completed, which supposedly included my circuit. Obviously that is not the case.]]> http://www.dslreports.com/forum/remark,8064432 Thu, 25 Sep 2003 16:35:04 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8064301 borborpa : Kat has said that Seattle has not yet been corrected, but will be in a week (or something). I'm sure she will let everyone know as soon as it's fixed, so we can see results.

[EDIT] Here's what I was looking for...
Said by KatOak See Profile
We performed some upgrades to the SEA POP on Wednesday, but the full upgrade isn't scheduled for a couple more weeks, I believe. We are working on an interim solution for this and the NYC POP, however.

--
There are no stupid questions, but there are a LOT of inquisitive idiots.[AIM - BoyBandsMakeUGay]
[text was edited by author 2003-09-25 19:45:52]
]]> http://www.dslreports.com/forum/remark,8064301 Thu, 25 Sep 2003 16:20:12 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8064282 Yourself : Sigh...yep Seattle is still a mess. Here is what I was seeing last night:

~45-60 ms to my first hop(usually 12-14)
20-50% Packet loss at my second hop
Speedtest results of 667/36 on a 1.5/384 line...

I use my line for gaming, so lately my service has been useless. :(

Self]]> http://www.dslreports.com/forum/remark,8064282 Thu, 25 Sep 2003 16:18:17 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8063518 LowJack : I'm in the same boat. Ever since a week or so ago, when they had the Seattle "maintenance" event, my line has been continually getting worse.

Usually my packet loss is 0% and my ping to the gateway is a solid 25ms.

[Tau-Ceti:~] cbishop% ping 216.254.16.1
PING 216.254.16.1 (216.254.16.1): 56 data bytes
64 bytes from 216.254.16.1: icmp_seq=0 ttl=63 time=41.149 ms
64 bytes from 216.254.16.1: icmp_seq=1 ttl=63 time=44.582 ms
64 bytes from 216.254.16.1: icmp_seq=2 ttl=63 time=31.596 ms
64 bytes from 216.254.16.1: icmp_seq=3 ttl=63 time=37.916 ms
64 bytes from 216.254.16.1: icmp_seq=4 ttl=63 time=38.922 ms
64 bytes from 216.254.16.1: icmp_seq=5 ttl=63 time=46.153 ms
64 bytes from 216.254.16.1: icmp_seq=6 ttl=63 time=45.293 ms
64 bytes from 216.254.16.1: icmp_seq=7 ttl=63 time=47.172 ms
64 bytes from 216.254.16.1: icmp_seq=8 ttl=63 time=44.933 ms
64 bytes from 216.254.16.1: icmp_seq=9 ttl=63 time=48.611 ms
64 bytes from 216.254.16.1: icmp_seq=10 ttl=63 time=41.876 ms
64 bytes from 216.254.16.1: icmp_seq=11 ttl=63 time=48.696 ms
64 bytes from 216.254.16.1: icmp_seq=12 ttl=63 time=48.73 ms
64 bytes from 216.254.16.1: icmp_seq=13 ttl=63 time=41.144 ms
64 bytes from 216.254.16.1: icmp_seq=14 ttl=63 time=42.072 ms
64 bytes from 216.254.16.1: icmp_seq=15 ttl=63 time=52.93 ms
64 bytes from 216.254.16.1: icmp_seq=16 ttl=63 time=32.681 ms
64 bytes from 216.254.16.1: icmp_seq=17 ttl=63 time=47.175 ms

It gets MUCH worse when I try to do anything. My speed tests are normal this morning, but my upload dropped from the 768 region last night to the 128 region during the worst of it.

I call Speakeasy, nothing but pissed off techs telling me "we don't garauntee ping times call us when it's 100+".

You are a "gaming" ISP, asshats. You better start garaunteeing them.

Some router log examples:

Sep/25/2003 10:26:35 Drop ICMP packet from WAN 216.254.197.65:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:26:15 Drop ICMP packet from WAN 216.254.223.158:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:25:47 Drop ICMP packet from WAN 216.254.220.123:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:25:23 Drop ICMP packet from WAN 216.255.194.31:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:25:00 Drop ICMP packet from WAN 216.254.16.1:0 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:23:53 Drop ICMP packet from WAN 216.254.214.137:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:23:47 Drop ICMP packet from WAN 216.254.209.138:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:23:36 Drop ICMP packet from WAN 216.254.3.30:8 216.254.16.31:0 Rule: Default deny

Another block:

ep/25/2003 10:01:40 Drop TCP packet from WAN 65.33.248.200:2743 216.254.16.31:135 Rule: Default deny
Sep/25/2003 10:01:19 Drop ICMP packet from WAN 216.254.220.158:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:01:01 Drop ICMP packet from WAN 216.254.160.229:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:00:44 Drop ICMP packet from WAN 216.254.162.64:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:00:18 Drop ICMP packet from WAN 216.254.161.247:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:00:04 Drop TCP packet from WAN 216.241.26.9:1606 216.254.16.31:135 Rule: Default deny
Sep/25/2003 10:00:01 Drop TCP packet from WAN 216.241.26.9:1606 216.254.16.31:135 Rule: Default deny
Sep/25/2003 09:59:39 Drop ICMP packet from WAN 216.254.234.23:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 09:59:38 Drop ICMP packet from WAN 217.1.168.39:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 09:58:41 Drop ICMP packet from WAN 216.254.209.4:8 216.254.16.31:0 Rule: Default deny

More ICMP requests today but yesterday there were nothing but port 135/137 requests from Speakeasy IPs.

MEH.]]> http://www.dslreports.com/forum/remark,8063518 Thu, 25 Sep 2003 14:45:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8060862 Bondman : Hi Kate:
Since the maintenance activity at the Chicago POP servicing the Detroit NAP things are very close to normal. Thanks for working on this. I held off responding to see if the fix would hold. The only strange thing I ran into which I have covered in another topic is the inability to resolve webmail.registeredsite.com and pop.registeredsite.com with Speakeasy DNS servers. From a Terminal Server session at a site that has a different ISP I can resolve these names to IP's.
Thanks again for your help!]]> http://www.dslreports.com/forum/remark,8060862 Thu, 25 Sep 2003 08:00:24 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8058639 paulhaskew : well as in regards to my area... only marginal packetloss/spikes, maybe 1 out of 500... the PDX/SEA POP Bridge is working fine for me at this point...

sorry to see you go catskinner...
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.]]> http://www.dslreports.com/forum/remark,8058639 Wed, 24 Sep 2003 22:34:06 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8054721 jesterekim : Damn sorry to hear that. :(]]> http://www.dslreports.com/forum/remark,8054721 Wed, 24 Sep 2003 16:22:53 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8048364 sh0V3L : IM AM so fed up i just hung up om my acc manager and ive had it with speakeasy at this point im looking for a way to get out without paying cancelation fees thats all i just want the f out they changed my ip block hoping for better its now worse they wont go back they arent willing to do anything they see no problems they think they can bs me well its not gonna fly its bye bye speakeasy ]]> http://www.dslreports.com/forum/remark,8048364 Tue, 23 Sep 2003 21:56:32 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8047828 koitsu : Not too sure what's with the ICMP echos -- those are classic "ping" responses. I'm almost 90% positive that a recent Win32 worm checks to see if a host is up this way, however. I forget which worm.

TCP port 135 would be W32/SoBig.F.

TCP port 445 would be NetBIOS or SMB shares.

TCP port 1433 -- Microsoft SQL server. Probably a trojan/virus of sorts looking to exploit MSSQL holes (and there are many).

TCP port 17300 -- Kuang2TheVirus. No idea what this is, other than a virus. ;-)

TCP port 80 (HTTP) is ""normal"" *cough cough*. I get these as well. They are almost always compromised or infected Windows machines running IIS, attempting to spread itself. Most of the requests I get are "GET / HTTP/1.0" requests, sometimes they're simply "GET /default.ida?blahblahblah HTTP/1.0" for the (old but still active) IIS worm.

TCP port 27374 -- SubSeven backdoor probe. Sub7 lets a remote individual transparently control your PC -- most importantly, it turns your workstation into a DDoS client. Bloody IRC kids...

The UDP requests don't mean much to me either. Possibly another worm, but possibly leftover UDP deliveries from a gaming server?

Sounds like you should check out the Internet Storm Centre sometime. Incredibly useful for how simple it is.

EDIT: Figured out what some of the ports were.
--
Making life hard for others since 1977.
[text was edited by author 2003-09-24 00:19:57]]]> http://www.dslreports.com/forum/remark,8047828 Tue, 23 Sep 2003 20:54:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8047793 metrodust : just got back in town and noticed this problem has finally started to hit me in the fact that my latency 5 times higher then it was before i went out of town. no trouble ticket in yet because i noticed it while at work and thought that maybe a router reset would fix it up.. but it didnt.. grrr!

[text was edited by author 2003-09-23 20:51:36]
]]> http://www.dslreports.com/forum/remark,8047793 Tue, 23 Sep 2003 20:49:24 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8040923 GamingGirl : It looks like the TCP requests are all wanting port 135 with the occasional 445, 1433, 17300, NetBIOS session, HTTP and 27374. The UDP requests are looking for mainly (NetBIOS name) and two hits for port 1434. The ICMP request are ICMP Echo Requests. These results are for the last 1000 hits. This was from 3pm this afternoon until now (10:53pm).

I don't know if this answers what you need, but if not, let me know =).
[text was edited by author 2003-09-23 01:55:23]
Click for full size
]]> http://www.dslreports.com/forum/remark,8040923 Tue, 23 Sep 2003 01:53:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8040854 koitsu : List off some firewall entries, port numbers (src+dst) and protocols and I'll probably be able to tell you what's what.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8040854 Tue, 23 Sep 2003 01:38:28 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8040794 macmouse : Although I haven't managed with anything this large scale, I can give you my own little expierence...

Ok, Back in the day, I ran an game (counter-strike) server on my good 'ol pentium 200 linux box. w/64MB ram. It started as an NAT/ipmasq machine (before those router boxes are now so cheap) and grew into an catch all for misc computer stuff...

Anyway, this "little computer that could" was running an half-life dedicated server in linux. I was actually able to get as far as 10 players, when finnaly I maxed out all CPU capacity on the machine. The problem wasn't with bandwith (it was an local LAN game) but latency (ping) dropped like an rock. I'm not talking about just latency in game- If I ran an ping to the machine through the 100bt network, I would get 500ms+ pings! From what I'm figuring, the cpu (io bus) was so busy that all of the IO devices (like network card) had to sit and *wait* until the cpu was free to then be able to process the information.

I would not be suprised under extreme conditions that this could result in packet loss. The network card has to wait so long that the computer on the other end "gives up".

In the end, an router (cisco) is an computer and specialized software and an *lot* of io (network) ports. Ethernet,fiber,dsl what-have-you. So there was simply so much stuff going through that the "cpu" in the router got full, and that was that.

Also (somewhat reading between the lines), kat pretty much said that the POP's were at capacity (more or less) and they were going to have to upgraded. So, with an sudden increase of network activity (virus/torjans, or even heck just a bunch of new users) was/is enough to push it over the edge.

Simular things have happened in the past with other companies *cough* comcast *cough* so speakeasy isn't alone. To their credit, they are *telling* us whats wrong and that they are working on it. With comcast/attbi/@home, they said *nothing* and that was "just the way it is" and *eventually* things would "magically" start working again, with no explanation.

I hope that answers your question :).]]> http://www.dslreports.com/forum/remark,8040794 Tue, 23 Sep 2003 01:25:17 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8040785 GamingGirl :
said by Carl F:

BTW, GamingGirl, is your upload stable enough for you to game reliably?

Cheers,

Carl F.


It used to be awesome. I would get stable 55-65 pings to the server I play on in San Antonio. As it stands now, I still see occasional dropped packets. PL will spike to 30% from zero but only on occasion whereas I was seeing consistent PL for the first few weeks of Sept. I do see increased latency compared to the first six months of my service but there's not much I can do regarding that as it's a "best effort" service. I blame the latency on my second hop since results on each hop thereafter are solid and rarely, if ever, fluctuate. My ping plotter looks much like it did before but without the heavy red areas. That second hop has some serious ping spike issues.

I did a lot of ftp transfers just last night though and my upload is very steady around 650 up (I am on the 1.5/768 plan).

As far as gaming, I was thoroughly pleased with my line during the first six months and now I'm just satisfied and hoping that whatever changed will reverse itself and go back to normal when these upgrades take place.

Regarding the worms... I am seeing serious "ping requests" on my firewall, upwards of 400 over a 3 hour period (like whoa!). These are all ICMP packets with the occasional TCP and UDP requests for certain ports.... I don't know if that makes sense (I'm not that technically inclined =p ) but I do see wayyy more hits than I ever have before. A lot come from users on the speakeasy network (about half) and the rest are from other sources. Does anybody know why I keep getting hit like that? Is it indeed caused by the worms out there?]]> http://www.dslreports.com/forum/remark,8040785 Tue, 23 Sep 2003 01:23:08 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8039879 anon : Hi jmberry -

Kat said several posts back that they've reworked their processes to improve communication between engineering and tech support.

Sounds from your post, much to my astonishment, that first tier CS is still working off the same old script.

Good luck,

Carl F.
]]> http://www.dslreports.com/forum/remark,8039879 Mon, 22 Sep 2003 23:22:52 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8037814 jmberry : While I'm glad I read through this entire thread, I'd have been a lot happier if more of this information had been shared w/me via the number of phone calls/emails/TT's I've submitted.

I'm in the last couple of days of my eval period for a T1, and am doing my best to come to the right decision in whether or not I am going to continue past that period with Speakeasy. I've heard (and read) a lot of very positive things about them, but am having issues with a T1 line having a 100ms lag on its first hop....

~JMB]]> http://www.dslreports.com/forum/remark,8037814 Mon, 22 Sep 2003 20:05:47 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8037443 anon :
Hi Kat -

I can see, and very much appreciate, that you're working hard to be responsive. Let me see if I can boil it down to a single question that I'd like to see answered in enough engineering detail that it makes some sense to me.

You said that Blaster et al. caused a .4X to .5X increase in traffic. I measured a .5X decrease in bandwidth on both download and upload. But I measured a 100X increase in variability (latencies, duration of stall events, etc.)

This is way out of whack, a very weird pair of symptoms. If I were to graph this, the relationship between traffic increase and bandwidth decrease would be linear, .5X demand increase, .5X bandwidth decrease. Yet the relationship between traffic increase and service stability and reliability would look like an exponential runaway or an asymptote, .5X demand increase, 100X reliability decrease.

If bandwidth were also collapsing nonlinearly, I would say that the traffic queueing management was thrashing terribly, with intermittent queue overflows: Kind of like the disk thrashing, the slowdown and the intermittent app failure that occurs when apps start page swapping like crazy. But all that thrashing takes real-time. The slowdown of applications goes nonlinear well before app failure occurs. Yet the traffic slowdown, the loss of bandwidth stayed linear, so two orders of magnitude more thrashing still took place in a linearly increased amount of time.

How did download and upload bandwidth decrease linearly, but service reliability decrease exponentially/asymptotically, especially if, as you've repeated many times, there was no form of traffic throttling in force?

If, as Customer Experience Manager, this is beyond your expertise to answer in detail--it would certainly be beyond my expertise, as a real-time embedded engineer with negligible networking experience--could you please bring one of your network architects into the loop to answer this?

Thanks,

Carl F.


]]> http://www.dslreports.com/forum/remark,8037443 Mon, 22 Sep 2003 19:29:48 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8036979 Yourself : or Seattle?....tomorrow is the one month anniversary of my trouble ticket being open for this issue.]]> http://www.dslreports.com/forum/remark,8036979 Mon, 22 Sep 2003 18:46:42 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8036961 falkclan : any further information on the Chicago POP?
--
Alan Falk]]> http://www.dslreports.com/forum/remark,8036961 Mon, 22 Sep 2003 18:44:22 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8035299 KatOak : I'm not sure what additional information you're looking for, but the issue is a hardware max-out on the number of packets per second that two of the routers in our SFO POP could process. We alleviated this issue by moving about 40% of the customers on these routers over to another router with considerably less load.

We don't consider this issue closed as we're still seeing about 1 - 3% packet loss at this POP, depending on the time of day. As I posted before, our Engineering team is continuing to investigate alternate improvements in the interim prior to the complete upgrade later this year.

Again, we made absolutely no changes to our SFO POP - have done no work on the impending complete upgrade scheduled for this fall. This issue was caused by our routers inability to handle the extreme increase in the number of packets per second as the result of recent Internet worms, which are still having a significant impact. Our abuse team is notifying and unbinding folks as appropriate in order to minimize our customers' contribution to this problem.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,8035299 Mon, 22 Sep 2003 15:30:10 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8029305 anon :
I'd also like to know what was wrong and what they did to fix it.

Could just be that they undid whatever it is that they did--or started to do--about a month ago, when my download streams started to collapse with much higher frequency. After all, we're all reporting that we're more or less back where we were.

Sounds to me like more of an 'Undo' than an 'Upgrade'.

BTW, GamingGirl, is your upload stable enough for you to game reliably?

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8029305 Sun, 21 Sep 2003 21:18:51 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8028940 GamingGirl : Yeah, other than extreme ping spikes on my second hop, I am seeing a whole lot of improvement. My packet loss is minimal and other than the ping spikes, I'm happy with it =). I also would like to know what they tinkered with to fix it as I'm curious that way. I cannot wait until it's 100% back to the way it was but for the meantime this will satisfy me =).]]> http://www.dslreports.com/forum/remark,8028940 Sun, 21 Sep 2003 20:44:25 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8027902 koitsu : Wow. I switched over to the ADSL line this morning to see how things were, and to use it while at work throughout the day. The graph is for SUNDAY, which means people are quite a bit less likely to be pounding on the network as much as during Mon-Fri.

I'm very impressed. The spikes/ramping you see in the graph are _entirely_ caused by me (streaming mp3s + remote desktop + file transfers); latency to my gateway has an average of 28ms, but that's simply because I did a lot more streaming today than I usually do. I'd say the average is more like 11-13ms. No socket stalling or Remote Desktop drops were encountered.

There is still some packet loss, but like Carl F. said, things are looking up for SFO. PLEASE do not consider this issue closed, however -- things are just looking better, that's all.

I'll have to see how the rest of the work week looks. If it stays this way.

P.S. -- Can we get a statement in regards to what exactly was the (primary) cause of this? Routers? Oversaturation? Equipment failure? Aliens? :-)
--
Making life hard for others since 1977.
Click for full size
SFO, 09/21
]]> http://www.dslreports.com/forum/remark,8027902 Sun, 21 Sep 2003 18:33:21 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8015541 sh0V3L : asdfghj !!!! i have sdsl and it may aswell be cable the way it is now .................]]> http://www.dslreports.com/forum/remark,8015541 Sat, 20 Sep 2003 02:06:22 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8013138 KatOak : We performed some upgrades to the SEA POP on Wednesday, but the full upgrade isn't scheduled for a couple more weeks, I believe. We are working on an interim solution for this and the NYC POP, however.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,8013138 Fri, 19 Sep 2003 20:12:16 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8012917 Yourself : Kat-

I got a call from tech support last week that stated the Seattle POP would be upgraded on this past Wednesday(9/17). Did that happen or is it on hold?

Self]]> http://www.dslreports.com/forum/remark,8012917 Fri, 19 Sep 2003 19:46:05 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8012543 anon :
15:20 PDT

So, my stream from KWAX (64Kbps) went to hell in a hand-
basket, but my stream from KUAT (132Kbps) seemed fine.

So I decided to run the characterization test I ran a
couple of weeks ago, at almost the exact same time of
day, to see more exactly where things stand.

For those of you who haven't looked at the thread on
SpeakEasy's own forums (to which I supplied a link on
page 4 of this DslForum thread), here's the procedure
I use.

On the SFO POP, SpeakEasy provides an applet that
instruments your current bandwidth. Because it's an
instantaneous measure, no single measurement will be
all that accurate. Moreover, it may have some additional
inefficiencies that causes it to underestimate bandwidth
by 10% or maybe even 20%. (For example, I'm unclear if
it counts IP packet headers.)

That said, this applet is a fine tool if you take
several samples back-to-back. 20 samples gives fine
statistical power. It's not a good tool for tweaking
10% or 20%. It's a great tool for estimating orders
of magnitude, that is, powers of 10.

The average bandwidth over 20 samples is fine for
estimating the order of magnitude of bandwidth delivered,
that is, am I getting closer to 600Kbps or 60Kbps.

The standard deviation over 20 samples is a great way
to estimate the order of magnitude of variability of
bandwidth delivery, that is, am I getting X Kbps,
plus-or-minus 10Kbps, 100Kbps or 1000Kbps.

For a 768/384 service like I have, the following apply.
Plus-or-minus 10Kbps is a very even stream, and should
be suitable for almost any purpose. Plus-or-minus
100Kbps is adequate only for low-bandwidth streaming,
and may be completely unsuitable for gaming. Plus-or-minus
1000Kbps is unsuitable for any real-time interactive
use.

What I'm going to show is 2 times the standard deviation,
because 97% of all packets should fall within this
plus-or-minus range.


SFO POP, 20 back-to-back samples, 768/384 Straight-Up

TWO WEEKS AGO, 6sep03 14:15 PDT
=====================================
Average DOWN: 345 Kbps
Average UP: 293 Kbps

2 x Std. Deviation DOWN: +/- 184 Kbps
2 x Std. Deviation UP: +/- 1742 Kbps (NOT a typo!!)

TODAY, 19sep03 15:00 PDT
======================================
Average DOWN: 633 Kbps
Average UP: 640 Kbps

2 x Std. Deviation DOWN: +/- 26 Kbps
2 x Std. Deviation UP: +/- 300 Kbps


SUMMARY: Twice as much bandwidth is being delivered,
in both directions, with almost an order of magnitude less
variability. The download side variability should be
reduced farther, though not as a highest priority
activity. The upload side still shows too much variability
for gamers, video conferencing and the like.

For my purposes, IF THIS IS STABLE, I'm in decent shape,
and should move down the triage list--but NOT off it.
Gamers and other interactive users should not allow
SpeakEasy tech support to use the strength of my report,
based on my download centered usage, to b.s. them that
everything is much better on the SFO POP, and what's
their problem. Upload bandwidth is good, but variability
is still unacceptable.

Cheers,

Carl F.
San Francisco



]]> http://www.dslreports.com/forum/remark,8012543 Fri, 19 Sep 2003 18:57:54 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8012312 KatOak : We are continuing to work on this issue - the majority of our customers in SFO, LAX and CHI areas have reported a much improved experience. I have a couple of reports otherwise and our Engineering team is investigating more interim improvements to all POPs, specifically SEA and NYC, that can be made prior to the full POP upgrades already scheduled.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,8012312 Fri, 19 Sep 2003 18:23:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8010951 anon :
12:15 PDT

I've been streaming 176Kbps from UMich for almost
four hours now.

Packet Loss is sitting at about 1%. Unrecoverable
packets, that is, lost packets that RealPlayer
retries but cannot recover in time, is sitting at
about .1%.

This is about one order of magnitude better than the
terrible performance of a week or two ago.

This is adequate performance, that is, RealPlayer
doesn't automatically downgrade my stream to a lower
bitrate. However, it's also not stellar, as I should
have lots of head-room on a 768/384 Straight Up,
with no other network activity in progress.

I can tell that it's not yet as good as it used to be,
because of the frequency with which I'm hearing the
stream glitch, every time a packet is unrecoverable.
In a 24/7 classical music stream, the glitches are
unmistakably audible. But I can live with this for a while,
in terms of SpeakEasy's triage, as long as: (1) It doesn't
get worse from here, and (2) SpeakEasy knows that they need
to do another iteration of performance improvements, before
the SFO POP will be whole.

More later,

Carl F.


]]> http://www.dslreports.com/forum/remark,8010951 Fri, 19 Sep 2003 15:33:05 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8010781 sh0V3L : I still see zer0 improvement check this out »/quality/nil/1166407

Im really getting tired of this i get email from tech support that pretty much calls me a liar

After the maintainence late tuesday night, we're hearing lots of reports that the problems in your area have much improved. ("WEEEEEEEEE!!!!!!!")

Port Status: Up
Actual Port Speed: 768 kbps
Margin: 15.5 dB
Upstream Cells Received from CPE: 67 ( 943187633 )
Downstream Cells Transmitted to CPE: 184 ( 1461619541 )
ATM HEC Errors: 0 ( 7998 )
Upstream Line Errors: 0 ( 958 )
Downstream Line Errors: 0 ( 667 )
Time Since Snapshot Counters Reset:
3 Min. 13 Sec.

Are you still seeing problems when you have just one computer plugged directly into your modem?

ANS : my modem has 1 wan port 1 lan um so ya . WTF does that have to do with pnaps lousy routing

Could you show me the 18 hop traceroute to yahoo?
----------------------------------------------------------------------

Thank you for your business,......... heh

The Speakeasy Crew
800.556.5829
one of my many imediate replys

Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\>tracert www.yahoo.com

Tracing route to www.yahoo.akadns.net [66.218.71.93]
over a maximum of 30 hops:

1 20 ms 10 ms 20 ms dsl092-028-001.sfo4.dsl.speakeasy.net
[66.92.28.1]
2 20 ms 10 ms 20 ms border5.g3-4.speakeasy-29.sfo.pnap.net
[63.251.53.2
3 10 ms 20 ms 20 ms core2.ge2-0-bbnet2.sfo.pnap.net
[63.251.63.66]
4 10 ms 20 ms 20 ms iar1-so-2-2-0.SanFrancisco.cw.net
[208.173.173.9]
5 10 ms 20 ms 20 ms acr2-loopback.SanFrancisco.cw.net
[206.24.210.62]
6 20 ms 30 ms 20 ms bpr2-as0-0.SanJoseEquinix.cw.net
[206.24.211.42]
7 * 20 ms 30 ms bpr1-ae0.SanJoseEquinix.cw.net
[208.173.54.85]
8 20 ms 30 ms 30 ms 208.173.54.74
9 20 ms 20 ms 30 ms so-5-0-0.gar1.SanJose1.level3.net
[209.244.3.137]
10 20 ms 20 ms 20 ms gige9-0.ipcolo3.SanJose1.Level3.net
[64.159.2.9]
11 140 ms 220 ms 211 ms unknown.Level3.net [64.152.69.30]
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

Ya Speakeasy Thats Excellent routing and such an improvement looks like you made a liar out of me !!!!!!!!...........NOT!!!!!!!!

Just fix it!!!! this issue is so old now ]]> http://www.dslreports.com/forum/remark,8010781 Fri, 19 Sep 2003 15:07:06 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8009105 anon : 8:00 PDT

It's early in the morning just yet, but things seem fine.
I'm streaming 176Kbps from UMich as I write.

Also, I'm realizing that what I reported last night as
stalling is likely to be the way a RealStream does flow
control, in some configurations.

As such, I'm getting progressively more optimistic. If
this stabilizes, the upgrade will have yielded a real
QOS improvement.

More later,

Carl F.
San Francisco
]]> http://www.dslreports.com/forum/remark,8009105 Fri, 19 Sep 2003 11:16:51 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8008456 MrCoal : Last night there was some real improvement in the Chicago POP. Then at 3:06am, the route changed, ping times doubled and packet loss started again.
Click for full size
Chicago POP
]]> http://www.dslreports.com/forum/remark,8008456 Fri, 19 Sep 2003 09:34:43 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8007456 koitsu : $!(*&$(*@ I'm SO GLAD someone is also experiencing the stalling. The packet loss is one thing, but I thought I was the only one experiencing socket stalls.

I've been on my cable modem since my last post, and despite the flippant latency, I've had nothing but a solid connection. *sigh*

Here's hoping Kat and engineering fix what's broke! *thumbs up*
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,8007456 Fri, 19 Sep 2003 03:14:56 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8007133 paulhaskew : here are the only places that i see any sort of packet loss... yes, i did happen to ping the router itself for my pop... cause it resolves to sea.speakeasy.net, and i have no clue where gate.tailie.net is... :S
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.
[text was edited by author 2003-09-19 01:42:26]
Click for full size
jango.outerfx.net to lax.speakeasy.net
Click for full size
jango.outerfx.net to pdx.speakeasy.net (sea.speakeasy.net)
]]> http://www.dslreports.com/forum/remark,8007133 Fri, 19 Sep 2003 01:41:16 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8007034 anon :
More data:

I opened a stream to UTenn at 176Kbps. Partial collapse
at some point to 132Kbps.

At 22:15 PDT, when I'm writing this, I'm seeing 1-2 sec.
stalls, very reliably every 10 seconds. That is, duty
cycle of 80% transmission and 20% dead time. Fetch-ahead
seems to be good enough that this does not result in
horrible degradation, but still some.

On the other hand, earlier today, I got almost flawless
transmission of a 96Kbps stream from Budapest, Hungary.

I'm not sure how to interpret these unsystematically
collected data points, but perhaps they will correlate
with QOS monitoring.

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8007034 Fri, 19 Sep 2003 01:21:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8004200 ykrad : Ditto. Although my problems haven't been as extensive (or extreme) as many others, it's still irritating to still have to point this out.

line quality test fails coming from both west and east coasts:
»/quality/nil/1165758]]> http://www.dslreports.com/forum/remark,8004200 Thu, 18 Sep 2003 20:24:42 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8003803 Endorphine : Same here.
I haven't seen any changes to the Seattle POP at all :(

Marc
Click for full size
]]> http://www.dslreports.com/forum/remark,8003803 Thu, 18 Sep 2003 19:45:30 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8003379 paulhaskew : mine is doing fine, but for some odd reason when i do a tracert to pdx.speakeasy.net, it comes back as sea.speakeasy.net...

also what does gate.tallie.net have to do with my connection??

i don't see this on anyone else's ping plotter posts... unless you guys are skipping the first hop... which i never do...

so where is the PDX pop? acutally in seattle??
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.]]> http://www.dslreports.com/forum/remark,8003379 Thu, 18 Sep 2003 19:04:54 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8002754 Yourself : I'm not sure if the Seattle POP received any upgrades or made any changes last night, but my connection remains as bad as previously noted in this thread.

Self]]> http://www.dslreports.com/forum/remark,8002754 Thu, 18 Sep 2003 18:04:20 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8001164 anon :
So here's another interesting data point.

KUAT, our preferred 132Kbps stream, has just gone into
fund-raising mode, so we've switched to our preferred
fallback, KWAX which offers a 64Kbps stream.

UDP packet loss has jumped up to almost 1%, 10x greater
than with KUAT, but still well within what RealPlayer's
packet recovery protocol can sustain.

BUT, KWAX has always been a more heavily subscribed and
somewhat flakier stream.

So, I'm beginning to feel like I'm getting back to the
point where I can see the internet beyond SpeakEasy,
warts and all, instead of constantly bumping up against
problems at SpeakEasy that obscured the internet from view.

Again, I'm looking for several days of stability before
I turn off the high-frequency acquisition radar and go
back to low-frequency search mode. Paradoxically, however,
this performance decrement in a place where I'd expect
it is another good early sign.

Cheers,

Carl F.
]]> http://www.dslreports.com/forum/remark,8001164 Thu, 18 Sep 2003 14:37:31 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8000934 GamingGirl : I can't wait to get home to see if there is improvement on my line. I actually heard from Speakeasy last night on my ticket and they ran tests and saw the packet loss for themselves. That made me feel a little better that they did acknowledge that I do have problems on my line. I tried to game a little last night and it was horrid... I was experiencing 10-15% packet loss so half of my shots wouldn't fire and I kept falling off of cliffs. This may have been previous to any work they had scheduled though. I cannot wait to be able to get a good game on again =).]]> http://www.dslreports.com/forum/remark,8000934 Thu, 18 Sep 2003 14:05:13 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8000921 whiteshp : Well I didn't notice any problems till today. From 12:30 till 4AM service to my SDSL was mostly out (longest outage ever for me). It has been up mostly today but still a LOT of little brief outages all through the day. I sure hope they get this new hardware/routing fixed soon! :/]]> http://www.dslreports.com/forum/remark,8000921 Thu, 18 Sep 2003 14:02:46 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8000724 anon :
So far today, service looks much improved. Based on 4
hours of streaming, UDP packet loss is around .01% (32
packets in 210,000), as opposed to 6% two weeks ago.
I also ran a concurrent 8MB download, which sustained
60KBytes/sec, without adversely impacting the stream.

It's still a bit early in the morning, just yet.
I'm holding my breath to see if this remains stable,
but it's definitely a good first step of forward progress.

More later,

Carl F.
San Francisco
]]> http://www.dslreports.com/forum/remark,8000724 Thu, 18 Sep 2003 13:40:34 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8000411 KatOak : Our engineering team is currently working with the vendor on this.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,8000411 Thu, 18 Sep 2003 13:04:42 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,8000086 surtin :
said by MrCoal See Profile:
Well, the maintenance on the Chicago POP has made packet loss WORSE


Yup it sure did, I'm especially upset since I didn't have any problem at all, now I'm downloading at at best 2kb/s now (early in the morning it was 5kb/s). This is pretty much like bein' on dial-up... oh wait I even downloaded faster when I WAS on dial-up.]]> http://www.dslreports.com/forum/remark,8000086 Thu, 18 Sep 2003 12:24:58 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7999398 MrCoal : Well, the maintenance on the Chicago POP has made packet loss WORSE (I am posting this on dial-up because I cannot connect via DSL)

I have had packet loss/connection problems for at least four months and Speakeasy has a very limited time to fix this before I cancel service.]]> http://www.dslreports.com/forum/remark,7999398 Thu, 18 Sep 2003 10:51:02 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7997332 sh0V3L : well i dont know what they did or if anything was done im still getting spikes up to 300 ms off the sfo pnap routing is still screwed up i say lies all lies sorry thats just how it asppears to me i would post a tracert or ping platter but i see no use as i have posted many and speakeasy is well aware of the problem that being PNAP i feel the nastys coming on so ill stop here ]]> http://www.dslreports.com/forum/remark,7997332 Thu, 18 Sep 2003 00:47:05 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7996689 Endorphine : As you may have noticed I said "over the life of the worm", I am aware the chart is over a month old. Just was pointing out the severity of the infection.

If you want recent data on port attacks a good site is the Internet Storm Center: »isc.incidents.org/

Marc :)]]> http://www.dslreports.com/forum/remark,7996689 Wed, 17 Sep 2003 23:18:30 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7996199 borborpa : FYI the graph on hackerwatch is a month old... :)]]> http://www.dslreports.com/forum/remark,7996199 Wed, 17 Sep 2003 22:25:23 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7995897 Endorphine : Kat,

Thanks again for the update on the maintenance updates. I would agree that there is quite a bit of worm traffic going on. My router catches 300+ hits alone on port 135 & 137. And this is on just my IP address!?!?! Last I checked there were over 1.5 million computers infected over the life of the worms (68,000 infections each hour!). If you don't believe me check this out: »www.hackerwatch.org/checkup/graph.asp

Any information on when the Seattle routers were going to be upgraded? I though you said previous that it was going to be tonight?

Thanks again.

Marc
[text was edited by author 2003-09-17 21:56:22]

[text was edited by author 2003-09-17 21:57:02]
Click for full size
]]> http://www.dslreports.com/forum/remark,7995897 Wed, 17 Sep 2003 21:54:54 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7995402 anon : Hi Kat -

I just got home and brought up my 132Kbps stream from KUAT.

I'll check-in in the morning with my observations. But
I also don't want to call this closed until I've been
stable for several days.

Also, for my money, you don't owe anyone any apologies.
As I said, you do your job very well, howsoever you
choose to categorize it, PR or otherwise. I hope you
found me to be pursuing my legitimate interests in a
fair-minded and reasonable fashion, howsoever challenging
that may have made your life on any given day.

Your frustration was understandable. At the same time, I
didn't want people to forget that this is a business
relationship, not a personal one, and so our sympathy
for you personally should not be allowed to translate to
sympathy and a free pass for SpeakEasy as a whole. That
said, I never took this personally, and I hope you didn't
either. It's been good and educating to do business with
you.

I'm definitely heartened to hear that your CEO has been
tracking this thread. I hope that means that he realizes
that he owes you an apology for keeping you on a tight
leash in an awkward position. Oh, and a week of comp-time,
totally. At least. ;-) (I also think he owes whomever you
report to directly a good wire-brushing behind the ears.)

Thanks for hanging in there with a high-stress encounter
for all of us. Relationships are like that sometimes.
Yet SpeakEasy will move higher on my list as a vendor than
it was before all this, if I can look back and say to
myself, honestly, They learned something, they let their
customers make an impact, and service is better than the
seemingly halcyon days before the problem surfaced.

Cheers,

Carl F.
San Francisco
]]> http://www.dslreports.com/forum/remark,7995402 Wed, 17 Sep 2003 21:05:52 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7995222 KatOak : Just finished meeting with Engineering - quick update:

Tonight we'll be performing maintenance in CHI and LAX that will improve performance for circuits in Chicago, Detroit, Dayton, Los Angeles and San Diego.

I'm compiling information on NYC - I have two recorded issues (yoonix and bhan21), and will gather data from our ticket system tomorrow to determine what kind of an impact we're seeing there. Upgrades to NYC are still being discussed.

At this time, we have no reports on our other POPs or NAPs, but are continuing to monitor.

We have sent out another round of notices on the Welchia worm and will be assessing reported IPs for another notification within the next couple of days.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7995222 Wed, 17 Sep 2003 20:47:46 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7995138 paulhaskew : people, just an FYI, the PDX Pop, is not being affected by this... it was till we had the outage while ago, i have been doing tests along with all of you to your own pops and the monitoring is still going on from my end :)

heh, NYC, CHI, LAX all are being tracked through my POP, PDX...

I have very little or no packetloss from my POP, sorry for making some of you confused... :|

as for communicating with the other pops, well i will post that info in a couple of hours :)
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.
[text was edited by author 2003-09-17 20:37:17]
]]> http://www.dslreports.com/forum/remark,7995138 Wed, 17 Sep 2003 20:36:13 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7994924 wattjg : We've had *no* packet loss recorded by our monitoring software since 2330 PDT yesterday. The software (mrtg) generates loss graphs on 1500 byte pings to the SFO POP.

We're in one of the 216 netblocks.

We're still observing and reporting quite a bit of W32.Welchia ICMP traffic, though. It hasn't decreased much.

Jim]]> http://www.dslreports.com/forum/remark,7994924 Wed, 17 Sep 2003 20:12:55 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7994841 KatOak : Bondman -

While the full upgrades to our POP will be completed in early next year, we will be making improvements ASAP to improve the performance of folks in the LAX, CHI and NYC POPs. This is the information that I will provide tomorrow morning after my meeting with the Director of Engineering in a few minutes. We are monitoring SFO to confirm that this morning's transition alleviated the packet loss there. SFO was, by far, the most impacted POP.

Mike is definitely aware of the issues and this thread - and is working with me on a resolution for this. I have addressed the issue of service credit for those that are experiencing severe service-impacting packet loss and will resolve this ASAP.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7994841 Wed, 17 Sep 2003 20:05:29 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7994581 Bondman : Kat:
I appreciate the info you have given. The concern I have is the July CEO update Speakeasy gave no time frame to fix the problems. You mentioned the term several months. I interpret that to mean 4 or more months away before all of this is done. Shouldn't one get compensation for services paid for but not delivered. I am very concerned with the announcement pushing for more customers before the proper service is in place.
»Little Fish, Big Pond
I am having major issues with browsing any time in the afternoon and evening hours. Why are you (Speakeasy) not acknowledging that there are problems with the Speakeasy Network status by naming POPs involved other than the general Worm statement? I have no complaints with you as a person trying to help people and you have personally helped me in the past. I want Mike Apgar to personally see all of these postings about the poor quality of service Speakeasy is providing customers and to state why we should not get some compensation for services not delivered.
[text was edited by author 2003-09-17 19:41:42]]]> http://www.dslreports.com/forum/remark,7994581 Wed, 17 Sep 2003 19:41:08 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7994393 KatOak : Hi all -

Update here on this issue:

The transition in the SFO POP is complete - if any customers in that POP reading this have the time to post information on their circuit's performance since 8AM PST, I'd be very interested to see what kind of impact on performance this has made.

The fact of the matter is, as referenced in my original post on this thread, the added load of the worm in August and early September maxed out equipment that we had already identified as in need of an upgrade. For those of you that did not receive this CEO update that Mike Apgar wrote to all customers in July of this year, it details all of our impending network upgrades that will begin with Seattle and continue over the next several months, with upgrade priority based on both present capacity and POP load. Additionally, the ADSL service upgrades we informed customers about a few weeks back will occur in tandem with these equipment upgrades.

I am meeting again with our Director of Engineering tonight to go over the interim upgrades that we will be making to our POPs in order to improve and/or maintain the service levels for our existing customers. I will transmit a synopsis of this to all staff and post an update of it here tomorrow, most likely mid-morning.

I regret that any of our customers felt that their experience and/or this issue was being neglected by Speakeasy and that we were not able to respond in a more visible manner sooner. We are very much focused on resolving this problem in the short-term while we deploy the equipment improvements that will solve this problem in the long-term. We have had some failures in communication between our engineers and support staff -- this morning we identified some ways in which we can improve this and I'm hopeful that these measures will be put into place ASAP to ensure that our front-line support staff is as pro-active and informative as possible if & when we are faced with a similar situation in the future.

Additionally, I'd like to apologize to anyone who interpreted my posts as self-righteous in any way - that was/is absolutely not my intention. It does get frustrating to have to regularly defend whether or not I am providing accurate, honest and up-to-date information in a forum that I have participated in for over 2 years - but that's my baggage. Also, my role with Speakeasy is not as Public Relations, so sometimes my tongue gets the best of me. In fact, that's why I quit doing PR. ;)

I'll post a more detailed update around our network improvements tomorrow.

Thanks,
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7994393 Wed, 17 Sep 2003 19:22:03 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7991202 MrCoal : Here's my contibution. It's been this way 24 hours a day for months!
Click for full size
Chicago POP
]]> http://www.dslreports.com/forum/remark,7991202 Wed, 17 Sep 2003 13:15:12 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7989826 anon : Hi Charter Pipeline Support Tech -

I agree absolutely that Kat Oak is in the difficult
position of being a PR soldier, executing the decisions
of others higher-up. I have tried to say as clearly as
possible that I think she does her job very well. Based
on her performance in this and SpeakEasy's own forums,
I'd hire her in a heartbeat for my own company, if I had
one.

For that reason, I really don't want to be a thorn in her
side. I feel bad that circumstances place a good,
competent and diligent person like Kat between me and
the kind of information that SpeakEasy would share, if
it really meant to have the kind of relationships with
its customers that it says it has and values having.

At the same time, I do mean--and perhaps enjoy a little
too much--to be a thorn in the side of the greasy-pole-
climbing directors and VPs at SpeakEasy who are making all
the bone-headed decisions about how to do damage control,
and who are then hiding behind the good soldiers at
SpeakEasy like Kat and the first tier CS techs.

Unfortunately, the only way I know to put pressure on
senior management is to put back-pressure on PR spun by
people like Kat, who are the public face of SpeakEasy's
spin machine. When people understand the techniques that
PR people use to look like they're doing their level best
to answer questions when in fact they're sharing precious
little information, the playing field between small
customer and big vendor gets a bit more leveled.

Kat is a nice and earnest person, not a sneering PR
asshole, so I feel bad that she's the target of my
backpressure--but not so bad that I'm willing to hang
back and accept protractedly poor service and feeble
explanations.

What I'd really like to see at this point is for the
responsible senior players at SpeakEasy to have the
integrity to get on-line themselves, instead of hiding
behind Kat and others. I personally find it
unconscionable that her management has left Kat hanging
out to dry, while they position themselves to have their
bonuses and options packages grow. But that's what
greasy pole climbers do: They clutch and crawl on top of
the bodies of the good people that they burn out, in their
overweening sense of entitlement to get to the top.

Cheers,

Carl F.


]]> http://www.dslreports.com/forum/remark,7989826 Wed, 17 Sep 2003 10:01:45 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7989371 derk : It's 8:37AM and I am in Cleveland OH. I go through a Cleveland NAP and through the Chicago POP.

Ping statistics for 209.123.109.175:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 49ms, Maximum = 49ms, Average = 49ms

I use either a 5260 Speedstream or Zyxel modem. I agree that it looks like the Chicago POP is oversold. I work at home and I have been experiencing the latency issue all week throughout the day.

I am glad my binding contract is over with Speakeasy. I am giving this problem a month before I switch to another alternative. Well, all I can say is that this was great DSL while it lasted, but NOTHING lasts forever...

:(]]> http://www.dslreports.com/forum/remark,7989371 Wed, 17 Sep 2003 08:40:57 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7989152 Bondman : Kat:
It is 7:15 in the morning in Southeast Michigan. Below is my ping time for DET.Speakeasy.net this morning.

Ping statistics for 64.81.159.2:
Packets: Sent = 73, Received = 73, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 14ms, Maximum = 22ms, Average = 16ms

I am using the same equipment etc. I get 2 twice the ping times and some packet loss mid-day and in the evening. I have been experiencing issues every night for the past couple of weeks. Either the Chicago POP has been over sold, equipment is failing at higher stress levels or groups of users are coming on mid-day and evening are using excessive bandwidth.
[text was edited by author 2003-09-17 07:43:33]
]]> http://www.dslreports.com/forum/remark,7989152 Wed, 17 Sep 2003 07:41:51 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7988667 koitsu : I experienced too many random whacko latency/stalls of up to 10 seconds while RD'ing into home (worked great until about noon, obviously nothing changed on my end) throughout the day. My day was from 07:00 until 17:00.

Came home around 17:30, and noted PP showing the usual heavy packet loss. I spent my evening playing GBA (primarily) and chatting on Messenger, checking my Email at my co-lo off and on (stalling did not happen, but the packet loss was still there). Back to the GBA for awhile, then decided to try to do some co-lo kernel work at around 23:30, shortly prior to bed. Major stalls, a couple sockets never even came back (!). PP shows nothing other than the usual behaviour at this time of night.

Just can't deal with it any longer. Call me when the SFO fix is deployed; I'm still not giving up on SE yet, but they have until the end of the month to provide a full prognosis of what the actual problem is (oversaturation, flaky interfaces, evil Junipers (sorry, I cannot stand those bloody things; pure software routing on border and backbone routers is just plain wrong), packet kids, or whatever it might be).

Once I get a factual explanation, I'll probably stick around for another month after that, just to see if it's solved. Otherwise, consider me gone before the end of the next billing cycle, as I'll move to SBC or DSL Extreme, as two people in my apt. complex (one in my building!) have DSL through those providers and their traceroutes are much more golden.

So I'm back to the cable modem for the next few days. Flippant 25-75ms latency sucks for gaming or anything fun like that, but at least the connection is solid and there aren't stalling sockets, so at a bare minimum I can get some co-lo work done.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7988667 Wed, 17 Sep 2003 03:53:35 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7987383 paulhaskew : that is way worse than what i am getting, i will post a 24 hour graph tonight when i get home... :) cheers...
[text was edited by author 2003-09-17 02:01:01]
Click for full size
]]> http://www.dslreports.com/forum/remark,7987383 Tue, 16 Sep 2003 23:30:18 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7987238 GamingGirl : This is what I saw when I got home tonight. I shut it down at 8pm as I didn't want my browsing to affect the results. It looks as if the packet loss really kicks in around the time everyone is booting up their systems in the morning and continues throughout the rest of the day. The ping spikes on my second hop are pretty much the only ones I see. My ping is fairly stable the rest of the way.

I will email you my information tonight Kat. =)
Click for full size
9/16/03
]]> http://www.dslreports.com/forum/remark,7987238 Tue, 16 Sep 2003 23:17:22 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7986308 paulhaskew : the PDX pop is no longer having the packet loss issues i was seeing over the past week. Not sure if this was corrected with the last two outages that happened this month...

Carl you speak with fluid grace, but remember she is the PR person for the company on these forums, and i beleive that is the reason why she is the only person posting here. As for not releasing exactly what the issue is, I can understand that as well, as the company i work for does the exact same thing. most of the time the EU/End User, you and me, do not get all the information we want. yes we are paying for a service and yes we should get answers to our questions, but you more than anyone from reading your posts, should know that information of this type is not always available to able to be released due to NDA/Contracts and the what not...

/END
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.]]> http://www.dslreports.com/forum/remark,7986308 Tue, 16 Sep 2003 21:43:38 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7984397 Bondman : Kat:
The service through the Detroit NAP is very poor. If it is going to take months to get this matter resolved how about some credit on our accounts.

Below is the ping results I just took moments ago.

Ping statistics for 64.81.159.2:
Packets: Sent = 53, Received = 53, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 25ms, Maximum = 64ms, Average = 39ms

At times I do get Packet loss. I used to get regularly pings of 16ms. I could not ping my server from a Charter School that I was at today. I have one of the old ADSL on a separate line that I had when I switched over from Flashcom. I am very leery of changing to line share to save $10 when I am not getting the service I am paying for now and then get stuck on a new year contract. If one does switch do they get a new modem and the phone filters?
[text was edited by author 2003-09-16 18:59:22]
]]> http://www.dslreports.com/forum/remark,7984397 Tue, 16 Sep 2003 18:47:56 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7983680 anon : Sorry for the duplicate post. I've asked to have one removed...

Carl F.
]]> http://www.dslreports.com/forum/remark,7983680 Tue, 16 Sep 2003 17:26:24 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7983599 anon : Hi Kat -

I label my inferences as such, to make it clear that I'm
guessing and that I know I'm guessing. I'm sure some of
my guesses are off the mark to some extent or another.
I'm also sure that some of my guesses are fairly close
to the mark. If I had anything close to adequately
complete information, I wouldn't have to guess! :^)

What follows, however, are not inferences, they're
behavioral observations.

1. You poke your head up from time to time to tell us
what the problem is definitely not. So far since
this issue arose in the last month, you've never
spoken-up to tell us anything definite and
comprehensive about what the problem is. (Blaster
was a stressor, not a root cause.)

2. You're being very clear on what seem like relevant
facts: No upgrades to the SFO POP, no bandwidth caps,
period! I believe these things are true. Period! :^)
But they are unresponsive factoids. They state some
tidbit that dramatically narrows the intent of what
someone wrote, and then responds to that narrowed
intent. I'm not a networking expert, so if I use
a phrase like "bandwidth capping" to describe
symptoms I've laid out in detail, I expect you to tell
me that my symptoms are more consistent with what
folks call "traffic throttling" (as Jester did), and
then go on to address the broader concern. Similarly,
I didn't say that the upgrade possibly gone bad was
on the SFO POP, and yet you respond narrowly to SFO,
when I--and your CEO--are both talking about an
infrastructure upgrade throughout SpeakEasy. You
use the rhetoric and cadences of a straight-talker,
but I'm finding that understanding you and others from
SpeakEasy depends on what the meaning of 'is' is,
and how that narrows and misdirects my attention.

3. When you offer a "debunking" factoid, you write with
emphasis and a slight edge of self-righteousness,
like a beleaguered person trying hard to be responsive.
However, having seen other posts from you that have been
substantively responsive in abundant detail, without
all the emphasis, you've evidenced that you're very
smart and articulate, and know the difference between
responsiveness and unresponsiveness. I infer that you
know that you're not currently offering us direct
responses, whatever emphatic edge you put on the
factoids you offer. Emphasis is what you're using to
backfill for whatever hard facts you're not sharing
(and perhaps are not being shared with you).

4. You've offered several of us escalation, which seems
like very proactive, can-do problem solving. But there
has been no follow-up from whomever you might have
escalated to. The best I can say is that you are
a proactive element in a largely inactive support
organization.

5. If you respond to nothing else, please respond to this.
It's been almost three weeks since I opened my first
trouble ticket.

WHO IS THE SINGLE POINT OF OWNERSHIP FOR MY PROBLEM, OR, WHY, AFTER SO MUCH TIME, DON'T I HAVE ONE WHO I CAN CONTACT DIRECTLY AND TROUBLE-SHOOT WITH CONJOINTLY?

You're giving off all of the emotional signals of
owning our problems, Kat, but in actual substance
you don't fully own any of our problems, and I can't
see that anyone does. You guys aren't hotwired to
each other like the Borg, so I can't expect nearly
as consistent an outcome, knowing that my problem is
somewhere in the SpeakEasy Collective. ;^) Talk to me
about my single point of ownership, Kat, and how I
arrange to work directly and collaboratively with her
or him.

Finally, Kat, please don't get me wrong. I think you're
doing a great job for SpeakEasy. If I had a high-tech
company with a substantial online presence and customer
contact, I'd hire you in a heartbeat. You're bright,
articulate, good-humored, and quickly connect well with
distressed customers. You should definitely be on
SpeakEasy's must-retain short list.

All I ask is that you don't treat me/us like we're too
naive to know that doing a great job for SpeakEasy is not
always the same as solving our problems. Sometimes the
body of facts is going to outrun your ability to spin
them. At that point, it's time to go back to your bosses
(as you may well be doing) and insist on developing and
deploying Plan B. I'm at the point where I'd like to see
Plan B in evidence.

The problem, I'm guessing, will be getting the people you
work for to embrace Plan B, or even that they need one.
I'm particularly under-whelmed at the agility of your
management team under crisis. Most every response I've
received has been either an unfulfilled promise, or else
something that would have been timely and forthcoming two
to three days earlier, instead of begrudgingly, belatedly
acknowledging to me what I'd already figured out for myself.

Best regards,

Carl F.
San Francisco



]]> http://www.dslreports.com/forum/remark,7983599 Tue, 16 Sep 2003 17:15:05 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7983562 anon : Hi Kat -

I label my inferences as such, to make it clear that I'm
guessing and that I know I'm guessing. I'm sure some of
my guesses are off the mark to some extent or another.
I'm also sure that some of my guesses are fairly close
to the mark. If I had anything close to adequately
complete information, I wouldn't have to guess! :^)

What follows, however, are not inferences, they're
behavioral observations.

1. You poke your head up from time to time to tell us
what the problem is definitely not. So far since
this issue arose in the last month, you've never
spoken-up to tell us anything definite and
comprehensive about what the problem is. (Blaster
was a stressor, not a root cause.)

2. You're being very clear on what seem like relevant
facts: No upgrades to the SFO POP, no bandwidth caps,
period! I believe these things are true. Period! :^)
But they are unresponsive factoids. They state some
tidbit that dramatically narrows the intent of what
someone wrote, and then responds to that narrowed
intent. I'm not a networking expert, so if I use
a phrase like "bandwidth capping" to describe
symptoms I've laid out in detail, I expect you to tell
me that my symptoms are more consistent with what
folks call "traffic throttling" (as Jester did), and
then go on to address the broader concern. Similarly,
I didn't say that the upgrade possibly gone bad was
on the SFO POP, and yet you respond narrowly to SFO,
when I--and your CEO--are both talking about an
infrastructure upgrade throughout SpeakEasy. You
use the rhetoric and cadences of a straight-talker,
but I'm finding that understanding you and others from
SpeakEasy depends on what the meaning of 'is' is,
and how that narrows and misdirects my attention.

3. When you offer a "debunking" factoid, you write with
emphasis and a slight edge of self-righteousness,
like a beleaguered person trying hard to be responsive.
However, having seen other posts from you that have been
substantively responsive in abundant detail, without
all the emphasis, you've evidenced that you're very
smart and articulate, and know the difference between
responsiveness and unresponsiveness. I infer that you
know that you're not currently offering us direct
responses, whatever emphatic edge you put on the
factoids you offer. Emphasis is what you're using to
backfill for whatever hard facts you're not sharing
(and perhaps are not being shared with you).

4. You've offered several of us escalation, which seems
like very proactive, can-do problem solving. But there
has been no follow-up from whomever you might have
escalated to. The best I can say is that you are
a proactive element in a largely inactive support
organization.

5. If you respond to nothing else, please respond to this.
It's been almost three weeks since I opened my first
trouble ticket.

WHO IS THE SINGLE POINT OF OWNERSHIP FOR MY PROBLEM, OR, WHY, AFTER SO MUCH TIME, DON'T I HAVE ONE WHO I CAN CONTACT DIRECTLY AND TROUBLE-SHOOT WITH CONJOINTLY?
You're giving off all of the emotional
signals of owning our problems, Kat, but in substance
you don't fully own any of our problems, and I can't
see that anyone does. You guys aren't hotwired to
each other like the Borg, so I can't expect nearly
as consistent an outcome, knowing that my problem is
somewhere in the SpeakEasy Collective. ;^) Talk to me
about my single point of ownership, Kat, and how I
arrange to work directly and collaboratively with her
or him.

Finally, Kat, please don't get me wrong. I think you're
doing a great job for SpeakEasy. If I had a high-tech
company with a substantial online presence and customer
contact, I'd hire you in a heartbeat. You're bright,
articulate, good-humored, and quickly connect well with
distressed customers. You should definitely be on
SpeakEasy's must-retain short list.

All I ask is that you don't treat me/us like we're too
naive to know that doing a great job for SpeakEasy is not
always the same as solving our problems. Sometimes the
body of facts is going to outrun your ability to spin
them. At that point, it's time to go back to your bosses
(as you may well be doing) and insist on developing and
deploying Plan B. I'm at the point where I'd like to see
Plan B in evidence.

The problem, I'm guessing, will be getting the people you
work for to embrace Plan B, or even that they need one.
I'm particularly under-whelmed at the agility of your
management team under crisis. Most every response I've
received has been either an unfulfilled promise, or else
something that would have been timely and forthcoming two
to three days earlier, instead of begrudgingly, belatedly
acknowledging to me what I'd already figured out for myself.

Best regards,

Carl F.
San Francisco



]]> http://www.dslreports.com/forum/remark,7983562 Tue, 16 Sep 2003 17:12:10 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7983452 falkclan : WOW
I had been forgetting to check into these forums :(

I'm glad I did, but last night I FINALLY put in a ticket to speakeasy

my normal ping to chi.speakeasy.net (2nd hop) from here in Kansas City was 18ms

now I get 18ms to the gateway hop (x.x.x.1) and like 50ms to chi.speakeasy.net

seems it's widespread, hoping for a fix
I emailed you Kat, with my specific ticket
thanks
--
Alan Falk]]> http://www.dslreports.com/forum/remark,7983452 Tue, 16 Sep 2003 16:55:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7982300 KatOak : Definitely - and we're still working on the issue as reported in NYC, CHI and LAX. If possible, can you provide me with your account name via email?

Thanks,
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7982300 Tue, 16 Sep 2003 14:27:48 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7982065 GamingGirl : I do appreciate your updates Kat. However, my problem is occurring on the LAX POP and despite my many attempts, not one person has addressed my concerns. I have updated my service ticket numerous times, I've sent email to a first level tech with his assurance that he would pass it to the AS tech assigned to my ticket. It has been over a week and the AS tech has not even bothered to call me back. When I call in to speak with him, he is always on another call. Covad has closed my ticket. Speakeasy has not provided me any reassurance that my problem is even valid. All I want is a usable connection and if that is not possible, I at least want an explanation about what you (as in Speakeasy) are doing to correct the problem. Telling me to hang in there while I am still being charged full price for a less than fully working connection is unsuitable, especially when these updates aren't even pertaining to me. I'm sure you could understand my frustration.]]> http://www.dslreports.com/forum/remark,7982065 Tue, 16 Sep 2003 13:55:59 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7981764 KatOak : Our POP upgrades will begin with Seattle - we have not installed new hardware in our SFO POP. As I posted on Friday, we will be transitioning a large portion of customers over to a new circuit and router tonight to alleviate the max on packets per second that we're seeing on one of our SFO routers.

Basically, anyone here can infer, imply or extrapolate to their heart's content regarding the cause of this or any other issue. That's pretty much one of the reasons that this forum is here - for open discussion of questions, concerns and experiences. But I have always provided straight information in this forum over the past two years and will continue to do so. Whether or not you elect to accept that information as fact is a personal decision.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7981764 Tue, 16 Sep 2003 13:17:09 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7980245 Chicago_DSL6 : If this doesn't get the wheels in motion...

Anyone experiencing these issues and can not seem to get any sort of resoltuion after a reasonable amount of time, visit »www.secfraud.com/ and email Marc Godino with your issues. He worked on a lawsuit against Earthlink in California and won.]]> http://www.dslreports.com/forum/remark,7980245 Tue, 16 Sep 2003 09:21:00 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7971996 anon : Here's a link to a different kind of data, and more
discussion, in SpeakEasy's own forums.

»forums.speakeasy.net/6/ubb.x?a=t···1608163

My take is that it's neither the loss of average bandwidth
nor the intermittent packet loss that's killing our service,
but the VARIABILITY, the burstiness and the stalls that
are so long that packet recovery schemes built on top
of UDP (such as RealPlayer's fairly robust method)
cannot possibly recover lost packets on time.

I agree that SpeakEasy is buying time at this point. I
have no access to information, and yet I am beginning to
feel as though I am characterizing the problem more
quickly than they are. Of course, this is ridiculous.
SpeakEasy has not provided such great service over the
years, and grown so impressively, with a bunch of idiots
on their staff. They are very bright people who have
access to much more information and who have much more
experience in interpreting their information. To think
that I'm out in front of them characterizing their
problem is silly.

I infer that SpeakEasy knows much more than they are
choosing to share at this point. The amount of
misinformation and scripted misdirection of attention from
a technical service department that used to be consistently
great is a strong indication that SpeakEasy is buying
time. Why keep running phy level tests with Covad, when
the problem is time-of-day sensitive? Is the concern that
after the sun comes-up and the line warms-up, that its
capacity drops precipitously? ;^) If I were Covad, I'd be
getting pretty annoyed at the amount of my tech support
resource being squandered by SpeakEasy just now.

SpeakEasy must be a much less fun place to work these days.
A crew that used to enjoy straightforward and direct
relations with customers now has to play poker for a living.
I certainly don't envy Kat Oak her job right now. What
a tough position to be in!

There's a memo from their CEO announcing a major
upgrade to Juniper equipment on their forum. Perhaps
the upgrade is not going well. Perhaps they are trying
out new traffic management policies with the new
equipment, and our midday packet loss is an unanticipated
side-effect. Perhaps SpeakEasy is testing the threshold
of pain in our segment of their market, to see how much
stability of traffic they can divert to big ticket
enterprise customers. To the extent SpeakEasy has
enterprise customers in the $10,000+ per month range, our
complaints are unlikely to be handled with the same
priority.

But again, this could just be an upgrade gone really bad.
After all, if SpeakEasy is not a big ticket customer for
Juniper, Juniper might have cocked-up the installation,
and is now tap-dancing for SpeakEasy the way that SpeakEasy
is tap-dancing for us.

My willingness to stick with SpeakEasy depends upon (1)
quick resolution of problems and (2) honest characterization
of problems when quick resolution is not possible. That
honest characterization acknowledges, in our business
relationship, that we are both incurring damage. When
problems are not forthrightly acknowledged, then SpeakEasy
makes the relationship all about their bottom line and
their saving face.

Please, SpeakEasy, get your network architects into this
discussion, and have them tell us what they know, and
how SpeakEasy plans to make all of us whole.

Cheers,

Carl F.
San Francisco ]]> http://www.dslreports.com/forum/remark,7971996 Mon, 15 Sep 2003 12:29:37 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7971814 koitsu : Latest update. From about 15:45 yesterday afternoon until 09:03 this morning.

Large square-wave ("ramping") effect you see at destination between 06:00 and 09:00 this morning caused by my co-lo provider doing some cabling maintenance.

P.S. -- The "packet stalling" problem I mention every so often was in full effect last night, particularly between 22:15 and 23:15. Graph should prove that beyond a doubt.
--
Making life hard for others since 1977.
[text was edited by author 2003-09-15 12:08:47]
Click for full size
SFO 09/14 - 09/15
]]> http://www.dslreports.com/forum/remark,7971814 Mon, 15 Sep 2003 12:05:40 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7971569 paulhaskew : wow, thats insane gaminggirl...

mine shows loss only on my 2nd hop, which is a speakeasy router...

other than that, no loss...

i have two routes going right now with ping plotter, only another 12-15 hours and it wll be all done...
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.]]> http://www.dslreports.com/forum/remark,7971569 Mon, 15 Sep 2003 11:24:28 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7969719 GamingGirl : Here is my ping plotter for one hour right before I called in for tech support (had to follow up on my support ticket ya know). I told the tech on the line that I wanted to send the .jpg to my advanced support technician (he was on the phone with Covad at the time...hmmm) so I ended up emailing it to the first level tech in the hopes that he would pass it on. He promised me that the AS tech assigned to my ticket would call me back... he never did =(. I have a feeling I'll get a call in the middle of the day tomorrow when I'm at work (isn't that how it always is?)... Anyhow, this ping plotter thing is fun =). I think I'll let it run tonight while I sleep and tomorrow while I work just to see how it does.
Click for full size
]]> http://www.dslreports.com/forum/remark,7969719 Mon, 15 Sep 2003 01:44:43 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7961493 paulhaskew : Tracing route to vnsc-pri.sys.gtei.net [4.2.2.1]
over a maximum of 30 hops:

1 20 ms 19 ms 19 ms gate.tailie.net [66.93.174.1]
2 * 19 ms 20 ms ge0-0-0.brd-1-sea.speakeasy.net [206.191.168.200
]
3 20 ms 19 ms 19 ms border26s.g3-2.speakeasy-14.sea.pnap.net [206.19
1.168.220]
4 20 ms 19 ms 18 ms core4.ge0-0-0-bbnet1.sea.pnap.net [206.253.192.1
44]
5 18 ms 19 ms 18 ms p6-0.sttlwa1-cr1.bbnplanet.net [4.25.88.17]
6 18 ms 21 ms 22 ms so-2-0-0.sttlwa1-hcr1.bbnplanet.net [4.24.10.238
]
7 20 ms 20 ms 21 ms p3-0.sttlwa2-br1.bbnplanet.net [4.24.11.201]
8 20 ms 20 ms 19 ms p1-0.sttlwa2-cr1.bbnplanet.net [4.24.11.194]
9 20 ms 18 ms 18 ms vnsc-pri.sys.gtei.net [4.2.2.1]

Trace complete.

there i did that tracert, gonna start pingplotter and see where it takes me for the next 48 hours... :)]]> http://www.dslreports.com/forum/remark,7961493 Sun, 14 Sep 2003 02:38:13 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7961286 GamingGirl : koitsu, you are very informative =). I appreciate that... you're helping me understand a little better about things that Speakeasy can't or won't take the time to explain to me. Thank you =).

I downloaded the ping plotter program a few nights ago so last night I let it run while I slept... attached are the results. I guess I'm not seeing the dropped packets as badly as you are but nonetheless I am still seeing poor performance.

Both Covad and Speakeasy refuse to admit that there is any kind of problem or change that could possibly have affected my service (go figure *sigh*) but if this remains to be the norm over the next week or two, I'm just going to wave goodbye, pay the early termination fee and find a more suitable provider.

It's just a shame that they must think that I have nothing better to do than sit on hold, wait for a tech, explain my problem and hope for a resolution. One thing I have learned through working in customer service is that you don't hear from the customer when things are going good, you only hear from them when things are going bad. To be told that there is nothing wrong is saddening at the least =(.

I truly hope that you all find the resolution you are looking for because I'm not patient enough (nor willing to pay the incredible fees each month) to sit around praying that things will return to normal. The least they could do is say "ma'am, I am sorry that you aren't happy with our service, we will do everything we can to make things right"... instead I hear "ma'am, we don't even want to hear about what's affecting you unless you can show proof since we can't seem to reproduce the problem with our 'tools'". In fact, I don't even hear that unless I call in since Advanced Support doesn't seem willing to pick up the phone and follow up. =(

I don't get paid by Speakeasy to research and find out what is going on with my line, rather I pay them to do that for me and I'm just not seeing enough from their side to warrant my payment. I am truly disappointed.
Click for full size
LAX to SFO
]]> http://www.dslreports.com/forum/remark,7961286 Sun, 14 Sep 2003 01:46:35 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7960815 koitsu : Following in the footsteps of the thread subject, included above is the latest Ping Plotter results over the past 48 hours (and then some).

I'm somewhat bothered by the fact that today -- Saturday -- shows no ramping usage spike from 09:00 until midnight (or, well, technically 21:33).

Just posting this for Kat and anyone else who's interested in how SFO is doing. That huge packet loss problem around 23:30 to midnight last night, I'm hoping, was caused either by Verio or possibly changes with InterNAP or SE.

That's all for now!
--
Making life hard for others since 1977.
Click for full size
SE, 09/12 to 09/13
]]> http://www.dslreports.com/forum/remark,7960815 Sun, 14 Sep 2003 00:33:45 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7958271 paulhaskew : yeah, i noticed that as well... i downloaded the free pingplotter yesterday and the tracert completed... :(

i will test some other stuff once i get home tonight...

here is something i did from work, ping test almost all day, from 1:30pm to 8:53pm...

Ping statistics for 66.93.174.127:
Packets: Sent = 26297, Received = 26290, Lost = 7 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 15ms, Maximum = 547ms, Average = 18ms

--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.
[text was edited by author 2003-09-13 23:53:26]
]]> http://www.dslreports.com/forum/remark,7958271 Sat, 13 Sep 2003 19:03:14 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7958082 koitsu : So to sum everything up, literally InterNAP is just a customer who happens to have service with multiple backbone providers (Global Crossing, Verio, Level 3, Alter, and a few other who's names are slipping my mind since I just woke up), then proceed to resell portions of their own POP, with their mysterious BGP setup deployed, to customers (like SE) for network connectivity.

Likewise, the reason for their entertaining merry-go-round routing procedures is a purely financial one due to the costs of aggregate bandwidth total per peering point (hence balancing out the traffic between all providers rather than shoving it all through one which happens to have the lowest hop count and more reliable link (at that time)).

What a bummer.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7958082 Sat, 13 Sep 2003 18:40:02 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7958019 koitsu : The examples you show above make it all the way to a Sprint peering point in Chicago (for BBR) and a peering point with Level 3 in (probably) their Santa Clara POP (for Yahoo).

I can't provide you traceroutes for comparison to either www.yahoo.com or www.broadbandreports.com because the route packets go from my co-lo to either of those sites are completely different. (Heck, my co-lo is 6 hops from www.yahoo.com, go figure).

Try tracerouting to a different destination. For example, 4.2.2.1 (which is a public nameserver hosted by GTEI):

Tracing route to vnsc-pri.sys.gtei.net [4.2.2.1]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.0.1
2 11 ms 11 ms 11 ms dsl081-051-001.sfo1.dsl.speakeasy.net [64.81.51.1]
3 11 ms 12 ms 12 ms border5.g3-4.speakeasy-29.sfo.pnap.net [63.251.53.219]
4 12 ms 11 ms 11 ms core1.ge0-0-bbnet1.sfo.pnap.net [63.251.63.1]
5 12 ms 12 ms 12 ms sl-gw13-sj-0-2-155M.sprintlink.net [160.81.100.1]
6 13 ms 12 ms 11 ms sl-bb21-sj-6-0.sprintlink.net [144.232.3.173]
7 13 ms 13 ms 12 ms sl-st20-sj-13-0.sprintlink.net [144.232.9.58]
8 15 ms 14 ms 13 ms interconnect-eng.SanJose1.Level3.net [209.245.146.245]
9 13 ms 14 ms 14 ms so-5-0-0.gar1.SanJose1.level3.net [209.244.3.137]
10 15 ms 15 ms 13 ms so-7-0-0.mp1.SanJose1.Level3.net [64.159.1.73]
11 78 ms 79 ms 79 ms so-0-2-0.bbr1.Atlanta1.level3.net [209.247.9.101]
12 78 ms 78 ms 79 ms pos8-0.hsa1.Atlanta1.Level3.net [209.247.9.166]
13 79 ms 81 ms 78 ms vlan521.public-msf1.Atlanta2.Level3.net [67.72.92.18]
14 78 ms 81 ms 79 ms vnsc-pri.sys.gtei.net [4.2.2.1]

One thing to note: it looks to me that Ping Plotter and Windows traceroute may be using different protocols (possibly ICMP vs. UDP) for their tests. I cannot traceroute to my co-lo using Windows tracert (dies at hop #5), while using Ping Plotter from the same box works fine. Hop #5, in this case, happens to be so-3-0-0.ar4.SFO1.gblx.net.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7958019 Sat, 13 Sep 2003 18:30:52 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7958011 sporkme :
said by koitsu See Profile:
InterNAP is who Speakeasy peers with. InterNAP peers with all sorts-of different providers; it's synonymous with the PAIX in Palo Alto (i.e. a central point for many backbone providers to connect to one another, hence making the Internet work).


It's not really synonymous. InterNAP buys *transit* from mutliple providers, likely at some fixed price point if they guarantee "X" number of megabits/sec average. At a peering point like PAIX, providers go there to exhange *peering*, which is *not* the same as paying for transit. They will agree on a pipe of some size, who will pay what portion of the interconnect, and agree that this is of mutual benefit to them as ISP X's customer's can now directly reach ISP Y's customers without either side paying each other for transit. These days it's a bit more complicated than that, but that's the theory at least.

InterNAP is simply a paying customer of a number of backbones. They have no network interconnecting their PoPs, each one is an "island". The original plan was to sell "premium connectivity", but ever since they've had financial difficulties, it's become a bit different. You now see more and more traffic going out the cheapest providers. One day we'll see Cogent in the mix. :)

said by koitsu See Profile:

InterNAP has some seriously magic BGP voodoo going on -- they drop/swap routes practically on an hourly basis. Even when the interface you're going through is absolutely 100% clean (e.g. prior to all of this packet loss crap), they will _still_ drop the route and switch you from, for sake of example, GBLX to ALTER for absolutely no reason. That's why I say it's "magic BGP voodoo." This is NOT how BGP is normally supposed to be used, and it looks to me like InterNAP simply uses some excessively anal settings, or simply plays spin-a-roo with their routes (I've heard of peering providers doing this as well, but it's not very common).


This is certainly on purpose, and it's so that they meet their contractual agreements with each provider without going over/under what they've agreed to buy from each provider. I would not be at all surprised to find a RouteScience box at each PoP twiddling all these paths for them. RouteScience, »www.routescience.com/, is a really nifty thing, but it's just a tool. It's generally used to *improve* performance, but it also has many options that let you say "we don't want to average more than 10Mb/s this month, we don't want to average more than 40Mb/x this month to Level3, but we'll go up to 100Mb/s to Verio, and Cogent is unlimited".

Hopefully one day SE will get more neteng people on board and think about simply multihoming themselves to 5 or 6 providers at each PoP and managing this themselves. It requires more time and talent, but in today's fluctuating bandwidth market, it makes sense to add and drop providers on a regular basis. If they are in good facilities with low x-connect charges, it's a win-win.
--
just a minute]]> http://www.dslreports.com/forum/remark,7958011 Sat, 13 Sep 2003 18:29:53 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7957703 wattjg : They may be blocking UDP but not ICMP echo requests. The only thing that's odd is that the Windows "tracert" uses ICMP. Unix traceroutes commonly use UDP by default. Both Windows and Unix "ping" use ICMP.

That's not consistent with your data, but I was able to "tracert" (Windows) to www.dslreports.com, but not "traceroute" (Unix/FreeBSD) to it. I can "ping" it from both machines.

Jim]]> http://www.dslreports.com/forum/remark,7957703 Sat, 13 Sep 2003 17:44:53 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7957665 paulhaskew : well they used to complete :)]]> http://www.dslreports.com/forum/remark,7957665 Sat, 13 Sep 2003 17:40:25 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7957453 Bondman : Paul:
I get timeouts on tracerts to WWW.BroadbandReports.com as well unfortunately but I am able to ping them. I think Speakeasy needs to push for better service from their backbone providers.

tracert www.broadbandreports.com

Tracing route to broadbandreports.com [209.123.109.175]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms ml350.server-tech.local [10.1.1.1]
2 17 ms 16 ms 16 ms dsl093-002-001.det1.dsl.speakeasy.net [66.93.2.1
]
3 30 ms 19 ms 32 ms border5.ge3-2.speakeasy-28.chg.pnap.net [64.94.3
5.212]
4 19 ms 19 ms 19 ms core2.fe0-1-bbnet2.chg.pnap.net [64.94.32.66]
5 19 ms 18 ms 20 ms POS3-1.GW1.CHI13.ALTER.NET [65.207.239.21]
6 20 ms 18 ms 18 ms 0.so-1-0-0.XL1.CHI13.ALTER.NET [152.63.69.178]
7 21 ms 20 ms 20 ms 0.so-2-2-0.XL1.CHI2.ALTER.NET [152.63.70.102]
8 19 ms 20 ms 19 ms POS6-0.BR2.CHI2.ALTER.NET [152.63.67.241]
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * *

ping www.broadbandreports.com

Pinging broadbandreports.com [209.123.109.175] with 32 bytes of data:

Reply from 209.123.109.175: bytes=32 time=53ms TTL=53
Reply from 209.123.109.175: bytes=32 time=52ms TTL=53
Reply from 209.123.109.175: bytes=32 time=53ms TTL=53
Reply from 209.123.109.175: bytes=32 time=63ms TTL=53

Ping statistics for 209.123.109.175:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 52ms, Maximum = 63ms, Average = 55ms

[text was edited by author 2003-09-13 17:28:49]
]]> http://www.dslreports.com/forum/remark,7957453 Sat, 13 Sep 2003 17:09:35 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7957013 paulhaskew : any idea why my tracerts no longer complete?]]> http://www.dslreports.com/forum/remark,7957013 Sat, 13 Sep 2003 16:00:32 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7953668 koitsu : InterNAP is who Speakeasy peers with. InterNAP peers with all sorts-of different providers; it's synonymous with the PAIX in Palo Alto (i.e. a central point for many backbone providers to connect to one another, hence making the Internet work).

InterNAP is a double-edged sword, and I'm sure Kat can talk a bit about why this is as well. It's fantastic -- when everything is working how it should be. Otherwise it's an absolute nightmare to debunk, since they seem to change routes almost at the drop of a coin.

InterNAP has some seriously magic BGP voodoo going on -- they drop/swap routes practically on an hourly basis. Even when the interface you're going through is absolutely 100% clean (e.g. prior to all of this packet loss crap), they will _still_ drop the route and switch you from, for sake of example, GBLX to ALTER for absolutely no reason. That's why I say it's "magic BGP voodoo." This is NOT how BGP is normally supposed to be used, and it looks to me like InterNAP simply uses some excessively anal settings, or simply plays spin-a-roo with their routes (I've heard of peering providers doing this as well, but it's not very common).

The rep. you spoke to was correct in his description of what's supposed to happen -- when there's a network outage or major anomaly (i.e. packet loss, CRC errors, line errors, or anything else that generally affects POS (Packet Over Sonet) links), BGP will automatically say "Link is screwed, dropping route automatically, moving to another peer." In about 90% of the cases out there, this works great. InterNAP simply deploys this along side a bunch of other stuff -- it's the other stuff which is causing your traceroutes to show different routes every hour, or sometimes as often as every 15 minutes. Oh, BGP stands for Border Gateway Protocol, by the way -- it's just a monitoring protocol that allow routers to make automatic configuration changes based upon rulesets to determine if a network is functioning "up to par" or not.

The best way to understand all of this is to get the utility called Ping Plotter and let it run for a few hours again a website you commonly visit (use www.news.com if you need a default). If you know how to use traceroute, then PP will make perfect sense to you -- except that it's a graphical representation of a continual traceroute. One of the coolest features of PP is that it detects and logs route changes, so you can see exactly where, when, and why a route was changed. The other thing it's useful for is showing actual graphs of packet loss and latency. Try it out, you might like it -- for *IX I prefer mtr (matt's traceroute), while for Windows I prefer Ping Plotter.

Hope this helps.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7953668 Sat, 13 Sep 2003 06:37:10 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7953653 koitsu : Sadly, Verio here in SFO is absolute garbage (I can vouch for this, as I worked for them for 3 years here in the Bay); packet loss madness. I'm presently seeing a consistent 5-6% loss from Verio. It does not fluxuate. It's good to know Verio's PDX POP is still standing (the head west coast network engineer lives there ;-) ).

The two best peering points through SFO InterNAP look to be Level3 and Alter. At least for where I'm commonly going those look to be the best.

I'll cover the "random route" thing that GamingGirl asked in the next post.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7953653 Sat, 13 Sep 2003 06:27:19 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7953436 paulhaskew : my stuff is getting routed sadly through level 3 and keeps dropping tons of packets with tracerts... it sucks...

AT&T is the carried for the PDX area... and they have problems big time... not sure why speakeasy chose to this provider as the backbone transport...

Verio/Sprintlink are nice... good ping times etc...

anywho... i dunno... i am sooo tired...

example...

traceroutes to both yahoo & here...

Tracing route to broadbandreports.com [209.123.109.175]
over a maximum of 30 hops:

1 19 ms 19 ms 19 ms gate.tailie.net [66.93.174.1]
2 39 ms 44 ms * ge0-0-0.brd-1-sea.speakeasy.net [206.191.168.200
]
3 17 ms 17 ms 18 ms border26s.g3-2.speakeasy-14.sea.pnap.net [206.19
1.168.220]
4 17 ms 17 ms 16 ms core1.ge2-0-bbnet1.sea.pnap.net [206.253.192.129
]
5 17 ms 17 ms 16 ms sl-gw11-sea-1-0.sprintlink.net [144.228.95.45]
6 16 ms 15 ms 15 ms sl-bb21-sea-9-3.sprintlink.net [144.232.6.117]
7 59 ms 59 ms 59 ms sl-bb25-chi-2-0.sprintlink.net [144.232.20.157]

8 59 ms 59 ms 59 ms sl-bb21-chi-13-0.sprintlink.net [144.232.26.89]

9 61 ms 61 ms 60 ms sl-st20-chi-15-1.sprintlink.net [144.232.20.80]

10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.

Tracing route to yahoo.com [66.218.71.198]
over a maximum of 30 hops:

1 20 ms 19 ms 19 ms gate.tailie.net [66.93.174.1]
2 * * * Request timed out.
3 19 ms 19 ms 19 ms border26s.g3-2.speakeasy-14.sea.pnap.net [206.19
1.168.220]
4 17 ms 18 ms 18 ms core2.ge2-0-bbnet1.sea.pnap.net [206.253.192.130
]
5 17 ms 17 ms 17 ms 12.124.173.37
6 17 ms 16 ms 16 ms gbr2-p60.st6wa.ip.att.net [12.123.44.118]
7 18 ms 18 ms 18 ms tbr1-p012601.st6wa.ip.att.net [12.122.12.161]
8 15 ms 16 ms 15 ms ggr1-p340.st6wa.ip.att.net [12.123.44.129]
9 33 ms 33 ms 32 ms att-gw.sea.level3.net [192.205.32.22]
10 34 ms 34 ms 34 ms ge-6-0-1.mp1.Seattle1.level3.net [209.247.9.45]

11 34 ms 34 ms 33 ms so-3-0-0.mp2.SanJose1.Level3.net [64.159.1.130]

12 33 ms 34 ms 34 ms gige10-2.ipcolo3.SanJose1.Level3.net [64.159.2.1
69]
13 36 ms 35 ms 34 ms unknown.Level3.net [64.152.69.30]
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.

They never complete anymore... :( can ping the sites though
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.]]> http://www.dslreports.com/forum/remark,7953436 Sat, 13 Sep 2003 03:47:02 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7953385 GamingGirl :
said by TrueAudio See Profile:
...and there seems to be allot of route changes going on. I haven't been keep much track of the packet loss, but I do notice it...


I've noticed a lot of routing changes as well. I asked the rep. that I spoke with earlier why I was seeing all these different routes to one of the websites I typically trace (out of curiosity of course =) ) and he told me that "sometimes when a router goes down the route will get changed but Speakeasy hasn't made any changes to the way things get routed". However, I am seeing a lot of sprintlink and att routes that I never saw before. I used to see mainly ALTER for my trips across the US. However, as I said in a previous post, I'm not technically inclined in the least so it could all still be the same but with a different name... what do I know =)]]> http://www.dslreports.com/forum/remark,7953385 Sat, 13 Sep 2003 03:25:30 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7953362 GamingGirl : Ok, this is what I found in my service ticket area when I logged into MySpeakeasy...

2003-09-12 17:00:30 Backhaul installed
Statused TT. ISP has reported Dropping Packets (Slow throughput, latency, packet loss). Training Starts: 0 ( 69 ) Reprovisioned loop per ISP request. Training Starts: 0 ( 70 ). This circuit has had 1 previous trouble tickets for RMA. TT will be placed into CPPV> TT will auto close w/in 24 hrs.
Not able to reproduce. Trace routes
die at our redback, and could be
condition of latency between redback
and customer. Please depro/repro
the port to see if this helps the
situation. Thank you.
DSLAM Trunk Status: OK
Technology: DMT8-2
Card Status: OK
Port Status: Up
Actual Port Rates: 1536 kbps Downstream / 768 kbps Upstream
Margin: 26.0 dB Downstream / 9.5 dB Upstream

2003-09-12 17:01:06 Backhaul installed
CLOSED-Pending Partner Verify

Unfortunately the reprovision didn't do anything =(. I see lots of latency at my second hop and dropped packets from that point on. What does it mean when they say "Trace routes
die at our redback, and could be condition of latency between redback and customer"?

I think I'm just out of luck =( Unfortunately, I still have six months on my contract so it doesn't look like I have many options at this point... too sad. =(]]> http://www.dslreports.com/forum/remark,7953362 Sat, 13 Sep 2003 03:18:44 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7952699 paulhaskew : agreed, i am out of the PDX pop, so far no issues as your guys are talking about...

only the two major outages that lasted long than 4 hours, once was the who dang pop went down, then AT&T screwed up everything...

Did that to the company i work for Charter, numerous times... :(

I hate seeing posts like these anywhere especially for people who use the same service as me, or the company I work for...

Oh, did I mention that tracerts no longer work correctly???

Have the time every other hop times out now... hmm...

My fingers are crossed for resolution of the issue you are all having...

As for my just look at my thread, Transfer Nightmare (still is by the way, being ignored still :()
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries.
[text was edited by author 2003-09-13 00:53:11]
]]> http://www.dslreports.com/forum/remark,7952699 Sat, 13 Sep 2003 00:52:14 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7952186 TrueAudio : Nobody wants to upgrade there systems till its absolutely necessary. Those routers and servers are super expensive.
Its just gets to a point where customers know what there talking about and experiencing, and start calling the company on it, and then they are forced to take some kind of action. Since they do a great deal of advertising from DSL reports I would think it would be in there best interest to keep the customers that post here happy, and with service they are expecting to receive.

I ordered this service because of the ads on this website. I saw allot of people praising the service and that really turned me on. Im 3 or 4 days into this service, and im already experiencing complete loss of connection for about 15-20 seconds, then it pops back, and there seems to be allot of route changes going on. I haven't been keep much track of the packet loss, but I do notice it.

Well... I hope they are upgrading around here(SFO). Its seems like they bay area has allot of packet loss on some of the major ISP's. Tons of data flowing out of Cali & Seattle.. that's for sure.

I guess we wait and see. Good luck to those who are having problems.
--
»www.mtndewbuzz.com]]> http://www.dslreports.com/forum/remark,7952186 Fri, 12 Sep 2003 23:43:48 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7950441 koitsu : *sigh* Shady technical support reps. Thumbs down.

First off, what the rep. was referring to was the fact that UDP traffic by nature is lossy -- there is absolutely no guarantee that the packet will reach it's destination. The protocol is basically a "send the packet and pray" medium. TCP, on the other hand, guarantees that your packet makes it to it's destination due to handshaking (TCP SYN/ACK). Most games use UDP for their actual transmission of movement, shots fired, blah blah blah, because of two reasons: 1) UDP is by far less intensive on a server (as far as processing goes), and 2) if some bloke running along happens to miss a delivery packet, big whoop, he'll send another one half a second later anyways. Now you know why gamers who have slow or saturated net connections "jump around" a lot, and why latency plays a big role.

The issue I have with what the rep. told you is that the problem we're all experience has _NOTHING_ to do with the kind-of traffic that's being propagated -- TCP, UDP, ICMP. It doesn't matter. The results are the same no matter which protocol you use: the POPs are a mess. The rep. shouldn't be giving you some schmooze about how they "can't do anything about UDP traffic." I get stalls and lag via ssh to my co-lo, which is _entirely_ TCP-based, and I still don't see them bending over backwards about it (because the problem has NOTHING to do with protocols! :-) ).

The fact that SE calls their "gamer package" a "package for gamers" is misleading. All of the packages are the same, just with different speeds or different account options (i.e. shell accounts, 10 Email boxes, and so on). There's absolutely no difference, DSL or network wise, between the SysAdmin package and the Gamer package. It's just marketing schmooze.

Your latency problem will probably get solved when they deploy what Kat said above in her most recent post. I can attest to seeing a _very_ marginal increase in stability since 9/9 (re: MSBlast terminating itself), but the problem (not to sound like a braggart) is definitely looking like oversaturation, which is what I claimed in the first place. *shakes head*

Ah well. We'll see how it goes. I'm glad the Seattle folks are getting the upgrade first since their POP is absolutely horrid. I just hope SFO and LAX are next.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7950441 Fri, 12 Sep 2003 20:18:27 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7950324 anon : "my feelings are this se knows its got problems ok but if they can buy a lil time by having you do the ping ploter thing they will play that game with you as they did me till i called their bluff. just tell them you cannot reproduce the symptoms and document them because its udp connectionless service when i told them thats when things started going my way so give that a shot"

Hiya Catskinner... well, that didn't work for me. I called and spoke to the first person, told him what was going on and he proceeded to tell me that there is nothing they could do for UDP traffic...something about it not being able to be corrected en route or something. He told me that it was my game that was causing it (huh?!). He wanted me to believe that it was the netcode or something... two years of never experiencing packet loss and then all of a sudden the game socks me in the back? I don't think so =(.

Of course I began to ask him why there is no way it could have to do with the network if the packets still use the network to reach me. He decided to transfer me to advanced support (finally!). The advanced support guy said that he sent 100 packets to me and saw no packet loss. When I explained about my game and the UDP traffic he said "we can't even count what happens in your game"... Ok, my big problem with this is, I pay for the so called Gamer Package. Like duh, I use my connection for gaming. When my connection becomes useless to me for that purpose, of course I want things fixed. For them to tell me that the problems I am experiencing in game (and to a lesser extent outside of it) are not relevant and deserve no time for discussion is ludicrous. Why sell me a package geared for gaming (knowing that UDP traffic is what consists of gaming) if they refuse to support me when I am having a problem with the very marketing tool they sold me? That's like me buying shampoo, finding out there's dirt inside and being told "I'm sorry, it may be called shampoo but we didn't say it worked like it!" =(

I'm not normally someone who gets upset easily but I feel ignored and I'm just very frustrated. I'm sure that if all I did was surf the net I wouldn't have much to complain about, but then again I wouldn't be paying 100 bucks a month to do it... sigh =(.

He did say that he is going to have Covad reprovision my line to see if that helps my latency (a whole other issue) so who knows, maybe that will work. ]]> http://www.dslreports.com/forum/remark,7950324 Fri, 12 Sep 2003 20:02:25 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7950195 GeminiCub4U : Kat can you please put me on the new router? Im a new customer and am experiencing latency and packet loss.
--
Reality TV has taken over me.]]> http://www.dslreports.com/forum/remark,7950195 Fri, 12 Sep 2003 19:49:28 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7949223 KatOak : Status:

Although we've been aggressively pursuing customers that have been reported as infected with the MSBlaster worm or its variants, the packet loss that is occurring as a result of this has decreased only marginally.

The Blaster worm & its variants basically scan everything within its class B subnet relentlessly, and we've a couple of these shared with other organizations with whom we've also been in contact with in order to cut down the traffic we're seeing as a result of this worm.

Packet loss is as a result of a maxing of the packets per second that can be processed by our routers. We will be upgrading all of our equipment so in the future we are able to filter/block these kinds of attacks based on more individualized rule sets, instead of the global filters we'd need to put in place right now.

We did attempt a filter on port 135 about 3 weeks ago, but had to pull it due to complaints from Exchange and Windows Filesharing users.

All recorded complaints of packet loss and latency in the SFO area have been customers on one of our routers - we will be transitioning 40% of the customers off of this router and circuit and onto a new router and circuit to alleviate the issue in the short term.

In Seattle, we will begin our POP upgrade next week and will have all new equipment in place within the next few weeks.

We have just a handful of issues in a couple of our other POPs, and will continue to monitor those and work with customers infected with this worm in order to alleviate any load that we're seeing.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7949223 Fri, 12 Sep 2003 17:54:14 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7948659 sh0V3L : i finaly got them to aknowledge the fact the sfo pops are borked fubar and then some lol.i recieved an email stating a temp fix in the next few days then a permanent one in 2 weeks as they do some upgrades i realized se was violating the tos for sdsl premium routing as it states 2hops to backbone.for 2 months or whatever it is since they did the last duct tape and tie wire fix at sfo its been minimum 3 hops so i had them by the...... heh you get the picture lol so we will se if they do as said.

said by GamingGirl:
I just got that ping plotter program and I'm wondering... what size packets should I use to accurately show the way my line reacts when playing online games (Unreal Tournament to be exact)?


hi gaming girl im an ut nut too well now ut2003 i ran ut servers for 3 yrs and now ut2003 for 1 yr or more.

i dont think the packet really matters, why? because ut is all udp as far as actual game play server to client. the ping is icmp and the initial handshake is tcp so the routers will handle udp different than the icmp ping.

my feelings are this se knows its got problems ok but if they can buy a lil time by having you do the ping ploter thing they will play that game with you as they did me till i called their bluff. just tell them you cannot reproduce the symptoms and document them because its udp connectionless service when i told them thats when things started going my way so give that a shot

frag ya l8r gg]]> http://www.dslreports.com/forum/remark,7948659 Fri, 12 Sep 2003 16:57:09 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7946250 Yourself : FYI....

I got a call last night from SE(yes they called me) to inform me that there would be a hardware upgrade/changeout taking place next Wednesday that should take care of my issues on the Seattle POP. I will update....

Self]]> http://www.dslreports.com/forum/remark,7946250 Fri, 12 Sep 2003 12:19:27 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7943864 koitsu : The default (56 bytes) probably isn't going to be good, but don't pick something absurdly large either (most routers will ignore large windowed ICMP packets). Pick something like 1020 bytes, or maybe 516.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7943864 Fri, 12 Sep 2003 02:51:41 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7943566 anon : I just got that ping plotter program and I'm wondering... what size packets should I use to accurately show the way my line reacts when playing online games (Unreal Tournament to be exact)?]]> http://www.dslreports.com/forum/remark,7943566 Fri, 12 Sep 2003 01:35:32 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7934567 anon : I'm not as technically savvy as most of you are but here are some traceroutes I did last night... I'm too tired to wait for them to complete tonight but this will show you that something wacky is going on as I've seen this consistantly for over a week now...

Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 11 ms 11 ms 11 ms dsl081-084-001.lax1.dsl.speakeasy.net [64.81.84.
1]
2 11 ms 13 ms 11 ms border12.fe4-0.speakeasy-32.lax.pnap.net [63.251
.223.91]
3 12 ms 13 ms 13 ms core1.ge2-0-bbnet1.lax.pnap.net [216.52.255.1]
4 14 ms 13 ms 13 ms sl-st20-la-10-0.sprintlink.net [144.232.154.205]

5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.

Trace complete.

Here's a trace to a website I visit based on the east coast...

Tracing route to www.iglnet.com [66.151.180.190]
over a maximum of 30 hops:

1 10 ms 11 ms 11 ms dsl081-084-001.lax1.dsl.speakeasy.net [64.81.84.
1]
2 * 11 ms 11 ms border12.fe4-0.speakeasy-32.lax.pnap.net [63.251
.223.91]
3 12 ms 15 ms 11 ms core1.ge2-0-bbnet1.lax.pnap.net [216.52.255.1]
4 10 ms 13 ms 11 ms gigabitethernet5-1-180.ipcolo2.LosAngeles1.Level
3.net [63.208.234.133]
5 13 ms 11 ms 11 ms unknown.Level3.net [209.244.10.245]
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.


Trace complete.

Here is to Speakeasy...

Tracing route to speakeasy.net [216.254.0.95]
over a maximum of 30 hops:

1 11 ms 11 ms 11 ms dsl081-084-001.lax1.dsl.speakeasy.net [64.81.84.
1]
2 390 ms 105 ms 337 ms border12.fe4-0.speakeasy-32.lax.pnap.net [63.251
.223.91]
3 11 ms 11 ms 11 ms core1.ge3-0-bbnet2.lax.pnap.net [216.52.255.65]

4 11 ms 11 ms 11 ms gigabitethernet5-1-180.ipcolo2.LosAngeles1.Level
3.net [63.208.234.133]
5 13 ms 13 ms 13 ms gigabitethernet5-0.core2.LosAngeles1.Level3.net
[209.244.10.45]
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 *

Obviously I stopped it before I had to go all the way to 30 hops but I think you get the picture.

I get spikes of up to 28% packet loss in game (at least that I've seen with my own eyes) no matter what time of day that I play. The Blaster worm is expired but yet I saw no difference. It's been five days since I opened my service ticket and I STILL haven't heard from advanced support. Is everyone off this week? Do any of you know what could cause this?

Oh well, I'm going to bed, I bet in the morning nothing will have changed =(. ]]> http://www.dslreports.com/forum/remark,7934567 Thu, 11 Sep 2003 02:33:44 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7934536 koitsu : He and I go through the same Speakeasy POP in SFO. See my PP graphs throughout all my Speakeasy threads (click on my username and look at my recent posts); I can guarantee you what he's seeing is identical to what I am.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7934536 Thu, 11 Sep 2003 02:23:11 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7934259 ykrad : off topic but; hooooly crap. that is one HELL of a run on!

on topic; yeah, pl sucks, especially for gamers. they're by far the most critical of broadband analyzers. catskinner, no offense but I suggest you post traceroutes with your comments so they're at least semi-understandable.

I also go through the Seattle POP and have experienced the pl but haven't been significantly impacted by it. My roomate plays SW:Galaxies and hasn't expressed any grief with it - except when SE was dropping altogether the other day(s)...

while I agree it is a pain, I still think SE is doing the best they can. They wouldn't be the #1 independant bandwidth reseller if they had a bunch of no-nothing network engineer monkeys on staff. Lets just have a little more faith. ]]> http://www.dslreports.com/forum/remark,7934259 Thu, 11 Sep 2003 01:29:19 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7932264 sh0V3L : Thats all fine and dandy I think speakeasy could care less about our ping plotter charts they know thier system is below par internap sucks routing is just rediculius i can see how spiky the network is i have a 768/768 sdsl circuit i run a game server ok im in sancarlos ca guys in sanfrancisco with a speakeasy dsl connection going through the same pop have a ping of 50 to 80 thats just way to high its 25 miles away yet a tracert reveals 11 or 12 hops internap just flat sux i trace a game server in sanjose again just 25 miles away 12 friggin hops abd a ping of 40 hell it shoould be 6 hops a ping of 15 to 20 max you know that i know that and SE Damn well knows it too i think the only reason they say send in ping plotter logs is to give you something to do so your not constantly calling sorry if this offends anyone its not ment as a flame its reality dont let them fool ya we must demand a cure and this is not a worm issue as i have been trying to rectify this for months as koitsu has to .

Speakeasy u know the cure apply it now please]]> http://www.dslreports.com/forum/remark,7932264 Wed, 10 Sep 2003 21:43:29 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7931761 koitsu : Where did you conclude I was "pinging a router and not a host at a POP?" Not a single graph of mine that I've posted here in the past 2 months has done that -- I test from endpoint to endpoint.
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7931761 Wed, 10 Sep 2003 20:54:18 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7929205 sporkme :
said by koitsu See Profile:
I realise SE doesn't filter, but I strongly urge reconsideration -- filtering TCP src port 135, or TCP dst port 135 at the border routers. Border routers are specifically deployed for this purpose, and it takes less processing time to drop a packet on a Cisco or Juniper than it does to route it.


A few problems with that... Blaster (or the next worm, or NACHI) doesn't really care if it's being filtered upstream. It will still blast away out your pc, out your modem, out the dslam, out Covad's ATM network, out SE's backhaul. SE does not have any real breathing room for this, so they get slammed. Unlike the cable guys, they can't filter back at your CPE to save their network.

And depending on IOS, filtering is much more CPU intensive. On 12.2T(mumble), adding 135 filters gobbled up 20% more CPU, and that is with CEF enabled.

But I don't think SE has Cisco or Juniper boxes, just a Redback and an FE or GigE handoff to InterNAP. And from what I've seen, the Redback doesn't get too fancy with filtering...

Another thing with Redback is that they prioritize ICMP replies *really low*, so pointing pingplotter at the router gives a less accurate picture than say pointing it to a DNS or game server at your pop. That also gives you more leverage if their network guy knows that pinging routers gives flawed results.

said by koitsu See Profile:

This is a route change, which is EXTREMELY prominent with InterNAP (I've seen them change routing tables every 30 minutes -- I have no idea what criteria they use to decide when to drop BGP with someone and switch, but it seems almost random).


While their PR stance is that they have all these transit connections for "robust connectivity", I think a big part of their decisions are based on closely meeting minimums with each provider to meet certain price points. Since they went in the crapper, you see almost 0% of traffic going out "expensive" paths. I think they put the rest of their money into automating route selection based on economic constraints rather than "optimum routing". No one likes to dis InterNap, but they really aren't that "premium" anymore. Sorry.

said by koitsu See Profile:

What I want to know is why the packet loss is induced upon my DSL gateway depending upon which route packets are taking at InterNAP.


Coincidence? Or the bgp scanner process is sucking up cpu after a change? Another reason not to ping a router, but a host at the POP instead.
--
just a minute]]> http://www.dslreports.com/forum/remark,7929205 Wed, 10 Sep 2003 16:06:08 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7928441 paulhaskew : heh, my modem just lost sync for the third time in two days... hmmm]]> http://www.dslreports.com/forum/remark,7928441 Wed, 10 Sep 2003 14:39:51 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7927160 Yourself : I have a feeling that my issue is not worm related, as I am on the Seattle POP. This morning(5:45 am)I am still showing up to 20-30% PL on the 2nd hop of my connection. In my opinion it sounds like a NOC(Internap) issue for us Seattle POP folks...I just wish we weren't still in the dark.

Self]]> http://www.dslreports.com/forum/remark,7927160 Wed, 10 Sep 2003 11:27:58 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7925644 bhan261 : www.pingplotter.com

Also, yesterday was the first day in a month that I didn't have several spikes of P/L. So maybe it was MSBlaster...I'll keep my fingers crossed.
]]> http://www.dslreports.com/forum/remark,7925644 Wed, 10 Sep 2003 07:19:11 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7924941 anon : I am out of the LAX pop and I am seeing continuous (throughout the day) packet loss... anywhere from 1% to 28%. I have the Gamer package and have never seen packet loss on my line before last week... obviously this is affecting my gaming =( and although I've contacted support, I'm still waiting for contact from advanced support (it's been three days now). If this truly was related to the worms going around, why is it that nobody else I game with is experiencing any problems?

And where do you get that ping plotter? ]]> http://www.dslreports.com/forum/remark,7924941 Wed, 10 Sep 2003 01:50:57 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7916493 koitsu : Hahaha. Thanks for the enthusiasm. ;-)

Probably not though, since I work today (Tue) and tomorrow. :-) I usually close Ping Plotter while using Remote Desktop, but of course the week before last was an exception since I absolutely needed to prove that something strange was afoot at the Circle K.

I'm hoping that the occasional packet loss shown in our graphs goes away. As far as Seattle goes, hoo boy, I'm really crossing my fingers on that one. Just when I thought it was back via SFO, SEA comes along and kicks me to the curb! ;-)
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7916493 Tue, 09 Sep 2003 09:40:02 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7916377 Chicago_DSL6 : I'll bet koitsu will find out before SE!]]> http://www.dslreports.com/forum/remark,7916377 Tue, 09 Sep 2003 09:22:14 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7915916 bhan261 : Yes...today is the day that we'll find out whether SE's engineers were correct or simply living in hope. Anyone want to make any bets?]]> http://www.dslreports.com/forum/remark,7915916 Tue, 09 Sep 2003 07:32:11 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7912789 koitsu : Latest check from my 1.5/384 ADSL.

The "huge red blotch" you see from about 16:12 until 17:00 or so is Verio. I verified this since PP logs route changes. Why are BGP route changes at InterNAP affecting my DSL gateway? This makes absolutely no sense to me what-so-ever.

The "smaller packet loss" lines shown are probably caused by MSBlast and SoBig.F. That I can justify.

But I still don't understand why latency shoots through the roof from 09:00 until midnight. That has absolutely nothing to do with MSBlast or SoBig.F, that's for sure. Plus, you can see a point in the graph above (a little after 11:00) where the latency dips. Unless all of these people (and there would have to be a LOT meeting this condition) using the 'net through SE don't leave their PCs from 00:01 until 08:59, and simultaneously suffer from MSBlast.

Tomorrow is 09/09. MSBlast stops working after that point in time. If what's being shown here (from everyone) continues past that, my vote is that it has absolutely nothing to do with MSBlast.

EDIT: Typos.
--
Making life hard for others since 1977.

[text was edited by author 2003-09-08 21:00:26]
Click for full size
SE SFO, 09/08
]]> http://www.dslreports.com/forum/remark,7912789 Mon, 08 Sep 2003 20:58:12 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7911928 Yourself : Kat-

Any update on the Seattle POP?

Self]]> http://www.dslreports.com/forum/remark,7911928 Mon, 08 Sep 2003 19:31:24 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7888853 Endorphine : Is the Seattle POP outage/problems related to the blaster worm, or the other reason you didn't tell us yet? :)
Click for full size
]]> http://www.dslreports.com/forum/remark,7888853 Sat, 06 Sep 2003 00:18:28 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7886317 brikholl : I'll 2nd those that don't think that the blaster (or variants) is the root cause of these issues. Partially, maybe, but not entirely.

I had been seeing problems before my initial call to support way back last month. I just waited a few days to call in, hoping it'd get better.

I've since left SE (was in the 30 day trial at the time) and am really surprised that not only is the issue still going on, but that it's worse now than it was when I had SE.

I've moved to another provider, and see NONE of these issues anywhere. What I find interesting is that at times I'd be routed with SE over just about the same path I take to places at my new ISP, and while I saw massive PL when at SE, I see none now.

Lastly, Kat does a great job, but the folks on the phone need to stop spouting the "it's your line, not us" stuff. Covad came out and checked my line, stating it's fine. My current ISP is running flawlessly on it. I'm glad to see that in Kat's post they seem to be heading in that direction, so that's a good thing.

Overall, I thought that for the premium price I was paying for SE, that the techs on the phone should have followed through with things better. Rather than always playing the "blame game" with Covad or my line. I mean, I didn't know Covad was coming out until I called SE to request a status update and was told "yeah, Covad is coming out tomorrow." No one called me to tell me that. I wouldn't have been home if I hadn't called in. :)

But, for the week or so the service worked "right" it was great. ;)]]> http://www.dslreports.com/forum/remark,7886317 Fri, 05 Sep 2003 19:08:33 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7884452 koitsu : Excellent update, Kat.

I did take router load and pipe saturation into mind when considering the situation. 30-50% sounds a little high, to be honest (I would guess more along the lines of 20-25%), but I won't argue because I simply don't have access to the networking equipment: period.

I realise SE doesn't filter, but I strongly urge reconsideration -- filtering TCP src port 135, or TCP dst port 135 at the border routers. Border routers are specifically deployed for this purpose, and it takes less processing time to drop a packet on a Cisco or Juniper than it does to route it. Don't respond with ICMP_UNREACH_PORT or ICMP_UNREACH_ADMIN_PROHIBIT: just silently drop the things, both directions. If there are customers who are ACTUALLY RELYING on RPC, unfiltered, across their connections, they need to get a grip on security and start using a VPN with IPsec.

If this is indeed the W32.Blaster.Worm, then I can conclude the following from my graphs:

If you notice, most of the heavy (80-100%, "large red blocks") packet loss appears in sections. If you examine the latency (black line), you'll see it takes a sharp ramp or dip simultaneously.

This is a route change, which is EXTREMELY prominent with InterNAP (I've seen them change routing tables every 30 minutes -- I have no idea what criteria they use to decide when to drop BGP with someone and switch, but it seems almost random). From examining my data files for all of this, I've found that ,ost of the heavy packet loss sections are caused when routing through Verio. Being as I worked for Verio for over 3 years, this comes as no surprise -- they refuse to filter ANYTHING ANYWHERE, regardless if their entire network is being hit (that includes DoS/DDoS), and they don't play very well with end-users, ISPs, or other peering providers.

I don't know what kind-of agreement you have with InterNAP, but it might be wise to request that Verio be taken out of the BGP table for SE netblocks until this can be resolved. The "lingering" packet loss you see in the other portions of my graphs I can accept being caused by the Blaster variants.

What I want to know is why the packet loss is induced upon my DSL gateway depending upon which route packets are taking at InterNAP. The only explanation I can come up with is that the DSL gateways have multiple interfaces, and while one interface may be saturated, the other is seemingly fine (only noticed on a BGP change). If this is the case, alright, but still pretty odd.

P.S. -- Don't forget W32/SoBig.F, which I guarantee is giving routers a work-out as well. Luckily the thing terminates itself on 09/09. W32.Blaster.Worm stops working between January 1st and August 15th (but good luck getting people to put up with this for another 4 months).
--
Making life hard for others since 1977.]]> http://www.dslreports.com/forum/remark,7884452 Fri, 05 Sep 2003 15:21:29 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7884444 bhan261 : I will admit that this isn't my area of expertise, but if it IS a result of the Blaster worm, wouldn't I be seeing similar problems on my work connection? (NYConnect T1) I'm not. Wouldn't my wife see it on her work connection? (DSL.net T1) She's not.

As I said, I am not an engineer so Blaster could very well be the problem. It just seems odd.]]> http://www.dslreports.com/forum/remark,7884444 Fri, 05 Sep 2003 15:20:55 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7884197 MrCoal : I think it is a cop out to blame these problems on the Blaster worm.

I have had intermittent severe packet loss for WEEKS, long before Blaster started appearing.

In my case, my connection will just suddenly start experiencing packet loss throughout the internet. This can occur anytime and lasts for several minutes, then just as mysteriously, the packet loss goes away and everything works great again.

Sometimes I can not even reach many sites. Lowering my MTU to 1400 helps but I still get intermittent packet loss.

I am just living with it, but there must be some underlying routing problem.
--

MrCoal
Crystal Lake, IL]]> http://www.dslreports.com/forum/remark,7884197 Fri, 05 Sep 2003 14:51:22 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7884012 yoonix : Thanks for a very useful and detailed post, Kat, good to see that Speakeasy is looking into this issue. And not so good to see that worm writers are responsible for this. :(]]> http://www.dslreports.com/forum/remark,7884012 Fri, 05 Sep 2003 14:23:56 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7883821 Yourself : Thanks for the update Kat!! Given the fact that I am on the Seattle POP, I am actually intrigued by the first paragraph of the update and look forward to the next bit of info(it's like a soap opera :) )

Self]]> http://www.dslreports.com/forum/remark,7883821 Fri, 05 Sep 2003 13:56:11 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7883595 KatOak : UPDATE

We met last night and this morning about the packet loss and latency reports in a few of our POPs. We have a separate, unique concern and overriding issue in our Seattle POP that is still under investigation, so I cannot speak to that at this time – but I can say that the issue under investigation is only being exacerbated by the root cause of the issue across our network.

The crux of it is this: The MSBlaster worm and variants have caused an increase in unnecessary traffic by 30% - 50%, depending on the POP. This is impacting Speakeasy’s network and the Internet as a whole – resulting in both monitored packet loss and increased latency Internet-wide.

The original report, by brikholl on 8/7, was due to the circuit transitions that we performed in SFO on 8/6 and 8/7. From the information he provided, it appeared that the connection returned to normal and then the packet loss and latency increase returned on 8/13, two days after the MS Blaster worm was identified and began propagation.

Due to the worm activity, you may see degraded service during peak hours. This is not due to bandwidth constraints but rather to maxing out how many packets the router can forward per second.

The monitored packet loss graphs displayed in this forum recently tell part of the story, but can be misleading. Routers and servers treat “ping” (ICMP traffic) with a very low “handle” priority. In practice, this means that if a given router is busy handling other traffic, it can delay the response or even decide not to respond (which appears as packet loss). While some of our POPs have high usage during a few hours in the evening, San Francisco is one of our largest markets and has a high concentration of heavy usage throughout the day. Add the unnecessary worm traffic to this usage pattern and the dropped packets will increase accordingly. The general rule is that as long as you are reaching your destination host with no packet loss, or only a small jump in latency, you are not being affected by the current state of things. Latency will probably be higher than usual until these worms die down.

Regarding the SBC connection posed by koitsu & brikholl awhile back – We read through these threads and it appears that SBC is also seeing a significant hit due to these worms. At one point, a rep states that they isolated this into one of the routers in one of their CO's and that it was resolved, but we have no further information on the state of this.

If SBC was having problems with equipment related specifically to their DSL services, our customers shouldn't have been impacted at all. However, if the problems were related to their backbone/Internet services, we may have seen some impact, assuming Internap was routing through SBC at the time – which is highly unlikely.

So, that’s the diagnosis – this is what we’re doing to fix it:

Our abuse team is notifying customers reportedly infected with this worm and will work with them to secure their network. We’re going to aggressively work this until we can reduce the number of infected computers on our network by at least 75%. Industry experts have stated that this worm has peaked and is on the decline, although we haven’t seen any real world evidence of that quite yet.

As you know, we are in the process of a major network upgrade and will be deploying equipment that allows us to be more resilient during situations like this in the future.

We have informed our support staff of this outstanding, global network issue and requested that while they are troubleshooting specific connections, they keep this in mind. We want to avoid any more of the “it’s your LAN, nothing we can do” and any unnecessary frustration on the part of the customer if at all possible, but still continue to troubleshoot connections to determine if there is another issue, unrelated to the worm, at hand.

We are continuing to monitor and work on this situation and I’ll post any additional updates here.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7883595 Fri, 05 Sep 2003 13:25:26 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7880725 koitsu : Agreed. I cannot even imagine how SDSL customers are putting up with this for how much SDSL costs. I've personally now moved to my Comcast modem until this can be resolved.

Graph shows what's happening; large gray portion is due to me simply not running PP during that time (I was doing some hardware work on my workstation).

I also began witnessing the "stalling socket" problem I've documented here on the forum a few times (which leads me to believe that the problems I had with it back when I was 1.5/768 were the beginning stages of this whole thing!). It also took me a good 10-15 full minutes to download a battle.net update for Warcraft 3 (something that should take about 2 minutes max), due to all the packet loss.

Sadly, if it's not fixed by the end / near the end of this month, I'm going to have to cancel. Two months of this is just unacceptable, and I'm sure Kat can understand our frustrations.

I'll power up my DSL bridge a couple times a week to see how things are doing. I may end up sticking a spare gateway I have (Netgear RP614) on my DSL line just so I can run mtr against it from my co-lo to see if there's ever any improvement (otherwise a tr just times out due to nothing listening on the IP).
--
Making life hard for others since 1977.
[text was edited by author 2003-09-05 02:27:51]
Click for full size
SE, evening 09/04
]]> http://www.dslreports.com/forum/remark,7880725 Fri, 05 Sep 2003 02:26:39 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7878520 sh0V3L : Im really fed up i have regulars complainig about lag ( i run a ut2003 game server btw ) i dont blame the my connection is getting worse by the min when will they take this serious ive had and have tickets open nothing gets done same thing everytime from our side things are fine lol what an answer so check this out »/quality/nil/1152730

that really suxorz the sour cho cho man FIX IT GUYS!!!!!!!!!!!!!!!!!! THIS is an sdsl 768/768 circuit here $$$$
GET IT RIGHT please
FED UP!!!!!!!]]> http://www.dslreports.com/forum/remark,7878520 Thu, 04 Sep 2003 21:54:24 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7876357 koitsu : Graph above is the past 13 hours -- while I was sleeping (and no, I don't leave any applications running like P2P apps and that sort-of thing). The fact that the PL goes away for hours then suddenly comes back again is proof to me that there's either an oversaturation or equipment problem somewhere. My vote is still the DSLAM. The other part that bothers me is the increasing average latency from 09:00 to midnight -- it doesn't make any sense why a network would increase in latency _over 20ms_ unless their equipment and/or network capacity was oversaturated. Just my opinion, of course.

I don't go the hosting route; I have actual co-location for that. :-) All I want here in the Bay is 1.5mbit down and 384kbit up (or higher -- but those are my minimum requirements). The download speed is pretty much a standard by now, and the upstream I mainly require for a) a fast Remote Desktop connection while at work, and b) being able to stream mp3s from my home box while at work (I use 96-320kbit VBR, so 256kbit upstream induces stuttering on occasion)).

So far SE has gone to great lengths to provide me with pretty much everything I need, including renegotiating my line speed (I was originally 1.5/768 syncing at 1.5/540 and had occasional sync problems, now I'm 1.5/384 -- but all of these network problems were still apparent throughout that fiasco too).
--
Making life hard for others since 1977.
Click for full size
09/04 @ 02:00 to 15:30
]]> http://www.dslreports.com/forum/remark,7876357 Thu, 04 Sep 2003 18:34:55 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7871460 Chicago_DSL6 : Koitsu, I hope they at least waive the cancellation fee. I am glad that there is an end user in these forums that DOES actually know what they are talking about. The fact that you have presented this much information about your problem and still not gotten your issue resolved in unreal. I never wanted to go to the dark side (SBC), but I called them yesterday for status on install (moved over the weekend) and they mentioned how ALL of their DSL customers are getting doubled downstream. I do not know what you need your DSL connection for, but if it isn't for hosting, SBC might be your most "reliable" and inexpensive choice.

Again, nice work koitsu and best of luck!]]> http://www.dslreports.com/forum/remark,7871460 Thu, 04 Sep 2003 09:29:52 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7870972 bhan261 : Koitsu,

I don't know whether you are a saint or an idiot to endure this kind of poor service without some sort of offer for reduction in fee from SE. I (and others) are having problems in NYC but nothing this extreme.

But you are right that a continuing flow of information about what's being done to investigate and resolve the problem will go a long way to maintaining the peace. Let's hope SE follows through accordingly.]]> http://www.dslreports.com/forum/remark,7870972 Thu, 04 Sep 2003 07:58:10 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7870653 warcorp : YIKES!
Nope can't beat that. We are broken, but not THAT broken....]]> http://www.dslreports.com/forum/remark,7870653 Thu, 04 Sep 2003 05:48:16 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7870631 koitsu : Can you beat this?

This has been going on since about 23:30 or so. Prior to that, it was what I'm used to seeing (flippant latency, occasional packet loss (3-4%)).

You know we're sad individuals when we're comparing results of broken connectivity! :-)
--
Making life hard for others since 1977.
Click for full size
]]> http://www.dslreports.com/forum/remark,7870631 Thu, 04 Sep 2003 05:37:03 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7870597 warcorp : Koitsu,
We seem to be having the same problem out of the Chicago POP as well so I feel your pain. Not nearly to the degree you are however and seems to be a problem starting at about 7:30 and going all the way till 4:00am ish the next day and it's regular as clockwork (for the past 4 or 5 weeks that I know of. This is the last 12 hours:
Click for full size
]]> http://www.dslreports.com/forum/remark,7870597 Thu, 04 Sep 2003 05:10:34 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7870578 koitsu : Very thankful for this, Kat.

You -- and this post of yours -- are literally the reason I am still a customer. I had just logged in to MySpeakeasy to file a cancellation (yes, after all the hard work you, myself, and lots of other reps have put into dealing with the issues I've brought forth), but remembered I hadn't checked the SE forum in a week or so. I'm thankful I did.

The problem I have with all of this is that it's confusing SE's engineering crew. I am an engineer (system and networking), and have been since 1991. I guess I'm having trouble accepting that an engineering group simply can't pinpoint what/where the problem is.

I know you've probably heard all of this before, but it's just that, well... we're going on, what, almost 5 weeks of this now (in regards to SFO)? It's becoming hard to stomach. I shake my head every day when I stare at my graphs and see the same repetitious behaviour and over between the hours of 09:00 and midnight. I keep waiting to see some change in it, even something as trivial as the Window-of-Doom being audited to something smaller (or heck, even LARGER!).

To make matters worse, as of this evening, things have seemingly become downright foul. It's nearly 2 in the morning and I'm witnessing packet loss (huge margins might I add, suddenly appearing out of no where in clumps of literally 35-40%!), as well as the flippant latency all of us have been reporting. It's the first time I've ever seen it happen at this late of an hour. I even tried power-cycling my DSL bridge: no love.

I'm presently on my Comcast connection, but good lord -- you can tell from the graph I've posted just how absolutely HORRID Comcast is right now here in the Bay. This has been a long-standing problem with AT&T and Comcast for over 3 years now, and it's getting worse. It's been confirmed as well: it's over-saturation due to the populus here.

Despite my comments, I'm going to continue sticking with SE for as long as I can. It's just that... *sigh*... SBC doesn't seem to be having problems like this, or rather, at least not to this degree. I'm sure you can understand my frustration.

Please keep us updated, even with small things like "there was an engineering meeting today about this." I just like knowing someone is at least acknowledging what we're seeing as legitimate, since we (customers) fought so hard to get SE Support to accept this issue in the first place.

I haven't forgotten about your orchids either, by the way. I'm still keeping my word on getting you some. I just happen to be swamped due to work, and a lazy person to bat.
--
Making life hard for others since 1977.
Click for full size
Comcast, Bay Area
]]> http://www.dslreports.com/forum/remark,7870578 Thu, 04 Sep 2003 04:59:42 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7869537 Endorphine : Thanks for the update Kat!

Hope they figure out what is going on with the border routers.]]> http://www.dslreports.com/forum/remark,7869537 Thu, 04 Sep 2003 00:21:27 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7866956 JonW : There are lots of problems all around the internet lately, I assume the multiple worms going around haven't quite been defeated.]]> http://www.dslreports.com/forum/remark,7866956 Wed, 03 Sep 2003 20:11:53 EDT Re: Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7864877 jaykaykay : Great and glad to hear it. While I have not posted to this forum about my packet loss and latency problems, I have done so on the site. I am glad to note that it is a wider problem than just my own, and since I haven't been able to spend much time here of late, I hadn't even checked this forum.
--
JKK:-) Age is a very high price to pay for my maturity. If I can't stay young, I can at least stay immature! ]]> http://www.dslreports.com/forum/remark,7864877 Wed, 03 Sep 2003 16:15:36 EDT Packet Loss/Latency Reports http://www.dslreports.com/forum/remark,7863751 KatOak : Just a quick update here:

Many of you have provided excellent information regarding the packet loss and/or latency that you are either monitoring or directly experiencing in your Point of Presence.

I have passed all account information to our Engineering team and they are thoroughly investigating this situation at this time. I'm hopeful that we'll have more information to provide within the next couple of days.

We really appreciate the time that many of you have spent tracking and reporting this problem as it has been extremely helpful.
--
Kat Oak
Speakeasy
kat@speakeasy.net]]> http://www.dslreports.com/forum/remark,7863751 Wed, 03 Sep 2003 14:18:55 EDT