KatOak
VIP
join:2001-09-10
Seattle, WA
| reply to KatOak
Re: Packet Loss/Latency Reports
I'm not sure what additional information you're looking for, but the issue is a hardware max-out on the number of packets per second that two of the routers in our SFO POP could process. We alleviated this issue by moving about 40% of the customers on these routers over to another router with considerably less load.
We don't consider this issue closed as we're still seeing about 1 - 3% packet loss at this POP, depending on the time of day. As I posted before, our Engineering team is continuing to investigate alternate improvements in the interim prior to the complete upgrade later this year.
Again, we made absolutely no changes to our SFO POP - have done no work on the impending complete upgrade scheduled for this fall. This issue was caused by our routers inability to handle the extreme increase in the number of packets per second as the result of recent Internet worms, which are still having a significant impact. Our abuse team is notifying and unbinding folks as appropriate in order to minimize our customers' contribution to this problem.
--
Kat Oak
Speakeasy
kat@speakeasy.net |
|
falkclan
join:2000-07-02
Olathe, KS | reply to KatOak
any further information on the Chicago POP?
--
Alan Falk |
|
Yourself
join:2000-08-10
Lynnwood, WA | or Seattle?....tomorrow is the one month anniversary of my trouble ticket being open for this issue. |
|
Carl F
@Stanford.EDU
| reply to KatOak
Hi Kat -
I can see, and very much appreciate, that you're working hard to be responsive. Let me see if I can boil it down to a single question that I'd like to see answered in enough engineering detail that it makes some sense to me.
You said that Blaster et al. caused a .4X to .5X increase in traffic. I measured a .5X decrease in bandwidth on both download and upload. But I measured a 100X increase in variability (latencies, duration of stall events, etc.)
This is way out of whack, a very weird pair of symptoms. If I were to graph this, the relationship between traffic increase and bandwidth decrease would be linear, .5X demand increase, .5X bandwidth decrease. Yet the relationship between traffic increase and service stability and reliability would look like an exponential runaway or an asymptote, .5X demand increase, 100X reliability decrease.
If bandwidth were also collapsing nonlinearly, I would say that the traffic queueing management was thrashing terribly, with intermittent queue overflows: Kind of like the disk thrashing, the slowdown and the intermittent app failure that occurs when apps start page swapping like crazy. But all that thrashing takes real-time. The slowdown of applications goes nonlinear well before app failure occurs. Yet the traffic slowdown, the loss of bandwidth stayed linear, so two orders of magnitude more thrashing still took place in a linearly increased amount of time.
How did download and upload bandwidth decrease linearly, but service reliability decrease exponentially/asymptotically, especially if, as you've repeated many times, there was no form of traffic throttling in force?
If, as Customer Experience Manager, this is beyond your expertise to answer in detail--it would certainly be beyond my expertise, as a real-time embedded engineer with negligible networking experience--could you please bring one of your network architects into the loop to answer this?
Thanks,
Carl F. |
|
jmberry
join:2003-07-15
Tucson, AZ
|  reply to KatOak
While I'm glad I read through this entire thread, I'd have been a lot happier if more of this information had been shared w/me via the number of phone calls/emails/TT's I've submitted.
I'm in the last couple of days of my eval period for a T1, and am doing my best to come to the right decision in whether or not I am going to continue past that period with Speakeasy. I've heard (and read) a lot of very positive things about them, but am having issues with a T1 line having a 100ms lag on its first hop....
~JMB |
|
Carl F
@speakeasy.n
| Hi jmberry -
Kat said several posts back that they've reworked their processes to improve communication between engineering and tech support.
Sounds from your post, much to my astonishment, that first tier CS is still working off the same old script.
Good luck,
Carl F. |
|
GamingGirl
join:2003-09-13
Folsom, CA
| reply to Carl F
said by Carl F:
BTW, GamingGirl, is your upload stable enough for you to game reliably?
Cheers,
Carl F.
It used to be awesome. I would get stable 55-65 pings to the server I play on in San Antonio. As it stands now, I still see occasional dropped packets. PL will spike to 30% from zero but only on occasion whereas I was seeing consistent PL for the first few weeks of Sept. I do see increased latency compared to the first six months of my service but there's not much I can do regarding that as it's a "best effort" service. I blame the latency on my second hop since results on each hop thereafter are solid and rarely, if ever, fluctuate. My ping plotter looks much like it did before but without the heavy red areas. That second hop has some serious ping spike issues.
I did a lot of ftp transfers just last night though and my upload is very steady around 650 up (I am on the 1.5/768 plan).
As far as gaming, I was thoroughly pleased with my line during the first six months and now I'm just satisfied and hoping that whatever changed will reverse itself and go back to normal when these upgrades take place.
Regarding the worms... I am seeing serious "ping requests" on my firewall, upwards of 400 over a 3 hour period (like whoa!). These are all ICMP packets with the occasional TCP and UDP requests for certain ports.... I don't know if that makes sense (I'm not that technically inclined =p ) but I do see wayyy more hits than I ever have before. A lot come from users on the speakeasy network (about half) and the rest are from other sources. Does anybody know why I keep getting hit like that? Is it indeed caused by the worms out there? |
|
macmouse
Premium
join:2002-05-30
Saratoga, CA
| reply to Carl F
Although I haven't managed with anything this large scale, I can give you my own little expierence...
Ok, Back in the day, I ran an game (counter-strike) server on my good 'ol pentium 200 linux box. w/64MB ram. It started as an NAT/ipmasq machine (before those router boxes are now so cheap) and grew into an catch all for misc computer stuff...
Anyway, this "little computer that could" was running an half-life dedicated server in linux. I was actually able to get as far as 10 players, when finnaly I maxed out all CPU capacity on the machine. The problem wasn't with bandwith (it was an local LAN game) but latency (ping) dropped like an rock. I'm not talking about just latency in game- If I ran an ping to the machine through the 100bt network, I would get 500ms+ pings! From what I'm figuring, the cpu (io bus) was so busy that all of the IO devices (like network card) had to sit and *wait* until the cpu was free to then be able to process the information.
I would not be suprised under extreme conditions that this could result in packet loss. The network card has to wait so long that the computer on the other end "gives up".
In the end, an router (cisco) is an computer and specialized software and an *lot* of io (network) ports. Ethernet,fiber,dsl what-have-you. So there was simply so much stuff going through that the "cpu" in the router got full, and that was that.
Also (somewhat reading between the lines), kat pretty much said that the POP's were at capacity (more or less) and they were going to have to upgraded. So, with an sudden increase of network activity (virus/torjans, or even heck just a bunch of new users) was/is enough to push it over the edge.
Simular things have happened in the past with other companies *cough* comcast *cough* so speakeasy isn't alone. To their credit, they are *telling* us whats wrong and that they are working on it. With comcast/attbi/@home, they said *nothing* and that was "just the way it is" and *eventually* things would "magically" start working again, with no explanation.
I hope that answers your question  . |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA | reply to GamingGirl
List off some firewall entries, port numbers (src+dst) and protocols and I'll probably be able to tell you what's what.
--
Making life hard for others since 1977. |
|
GamingGirl
join:2003-09-13
Folsom, CA
| It looks like the TCP requests are all wanting port 135 with the occasional 445, 1433, 17300, NetBIOS session, HTTP and 27374. The UDP requests are looking for mainly (NetBIOS name) and two hits for port 1434. The ICMP request are ICMP Echo Requests. These results are for the last 1000 hits. This was from 3pm this afternoon until now (10:53pm).
I don't know if this answers what you need, but if not, let me know =).
[text was edited by author 2003-09-23 01:55:23] |
|
metrodust
Hey Thats Mine
join:1999-12-10
Seattle, WA
| reply to Bondman
just got back in town and noticed this problem has finally started to hit me in the fact that my latency 5 times higher then it was before i went out of town. no trouble ticket in yet because i noticed it while at work and thought that maybe a router reset would fix it up.. but it didnt.. grrr!
[text was edited by author 2003-09-23 20:51:36] |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| reply to GamingGirl
Not too sure what's with the ICMP echos -- those are classic "ping" responses. I'm almost 90% positive that a recent Win32 worm checks to see if a host is up this way, however. I forget which worm.
TCP port 135 would be W32/SoBig.F.
TCP port 445 would be NetBIOS or SMB shares.
TCP port 1433 -- Microsoft SQL server. Probably a trojan/virus of sorts looking to exploit MSSQL holes (and there are many).
TCP port 17300 -- Kuang2TheVirus. No idea what this is, other than a virus. 
TCP port 80 (HTTP) is ""normal"" *cough cough*. I get these as well. They are almost always compromised or infected Windows machines running IIS, attempting to spread itself. Most of the requests I get are "GET / HTTP/1.0" requests, sometimes they're simply "GET /default.ida?blahblahblah HTTP/1.0" for the (old but still active) IIS worm.
TCP port 27374 -- SubSeven backdoor probe. Sub7 lets a remote individual transparently control your PC -- most importantly, it turns your workstation into a DDoS client. Bloody IRC kids...
The UDP requests don't mean much to me either. Possibly another worm, but possibly leftover UDP deliveries from a gaming server?
Sounds like you should check out the Internet Storm Centre sometime. Incredibly useful for how simple it is.
EDIT: Figured out what some of the ports were.
--
Making life hard for others since 1977.
[text was edited by author 2003-09-24 00:19:57] |
|
sh0V3L
join:2002-02-16
San Carlos, CA
| IM AM so fed up i just hung up om my acc manager and ive had it with speakeasy at this point im looking for a way to get out without paying cancelation fees thats all i just want the f out they changed my ip block hoping for better its now worse they wont go back they arent willing to do anything they see no problems they think they can bs me well its not gonna fly its bye bye speakeasy |
|
jesterekim
join:2000-05-25
Flushing, NY | reply to KatOak
Damn sorry to hear that.  |
|
paulhaskew
Unoffical Dominos Spokesman
join:2002-01-10
Vancouver, WA
clubs:
| reply to KatOak
well as in regards to my area... only marginal packetloss/spikes, maybe 1 out of 500... the PDX/SEA POP Bridge is working fine for me at this point...
sorry to see you go catskinner...
--
I post for myself, from myself. Statements made do not necessarily reflect the views of my employer, Charter Communications, or any of its subsidiaries. |
|
Bondman
join:2001-08-24
Livonia, MI
|  reply to KatOak
Hi Kate:
Since the maintenance activity at the Chicago POP servicing the Detroit NAP things are very close to normal. Thanks for working on this. I held off responding to see if the fix would hold. The only strange thing I ran into which I have covered in another topic is the inability to resolve webmail.registeredsite.com and pop.registeredsite.com with Speakeasy DNS servers. From a Terminal Server session at a site that has a different ISP I can resolve these names to IP's.
Thanks again for your help! |
|
LowJack
join:2000-07-19
Seattle, WA
| reply to KatOak
I'm in the same boat. Ever since a week or so ago, when they had the Seattle "maintenance" event, my line has been continually getting worse.
Usually my packet loss is 0% and my ping to the gateway is a solid 25ms.
[Tau-Ceti:~] cbishop% ping 216.254.16.1
PING 216.254.16.1 (216.254.16.1): 56 data bytes
64 bytes from 216.254.16.1: icmp_seq=0 ttl=63 time=41.149 ms
64 bytes from 216.254.16.1: icmp_seq=1 ttl=63 time=44.582 ms
64 bytes from 216.254.16.1: icmp_seq=2 ttl=63 time=31.596 ms
64 bytes from 216.254.16.1: icmp_seq=3 ttl=63 time=37.916 ms
64 bytes from 216.254.16.1: icmp_seq=4 ttl=63 time=38.922 ms
64 bytes from 216.254.16.1: icmp_seq=5 ttl=63 time=46.153 ms
64 bytes from 216.254.16.1: icmp_seq=6 ttl=63 time=45.293 ms
64 bytes from 216.254.16.1: icmp_seq=7 ttl=63 time=47.172 ms
64 bytes from 216.254.16.1: icmp_seq=8 ttl=63 time=44.933 ms
64 bytes from 216.254.16.1: icmp_seq=9 ttl=63 time=48.611 ms
64 bytes from 216.254.16.1: icmp_seq=10 ttl=63 time=41.876 ms
64 bytes from 216.254.16.1: icmp_seq=11 ttl=63 time=48.696 ms
64 bytes from 216.254.16.1: icmp_seq=12 ttl=63 time=48.73 ms
64 bytes from 216.254.16.1: icmp_seq=13 ttl=63 time=41.144 ms
64 bytes from 216.254.16.1: icmp_seq=14 ttl=63 time=42.072 ms
64 bytes from 216.254.16.1: icmp_seq=15 ttl=63 time=52.93 ms
64 bytes from 216.254.16.1: icmp_seq=16 ttl=63 time=32.681 ms
64 bytes from 216.254.16.1: icmp_seq=17 ttl=63 time=47.175 ms
It gets MUCH worse when I try to do anything. My speed tests are normal this morning, but my upload dropped from the 768 region last night to the 128 region during the worst of it.
I call Speakeasy, nothing but pissed off techs telling me "we don't garauntee ping times call us when it's 100+".
You are a "gaming" ISP, asshats. You better start garaunteeing them.
Some router log examples:
Sep/25/2003 10:26:35 Drop ICMP packet from WAN 216.254.197.65:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:26:15 Drop ICMP packet from WAN 216.254.223.158:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:25:47 Drop ICMP packet from WAN 216.254.220.123:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:25:23 Drop ICMP packet from WAN 216.255.194.31:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:25:00 Drop ICMP packet from WAN 216.254.16.1:0 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:23:53 Drop ICMP packet from WAN 216.254.214.137:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:23:47 Drop ICMP packet from WAN 216.254.209.138:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:23:36 Drop ICMP packet from WAN 216.254.3.30:8 216.254.16.31:0 Rule: Default deny
Another block:
ep/25/2003 10:01:40 Drop TCP packet from WAN 65.33.248.200:2743 216.254.16.31:135 Rule: Default deny
Sep/25/2003 10:01:19 Drop ICMP packet from WAN 216.254.220.158:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:01:01 Drop ICMP packet from WAN 216.254.160.229:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:00:44 Drop ICMP packet from WAN 216.254.162.64:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:00:18 Drop ICMP packet from WAN 216.254.161.247:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 10:00:04 Drop TCP packet from WAN 216.241.26.9:1606 216.254.16.31:135 Rule: Default deny
Sep/25/2003 10:00:01 Drop TCP packet from WAN 216.241.26.9:1606 216.254.16.31:135 Rule: Default deny
Sep/25/2003 09:59:39 Drop ICMP packet from WAN 216.254.234.23:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 09:59:38 Drop ICMP packet from WAN 217.1.168.39:8 216.254.16.31:0 Rule: Default deny
Sep/25/2003 09:58:41 Drop ICMP packet from WAN 216.254.209.4:8 216.254.16.31:0 Rule: Default deny
More ICMP requests today but yesterday there were nothing but port 135/137 requests from Speakeasy IPs.
MEH. |
|
Yourself
join:2000-08-10
Lynnwood, WA
| Sigh...yep Seattle is still a mess. Here is what I was seeing last night:
~45-60 ms to my first hop(usually 12-14)
20-50% Packet loss at my second hop
Speedtest results of 667/36 on a 1.5/384 line...
I use my line for gaming, so lately my service has been useless.
Self |
|
borborpa
Slipping Slowly Into Oblivion
Premium
join:2002-02-20
New Cumberland, PA
clubs:
 ·Speakeasy
| Kat has said that Seattle has not yet been corrected, but will be in a week (or something). I'm sure she will let everyone know as soon as it's fixed, so we can see results.
[EDIT] Here's what I was looking for...
Said by KatOak 
We performed some upgrades to the SEA POP on Wednesday, but the full upgrade isn't scheduled for a couple more weeks, I believe. We are working on an interim solution for this and the NYC POP, however.
--
There are no stupid questions, but there are a LOT of inquisitive idiots.[AIM - BoyBandsMakeUGay]
[text was edited by author 2003-09-25 19:45:52] |
|
Yourself
join:2000-08-10
Lynnwood, WA
| said by borborpa :
Kat has said that Seattle has not yet been corrected, but will be in a week (or something). I'm sure she will let everyone know as soon as it's fixed, so we can see results.
The problem with that is that the tech support rep I spoke with 2 days ago said the interim fix had already been completed, which supposedly included my circuit. Obviously that is not the case. |
|
