Comcast XFINITY → Sometimes it pays to ask . . . pings cut in half

I noticed a somewhat odd high ping jump on sl-williamscom-10-0.sprintlink.net anytime I ran a traceroute to DSLR. I posted an example and asked for input in this thread. Although I didn't receive as many examples in that thread as I would have liked, it prompted me to also search the forum for examples . . . which I found.

»Re: HEEELLLLP!!!!
»Re: Slow throughput /High latency in Maryland tonight?
»Re: RANT: Comsucks in NJ (McGuire AFB)
»Re: The End of Sprintlink?
»Re: sprint routers are at it again...crazy pings

A WHOIS on sl-williamscom-10-0.sprintlink.net
[144.223.72.30] indicated it was a Sprintlink.net IP, but on contacting them, they replied that it actually belonged to Williams Communications Group . . . which I should have seen. Anyway, I contacted Williams Communications Group with a simple question (edited for brevity).

quote:

---

ME: According to Sprintlink.net, sl-williamscom-10-0.sprintlink.net belongs
to you . . . Williams Communication Group.

I'm hoping you can help me . . .
Why is there such a HUGE ping increase on hop 15?
This is an ongoing and persistent problem, not an isolated incident.

I'm not the only one who experiences this . . .
»Re: HEEELLLLP!!!!
»Re: Slow throughput /High latency in Maryland tonight?
»Re: RANT: Comsucks in NJ (McGuire AFB)
»Re: The End of Sprintlink?
»Re: sprint routers are at it again...crazy pings

C:\WINDOWS>tracert dslreports.com
Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 12 ms 10 ms 10 ms 10.159.32.1
2 12 ms 15 ms 11 ms 172.30.5.209
3 11 ms 12 ms 12 ms 172.30.5.238
4 14 ms 12 ms 12 ms 172.30.5.174
5 15 ms 16 ms 14 ms 172.30.5.110
6 13 ms 14 ms 13 ms 172.30.5.60
7 14 ms 13 ms 14 ms 172.30.4.108
8 15 ms 16 ms 19 ms 172.30.10.177
9 14 ms 16 ms 17 ms 172.30.10.186
10 23 ms 19 ms 17 ms 68.34.76.50
11 17 ms 16 ms 18 ms sl-gw26-rly-2-0.sprintlink.net [144.232.238.233]
12 17 ms 16 ms 18 ms sl-bb20-rly-3-3.sprintlink.net [144.232.14.33]
13 19 ms 17 ms 17 ms sl-bb26-rly-9-0.sprintlink.net [144.232.7.250]
14 18 ms 17 ms 19 ms sl-gw33-rly-9-0.sprintlink.net [144.232.14.98]
15 56 ms 49 ms 49 ms sl-williamscom-10-0.sprintlink.net [144.223.72.30]
16 55 ms 60 ms 56 ms hrndva1wcx2-oc48.wcg.net [64.200.95.73]
17 54 ms 55 ms 54 ms nycmny2wcx2-oc48.wcg.net [64.200.240.45]
18 55 ms 57 ms 55 ms nycmny2wcx1-netaccess-atm.wcg.net [64.200.86.178]
19 58 ms 56 ms 57 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
20 53 ms 54 ms 55 ms dslreports.com [209.123.109.175]

THEM: Your return path was traversing through Dallas.. You should now be exiting our network on the east coast..

Please try again..

Thanks,

{Name Deleted}
Network Engineering
IP/Data Services
WilTel Communications

---

Well, I did try again . . . and now pings through their network have been cut in half.

C:\WINDOWS>tracert dslreports.com
Tracing route to dslreports.com [209.123.109.175] over a maximum of 30 hops:
1 10 ms 9 ms 10 ms 10.159.32.1
2 11 ms 11 ms 11 ms 172.30.5.209
3 11 ms 11 ms 12 ms 172.30.5.238
4 11 ms 11 ms 12 ms 172.30.5.174
5 12 ms 14 ms 17 ms 172.30.5.110
6 13 ms 14 ms 13 ms 172.30.5.60
7 18 ms 15 ms 15 ms 172.30.4.108
8 14 ms 14 ms 23 ms 172.30.10.177
9 14 ms 17 ms 15 ms 172.30.10.186
10 22 ms 17 ms 18 ms 68.34.76.50
11 16 ms 21 ms 17 ms sl-gw26-rly-2-0.sprintlink.net [144.232.238.233]
12 22 ms 18 ms 18 ms sl-bb20-rly-3-3.sprintlink.net [144.232.14.33]
13 17 ms 19 ms 17 ms sl-bb26-rly-9-0.sprintlink.net [144.232.7.250]
14 21 ms 18 ms 17 ms sl-gw33-rly-9-0.sprintlink.net [144.232.14.98]
15 20 ms 21 ms 21 ms sl-williamscom-10-0.sprintlink.net [144.223.72.30]
16 26 ms 25 ms 26 ms hrndva1wcx2-oc48.wcg.net [64.200.95.73]
17 25 ms 27 ms 30 ms nycmny2wcx2-oc48.wcg.net [64.200.240.45]
18 27 ms 28 ms 29 ms nycmny2wcx1-netaccess-atm.wcg.net [64.200.86.178]
19 28 ms 28 ms 28 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
20 27 ms 27 ms 26 ms dslreports.com [209.123.109.175]

I'm not really sure what the Williams NOC meant when he said "Your return path was traversing through Dallas" . . . but since I'm not anywhere near Dallas, I did see the traceroute improvement immediately.

Can anyone confirm that their traceroutes to DSLR through sl-williamscom-10-0.sprintlink.net have also improved, most specifically the posters in the original examples;

Estandon in Salisbury, MD
Spector42 in Owings Mills, MD
aadic in Trenton, NJ
eco in Wilmington, DE
eliteone in Reading, PA
(example was actually a post by commtech, which gave me clue that it was a Williams Communication Group problem)

If I'm full of BS on this, please tell me so . . .

said by newview:

---

THEM: Your return path was traversing through Dallas.. You should now be exiting our network on the east coast..

Please try again..
I'm not really sure what the Williams NOC meant when he said "Your return path was traversing through Dallas" . . . but since I'm not anywhere near Dallas, I did see the traceroute improvement immediately.

---

One of the issues with a tracert is that it only shows you the path one way. Your return data path may be through a completely different route that you never see. What the Williams NOC is telling you is that they had a bad router table that had a return data path routed through Dallas when some other route would have been better.

Nice job researching and getting this issue corrected.

Now THAT'S going over Comcast's head. Good job.

said by newview:

---

I'm not really sure what the Williams NOC meant when he said "Your return path was traversing through Dallas" . . .

---

ahhh.. sprintlink's asymmetric routing strikes again..

basically what is happening is the ping packets take a different path getting back to you than they took when going to the destination.

without getting into too much detail, basically someone messed up a routing table (or don't use BGP - unlikely since this is between two large carriers) and as a result outbound traffic will take a drastically different path to get back to you instead of going back a similar path.

sprintlink is notorious for this.. had similar routing problems going back to 1995 and I doubt they will ever go away..

a way one can see this in action would be to try one of the many sites found at »www.traceroute.org/ - do a traceroute to your IP then do a traceroute TO the server, some you will see a totally different return route.

it in itself is not a problem, it's when one of the routers on the return is saturated/overloaded/broke - all you see are the high pings on that one hop and never see the true source of the bottleneck.

forget about troubleshooting, unless you have admin priv on the remote site you are tracing the route, or you are in contact with a tier 1 NOC, it will be near impossible to track down (believe me, I've been there before and it ain't fun tracking down why a link that is only 3% util is giving 400ms pings because some bozo fubared up a routing table half way across the country).


[text was edited by author 2003-01-22 17:44:33]

Help!

I am getting mid teens now consistantly!

I must say, excellent job. You not only solved your own issue but certainly helped a ton of other users wondering about the same problem but not knowing what to do about it.

look at this horrible mess:
C:\>tracert www.dslreports.com

Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 70 ms 70 ms 30 ms 10.177.0.1
2 80 ms 70 ms 70 ms 172.30.184.81
3 40 ms 30 ms 31 ms 12.118.168.17
4 50 ms 110 ms 30 ms gbr2-p51.hs1tx.ip.att.net [12.123.212.26]
5 70 ms 40 ms 80 ms gbr3-p40.dlstx.ip.att.net [12.122.2.97]
6 120 ms 70 ms 50 ms gbr4-p20.attga.ip.att.net [12.122.2.90]
7 50 ms 60 ms 110 ms gbr4-p30.wswdc.ip.att.net [12.122.2.226]
8 180 ms 50 ms 90 ms gbr2-p100.wswdc.ip.att.net [12.122.1.186]
9 91 ms 150 ms 70 ms tbr1-p012801.wswdc.ip.att.net [12.122.11.165]
10 80 ms 71 ms 70 ms tbr1-p013701.n54ny.ip.att.net [12.122.10.53]
11 131 ms 100 ms 110 ms gar1-p3100.nwrnj.ip.att.net [12.123.214.181]
12 60 ms 71 ms 120 ms 12.119.140.30
13 60 ms 110 ms 180 ms 1189.at-0-1-0.gbr1.oct.nac.net [209.123.11.69]
14 130 ms 140 ms 80 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
15 60 ms 100 ms 120 ms www.dslreports.com [209.123.109.175]

Trace complete.

Yes I am seeing the improvement as well. Pings to DSLR used to be in the 50s now im seeing aronud 24-26ms avg. Actually my bud items from this board contacted me and told me about the ping drop so I did a few traces myself and see that to most locations the ping was cut in half. Nice goin man!
[text was edited by author 2003-01-23 21:15:34]

My ping is still crappy and irregular. Any other ideas?

Those pings are no so bad. It looks like your average is around 30ms and never goes above 50ms. That little bit of packet loss looks nasty though.

My improvement is dramatic.....thanks to whomever made the decision to switch me OFF sprintlink.....and back to att

Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 7 ms 7 ms 8 ms 10.75.224.1
2 8 ms 9 ms 8 ms 172.30.103.65
3 10 ms 9 ms 9 ms 172.30.100.177
4 10 ms 9 ms 16 ms 172.30.100.186
5 9 ms 11 ms 9 ms 68.48.0.50
6 27 ms 44 ms 11 ms 12.126.168.5
7 11 ms 11 ms 11 ms tbr1-p012201.wswdc.ip.att.net [12.123.9.74]
8 15 ms 16 ms 15 ms tbr1-p013701.n54ny.ip.att.net [12.122.10.53]
9 16 ms 14 ms 16 ms gar1-p3100.nwrnj.ip.att.net [12.123.214.181]
10 15 ms 15 ms 16 ms 12.119.140.30
11 16 ms 19 ms 16 ms 1189.at-0-1-0.gbr1.oct.nac.net [209.123.11.69]
12 16 ms 16 ms 15 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
13 17 ms 15 ms 15 ms www.dslreports.com [209.123.109.175]

Trace complete.
[text was edited by author 2003-01-23 23:04:57]

Wow, nice work. I can't explain to you in words that YOU, got to the bottom of something people who are Comcast customers affected by this problem and have WRITTEN routing algorithms and proven them to be efficient were not able to take a peak at, diagnose and get to the bottom of. Nice work to you, and shame on Comcast, all you have to be is nice sometimes. If only their engineers were as nice as the people they send out to customers.

WOW . . . looks like some happy people! I guess my email to Williams NOC with my simple question solved some problems for a few of you. It was really no big deal, it only affected subscribers in Maryland, Delaware, New Jersey, Virginia and Pennsylvania.

Maybe I should send an invoice to Comcast for services rendered.

And now for my next trick . . .

No my friend, it was a big deal :P.

I forwarded it too both of them. Can't wait for the responses...

said by Estandon:

---

I forwarded it too both of them. Can't wait for the responses...

---

Please let me know what they say . . . thanks.

Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 9 ms 9 ms 11 ms 10.90.8.1
2 10 ms 32 ms 10 ms 172.30.140.81
3 10 ms 7 ms 12 ms 172.30.140.118
4 10 ms 13 ms 8 ms 68.80.0.230
5 9 ms 9 ms 10 ms POS4-0.hsipaccess2.Philadelphia1.Level3.net [63.
208.100.69]
6 10 ms 11 ms 9 ms ge-6-2-1.mp2.Philadelphia1.level3.net [209.247.9
.37]
7 11 ms 11 ms 12 ms so-4-3-0.mp2.NewYork1.Level3.net [209.247.8.66]

8 16 ms 11 ms 11 ms gigabitethernet5-1.core1.NewYork1.Level3.net [64
.159.17.101]
9 18 ms 12 ms 13 ms unknown.wcg.net [209.244.160.138]
10 16 ms 19 ms 14 ms nycmny2wcx1-netaccess-atm.wcg.net [64.200.86.178
]
11 15 ms 17 ms 13 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
12 14 ms 13 ms 13 ms www.dslreports.com [209.123.109.175]

Trace complete.

Wow
my traceroutes are abysmal!
traceroute to dslreports.com (209.123.109.175), 30 hops max, 40 byte packets
1 192.168.1.254 (192.168.1.254) 1.662 ms 1.238 ms 1.351 ms
2 10.56.216.1 (10.56.216.1) 11.747 ms 11.564 ms 10.607 ms
3 172.30.155.81 (172.30.155.81) 11.438 ms 12.171 ms 11.641 ms
4 172.30.158.45 (172.30.158.45) 13.923 ms 18.410 ms 15.103 ms
5 172.30.158.58 (172.30.158.58) 22.130 ms 12.375 ms 13.642 ms
6 172.30.158.115 (172.30.158.115) 12.172 ms 13.974 ms 14.150 ms
7 68.42.244.162 (68.42.244.162) 18.492 ms 18.190 ms 17.263 ms
8 12.119.243.37 (12.119.243.37) 21.655 ms 13.674 ms 16.468 ms
9 gbr1-p20.dtrmi.ip.att.net (12.123.139.30) 13.005 ms 13.552 ms 13.804 ms
10 tbr2-p012502.phlpa.ip.att.net (12.122.3.5) 28.823 ms 32.444 ms 26.699 ms
11 tbr1-p013601.phlpa.ip.att.net (12.122.9.165) 27.355 ms 27.056 ms 27.616 ms
12 tbr2-p012301.n54ny.ip.att.net (12.122.10.97) 33.819 ms 32.635 ms 28.904 ms
13 gar1-p340.nwrnj.ip.att.net (12.123.214.185) 28.978 ms 29.854 ms 35.228 ms
14 12.119.140.30 (12.119.140.30) 29.182 ms 32.418 ms 33.197 ms
15 1189.at-0-1-0.gbr1.oct.nac.net (209.123.11.69) 34.137 ms 31.778 ms 38.472 ms
16 0095.gi-1-1.msfc1.oct.nac.net (64.21.102.2) 33.776 ms 30.364 ms 35.652 ms
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Wow it worked....Thanks!

Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 45 ms 115 ms 119 ms 10.63.216.1
2 248 ms 79 ms 164 ms 172.30.155.17
3 117 ms 83 ms 72 ms 172.30.155.46
4 65 ms 135 ms 56 ms 172.30.154.186
5 91 ms 71 ms 79 ms 172.30.158.123
6 77 ms 75 ms 67 ms 68.42.244.186
7 89 ms 80 ms 101 ms 12.119.243.37
8 150 ms 195 ms 76 ms gbr1-p20.dtrmi.ip.att.net [12.123.139.30]
9 116 ms 81 ms 245 ms gbr1-p30.phlpa.ip.att.net [12.122.3.66]
10 169 ms 217 ms 74 ms tbr1-p012501.phlpa.ip.att.net [12.122.12.97]
11 76 ms 253 ms 170 ms tbr2-p012301.n54ny.ip.att.net [12.122.10.97]
12 307 ms 105 ms 86 ms gar1-p340.nwrnj.ip.att.net [12.123.214.185]
13 85 ms 84 ms 123 ms 12.119.140.30
14 108 ms 94 ms 118 ms 1189.at-0-1-0.gbr1.oct.nac.net [209.123.11.69]
15 112 ms 179 ms 74 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
16 115 ms 118 ms 96 ms www.dslreports.com [209.123.109.175]

Trace complete.

WOW thanks man, this is ownage! nah just kiddin, my pings have been like this for the past few months and comcast wont do anything until they get enough people on the complaint list for the problems in my area. the guy actually told me they would not put in any new nodes or split any current ones until they had enough complaints.

strugglin in michigan, drooling over your pings

Looking good here:

Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:

1 7 ms 7 ms 7 ms 10.129.192.1
2 8 ms 8 ms 7 ms 172.30.13.81
3 7 ms 7 ms 8 ms 172.30.13.123
4 7 ms 7 ms 7 ms 172.30.13.59
5 8 ms 7 ms 8 ms 68.54.81.2
6 8 ms 11 ms 8 ms sl-gw26-rly-2-0.sprintlink.net [144.232.238.233]
7 8 ms 7 ms 8 ms sl-bb20-rly-3-3.sprintlink.net [144.232.14.33]
8 8 ms 7 ms 7 ms sl-bb26-rly-9-0.sprintlink.net [144.232.7.250]
9 8 ms 9 ms 7 ms sl-gw33-rly-9-0.sprintlink.net [144.232.14.98]
10 12 ms 10 ms 12 ms sl-williamscom-10-0.sprintlink.net [144.223.72.30]
11 17 ms 18 ms 16 ms hrndva1wcx2-oc48.wcg.net [64.200.95.73]
12 18 ms 17 ms 18 ms nycmny2wcx2-oc48.wcg.net [64.200.240.45]
13 23 ms 18 ms 19 ms nycmny2wcx1-netaccess-atm.wcg.net [64.200.86.178]
14 18 ms 19 ms 17 ms 0095.gi-1-1.msfc1.oct.nac.net [64.21.102.2]
15 19 ms 19 ms 17 ms www.dslreports.com [209.123.109.175]

Trace complete.

Here is the reply from my comcast source who is in charge of the problem in our area...
Well the routing problem your friend made you aware of was definitley a
problem, However the upgrades we are doing will and have made a diffence for
you and your area, we also will continue to monitor and make other upgrades
in future needs. Are your pings to your first hop lower than what you first
stated, and the erratic pings how erratic are they. Are they consistent
through out the day or vary from peak hours to non peak hours, I'm still
working on this problem for you.