scr4t
join:2003-03-11
Columbia, MO
 ·Charter Pipeline
1 edit |  CenturyTel / Lightcore peering issue?
Could one of the CenturyTel (or Lightcore...) techs/employees please comment on this?
Before approximately two weeks ago I was routed from CenturyTel -> Lightcore -> Level3 -> destination (give or take a hop) for most locations. Since then, the route seems to be CenturyTel -> Lightcore -> UNKNOWN (timing out) -> destination.
So, to the point:
Did CenturyTel/Lightcore drop Level3 somewhere along the path in favor of XO Communications or some other provider? If so, would it be possible to find out which network is dropping ICMP packets? A sample traceroute of this is below, taken around 5 p.m. I've been seeing ping times above 200ms in the evening, and can't tell exactly where the trouble lies. The hops prior to the "mystery network" all seem to be fairly steady around 25ms. Related to this, does CT switch to cheaper routing during times of expected high usage? I'm not familiar with how things like this would work, exactly.
Also, a specific question about Columbia, MO:
Recently my ping to the first CT IP has doubled from under 10 to near 20ms. This isn't huge, but every bit adds up when combined with the routing issue previously mentioned. Is Columbia experiencing a heavy load due to the upcoming finals week, or am I possibly looking at a different issue?
Thanks!
|
|
dsldude08
Premium,VIP
join:2008-01-03
La Crosse, WI
 ·CenturyLink
| You could try Wireshark »www.wireshark.org/download.html
See how that goes. As for switching routes for cheaper bandwidth, that is possible too, but I cannot confirm that. If there is a fiber cut or issue somewhere along the line the routes can be changed also, or if there is unscheduled emergency maintenance or even regular maintenance and so forth.
As for the jump to the first hop in latency due to finals week, that is also possible but I cannot confirm either.
My apologies for any inconvenience this may be causing for you but the Wireshark should pinpoint where the packets are being dropped and as to what IP it is related to, please run it and see what you can come up with.
Thanks. |
|
scr4t
join:2003-03-11
Columbia, MO
·Charter Pipeline
| Thanks for the reply 
I ran Wireshark while doing a traceroute (I'm assuming that is what you were recommending) but unfortunately came up with the same results. Where the early hops properly show the TTL exceeded message from each router, nothing is sent back from the "mystery network" (TTL = 5). |
|
iansltx
join:2007-02-19
Golden, CO
·Comcast
·Qwest.net
 ·magicjack.com
 ·BeeCreek Communica..
·Sprint Mobile Broa..
| Just a random idea: trace to random looking glass servers (L3, Cogent) tntil you find one that exhibits the same packet dropping behavior. You've then found the provider that's dropping your packets. Look up LightCore's AS on Robtex to figure out which carriers you need to test... |
|
scr4t
join:2003-03-11
Columbia, MO
·Charter Pipeline
1 edit | reply to scr4t
All the college students have left and my first hop latency is still twice what it should be. I'm dreading the call to tech support to explain this one.
Also, I'm still experiencing bad routing in comparison to what I had about a month ago. Still worth noting is hop number 5.
*edit: At the very least I'm hoping CenturyTel/Lightcore can figure out why traffic is being routed from their Chicago location to Little Rock, Shreveport, Dallas, Denver, and then back to Chicago. :)
To NYC:
To Atlanta:
To Chicago (routing A):
To Chicago (routing B):
|
|
scr4t
join:2003-03-11
Columbia, MO
·Charter Pipeline
| reply to scr4t
It looks like things might have improved! I'm back to ~25ms with most Chicago locations, and 50 or so elsewhere.
One minor issue is that I still have strange routing (through Denver) to select CHI servers. Hopefully that will work itself out. |
|
scr4t
join:2003-03-11
Columbia, MO | reply to scr4t
Well, that lasted a little under two weeks. Back to crap. |
|
iansltx
join:2007-02-19
Golden, CO | What are the traceroutes like these days? Maybe try a reverse traceroute and post wht you're seeing there. SOftLayer and FDCServers (lg.softlayer.com and lg.fdcservers.net) would be good places to try from. |
|
scr4t
join:2003-03-11
Columbia, MO
·Charter Pipeline
3 edits | Thanks for the suggestion (and links).
I'm not sure what these reverse traceroutes suggest...
From my end it looks like I get to Lightcore CHI without going through Savvis in Dallas.
»lg.level3.net (Chicago location):
»lg.softlayer.com/ (WDC location):
»lg.fdcservers.net (CHI location):
|
|
iansltx
join:2007-02-19
Golden, CO
·Comcast
·Qwest.net
·magicjack.com
·BeeCreek Communica..
·Sprint Mobile Broa..
| Reverse traceroute shows where unexpected increases in ping might be coming from, stuff that might or might not be out of CenturyTel's control. In this case, providers are favoring Savvis over whatever else CenturyTel uses in Chicago. Problem is, CenturyTel doesn't interconnect with Savvis in Chicago, but they do in Dallas. Hence the latency jump.
It's hard to explain, but incoming packets don't necessarily take the same route as outgoing ones; when two backbones are each trying to keep packets on their own network as long as possible this is what happens. If Lightcore interconnected with Savvis in Chicago then you'd have lower pings. Or if they connected to a network that the various web hosting companies preferred to Savvis, the same thing would happen.
Hope that clears things up. I dislike asymmetric routing as much as the next guy, but looks like that's what's going on here. |
|
iansltx
join:2007-02-19
Golden, CO
·Comcast
·Qwest.net
·magicjack.com
·BeeCreek Communica..
·Sprint Mobile Broa..
| reply to scr4t
For some light reading check out
»www.robtex.com/as/as22561.html
That's the ASN for Lightcore. Looks like thee's a decent amount of Level3 in there, but there's also a heaping helping of XO and Savvis. the % numbers are percentage of IP space reached through that ASN as far as I can understand. So therein lies your latency problem.
FWIW, Comcast is a bit worse with their routing. Denver traffic often goes to LA before switching to Savvis or the destination network. Just sayin'... |
|
scr4t
join:2003-03-11
Columbia, MO
1 edit | Ah, thanks for the info. What is most annoying about this is how the routing has kept switching between good and bad over the past month or so... welcome to the Internet, I suppose. Maybe the situation will work itself out eventually. |
|
scr4t
join:2003-03-11
Columbia, MO
·Charter Pipeline
3 edits | reply to scr4t
Okay, somehow it seems like my routing is dependent upon the IP I receive. All of them (with the possible exception of the 99.x IP) show up as CenturyTel's in a WHOIS.
Examples:
72.161.240.x = ~50ms to Chicago, 70+ ms to TX and the east
69.29.73.x = ~50ms to Chicago, 70+ ms to TX and the east
69.29.63.x = under 30ms to Chicago, ~50 ms to TX and the east
72.161.205.x = under 30ms to Chicago, ~50 ms to TX and the east
Yesterday I was receiving a 99.x.x.x IP that gave sub-30ms to Chicago.
All traceroutes look the same, but I still have the dropped ICMP packets in Chicago (after bb-chci-jp-01-ae-0.core.centurytel.net), so the routing is obviously different in that area.
I'm going to keep track of which IPs give the superior routing, and those that don't. Maybe this is something that CenturyTel would be willing to check out. |
|
iansltx
join:2007-02-19
Golden, CO | Again, run a reverse traceroute at places like FDCServers and SoftLayer. That's probably where you're getting a latency drop on your different IP. |
|
