dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2776
share rss forum feed

austinfish

join:2002-11-22
Austin, TX

Global Crossing Packet Loss

I'm noticing extremely slow download speeds and packet loss with almost any traffic that goes through Global Crossing's backbone. This started Tuesday and has continued to get worse. Running a ping to globalcrossing.com shows about 10% packet loss, but it comes in bursts of 30-40% loss. Download speeds are only 15-30 KB/sec on sites through Global Crossing.

I ran several traceroutes to globalcrossing.com and the tracert looks good to ATT's last router, but the first Global Crossing router seems to frequently have packet loss.

8 72 ms 72ms 72 ms 151.164.251.162
9 111 ms * 110ms 206.132.195.69

I don't see any issues to Level 3. There is no packet loss at all to level3.com and better latency (71 ms) compared to 110 ms for Global Crossing.

Is there anything that customers can do to influence routing? Is ATT aware of the issue?



Gaff
Just like the gypsy woman said

join:1999-09-05
North TX, US

Posting in Direct (»AT&T Direct) is probably the best way to get this resolved.
--
My PC Gaming Blog
»www.gaffonline.com



NormanS
I gave her time to steal my mind away
Premium,MVM
join:2001-02-14
San Jose, CA
kudos:11
Reviews:
·SONIC.NET
·Pacific Bell - SBC
reply to austinfish

said by austinfish:

I ran several traceroutes to globalcrossing.com and the tracert looks good to ATT's last router, but the first Global Crossing router seems to frequently have packet loss.

What about the destination? If you see no packet loss at the far end, then you are only seeing the intermediary routers not responding to trace route.

Trace route is really pretty useless, by itself, and should be considered for the whole route, not just a part of it.
--
Norman
~Oh Lord, why have you come
~To Konnyu, with the Lion and the Drum

austinfish

join:2002-11-22
Austin, TX

I'm seeing packet loss through ping to the destination and I don't think it's my local connection. A ping to globalcrossing.com and to pair.com (my web host that routes through globalcrossing) have consistent 10% packet loss and fairly high latency (110ms). A ping to level3.com has no packet loss and about 72 ms latency. There's also no packet loss with Google or Yahoo, but they appear to have direct connections to ATT. It seems to possibly be with ATT's connection to Global Crossing. I see the packet loss on different Global Crossing routers, for example, 64.210.21.150 and 206.132.195.69.



NormanS
I gave her time to steal my mind away
Premium,MVM
join:2001-02-14
San Jose, CA
kudos:11
Reviews:
·SONIC.NET
·Pacific Bell - SBC

All I am saying is that you have to look at the trace route from end to end. If you see 33% packet loss at the seventh hop, and 66% packet loss at the tenth hop, but only 10% packet loss at the destination, then you only have 10% packet loss. Maybe.

It would be curious to know what your destination is; others might want to take a look as well.
--
Norman
~Oh Lord, why have you come
~To Konnyu, with the Lion and the Drum



Napsterbater
Meh
Premium,MVM
join:2002-12-28
Milledgeville, GA
Reviews:
·Windstream

said by NormanS:

All I am saying is that you have to look at the trace route from end to end. If you see 33% packet loss at the seventh hop, and 66% packet loss at the tenth hop, but only 10% packet loss at the destination, then you only have 10% packet loss. Maybe.

It would be curious to know what your destination is; others might want to take a look as well.

A Full tracert would help.
--
ASUS M4A79T Deluxe | AMD Phenom II x3 720 BE AM3 w/4 Cores @ 3.41Ghz(OC) | 4Gb DDR3 Memory @ 1600mhz | Sapphire ATI HD4870 1GB 800mhz/1000mhz(OC) | 2x500GB HDD's Raid 0 | Windows 7 Ultimate x64 Build 7600 (RTM) | Windstream DSL 12m (14.9m Sync)/766k


McSummation
Mmmm, Zeebas Are Tastee.
Premium,MVM
join:2003-08-13
Fort Worth, TX
kudos:2
reply to austinfish

Do you know where 206.132.195.69 is physically located?

I'm getting latency times similar to yours. However, I'm not getting any lost packets on ping to that location.

BTW, to the folks not in CenTex, the OP and I should be going out on the same backbone as we're only a few miles apart.


austinfish

join:2002-11-22
Austin, TX

It's slightly better today. I have to let the ping run multiple tests and after about 100, typically around 4-7% are dropping to 206.132.195.69. This seems to be my route going out:
6 22 ms 23 ms 23 ms 151.164.103.116
7 33 ms 33 ms 34 ms 151.164.101.82
8 71 ms 71 ms 71 ms 151.164.251.162
9 110 ms 109 ms 109 ms 206.132.195.69

It appears to be all ATT until hop 9 at the Global Crossing router, which is where I see the packet loss. There's still no packet loss pinging to Google, Yahoo, or Level 3. I'm not sure where 206.132.195.69, I wasn't able to get a host name and geolocation sites show different locations (Kansas City and New York City).



Napsterbater
Meh
Premium,MVM
join:2002-12-28
Milledgeville, GA

Just show the whole trace.

What are the ping times after 206.132.195.69?



NormanS
I gave her time to steal my mind away
Premium,MVM
join:2001-02-14
San Jose, CA
kudos:11
Reviews:
·SONIC.NET
·Pacific Bell - SBC
reply to austinfish

Just to show (routing from S.F. Bay Area, so not what you will see; but I've seen some folks claim the second from the last hop of evidence of packet loss) what I see to 'pair.com':

Tracing route to pair.com [66.39.3.7]
over a maximum of 30 hops:
 
  1     8 ms     8 ms     9 ms  adsl-69-105-119-254.dsl.pltn13.pacbell.net [69.105.119.254]
  2     9 ms     9 ms     9 ms  dist1-vlan50.pltn13.pbi.net [64.164.97.66]
  3     9 ms     9 ms     9 ms  ag2-10g0-12-0-0.pltnca.sbcglobal.net [151.164.102.250]
  4    10 ms    10 ms    11 ms  ppp-151-164-52-213.rcsntx.swbell.net [151.164.52.213]
  5    20 ms    19 ms    19 ms  po-4-100-10G.ar2.SJC2.gblx.net [64.215.195.241]
  6    85 ms    75 ms    75 ms  64.210.21.150
  7     *        *        *     Request timed out.
  8    77 ms    77 ms    76 ms  www.pair.com [66.39.3.7]
 
Trace complete.
 
This one is actually pretty short on hops, as trace routes go. And, despite one host name implying routing through Texas, this one is actually entirely in the south S.F. Bay Area of California. Up to the Global Crossing connection in hop 5.

If I were seeing issues beginning at hop 5, and carrying through to hop 8, that would be a Global Crossing problem. Seeing the entire trace route is most helpful.

--
Norman
~Oh Lord, why have you come
~To Konnyu, with the Lion and the Drum

austinfish

join:2002-11-22
Austin, TX

This is the full traceroute to pair.com. I notice about the same packet loss to 64.210.21.150 as to pair.com (4-8%). I'm not getting a ping response from 151.164.101.82 or 151.164.251.150, so I'm not completely sure whether the problem is internal in AT&T or at the connection to Global Crossing. Hops 3-5 seem to only show an IP some of the time:

1 * * * Request timed out.
2 31 ms 24 ms 23 ms 99-156-84-2.lightspeed.austtx.sbcglobal.net [99.156.84.2]
3 24 ms 24 ms 23 ms 75.8.139.8
4 * * * Request timed out.
5 25 ms 24 ms 25 ms 75.8.128.14
6 23 ms 22 ms 22 ms bb1-g10-0.aus2tx.sbcglobal.net [151.164.92.90]
7 34 ms 34 ms 35 ms 151.164.101.82
8 72 ms 72 ms 71 ms 151.164.251.150
9 110 ms 110 ms 114 ms 64.210.21.150
10 * * * Request timed out.
11 110 ms 112 ms 111 ms www.pair.com [66.39.3.7]


I don't have any packet loss to google.com. A tracert for Google is:

1 * * * Request timed out.
2 29 ms 27 ms 30 ms 99-156-84-2.lightspeed.austtx.sbcglobal.net [99.156.84.2]
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 24 ms 22 ms 22 ms 151.164.103.116
7 33 ms 33 ms 42 ms 151.164.101.82
8 33 ms 33 ms 33 ms 72.14.223.13
9 34 ms 33 ms 71 ms 72.14.233.77
10 34 ms 33 ms 33 ms 209.85.250.77
11 34 ms 34 ms 33 ms 74.125.227.51



PFH
Bully For You

join:2003-06-26
Saint Louis, MO
reply to austinfish

Be forewarned that some carriers, when their routers are under significant load, downgrade or drop ICMP/ping/tracert requests in favor of 'real' traffic. I can't remember offhand the discussion of which hauler was reported to be doing this.

Your stats, in that case, may not be very meaningful.



Napsterbater
Meh
Premium,MVM
join:2002-12-28
Milledgeville, GA
reply to austinfish

Notice the last hops don't show any loss.

Try a patchping and post the results.



annoyed

@sbcglobal.net

same problem here with global crossing. using att dsl from connecticut with latency and packet loss. obviously att tech support is useless. my ping and pathping results:

C:\WINDOWS>ping 146.82.33.238 (this is a gblx hop to the host i'm trying to reach)

Pinging 146.82.33.238 with 32 bytes of data:

Reply from 146.82.33.238: bytes=32 time=188ms TTL=242
Reply from 146.82.33.238: bytes=32 time=186ms TTL=242
Request timed out.
Reply from 146.82.33.238: bytes=32 time=188ms TTL=242

Ping statistics for 146.82.33.238:
Packets: Sent = 4, Received = 3, Lost = 1 (25% loss),
Approximate round trip times in milli-seconds:
Minimum = 186ms, Maximum = 188ms, Average = 187ms

---------------------

C:\WINDOWS>pathping 146.82.33.238

Tracing route to PCCW-GLOBAL-INC.Gi4-5.asr1.PTY1.gblx.net [146.82.33.238]
over a maximum of 30 hops:
0 vatest [10.0.0.100]
1 10.0.0.1
2 192.168.1.254
3 adsl-99-96-75-254.dsl.wlfrct.sbcglobal.net [99.96.75.254]
4 * * *
Computing statistics for 100 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 vatest [10.0.0.100]
0/ 100 = 0% |
1 1ms 0/ 100 = 0% 0/ 100 = 0% 10.0.0.1
0/ 100 = 0% |
2 1ms 0/ 100 = 0% 0/ 100 = 0% 192.168.1.254
0/ 100 = 0% |
3 10ms 0/ 100 = 0% 0/ 100 = 0% adsl-99-96-75-254.dsl.wlfrct.sbcgl
obal.net [99.96.75.254]
100/ 100 =100% |
4 --- 100/ 100 =100% 0/ 100 = 0% vatest [0.0.0.0]

Trace complete.


austinfish

join:2002-11-22
Austin, TX

I'm glad to hear that someone else is having the same problem, but AT&T hasn't done anything to help. In fact, it's even worse now. Now, I'm getting 10-20% packet loss to the Global Crossing router (206.132.195.69) and to globalcrossing.com. I also tried 146.82.33.238 with the same results (13% loss) If I run ping with 100 echos, it will generally run 10-15 tests in a row successfully and then error on several packets within a short duration of time.

I have a file hosted on pair.com for my software product at »www.acrovista.com/bellcommander/blcmd500.exe

It's only downloading at 8-15 KB/sec (not much better than dialup). Though, I also have it on Amazon Cloudfront at »download.acrovista.com/blcmd500.exe and it's downloading at about my maximum speed 1.18 MB/sec.


austinfish

join:2002-11-22
Austin, TX

Also, to rule out it being a pair.com issue, I also get the extremely slow download rates to globalcrossing. This PDF downloads at the same 8-15KB/sec for me - »www.globalcrossing.com/docs/ente···line.pdf



shaffer

@sbcglobal.net

I'm experiencing the same issue. I'm an AT&T U-Verse customer trying to get to my servers hosted at a datacenter nearby and they must pass through an AT&T - Global Crossing link. For me the problem is on my return path. From my server, everything on Global Crossing pings fine, but as soon as it jumps to the AT&T network I get 10-15% packet loss.

Also, from my U-Verse connection, I also see the problem going to the globalcrossing.com website as well as some other sites such as Facebook which traverse a similar circuit.



annoyed

@sbcglobal.net

now this morning (in connecticut) i'm getting considerably better results. looking at some other threads, some users claim that att has migrated a portion of their home clients to old equipment that grinds to a halt when usage rises. will test the speed in the pm to see if the theory holds. anyways, here are my gblx ping results today, like 300% faster and no timeouts so far:

C:\WINDOWS>ping 146.82.33.238

Pinging 146.82.33.238 with 32 bytes of data:

Reply from 146.82.33.238: bytes=32 time=78ms TTL=241
Reply from 146.82.33.238: bytes=32 time=76ms TTL=241
Reply from 146.82.33.238: bytes=32 time=77ms TTL=241
Reply from 146.82.33.238: bytes=32 time=77ms TTL=241

Ping statistics for 146.82.33.238:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 76ms, Maximum = 78ms, Average = 77ms



annoyed

@sbcglobal.net

the sucking has begun. guess users are waking up around the country and getting on facebook and youtube and gaming sites.

C:\WINDOWS>ping 146.82.33.238

Pinging 146.82.33.238 with 32 bytes of data:

Reply from 146.82.33.238: bytes=32 time=80ms TTL=241
Reply from 146.82.33.238: bytes=32 time=77ms TTL=241
Reply from 146.82.33.238: bytes=32 time=84ms TTL=241
Reply from 146.82.33.238: bytes=32 time=94ms TTL=241

Ping statistics for 146.82.33.238:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 77ms, Maximum = 94ms, Average = 83ms


austinfish

join:2002-11-22
Austin, TX

The problem seems to have been fixed. I don't have any packet loss to Global Crossing now and downloads are at normal speeds. Doing a traceroute, there was about 40 ms of latency in AT&T's network between 151.164.101.82 and 151.164.251.162, but now there doesn't seem to be any. This results in about 87 ms ping times to pair.com compared to 112 ms before.



annoyed

@sbcglobal.net

not here. getting worse as we move into the evening:

C:\WINDOWS>ping 146.82.33.238

Pinging 146.82.33.238 with 32 bytes of data:

Reply from 146.82.33.238: bytes=32 time=193ms TTL=241
Reply from 146.82.33.238: bytes=32 time=376ms TTL=241
Reply from 146.82.33.238: bytes=32 time=180ms TTL=241
Reply from 146.82.33.238: bytes=32 time=184ms TTL=241

Ping statistics for 146.82.33.238:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 180ms, Maximum = 376ms, Average = 233ms


austinfish

join:2002-11-22
Austin, TX

I checked and I still see about 5% packet loss to this router. I tested and saw 1% packet loss out of 100 echos to globalcrossing.com. For me, both route through 151.164.251.162 (AT&T), but 206.132.195.69 (GC) seems to be better than 146.82.33.238.



annoyed

@sbcglobal.net

thanks for checking. performance continues to go downhill as the day wears on. look at my speed to LA. this is a disgrace.
»www.speedtest.net/result/1212113324.png


daveinct

join:2011-03-25
reply to austinfish

Hi Guys,

This thread came to my attention today, and I thought I should share some information with you. I am a senior network/sys admin at a large hosting provider in CT. We are multi-homed, with Global Crossing being one of our network carriers. I also happen to be a residential AT&T Uverse customer. A couple months ago I began noticing packet loss and very slow download speeds from servers where I work to my home. This was especially noticeable since I frequently use SSH from home to work. Anyone who uses SSH knows that packet loss or latency is very visible when typing commands on a server. Further investigation showed that my traffic to/from AT&T Uverse was being routed via our Global Crossing circuit. The problem was generally not noticeable until peak traffic usage times on the Internet (evenings), which was a sign that a circuit is saturated. Since we are a Global Crossing customer, I was able to contact them to get their take on the issue. Their network engineers logged into the routers in question, and confirmed that the SBC/Global Crossing peering interface in the northeast was saturated (overloaded). SBC (AS7132) handles traffic for AT&T Uverse and DSL, at least in Connecticut. Keep in mind that SBC is actually a separate network from AT&T (AS7018).

I also contacted AT&T support, and worked my way up to their escalations team, and ultimately established an e-mail contact on their core networking group. They confirmed that they are aware of the situation, and had no eta on when it would be resolved. I hounded them on this and pretty much got no where.

Since we are multi-homed, I got around this problem by forcing traffic destined to SBC (AS7132) to leave our network via another network carrier, instead of Global Crossing. This resolves problems for our customers who happen to be on AT&T connections in CT. However, this does nothing to resolve the larger problem which is plaguing access to a large chunk of the Internet from AT&T residential connections, when their routing path includes Global Crossing.

Basically you have a peering arrangement between SBC and Global Crossing. SBC may actually be paying Global Crossing for Internet transit, but I am not positive on that (if so, that could explain why they aren't fixing it faster - they might not want to pay for more bandwidth). When there is peering between two providers, they typically peer at multiple locations around the country. On the East coast SBC peers with Global Crossing at least in New Jersey (or possibly Virginia), and also in Washington DC. From what I was told by Global Crossing engineers, SBC is not announcing all of their routes/prefixes at the Washington DC peering point. As a result of this, more traffic is flowing through the Northeast peering point, contributing to the saturation.

Both SBC and Global Crossing core networking engineers are well aware of the issue, but don't seem to be in any rush to fix it. Global Crossing is putting the blame on SBC for mis-configured BGP route advertisements, and says that they can not force SBC to fix that problem, which does make some sense. Bottom line is SBC either needs to re-configure their routing advertisements or get a bigger pipe with Global Crossing.

On a side note, some technical users may be interested to know that SBC has a public route server, which can be accessed via telnet: route-server.sbcglobal.net

My best recommendation is for everyone affected to report the problem in detail to AT&T support. They should also reference this forum thread, so that AT&T has a record of how many people are complaining. This should hopefully put more pressure on them to fix it quicker.

-Dave



NormanS
I gave her time to steal my mind away
Premium,MVM
join:2001-02-14
San Jose, CA
kudos:11
Reviews:
·SONIC.NET
·Pacific Bell - SBC
reply to annoyed

said by annoyed3:

thanks for checking. performance continues to go downhill as the day wears on. look at my speed to LA. this is a disgrace.
»www.speedtest.net/result/1212113324.png

Which has what to do with Level 3 Global Crossing packet loss?
--
Norman
~Oh Lord, why have you come
~To Konnyu, with the Lion and the Drum