 | [AZ] Huge latency and packet loss issues in the Valley Area Hello,
Just today, I've noted very severe packet loss issues in my conncetion, resulting in unacceptable (> 1000ms) latency in any sort of online gaming. This issue is not limited to me or my own connection ; all gamers that I know connecting to the same server who live in the Valley area, and those who live in the Valley only, are experiencing the same night-and-day decline in performance.
I believe I have identified the source of the sudden problems, but a bit of background is needed. About a month ago, persistent issues arose from Cox's use of Cognet-hosted servers in order to route my signal to, amongst other places, the World of Warcraft server to which I connect in Chicago. Long story short, those servers, for whatever reason, are associated with poor performance and a lot of latency. After multiple calls, Cox rerouted the path to this server through hops that did not appear to be Cognet-hosted. This change immediately cleared up the persistent latency issues for not only myself but also for several other people living in the Valley area who had been reporting similar issues.
Just today (9 January), my tracerts revealed that Cox has swapped back to Cognet servers. The extreme latency and packet loss have immediately returned with this change, for both myself and other gamers that I have spoken to who live in the Valley. Only the inital and terminal hops on the Cognet network are the same IP's as they were a month ago. Unfortunately, the first hop, 154.54.13.129, happens to be the server that was known to be causing issues the last time around.
I know that there are Cox technicians who browse these boards, and I'd like to know if possible why Cox swapped back to the terrible Cognet servers in the first place. Additionally, I'd like this change to be reverted, because at present, the poor connection precludes any sort of actual gaming. There are no suitable ISP alternatives in my area.
The following image displays the acceptable trace route at top, taken from a PingPlotter sample recorded yesterday evening, and the Cognet trace route at bottom from tonight. I have chosen trace routes to my Chicago-based World of Warcraft server because I know my signal is routed through the Cognet network to get there. I am experiencing no issues with connecting to servers that do not entail routing my signal through said network.

If a comparison would be needed, here is an old traceroute from 4 December when similar lag issues were occuring prior to my swap off the Cognet servers. Cognet hops that I have used previously are bolded.
1 NELTHARION [192.168.1.1] 2 10.100.64.1 3 70.169.77.212 4 chndcorc01-te-0-4-0-2.ph.ph.cox.net [70.169.72.52] 5 chnddsrj01-ae2.0.rd.ph.cox.net [70.169.76.229] 6 langbprj02-ae2.rd.la.cox.net [68.1.1.19] 7 te7-5.ccr02.lax05.atlas.cogentco.com [154.54.13.129] 8 te0-1-0-1.ccr22.lax01.atlas.cogentco.com [130.117.50.89] 9 te0-3-0-6.ccr22.iah01.atlas.cogentco.com [154.54.3.185] 10 te0-4-0-6.ccr22.dfw01.atlas.cogentco.com [154.54.0.206] 11 te0-2-0-6.ccr22.mci01.atlas.cogentco.com [154.54.2.230] 12 te0-5-0-4.ccr22.ord01.atlas.cogentco.com [154.54.45.150] 13 te0-5-0-3.ccr22.ord03.atlas.cogentco.com [154.54.43.234] 14 att.ord03.atlas.cogentco.com [154.54.12.86] 15 cr1.cgcil.ip.att.net [12.122.84.54] 16 gar2.clboh.ip.att.net [12.122.133.197] 17 12.122.251.22 18 63.240.130.210
Note that even in the best circumstances, the Cognet route would have an extra hop, resulting in unneccessary latency. The top image corresponds with a ping of about 77ms to the server, whereas the bottom one results in a ping of 900-1200ms. The apparent packet loss after 63.240.130.210 is the result of pings past that hop being blocked and is not indicative of a problem.
I also find it interesting that the Cox server 68.1.1.19 shows consistent 40-50 % packet loss with PingPlotter, regardless of my actual connection performance. I have never gotten less than 40 % packet loss at this address at any time of day or on any connection. This particular hop does not appear to affect my connection quality at all, but as I have never recorded any instance where there was not packet loss at this address, I have little basis for comparison.
For reference for any Cox tech that may come accross this thread, I am presently on the Ultimate package. I can provide further details if it will help bring about a resolution of this problem. |
|
|
|
 | I have been experiencing this for everything I do, not just gaming. I have been through the whole power cycle the cable modem, etc bit.
I even thought that maybe my wireless connection was the issue, but even connecting up with ethernet doesn't change the issue. Even the speed tests via Cox's website are showing a huge amount of latency. |
|
 | reply to Shina Posting this for the benefit of anyone else that might have had this issue (it's rather specific as the signal would have to be routed through Cognet servers but for those who have it, it's devastating):
Spoke for two hours today on the phone with Cox "Tier II" technicians about the issue. Was informed that the cause of the swap back could be due to some sort of software update in the area that inadvertently rolled back the pathing specification, because the change that fixed it wasn't "saved" permanently and thus could be overwritten. He didn't claim with authority that this is what did happen, but implied it was a very possible cause. That said, none of the technicians I was able to reach knew how to switch the pathing back to something that works, knew of no one who knew how to do this, and could not find any notes on file about the incident from a month ago.
And yet, when this issue occured a month ago, someone at Cox was indeed able to get the routing switched to resolve the issues.
I basically took the same steps on the phone as I did last time when a resolution was attained. I have no idea what to do and am at my wit's end with this problem ; if anyone has an idea of how to proceed I would greatly appreciate the advice. |
|
 | how did you get to talk to a Tier II technician? I am having this same issue sadly and would like to call to vent/report as well. |
|
 | It took a good hour to get to one, but I have a history of legitimate internet issues and have always made a point of calling as soon and as often as possible until they are resolved because they directly impact my ability to make a living.
Chances are because this is an issue that they've had before, and fixed, and because I had extensive documentation of the nature of the problem (meaning the "reset your router and modem" step could be skipped), they put me through to a higher-level tech.
I'd like to emphasise that the tech I spoke to, whilst shedding some light on a probable cause for the change that triggered the issues, did not know how to fix it, nor did he know who to refer me to that could. My internet (as well as that of other people in the area) is not any better at present at all. |
|
 | reply to Shina So looking into this further I did a traceroute to another server and see that my second hop seems to be where the delay is happening.
tracert 70.84.211.97
Tracing route to 61.d3.5446.static.theplanet.com [70.84.211.97] over a maximum of 30 hops:
1 7 ms 1 ms 1 ms xx.xx.xx.xx 2 1117 ms 1133 ms * 10.132.160.1 3 896 ms 868 ms 715 ms ip68-2-7-81.ph.ph.cox.net [68.2.7.81] 4 785 ms 789 ms 831 ms chndcorc01-te-0-15-0-0.ph.ph.cox.net [70.169.72. 56] 5 * 977 ms 1027 ms 70.169.75.153 6 1320 ms 1156 ms 1189 ms 68.1.5.129 7 1084 ms 1123 ms 1104 ms xe-1-0-0.bbr01.eq01.pal01.networklayer.com [50.9 7.16.17] 8 1137 ms 1179 ms 1215 ms ae3.bbr02.eq01.sjc02.networklayer.com [173.192.1 8.242] 9 991 ms 991 ms 899 ms ae0.bbr02.cs01.lax01.networklayer.com [173.192.1 8.151] 10 837 ms 847 ms 524 ms ae7.bbr01.cs01.lax01.networklayer.com [173.192.1 8.166] 11 462 ms 873 ms 877 ms ae19.bbr01.eq01.dal03.networklayer.com [173.192. 18.140] 12 972 ms 1007 ms 1085 ms po31.dsr02.dllstx3.networklayer.com [173.192.18. 227] 13 1172 ms 1237 ms 1296 ms po32.dsr02.dllstx5.networklayer.com [70.85.127.1 10] 14 827 ms 825 ms 854 ms 61.d3.5446.static.theplanet.com [70.84.211.97]
Trace complete.
Now if I trace back from that server to the first internet routable Cox address, 68.2.7.81, the latencey is non existent, comparitively speaking.
Tracing route to 68.2.7.81 [68.2.7.81]...
hop rtt rtt rtt ip address fully qualified domain name 1 1 1 1 70.84.211.97 2 0 0 0 70.87.254.1 3 0 0 0 70.85.127.105 4 0 0 0 173.192.18.224 5 0 0 0 70.167.150.133 6 50 50 50 68.1.3.115 7 59 48 59 70.169.75.158 8 43 42 43 70.169.73.67 9 52 52 52 68.2.7.86 10 54 54 54 68.2.7.81 Trace complete |
|
 | reply to Shina Shina,
I to have Cox, and live in the Valley. I have same issues that started Jan 9th Connecting to Blizzard's Chicago Data server. My ping plotter results look nearly identical to yours so I won't bother posting them. I have spoken to 4people, and had someone come out to my house, and all they say is "your levels look good"
I also remember the issues a month ago with cognet servers, and am baffled that they are using them again.... I |
|
 | Prishe -
You may find this thread interesting »us.battle.net/wow/en/forum/topic···8?page=1
I did not start it, but it does describe our issue. I do play on the Chicago datacentre and that does to be the most affected server cluster. It's not the only one, but it appears to make the most use of Cognet, which seems to result in the worst packet loss/latency issues. |
|
 | reply to Shina Yea I saw that thread, It is reassuring to know everyone in AZ is being affected...
Can't wait to endlessly call Cox tomorrow, until they give me a tech that doesn't say "Unplug your modem, wait 30sec, plug it back in"
Nothing worse then being able to to do the peon's job better then them  |
|
 | reply to g33kd0m With that high of a latency at your second hop you probably have a signal level problem.. |
|
 | reply to Shina Shina,
Could you register and PM so I can look into this for you? Also, as an FYI any Tier I or Tier II tech support rep won't be able to make any changes to backbone peering routers so continually calling them won't do much good. The engineers are the ones that have access to investigate and correct any issues on those routers which we'll need to have investigate.
Thanks, -- Chris@Cox Communications Arizona |
|
 Hazmat join:2007-07-10 Laveen, AZ | reply to Shina I have the same issue.
Tracing route to 63.240.161.189 over a maximum of 30 hops
1 1 ms 1 ms 1 ms 192.168.1.1 2 * * * Request timed out. 3 9 ms 9 ms 9 ms wsip-184-178-205-34.ph.ph.cox.net [184.178.205.34] 4 13 ms 11 ms 11 ms 70.169.75.84 5 * 10 ms 9 ms 70.169.75.153 6 * * 25 ms langbprj02-ae2.rd.la.cox.net [68.1.1.19] 7 22 ms 21 ms 22 ms te0-0-0-21.ccr22.lax05.atlas.cogentco.com [38.104.84.13] 8 21 ms 24 ms 20 ms 154.54.88.198 9 58 ms 57 ms 57 ms te0-2-0-3.ccr22.iah01.atlas.cogentco.com [154.54.45.1] 10 64 ms 64 ms 63 ms te0-0-0-6.ccr22.dfw01.atlas.cogentco.com [154.54.3.177] 11 73 ms 73 ms 80 ms te0-3-0-6.ccr22.mci01.atlas.cogentco.com [154.54.46.209] 12 85 ms 85 ms 86 ms te0-5-0-4.ccr22.ord01.atlas.cogentco.com [154.54.45.150] 13 85 ms 86 ms 85 ms te0-1-0-5.ccr22.ord03.atlas.cogentco.com [154.54.5.2] 14 98 ms 99 ms * 192.205.37.173 15 102 ms 99 ms 99 ms cr1.cgcil.ip.att.net [12.122.84.54] 16 * * 97 ms gar3.cgcil.ip.att.net [12.122.132.213] 17 100 ms 98 ms 100 ms 12.122.251.22 18 104 ms * 100 ms 63.240.130.214 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out.
Trace complete.
This all started when Cox switched to using Cognet-hosted servers to route signals, and I don't appear able to convince someone that the path needs to be re-routed, or they to stop using Cognet.
My only recourse appears to drop Cox, but I thought I would come here first. Have received help before, maybe lightning can strike twice. |
|
 | reply to AZHSISUPPRT2 having the same issue as well as many other people in the area it seems. i can post my tracert / pingpath if necessary. |
|
 Cifer join:2012-06-07 Mesa, AZ | reply to Shina I've been having the same issue as well. I'm connecting to the LA datacenter for WoW, and the ping is still terrible.
My traceroute:
1 1 ms 1 ms 1 ms HELMSDEEP [192.168.1.1] 2 8 ms 8 ms 9 ms 10.52.64.1 3 11 ms 10 ms 10 ms wsip-184-178-205-96.ph.ph.cox.net [184.178.205.96] 4 12 ms 15 ms 11 ms 70.169.73.45 5 10 ms 10 ms 11 ms mcdldsrj01-ae2.0.rd.ph.cox.net [70.169.76.225] 6 * * * Request timed out. 7 22 ms 25 ms 25 ms te0-7-0-21.ccr22.lax05.atlas.cogentco.com [154.54.13.129] 8 153 ms 155 ms 152 ms att.lax05.atlas.cogentco.com [154.54.11.10] 9 164 ms 167 ms * cr2.la2ca.ip.att.net [12.122.84.218] 10 162 ms 170 ms 163 ms gar20.la2ca.ip.att.net [12.122.128.181] 11 167 ms 165 ms 165 ms 12-122-254-238.attens.net [12.122.254.238] 12 159 ms 161 ms 160 ms mdf001c7613r0003-gig-12-1.lax1.attens.net [12.12 9.193.254] 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out.
Trace complete.
At this point I am considering using Centurylink if the problem continues. While their service is slower than what is offered by Cox, they are far more consistent and do not have the problems related to routing through terrible servers like Cogentco's. |
|
 NormanSPremium,MVM join:2001-02-14 San Jose, CA kudos:9 Reviews:
·SONIC.NET
·Pacific Bell - SBC
| reply to Shina said by Shina :Note that even in the best circumstances, the Cognet route would have an extra hop, resulting in unneccessary latency. Because Internet routers have near wirespeed across the ports, hop count is irrelevant; there is no appreciable increase in latency.
I also find it interesting that the Cox server 68.1.1.19 shows consistent 40-50 % packet loss with PingPlotter, regardless of my actual connection performance. I have never gotten less than 40 % packet loss at this address at any time of day or on any connection. This particular hop does not appear to affect my connection quality at all, but as I have never recorded any instance where there was not packet loss at this address, I have little basis for comparison. Maybe interesting, but of little analytic significance. Intermediary routers frequently ignore response to diagnostic packets.
Of most interest, IMHO, is that Cogentco appears to have capacity issues on the AT&T interface. As if Cogentco can't handle all of the traffic going to the AT&T transit network. I am not sufficiently savvy to know why the routing was switched from XO to Cogentco. Might be economics. -- Norman ~Oh Lord, why have you come ~To Konnyu, with the Lion and the Drum |
|
 NormanSPremium,MVM join:2001-02-14 San Jose, CA kudos:9 Reviews:
·SONIC.NET
·Pacific Bell - SBC
| reply to Cifer said by Cifer:I've been having the same issue as well. I'm connecting to the LA datacenter for WoW, and the ping is still terrible. I have the same observation for you as I made to Shina: There appears to be congestion at the Cogentco-AT&T exchange.
I don't see your destination host name listed, so I can't look from my location. My ISP routes through Level 3 to the AT&T transit network, and I find it odd that my latency jumps from ~30 ms to ~80 ms going from Level 3 transit to AT&T transit in San Francisco, California. I've seen other WoW players on other ISPs with latency issues over the years; usually seems as if Blizzard Entertainment doesn't buy enough capacity from AT&T to handle their load. -- Norman ~Oh Lord, why have you come ~To Konnyu, with the Lion and the Drum |
|
 | reply to Hazmat The issue is not cox or cox's peering. The problem is the server provider is using ATT. ATT appears to have capacity issues.. I tracerouted to 63.240.161.189 from he.net in LA..
1 1 ms 1 ms 1 ms 10gigabitethernet1-1.core1.pao1.he.net (184.105.213.66) 2 1 ms 1 ms 13 ms sjo-bb1-link.telia.net (213.248.86.53) 3 5 ms 3 ms 3 ms 192.205.33.49 4 67 ms 56 ms 64 ms cr1.sffca.ip.att.net (12.122.200.10) 5 58 ms 66 ms 56 ms cr1.cgcil.ip.att.net (12.122.4.122) 6 81 ms 63 ms 132 ms gar2.clboh.ip.att.net (12.122.133.197) 7 66 ms 54 ms 57 ms 12.122.251.50 8 66 ms 53 ms 53 ms 63.240.130.214 9 * * * ? 10 * * * ?
Even if cox peered directly with ATT you would still see these problems. Go ahead and go to centurylink, you will find their support could careless or say they care but have no power to change anything. With COX at least they have some employees that attempt to address the issues of customers on this forum.. |
|
 | reply to Shina Having the same issue with Cox Comm and WoW in the Benson area (45 miles east of Tucson). This only happens with WoW, none of my other games, programs, computers are having this issue. I normally connect to the Rampage battle group, but for grins I connected to Stormstrike and while the latency was manageable it was definitely higher than it was two weeks ago. |
|
 CCTVTechPremium join:2003-04-23 Phoenix, AZ | reply to Shina Same problem, I am seeing 24% packet loss at my Glendale office. |
|
 odogCable Centric Vendor BiasedPremium,VIP join:2001-08-05 Atlanta, GA kudos:9 Reviews:
·Comcast
| reply to Shina
FYI, this isn't a Cox issue per se. This is an internet issue that happens periodically.
If you look at the good trace it goes like this.
COX.phx....COX.la....XO.la....ATT.la....ATT.saltlake....ATT.denver....ATT.chi....Bliz
Bad trace
COX.phx....COX.la....Cogent.la....Cogent.iah?...Cogent.dallas....Cogent.mci?....Cogent.ord?....ATT.chi
This issue is happening as a peering issue between cogent and AT&T. The input interface going from cogent to AT&T is overloaded. This is probably because cogent and AT&T only have an agreement to allow so much peering between them. Cogent isn't known as "good" peering partner since their peering relationships are generally so lopsided. They get into peering spats with other tier1 providers about once a year due to this.(Level 3 cut them off last year, sprint cut them off before that etc.)
BGP is very complicated protocol, with MANY different variables. But first and foremost is uses something called AS-PATH to determine route selection.(this is a gross simplification mind you) since these routes have the same amount of AS-PATH something else is at work.
I'm going to ping some of the more ninja level routing/bgp guys and see if they can do anything. Please be aware this is no guarantee anything will change or improve by our hand. |
|