rich44 join:2003-01-24 Butte, MT |
to koolkid1563
Re: Massive Slowdowns?I am having the exact same problems at the exact same hop, very frustrating. |
|
|
to bigburd
Seemed to stop at roughly 12:20am MST this morning. The tech was saying that sounds like one of their core routers. Seems like it might be getting overloaded during peak usage time |
|
1 edit |
to bigburd
I hope they plan on increasing capacity at that tier. This is getting a little old. |
|
|
Last Result: Download Speed: 1697 kbps (212.1 KB/sec transfer rate) Upload Speed: 297 kbps (37.1 KB/sec transfer rate)
traceroute to www.l.google.com (74.125.155.105), 30 hops max, 38 byte packets 1 host-10-252-148-1.but-mt.client.bresnan.net (10.252.148.1) 8.629 ms 9.482 ms 11.280 ms 2 butmtfh2hb6-GE-0-0-0-U120.int.bresnan.net (69.144.94.122) 11.268 ms 11.870 ms 8.040 ms 3 hlnmt002hb7-SO-1-2-0-U10.int.bresnan.net (69.144.94.161) 14.875 ms 19.182 ms 21.909 ms 4 mslmtfh1hb7-GE-0-0-0-U432.int.bresnan.net (69.144.26.101) 19.495 ms 21.418 ms 17.403 ms 5 host-72-175-110-3.bln-mt.client.bresnan.net (72.175.110.3) 27.033 ms 24.541 ms 29.149 ms 6 host-72-175-110-10.bln-mt.client.bresnan.net (72.175.110.10) 86.303 ms 86.318 ms 99.119 ms 7 66.62.136.57 (66.62.136.57) 114.763 ms 106.305 ms 102.478 ms 8 six.sea01.google.com (206.81.80.17) 111.168 ms 115.358 ms 116.826 ms 9 209.85.249.34 (209.85.249.34) 220.997 ms 209.85.249.32 (209.85.249.32) 119.319 ms 128.705 ms 10 72.14.239.12 (72.14.239.12) 130.122 ms 209.85.250.126 (209.85.250.126) 127.412 ms 132.221 ms 11 216.239.48.32 (216.239.48.32) 129.776 ms 209.85.250.146 (209.85.250.146) 123.194 ms 130.496 ms 12 216.239.48.139 (216.239.48.139) 132.268 ms 124.658 ms 64.233.174.131 (64.233.174.131) 130.479 ms 13 216.239.49.166 (216.239.49.166) 137.087 ms 216.239.49.246 (216.239.49.246) 140.038 ms 134.581 ms 14 px-in-f105.google.com (74.125.155.105) 133.922 ms 137.164 ms 134.054 ms
traceroute to www.l.google.com (74.125.155.106), 30 hops max, 38 byte packets 1 host-10-252-148-1.but-mt.client.bresnan.net (10.252.148.1) 15.494 ms 9.749 ms 18.352 ms 2 butmtfh2hb6-GE-0-0-1-U96.int.bresnan.net (69.144.94.98) 10.884 ms 13.651 ms 8.904 ms 3 hlnmt002hb7-SO-1-2-0-U10.int.bresnan.net (69.144.94.161) 15.271 ms 23.853 ms 15.981 ms 4 mslmtfh1hb7-GE-0-0-0-U432.int.bresnan.net (69.144.26.101) 17.731 ms 22.851 ms 16.533 ms 5 host-72-175-110-3.bln-mt.client.bresnan.net (72.175.110.3) 24.140 ms 21.085 ms 34.702 ms 6 host-72-175-110-10.bln-mt.client.bresnan.net (72.175.110.10) 85.746 ms 87.705 ms 86.586 ms 7 66.62.136.57 (66.62.136.57) 115.702 ms 112.547 ms 122.162 ms 8 six.sea01.google.com (206.81.80.17) 110.201 ms 109.879 ms 111.263 ms 9 209.85.249.32 (209.85.249.32) 110.544 ms 209.85.249.34 (209.85.249.34) 113.363 ms 209.85.249.32 (209.85.249.32) 123.072 ms 10 209.85.250.126 (209.85.250.126) 151.233 ms 72.14.239.12 (72.14.239.12) 121.152 ms 209.85.250.126 (209.85.250.126) 124.493 ms 11 209.85.250.146 (209.85.250.146) 126.705 ms 216.239.48.34 (216.239.48.34) 129.596 ms 209.85.250.146 (209.85.250.146) 130.682 ms 12 64.233.174.123 (64.233.174.123) 120.175 ms 134.142 ms 216.239.48.137 (216.239.48.137) 128.242 ms 13 209.85.254.150 (209.85.254.150) 129.950 ms 209.85.254.146 (209.85.254.146) 140.858 ms 126.393 ms 14 px-in-f106.google.com (74.125.155.106) 122.419 ms 117.150 ms 118.857 ms
Not as obvious tonight but you can still clearly see the latency issue in Billings. C'mon Bresnan get your game face on and fix this! |
|
rich44 join:2003-01-24 Butte, MT |
rich44
Member
2009-Sep-30 12:01 am
Its as bad tonight as ever.
Last time i saw this problem with bresnan it took atleast 2 months to go away. |
|
MikroTik CCR1036-8G-2S+ MikroTik hAP AC
4 edits |
to bigburd
|
|
rich44 join:2003-01-24 Butte, MT |
rich44
Member
2009-Sep-30 12:35 am
The problems seems to be at the very last hop before leaving bresnan's network. Thus their internal speed test checks out just fine. |
|
1 edit |
to bigburd
Jayyyzzuuuzz...problems again tonight...so much for gaming. This is a traceroute right off the firewall interface:
firewall (root) ~ $ traceroute www.yahoo.com traceroute to www-real.wa1.b.yahoo.com (209.131.36.158), 30 hops max, 38 byte pa ckets 1 * * * 2 host-69-144-26-145.static.bresnan.net (69.144.26.145) 8.783 ms 7.023 ms 6 .399 ms 3 mslmtfh1hb7-SO-0-3-0-U0.int.bresnan.net (69.144.26.69) 15.863 ms 12.253 ms 12.368 ms 4 host-72-175-110-3.bln-mt.client.bresnan.net (72.175.110.3) 14.642 ms 23.44 6 ms 13.245 ms MPLS Label=301696 CoS=3 TTL=1 S=0 5 host-72-175-110-10.bln-mt.client.bresnan.net (72.175.110.10) 195.998 ms 19 9.439 ms 200.716 ms 6 66.62.136.49 (66.62.136.49) 208.035 ms 242.227 ms 220.271 ms 7 six.yahoo.com (206.81.80.98) 233.320 ms 213.067 ms 217.159 ms 8 so-4-0-0.pat1.pao.yahoo.com (216.115.110.42) 241.833 ms 228.001 ms 224.77 1 ms 9 ae0-p140.msr1.sp1.yahoo.com (216.115.107.49) 219.112 ms ae1-p150.msr2.sp1.y ahoo.com (216.115.107.77) 221.164 ms ae0-p140.msr1.sp1.yahoo.com (216.115.107.4 9) 223.574 ms 10 te-9-1.bas-a2.sp1.yahoo.com (209.131.32.23) 53.160 ms te-8-1.bas-a2.sp1.yah oo.com (209.131.32.19) 50.847 ms te-8-1.bas-a1.sp1.yahoo.com (209.131.32.17) 5 3.036 ms |
|
MikroTik CCR1036-8G-2S+ MikroTik hAP AC
|
Its strange that I don't see the same spike on that address even minutes after you ran yours:
traceroute to sea.speakeasy.net (66.93.87.2), 30 hops max, 38 byte packets 1 10.252.148.1 (10.252.148.1) 6.543 ms 6.252 ms 7.783 ms 2 butmtfh2hb6-GE-0-0-1-U96.int.bresnan.net (69.144.94.98) 7.355 ms 6.903 ms 6.914 ms 3 hlnmt002hb7-SO-1-2-0-U10.int.bresnan.net (69.144.94.161) 11.860 ms 12.440 ms 12.886 ms 4 mslmtfh1hb6-GE-1-3-0-U432.int.bresnan.net (69.144.26.102) 14.892 ms 15.911 ms 14.932 ms 5 host-72-175-110-1.bln-mt.client.bresnan.net (72.175.110.1) 17.397 ms 16.799 ms 17.402 ms 6 host-72-175-110-10.bln-mt.client.bresnan.net (72.175.110.10) 19.372 ms 21.913 ms 21.389 ms 7 66.62.136.57 (66.62.136.57) 82.823 ms 91.604 ms 85.244 ms 8 ge-7-10.car2.Seattle1.Level3.net (4.53.146.97) 79.101 ms 86.366 ms 84.269 ms 9 SPEAKEASY-I.car4.Seattle1.Level3.net (209.247.91.170) 83.700 ms 82.678 ms 82.488 ms 10 dns.sea1.speakeasy.net (66.93.87.2) 86.114 ms 81.317 ms 83.298 ms |
|
1 edit |
to bigburd
Here is another one:
firewall (root) ~ $ traceroute www.yahoo.com traceroute to www-real.wa1.b.yahoo.com (209.131.36.158), 30 hops max, 38 byte packets 1 * * * 2 host-69-144-26-145.static.bresnan.net (69.144.26.145) 7.188 ms 8.796 ms 6.382 ms 3 mslmtfh1hb7-SO-0-3-0-U0.int.bresnan.net (69.144.26.69) 11.988 ms 13.276 ms 14.959 ms 4 host-72-175-110-3.bln-mt.client.bresnan.net (72.175.110.3) 12.882 ms 12.423 ms 12.804 ms MPLS Label=301696 CoS=0 TTL=1 S=1 5 host-72-175-110-10.bln-mt.client.bresnan.net (72.175.110.10) 104.217 ms 111.064 ms 116.216 ms 6 66.62.136.49 (66.62.136.49) 126.532 ms 126.092 ms 122.249 ms 7 six.yahoo.com (206.81.80.98) 163.162 ms 139.298 ms 132.854 ms 8 so-4-0-0.pat1.pao.yahoo.com (216.115.110.42) 147.991 ms 152.654 ms 156.123 ms 9 ae1-p150.msr2.sp1.yahoo.com (216.115.107.77) 153.142 ms ae0-p140.msr1.sp1.yahoo.com (216.115.107.49) 147.478 ms ae1-p150.msr2.sp1.yahoo.com (216.115.107.77) 214.084 ms 10 te-8-1.bas-a1.sp1.yahoo.com (209.131.32.17) 53.567 ms te-9-1.bas-a2.sp1.yahoo.com (209.131.32.23) 51.884 ms 51.036 ms 11 * * |
|
|
|
Since you ran your first one I had this running which is very strange:
--- 72.175.110.10 ping statistics --- 201 packets transmitted, 201 packets received, 0% packet loss round-trip min/avg/max = 15.053/19.938/34.119 ms |
|
|
to bigburd
I am willing to be that you are coming in on a different port than we are on the device. I am willing to be that the interface is overloaded. |
|
MikroTik CCR1036-8G-2S+ MikroTik hAP AC
|
Could be...Doesn't help the next hop any though. Wonder if a majority of the load is coming off of the port/int that you are on and its just killing the whole system?
--- 72.175.110.10 ping statistics --- 279 packets transmitted, 279 packets received, 0% packet loss round-trip min/avg/max = 13.933/18.382/57.080 ms
--- 66.62.136.49 ping statistics --- 277 packets transmitted, 276 packets received, 0% packet loss round-trip min/avg/max = 68.024/92.341/134.275 ms |
|
|
to bigburd
I am not sure, but I am almost sure that they have SNMP monitoring on these network devices. They should know by now if they have a hardware problem or a capacity issue. |
|
MikroTik CCR1036-8G-2S+ MikroTik hAP AC
1 edit |
Knowing is only part of the solution though...Doing or actually fixing is a whole different ballgame especially with Bresnan in my experience.
We shall see, but I bet you are right...or hope anyways. Any good network staff would have SNMP monitoring or some other flavor (proprietary?) of monitoring of load/downtime/etc. |
|
|
to bigburd
my question is why does all of Montana have to go through Billings? It's most likely an overload at that ip, since it looks like everyone one of you go through the 72.175.110.10 address. |
|
MikroTik CCR1036-8G-2S+ MikroTik hAP AC
|
Not just Montana. Don't know if Northern Wyoming goes through that particular IP but Bresnan in Powell went down through Cheyenne then through Billings. Wouldn't surprise me if most if not all of their traffic went through Billings which is just asking for trouble. |
|
1 edit |
to bigburd
Here we go again, pay special attention to the bottom graph as it is our problem router. |
|
etlund join:2006-02-02 Helena, MT |
etlund
Member
2009-Sep-30 10:26 pm
Hey alphainfinit, What are you using to plot those ping times? I like those graphs! Well, except for the fact that they're showing us all how sucky Bresnan has become! |
|
|
to bigburd
ftrace.exe -d -i 1 google.com Tracing route to 74.125.67.100 with TTL of 32: 1 0ms 192.168.1.1 2 * Request timed out. 3 15ms 69.144.26.161 4 16ms 69.144.26.181 5 18ms 72.175.110.1 6 453ms 72.175.110.10 7 466ms 66.62.136.109 8 465ms 206.81.80.17 9 462ms 209.85.249.34 10 503ms 216.239.43.81 11 526ms 72.14.239.90 12 527ms 209.85.254.247 13 535ms 209.85.255.190 14 523ms 74.125.67.100 |
|
|
Uncle_Barley to bigburd
Anon
2009-Sep-30 11:19 pm
to bigburd
I'm jumping on the bandwagon....I've been noticing "issues" for a good month now, but just assumed it was something Bresnan knew about and was working on. Now I'm just tired of it. Pay attention to #5. It's definitely Bresnan.
Tracing route to www.l.google.com [74.125.155.99] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 192.168.1.1 2 7 ms 7 ms 8 ms host-10-24-128-1.msl-mt.client.bresnan.net [10.2 4.128.1] 3 8 ms 8 ms 6 ms host-72-175-110-30.bln-mt.client.bresnan.net [72 .175.110.30] 4 7 ms 8 ms 10 ms host-72-175-110-9.bln-mt.client.bresnan.net [72. 175.110.9] 5 440 ms 432 ms 431 ms host-72-175-110-10.bln-mt.client.bresnan.net [72 .175.110.10] 6 430 ms 429 ms 429 ms 66.62.136.57 7 421 ms 430 ms 434 ms six.sea01.google.com [206.81.80.17] 8 434 ms 434 ms 427 ms 209.85.249.32 9 425 ms 410 ms 414 ms 72.14.239.12 10 412 ms 411 ms 411 ms 209.85.250.146 11 409 ms 407 ms 406 ms 216.239.48.167 12 422 ms 411 ms 410 ms 209.85.254.150 13 400 ms 408 ms 410 ms px-in-f99.google.com [74.125.155.99] |
|
1 edit |
bigburd
Member
2009-Sep-30 11:26 pm
They have something changed. All new route, all new addresses. The real kicker is if you notice now the latency spike happens on the outside interface of the Billings router. I think they are trying to shape the traffic and hide the fact it is happening.
Last Result: Download Speed: 492 kbps (61.5 KB/sec transfer rate) Upload Speed: 127 kbps (15.9 KB/sec transfer rate)
traceroute to www.l.google.com (74.125.155.147), 30 hops max, 38 byte packets 1 host-10-252-148-1.but-mt.client.bresnan.net (10.252.148.1) 8.156 ms 7.008 ms 8.601 ms 2 butmtfh2hb6-GE-0-0-0-U120.int.bresnan.net (69.144.94.122) 12.289 ms 9.335 ms 8.680 ms 3 hlnmt002hb7-SO-1-2-0-U10.int.bresnan.net (69.144.94.161) 15.346 ms 13.091 ms 13.007 ms 4 blnmtfh1hb7-GE-0-0-0-U433.int.bresnan.net (69.144.94.201) 21.747 ms 22.106 ms 19.299 ms 5 blnmtdc3hb6.int.bresnan.net (69.145.247.67) 24.490 ms 23.606 ms 23.139 ms 6 66.62.160.25 (66.62.160.25) 521.773 ms 530.703 ms 531.996 ms 7 den1-core-02.360.net (66.62.4.66) 66.397 ms 70.226 ms 64.396 ms 8 lax1-core-01.360.net (66.62.3.41) 222.280 ms 199.846 ms 145.584 ms 9 66.62.6.195 (66.62.6.195) 527.485 ms 525.002 ms 533.453 ms 10 core1-0-1-0.lax.net.google.com (198.32.146.46) 527.175 ms 542.535 ms 536.948 ms 11 216.239.43.14 (216.239.43.14) 535.939 ms 535.026 ms 540.926 ms 12 72.14.232.85 (72.14.232.85) 546.100 ms 540.125 ms 543.803 ms 13 209.85.250.144 (209.85.250.144) 542.236 ms 578.798 ms 553.387 ms 14 64.233.174.121 (64.233.174.121) 557.122 ms 564.761 ms 64.233.174.129 (64.233.174.129) 571.062 ms 15 209.85.254.146 (209.85.254.146) 545.986 ms 508.506 ms 523.225 ms 16 px-in-f147.google.com (74.125.155.147) 521.242 ms 515.189 ms 520.519 ms |
|
|
to bigburd
I don't think they are trying to hide anything, but this is absolutely HORRIFIC: |
|
MikroTik CCR1036-8G-2S+ MikroTik hAP AC
|
to bigburd
I am also seeing a lot of packet loss on the hop just before leaving their network which reflects on all hops past that, though on either side of the lost packet the latency never spikes at all. Before they renumbered their equipment's IP scheme I wasn't seeing the loss as bad as I am now... |
|
|
Kissler
Anon
2009-Oct-1 7:02 am
Just had the most depressing agent on the phone with me. She first stated that since it's late at night, and people are logging on to the internet, it's going to be slow. then back tracked and blamed my node, then backtracked and blamed the modem and the outage....WTF. |
|
Kissler |
to koolkid1563
Remember that outage, all these changes happened after it. I don't think they fixed the problem, just re-routed it to an already congested line. It could be that it's now operating at max during the night hours, which would explain the ping issues (all that traffic on one line) |
|
|
Uncle_Barley to bigburd
Anon
2009-Oct-1 10:57 am
to bigburd
I called in last night to at least get a ticket started. The guy I talked to admitted that there seemed to be a problem and that his engineer was "on it". I just provided him with the tracert and the ip address of the location that had the high latency and he said they would work on getting it resolved. He didn't try to dodge anything or attempt to entertain me with all the useless remedial 1st level support tactics. This may be because they were swamped with calls considering my wait time was around 30 min before I actually talked to the guy. All I can say is....we'll see. |
|
|
to bigburd
I will be calling tonight and asking for a credit on my bill. I am a triple play customer and these problems have basically made the internet unusable at night for the last week. I will post more graphs if the problem occurs again this evening. |
|
|
Last Result: Download Speed: 6750 kbps (843.8 KB/sec transfer rate) Upload Speed: 395 kbps (49.4 KB/sec transfer rate)
best speed I've had after 5pm in a long time. Everyone else experiencing this? |
|
bigburd |
to alphainfinit2
They gave me $50 credit SO FAR but I am definitely not done complaining. I've lost a TON of work and play time over this and $50 doesn't even put a scratch in the paint as far as I'm concerned. |
|