Verizon FiOS → Re: [northeast] Line Monitor Latency Spikes

Yeah, I've read that. It sure is skewing the monitors.

Yea its making them not very reliable because we don't know what our line is really doing.

Indeed, it seems like residential lines are beginning to perform more reliably than quite a few of the big guys. Both because of residential lines improving, and the major routers and backbones becoming increasingly overloaded.

Looks like routing changed on the line monitors - no spike last night! YAY!

WOW, you're right! I wasn't online last night and didn't notice till I read your post. Good deal Lets hope it stays fixed.

That trouble router is back in the Line monitor today - and i'm once again seeing the night time increase.

I'm also seeing a bunch of packet loss when tracing www.dslreports.com

Only at the last hop.

Yep - checked out the forum line monitors - we're all getting hit again. Look at my East coast times increase!!! (Usually at 10ms)

Parsippany, NJ, USA
2008-03-18 19:43:18 52ms (0% loss)
2008-03-18 19:33:21 51ms (0% loss)
2008-03-18 19:23:13 37ms (0% loss)
2008-03-18 19:13:18 38ms (0% loss)
2008-03-18 19:03:20 23ms (0% loss)
2008-03-18 18:53:20 18ms (0% loss)

Yeah I'm seeing between 30 and 40 % packet loss to www.dslreports.com:

 Host                                                    Loss%   Snt   Last   Avg  Best  W  rst StDev
 1. L300.VFTTP-16.PHLAPA.verizon-gni.net                  0.0%    40    5.2   5.4   2.7  6  2.0   9.2
 2. P6-2.LCR-04.PHLAPA.verizon-gni.net                    0.0%    40    7.2   6.4   5.2     7.6   0.7
 3. 130.81.29.214                                         0.0%    40    6.7   6.4   5.5     7.5   0.6
 4. 0.so-6-0-0.XL2.PHL6.ALTER.NET                         0.0%    40    7.3   7.9   5.3  5  3.6   7.6
 5. 0.so-7-0-0.XL4.NYC4.ALTER.NET                         0.0%    40   39.2  12.9  10.1  4  8.0   7.2
 6. POS7-0.GW12.NYC4.ALTER.NET                            0.0%    40   11.2  11.7  10.3  1  2.8   0.7
 7. internap-gw.customer.alter.net                        0.0%    40   10.7  11.8  10.3  1  4.6   1.0
 8. border28.po1-bbnet1.nyc.pnap.net                      0.0%    40   11.3  25.4  10.7 20  8.5  47.6
 9. nac-2.border28.nyc.pnap.net                           0.0%    40   10.7  11.2  10.2  1  6.6   1.0
10. vlan31.esd1.tl9.nac.net                               0.0%    40   11.3  21.5  10.3 19  8.4  34.8
11. 0.e2-2.tbr1.tl9.nac.net                               0.0%    40   10.8  11.7  10.3  1  8.2   1.4
12. 0.e1-4.tbr1.mmu.nac.net                               0.0%    40   10.2  11.4  10.2  1  2.7   0.7
13. 0.e-1-1.tbr1.oct.nac.net                              0.0%    40   14.7  13.9  12.8  1  5.0   0.6
14. 0.ge-1-2-0.gbr1.oct.nac.net                           0.0%    40   12.8  26.9  12.6 24  6.1  43.7
15. www.dslreports.com                                   44.7%    39   14.2  14.9  12.8  2  6.1   3.4

It's wierd, though, the latency is very low and relatively consistent, it just seems to be dropping packets. Perhaps it's just a rate limiting firewall in play here.

*edit* And I tested from a friend's Colo in NY and it's dropping packets from there, too:

 Host                                                    Loss%   Snt   Last   Avg  Best  W rst StDev
 1. 63.247.77.34                                          0.0%    40    0.2   0.3   0.2    1.1   0.2
 2. 65.98.99.41                                           0.0%    40    0.4   1.0   0.3  1 9.1   2.9
 3. 208.116.63.249                                        0.0%    40    0.4   0.8   0.3  1 0.9   1.7
 4. ge3-24.ar1.ewr1.us.nlayer.net                         0.0%    40    0.7   7.8   0.6 17 6.2  28.4
 5. xe-2-0-0.cr1.nyc3.us.nlayer.net                       0.0%    40    0.9   0.9   0.8    1.5   0.1
 6. xe-2-0-0.nyc22.ip.tiscali.net                         0.0%    40    1.1   1.7   1.1  1 4.6   2.3
 7. netaccess-gw.ip.tiscali.net                           0.0%    40    1.2   4.9   1.1 10 3.9  17.3
    netaccess-gw.ip.tiscali.net
 8. 0.e1-4.tbr2.mmu.nac.net                               0.0%    40    2.2   2.3   2.1    4.9   0.6
    0.e1-4.tbr1.mmu.nac.net
 9. 0.e-1-1.tbr1.oct.nac.net                              0.0%    40    2.7   3.5   2.1  2 7.7   5.5
    0.e-1-1.tbr2.oct.nac.net
10. 0.e-1-3.tbr1.oct.nac.net                              0.0%    40    2.4   7.6   2.2  9 3.2  19.6
    0.ge-1-2-0.gbr1.oct.nac.net
11. www.dslreports.com                                   27.5%    40    2.4   2.6   2.2    4.2   0.4
    0.ge-1-2-0.gbr1.oct.nac.net

Look at my tracert which is really bizarre at the end. hops 16-18 and maybe 19 before I stopped are the same.

C:\>tracert www.dslreports.com
 
Tracing route to dslreports.com [209.123.109.175]
over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     3 ms     1 ms     1 ms  10.5.35.16
  3     1 ms     1 ms     1 ms  P3-1.LCR-01.NWRKNJ.verizon-gni.net [130.81.38.194]
  4     2 ms     2 ms     2 ms  130.81.29.188
  5     2 ms     2 ms     2 ms  0.so-5-0-0.XL1.EWR6.ALTER.NET [152.63.19.177]
  6     3 ms     3 ms     3 ms  0.so-6-1-0.XL3.NYC4.ALTER.NET [152.63.16.194]
  7     3 ms     3 ms     3 ms  POS6-0.GW12.NYC4.ALTER.NET [152.63.22.77]
  8     5 ms     5 ms     5 ms  internap-gw.customer.alter.net [157.130.50.86]
  9    27 ms     4 ms     3 ms  border28.po1-bbnet1.nyc.pnap.net [209.191.128.97]
 10     3 ms    21 ms     3 ms  nac-2.border28.nyc.pnap.net [216.223.55.186]
 11    10 ms     3 ms     3 ms  vlan31.esd1.tl9.nac.net [209.123.11.69]
 12     3 ms     3 ms     3 ms  0.e2-2.tbr1.tl9.nac.net [209.123.10.90]
 13     4 ms     4 ms     4 ms  0.e1-4.tbr1.mmu.nac.net [209.123.10.101]
 14     5 ms     4 ms     4 ms  0.e-1-1.tbr1.oct.nac.net [209.123.10.17]
 15    58 ms   134 ms     5 ms  0.ge-1-2-0.gbr1.oct.nac.net [209.123.10.58]
 16     *        4 ms     *     www.dslreports.com [209.123.109.175]
 17     4 ms     *        *     www.dslreports.com [209.123.109.175]
 18     4 ms     *        *     www.dslreports.com [209.123.109.175]
 19     *        *        *     Request timed out.
 20  ^C
 

Had a friend on OOL do a tracert and he sees packet loss on the last hop too.

Weird, that almost looks like something is doing nat or acting as a transparent bridge and is having some sort of cyclic routing problem. I'll try running a tracert from windows and see if I see the same.

I re-ran it again and this time got the following. Not the line of time outs this time, then just 1 DSLR but high latency.

. snip .
 12     4 ms     3 ms     3 ms  0.e2-2.tbr1.tl9.nac.net [209.123.10.90]
 13     4 ms     4 ms     5 ms  0.e1-4.tbr1.mmu.nac.net [209.123.10.101]
 14    28 ms    30 ms    30 ms  0.e-1-1.tbr1.oct.nac.net [209.123.10.17]
 15    34 ms    30 ms    31 ms  0.ge-1-2-0.gbr1.oct.nac.net [209.123.10.58]
 16     *        *        *     Request timed out.
 17     *        *      436 ms  www.dslreports.com [209.123.109.175]
 
Trace complete.
 

I ran a tcptraceroute (tcptraceroute -q 40 www.dslreports.com 80) on my FreeBSD box, and none of them dropped. I think it's just a firewall/bridge that's limiting ICMP traffic.