dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
11975
Nicko1us
join:2012-12-23
Washington, DC

Nicko1us

Member

Evening FIOS Slowdown!

This evening, as with the last several evenings, my FIOS internet connection has slowed to nearly a complete stop.

I think the problem, as the info below illustrates, is probably with Verizon FIOS, and not with my setup.

Below are my FIOS upload and download speeds measured over the course of the day, using the "Verizon Broadband Speed Test."

As you can see, things are fine until sometime after 7 pm, when things slow to a crawl. This has been the pattern over about a week:

Time, Dec. 17, 2012 Download Speed mbps Upload Speed mbps
7:00:00 AM 15.6 6.0
8:00:00 AM 15.6 6.0
9:00:00 AM 15.6 6.0
11:00:00 AM 15.6 6.0
1:00:00 PM 15.6 6.0
3:00:00 PM 15.6 6.0
5:00:00 PM 15.6 6.0
6:00:00 PM 15.6 6.0
7:00:00 PM 15.6 6.0
8:00:00 PM 3.2 6.0
8:10:00 PM 1.1 5.9
8:20:00 PM 0.3 5.3
8:35:00 PM 0.02 0.3

I have run the Verizon “Optimizer” program.

My computer is directly connected to my router via Ethernet, although I get the identical results using my wireless connection.

I have rebooted both the router and the external box where the FIOS cable comes into our house.

I regularly use “Microsoft Security Essentials” to protect and detect against computer viruses.

The program "tracert" run at 7 am when things were fine (FIOS speeds were as advertised) gave the following good results:

C:\Documents and Settings\Mark Brown>tracert verizon.net
Tracing route to verizon.net [206.46.232.39] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 3 ms 4 ms 3 ms L100.WASHDC-VFTTP-138.verizon-gni.net [74.96.205 .1]
3 5 ms 7 ms 8 ms G1-5-1-6.WASHDC-LCR-21.verizon-gni.net [130.81.2 16.106]
4 6 ms 6 ms 5 ms ae0-0.RES-BB-RTR1.verizon-gni.net [130.81.209.19 8]
5 45 ms 46 ms 44 ms so-4-0-0-0.DFW01-BB-RTR1.verizon-gni.net [130.81 .17.41]
6 45 ms 45 ms 46 ms so-1-0-0-0.DFW03-CORE-RTR1.verizon-gni.net [130. 81.20.157]
7 46 ms 45 ms 45 ms po1.ctn-border1.vzlink.com [206.46.225.85]
8 45 ms 45 ms 46 ms 206.46.225.210
9 46 ms 45 ms 45 ms 206.46.228.129
10 45 ms 46 ms 45 ms 206.46.232.39
Trace complete.

Running tracert again at 8:10 pm when FIOS had slowed to a crawl gave the following:

Tracing route to verizon.net [206.46.232.39] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 165 ms 14 ms 28 ms L100.WASHDC-VFTTP-138.verizon-gni.net [74.96.205 .1]
3 * 33 ms 33 ms G1-5-1-6.WASHDC-LCR-21.verizon-gni.net [130.81.2 16.106]
4 24 ms 30 ms 53 ms ae0-0.RES-BB-RTR1.verizon-gni.net [130.81.209.19 8]
5 59 ms 79 ms 62 ms so-4-0-0-0.DFW01-BB-RTR1.verizon-gni.net [130.81 .17.41]
6 57 ms 62 ms 116 ms so-1-0-0-0.DFW03-CORE-RTR1.verizon-gni.net [130. 81.20.157]
7 81 ms 63 ms 74 ms po1.ctn-border1.vzlink.com [206.46.225.85]
8 53 ms 59 ms 57 ms 206.46.225.210
9 69 ms 56 ms 76 ms 206.46.228.129
10 61 ms 55 ms 61 ms 206.46.232.39

Trace complete.

Here is additional info, from 8 am when things were running fine, about my setup from your program Verizon Broadband Speed Test:

Broadband Speed Test (12-17-12 8:00 am)

Analysis information:

Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
SendBufferSize set to [261360]
running 10s outbound test (client to server) . . . . . 6.02Mb/s
running 10s inbound test (server to client) . . . . . . 15.60Mb/s

------ Client System Details ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.6.0_26

------ Web100 Detailed Analysis ------
Client Receive Window detected at 263856 bytes.
10 Mbps Ethernet link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 132.48 msec; the Packet size = 1452 Bytes; and
No packet loss was observed.
This connection is receiver limited 33.62% of the time.
This connection is sender limited 65.63% of the time.
This connection is network limited 0.74% of the time.

Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: ON
Information: Network Middlebox is modifying MSS variable
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
Server says [108.XX.XX.XX] but Client says [192.XX.XX.XX]

A phone call to Verizon technical support was not that useful – the technician I spoke to agreed that this info is consistent with simple internet congestion (too many people trying to watch TV via FIOS internet!)

He suggested it could be hackers getting into my wireless router (but he agreed that shouldn’t affect the Verizon speed test, which only measures the connection speed between my hard-wired router and Verizon internet). But, logging into my router, I see no indication of anyone hooking up to my wireless network, which is protected anyway with WPA-2 encryption.

The bottom line is that it appears to me that my Verizon FIOS service slows in the evenings because of too much traffic, presumably other people also using it for video services.

That is, I'm paying for FIOS but not even getting DSL!

Any suggestions would be appreciated.


guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

Its clear there is an issue at the second hop to the gateway router ( 74.96.205 .1) ping jumps to 165ms when it should be 3 - 4 ms

Most likely the PON card at the Central Office is the cause, but I would suggest you call for a replacement router first

Its not normal and FIOS doesn't have these types of slow downs, so something at the CO is likely the cause. So your going to have to call in again and try and explain you have trace routes showing its the CO

slow1234
@verizon.net

slow1234 to Nicko1us

Anon

to Nicko1us
me too
RolteC
The Need for Speed
join:2001-05-20
New York, NY

RolteC to Nicko1us

Member

to Nicko1us
Its doing it for me first not. First hop is at an average of 38ms.

Download speed to the Verizon fios300 server is 4-6Mbit and uploads are between 12 and 48.

What in the world is going on?!!!
patt2k
join:2009-01-16

1 edit

patt2k to Nicko1us

Member

to Nicko1us
at first I did not believie this could happen but it is !

how do I make them fix this issue? call and request credit or what?

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Patryk>tracert google.com

Tracing route to google.com [74.125.228.101]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms READYSHARE [192.168.1.1]
2 1 ms 1 ms 1 ms 192.168.11.1
3 5 ms 7 ms 7 ms L100.NYCMNY-VFTTP-XXXXXXXXXX.verizon-gni.net [70.104.XX
5.1]
4 21 ms 20 ms 15 ms G0-5-4-5.NYCMNY-LCR-22.verizon-gni.net [130.81.4
8.190]
5 29 ms 32 ms 9 ms so-6-1-0-0.NY5030-BB-RTR2.verizon-gni.net [130.8
1.151.224]
6 33 ms 281 ms * 0.xe-5-1-1.XT2.NYC9.ALTER.NET [152.63.4.69]
7 90 ms 42 ms 32 ms 0.xe-2-1-6.XT2.NYC4.ALTER.NET [152.63.3.46]
8 15 ms 17 ms 11 ms TenGigE0-7-0-0.GW8.NYC4.ALTER.NET [152.63.22.45]

9 * * * Request timed out.
10 18 ms 17 ms 24 ms 72.14.238.232
11 14 ms 20 ms 44 ms 209.85.251.88
12 64 ms 64 ms 65 ms 209.85.249.11
13 64 ms 60 ms 54 ms 72.14.236.153
14 63 ms 62 ms 65 ms 72.14.238.253
15 59 ms 62 ms 65 ms iad23s08-in-f5.1e100.net [74.125.228.101]

Trace complete.

Dareius
It's a beautiful day in the neighborhood
Premium Member
join:2002-11-12
Elmhurst, NY

Dareius to Nicko1us

Premium Member

to Nicko1us
Definitely a rough weekend reading the newspapers online. The pages are taking forever to download.

I would assume it is a known issue, but we may have to wait a bit due to the upcoming holiday and all. I do not expect anything to be fixed until after the New Year.

PhoenixDown
FIOS is Awesome
Premium Member
join:2003-06-08
Fresh Meadows, NY

PhoenixDown to Nicko1us

Premium Member

to Nicko1us
If you haven't done so, call in an open a ticket.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

said by PhoenixDown:

If you haven't done so, call in an open a ticket.

For what? to complain that some other company's networks or server is slow?
cvinatoru
join:2008-11-16
Middle Village, NY

cvinatoru

Member

It is VERIZON's fault.

I have same problem with slow speeds at night for the past 6 weeks.

Even VOD doesn't work.

It seems that VERIZON oversubscribed FIOS.

May be it is time to look at other alternatives in the area. TWC will be happy for my bussiness.

louist
And So It Goes
Premium Member
join:2001-12-01
Oakland Gardens, NY

louist to RolteC

Premium Member

to RolteC
Roltec and Dareius:

You are geographically close to me. I didn't test around 12/23 but I did before and after that date. Did not see deteriorated speeds here. Have your problems improved since?

Here are my results over a few months for comparison. Seem pretty even over time.

»/testh ··· 06/2b784
serge87
join:2009-11-29
New York

1 recommendation

serge87 to cvinatoru

Member

to cvinatoru
said by cvinatoru:

It seems that VERIZON oversubscribed FIOS.

Doubtful. A small vocal minority are seeing this issue which could all be down to a bad PON card at the CO while many others(myself included) have full speed at all times of the day and do not see this problem.
said by cvinatoru:

May be it is time to look at other alternatives in the area. TWC will be happy for my bussiness.

Everyone knows cable ISPs are known for not oversubscribing. Good luck.
RolteC
The Need for Speed
join:2001-05-20
New York, NY

RolteC to louist

Member

to louist
louist See Profile

Last night it was happening again. Not as bad though because at least VOD worked without a problem.

Speeds were random around 8-20Mbit/sec (to any server/website).

But while downloading, especially if VOD was going on, or more than one person was downloading a file, which also happened all throughout last night, my first hop went to 150-200ms. This was while my downloads in total where hovering at around 1/3 to 2/3 max speed according to SNMP polling on my switch.

I do not understand what is wrong or why the connection is being throttled back or so it seems.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

said by RolteC:

louist See Profile
I do not understand what is wrong or why the connection is being throttled back or so it seems.

VOD can use upwards of 20mbs, Verizon has "fluff" of ~10mbs, so getting 2/3rd of your line speed with VOD is normal and with one downloading un-trottled and VOD, your using all available bandwidth and the will effect ping testing.
RolteC
The Need for Speed
join:2001-05-20
New York, NY

RolteC

Member

Maybe I exaggerated. But I will also state that without the VOD going, it was still at max 2/3 bandwidth with multiple streams, or sub 30Mbit on a single stream with my pings to the first hop going to 150-200ms

How can you explain that one? Then after 12:30am it somehow goes to normal again? I can do 154Mbit/sec and my ping goes to maybe 20ms to the first hop which is normal while saturating the line as you are explaining. I know how much bandwidth this stuff uses, I see it going through the network.

2/3*150 is 100Mbit/sec. Which is fine, that was around what I saw on my WAN port via SNMP when having more then one desktop/laptop streaming last night. Even with the single VOD going on and WITHOUT IT (which does roughly 14Mbit/sec average), the results were the same.

One thing I failed to mention. Menu options on the box when trying to navigate to each section, just to find a movie took 3-7 seconds and sometimes multiple touches of the remote, that's how lagged it was even before trying to do the VOD.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

First VOD,

Its the holiday, I suspect many are sitting home and using VOD and causing peak demands on the servers. Its not surprising that this is slowing down the response of the VOD system.

Next, your pings. Pings are low priority traffic and for the most part, when routers ( your edge router in this case ) is busy dealing with traffic, its low priority to respond as its only done as a back ground task.

Now when your streaming and downloading, you have all sorts of things going on in the background with TCP, your going to be rate limited by the acknowledgement rate from the servers, unless its UDP traffic, which downloads don't use as its not guaranteed delivery protocol. A slow download server can and do intentionally throttle the ACK's to manage the incoming request, bottom line is the content server is the decision maker on the rate you get data.

Christmas break, everyone is downloading, streaming, activating ect and most places your getting the content from is overloaded, to an end user, this may seem like its Verizon, but its not, its the content providers being maxed out.

Your provisioned line and while its within the Verzon network will function at your published speed, but its almost certain, where you getting content from is some where else.
RolteC
The Need for Speed
join:2001-05-20
New York, NY

1 edit

RolteC

Member

Click for full size
Example Night - no VOD
Click for full size
Example Night - no VOD
I do not need to be lectured about how networks work. Being certified and also being confirmed by Verizon that there are issues is enough. I am only sharing them here.

The issue has been happening since the beginning of November, but randomly. It has nothing to do with the Level3 ping issue everyone is seeing. It has to do with some nights of the week, apparently specifically for me/some of my neighbors, everything goes down the drain in terms of bandwidth, ping, VOD usability, and web browsing. Forget my siblings playing online games - they just make things worse coming at me asking why immediately.

The other week (happened twice before but as an example). I was on the phone with Verizon as it was happening. No downloads at all, they saw the same packet loss and 20-80ms ping times to the first hop. Its being investigated. Always specific times of the night and not every single night. My thoughts are over subscription on my PON card. Still waiting on Verizon to come back to me with details. Its during those nights that VOD sometimes doesnt work, its during those nights that online gaming for others doesn't vibe, and forget downloading, it doesnt go anywhere and only increases the ping. This specific example while on the phone with Verizon, I downloaded at 12Mbit/sec and my pings went to 300-450ms. Even the tech on the phone didn't know what to say. So far that has been the worst of it which is why I get Verizon involved. Last night was nothing compared to those few nights though.

Edit:
I added the graphs above of one of the worst nights. Lasted hours, with ping getting worse - meaning everything getting worse. So I tried downloading, only went around 5mbit/sec. I tried downloading more files, went to maybe 30. But then I tried having more then one PC download and you can see the max it was doing. Look at what it did to my ping.

Now, once the congestion ended, look towards the end for the spike of traffic. Notice, NO spike in ping - this is suppose to be normal, and it is after 1am till about 5pm everyday. But that's it.

Dareius
It's a beautiful day in the neighborhood
Premium Member
join:2002-11-12
Elmhurst, NY

Dareius to louist

Premium Member

to louist
said by louist:

Roltec and Dareius:

You are geographically close to me. I didn't test around 12/23 but I did before and after that date. Did not see deteriorated speeds here. Have your problems improved since?

Here are my results over a few months for comparison. Seem pretty even over time.

»/testh ··· 06/2b784

I checked my connection overnight and it seems the issues over the holiday weekend have been fixed. My pages load just fine on the New York Daily News and New York Times website.

If the problem reoccurs, I will post some tracer routes and see where the ping/packet loss is coming from. I would like to say that the problem was not on the Verizon network because I tested my speeds every night on different speed test sites and was receiving all or more of what I paid for. I am guessing the issue may have been on a backbone server.
patt2k
join:2009-01-16

patt2k to Nicko1us

Member

to Nicko1us
Wow today is horrible just horrible...

I opened a ticket on Verizon's Direct Forum I expect credit for this will be calling verizon in few days as well.

It is always at or after 7 pm :O




I know when speeds are fine they reach 83-84 mbps

You guys claim it's other content providers no it's everywhere... I tried every speedtest possible those are just 2 of them

RyanB186
join:2001-05-05
New York, NY

RyanB186

Member

Same problem for me. I'm on the 75 mbps fios package and I'm getting 15 - 20 mbps right now. I'm in Manhattan.
patt2k
join:2009-01-16

patt2k

Member

said by RyanB186:

Same problem for me. I'm on the 75 mbps fios package and I'm getting 15 - 20 mbps right now. I'm in Manhattan.

I am in Queens I wonder what that means it cant be some broken card in the CO.
cvinatoru
join:2008-11-16
Middle Village, NY

cvinatoru

Member

I am in Queens too (Middle Village) and
this is what i get right now. On a 25/25 connection.

Your download
speed is

0.377 Mbps
Your upload
speed is

12.524 Mbps

On Verizon own speed test.
patt2k
join:2009-01-16

patt2k to Nicko1us

Member

to Nicko1us
Cvinatoru I am in RDG Queens dont use Verizons speedtest site, it is really no good.

The best site for me to test is speedtest.comcast.net Plainfield,NJ server.
McBane
join:2008-08-22
Wylie, TX

McBane

Member

Wonder if we're seeing that over-congestion of BPON we predicted with the new Quantum speeds.
serge87
join:2009-11-29
New York

serge87

Member

said by McBane:

Wonder if we're seeing that over-congestion of BPON we predicted with the new Quantum speeds.

I wish some of the posters could post if they are on BPON or GPON and if their issues started before or after Hurricane Sandy. It would be much easier to see a pattern. Many NYC/NJ posters are seeing this problem.
RolteC
The Need for Speed
join:2001-05-20
New York, NY

RolteC to Nicko1us

Member

to Nicko1us
I am on GPON.

I started seeing the Alter.net / Level 3 issue everyone is seeing after Sandy. Maybe not everyone noticed it due to their specific usage patterns / Sandy being priority on TV and people without power everywhere made it less stressful on the networks for a few days most likely.

As for my more direct problem that happens 1-2 a week when my direct first hop goes up in ping and VOD doesn't work, that I suspect is a PON card problem at the CO.
patt2k
join:2009-01-16

patt2k to Nicko1us

Member

to Nicko1us
after sandy of course and I am not sure I have the residental fiber box.
guyjr
join:1999-08-30
USA

guyjr to Nicko1us

Member

to Nicko1us
I had this exact issue at this time of year two years ago. I lucked out and managed to get a Verizon rep who opened a chat session with a level 3 tech while I was on the phone. They were able to confirm my bandwidth issues at that exact time of night, and opened a ticket. A couple weeks later the problem was fixed, tho I never did get any confirmation from them about what the cause was.

I don't like spreading FUD, but my personal belief is one that I saw posted a while ago here, and that is it is possible that, even tho Verizon aren't necessarily "over" subscribing, that the sheer volume of people hitting the FiOS network over major holidays such as this past one, does cause any weakness in their network to show. I believe that the BPON cards overheat under such load, and possibly cause damage which needs replacing eventually.
brianiscool
join:2000-08-16
Tampa, FL

brianiscool to Nicko1us

Member

to Nicko1us
Must be us hungry bandwidth user on the 155Mbps - 300Mbps tier.

Tomel
join:2007-11-07
Midlothian, VA

Tomel to Nicko1us

Member

to Nicko1us
I just posted on Verizon Direct. Am getting very slow download speeds this afternoon. Provisioned for 15/5, but getting 2-3 Mbs download. Sometimes below 1 Mbs. Have almost always gotten the 15/5 or better.
rmdavis78
join:2002-03-04
Richmond, VA

rmdavis78

Member

I'm on the phone with VZ support now, no outtages reported in Midlothian. On the FIOS community forums there is a report of slow speeds in Leesburg from last night. The support is shipping me out a new router so maybe I'll get a Rev I, but so far ONT recycles/router recycles/etc. nothing is getting me above 4-5 Mbps download. Upload seems fine, very consistent 35-40 Mbps.