| [FiOS] Extremely High Latency in Tigard Oregon - Support is usel
Latency from my office to friends house |
I must say, the cutover from Verizon FiOS to Frontier FiOS went very poorly (some of that was Verizon not keeping on top of increasing bandwidth needs before the sale to Frontier, but most of it was Frontier not having enough resources to get a handle on things quickly).
That being said, once fully on the Frontier network things were better for a while, but I still have seen packet loss / high latency around oversubscription of aggregation devices back at the central offices. On more than one occasion these issues have gone on for days/weeks without being fixed.
Just this evening I discovered an extremely high latency situation that I presume is due to links from the TelLabs aggregation devices back at the CO to their edge/core Internet routers being down. This is causing a friend of mine's Internet connection to be at 160ms ping times to my office vs. the normal 16ms.
To be clear- This is a massive degradation of his connectivity to the Internet. While it is still technically "working", VoIP across this link, or any kind of gaming would be horrible right now. Not to mention web browsing being noticeably slow. In my world, this is absolutely unacceptable and should be treated as network-down which justifies after-hours attention.
To prove my point, take a look at these MTR outputs from my house, to my friends house (both on FiOS). Note that my Internet is doing OK at the moment, but my friends connection is having high latency to many (all?) destinations on the Internet. I am using the example of connectivity between our two houses being horrible since there is no way Frontier can blame this on anyone else's network (as is often the case on the Internet).
From my house to my friends house:
(last octet changed to xxx for privacy)
My traceroute [v0.75]
Vyatta01 (0.0.0.0) Fri Aug 10 21:15:55 2012
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. static-50-53-0-1.bvtn.or.frontiernet.net 0.0% 5337 4.4 3.4 2.7 163.5 5.4
2. 50.38.7.73 0.0% 5337 4.8 5.1 2.7 114.0 9.1
3. 50.38.7.22 0.0% 5337 2.7 3.6 2.5 10.6 0.8
4. 184.19.246.4 0.0% 5336 41.1 4.5 2.6 60.8 5.0
5. G15-0-1-120.PTLDOR-VFTTP-20.ncnetwork.net 0.0% 5336 236.4 181.5 85.6 428.0 35.0
6. static-50-53-64-xxx.bvtn.or.frontiernet.net 0.0% 5336 241.7 185.8 93.5 311.0 34.6
From my friends house back to mine:
(last octet changed to xxx for privacy)
My traceroute [v0.75]
Vyatta02 (0.0.0.0) Fri Aug 10 21:17:33 2012
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. static-50-53-64-1.bvtn.or.frontiernet.net 0.0% 8912 2.8 3.1 2.7 174.6 4.1
2. 184.19.244.36 0.0% 8912 245.0 163.5 66.0 293.5 38.9
3. xe--11-3-0---0.car01.bvtn.or.frontiernet.net 0.0% 8911 244.5 167.2 62.9 334.7 40.4
4. 50.38.7.70 0.0% 8911 261.4 167.0 65.3 350.3 39.3
5. static-50-53-7-xxx.bvtn.or.frontiernet.net 0.0% 8911 268.4 169.5 66.1 306.5 39.5
For those not used to reading Traceroute output (or MTR in this case), take a look at the "average" ping time columns. Note that on the Traceroute outbound from my friends house, the latency to the first hop looks great (3.1 ms). This proves that the issue is NOT with anything in my friends house, or between his house and the CO. The signal is making it there and back without issue.
Then you see at the second hop the average latency shoots up to 163.5 ms. This is utterly abysmal. That's like dial-up latency. Of course every hop past that also shows that latency. This indicates to me that there are some massively oversubscribed links between Frontier's first hop device and the second hop router.
Because Frontier has had issues in the past, I have monitoring tools setup at my office to ping my FiOS connection as well as all my local friends FiOS connections every 30 seconds and graph the loss amounts as well as latency. By looking back at these graphs, I can see this issue started rearing it's head at midnight last night, but went away as traffic dropped off later that night. It then came back this evening as traffic started to ramp back up again.
This looks to me like some bonded network links are offline which is massively overloading the remaining links and driving up the latency due to buffering.
So the real reason to post this is for me to complain in a public place about Frontier's ability to respond to an issue like this. Basically they are structurally incapable. Here is a log of my experience tonight trying to get a hold of a network engineer to troubleshoot this with them:
I called Tech Support and the good news is that I got a lady immediately. I must say though, the tone of voice I get from the folks that work there makes me think they really don't want to be talking to me. After giving them my phone numbers (neither of which are Frontier numbers since I only have Internet), they could not locate my account. I then gave them my SSN (which scares me) and they still could not locate it. Since they won't send me paper bills without paying extra per month, I don't even know what my Frontier account number is (post Verizon cutover).
So I ask them "Can you look me up by address?", and to that the answer is NO. Tech Support can not look up accounts by address. They have to call Customer Care to find my account number. That's just dumb, but ok. She says she is going to call over there to get my number and asks me to wait, but then I get ringing all of a sudden and I am *transferred* to Customer Care, not conferenced in.
Ok, whatever, I will roll with it.... So I ask the Customer Care rep for my account number and then ask to be transferred back to Tech Support. She comes back and says she has someone from Tech Support on the line, but when she transfers me I get hung up on. That's TWO "accidental" failures in a ROW. NOT ACCEPTABLE!
So I then call back a bit later after dinner (funny enough, I get the same Tech Support rep that ditched me earlier). This time I have my account number however and she can find the account.
I explain that I am a Network Engineer and that I have a serious problem that needs troubleshooting with their Network Engineers and at this point I am massively over her head (they don't staff front line people that have any genuine network skills from what I can tell). At this point she informs me that their Network Group folks are off-shift and there is nothing she can do for me at this time other than filling out a request for them to call me back tomorrow.
I specifically clarify with her at this point that *THEY DO NOT HAVE AN ESCALATION PROCEDURE* and she confirms this. So if things are genuinely broken for residential subscribers Fronter BY POLICY will *NOT* troubleshoot them after hours. Mind you this is NOT just an issue with one persons Internet. This is an issue deeper in their network and is probably impacting hundreds if not thousands of customers.
I find this utterly unacceptable as a network provider. Yes, granted this is Residential service we are talking about (not business), but still, their monitoring systems should have caught this and their NOC should be troubleshooting it proactively and calling out to Network Engineers as needed.
I also should note that this same CO device probably also provides service for Business FiOS connections, and they are ignoring that just the same.
I am posting this in hopes to publicly embarrass Frontier in the hopes that it makes them better in the future. There is always room for improvement.
To be clear, I am most frustrated by the utter and complete lack of escalation ability by their front line technical support. If *I* as a professional Network Engineer can not get my issue looked at successfully (when I have already done significant diagnostics on it) then the average customer is never going to be properly taken care of.
-Eric |
|
| Re: [FiOS] Extremely High Latency in Tigard Oregon - Support is
Smokeping Latency from my office to another friends house |
Yes, you are reading that correctly. That is 1.2 ms ROUND TRIP latency that is 100% consistent 24 hours a day.
Everyone in the neighborhood has 100 megabit Internet connections, and if you want to pay a little more you can get a gigabit connection. 
Mind you, my office does *not* get Internet from the same ISP as my friends house. The reason that the latency is so amazing is (in part) because both companies exchange traffic on the local peering exchange NWAX in Portland Oregon (i.e. it does not have to leave Oregon and come back).
Massively high speed Internet to residential homes IS FEASIBLE for reasonable prices and with high reliability. It's just a matter of spending the money and properly engineering / operating the network.
-Eric |
|
| I'm having the same issue around 135th & Walnut Street. You're not alone FiOS buddy. |
|
jws74Premium
join:2005-02-17
Gresham, OR | reply to eprosenx
Not the least bit surprised. I had an issue after I moved (3 physical blocks, but that was far enough to move out of BPON territory into GPON territory) where they rolled a truck for something they could have fixed remotely... but try and get anyone on the phone that speaks at that level.
"Hi, I have a problem related to the MAC address limitation in the ONT, could you please connect me to someone in networking support that can address that for me?"
"Oh, that setting can only be changed from the customer side of the ONT. I'll roll a truck for you in a few days."
Pathetic. |
|
|
|
| reply to eprosenx
Better this AM, but last night was HORRIBLE |
i.e. I would put money on it coming back again later this afternoon/evening when traffic levels go back up at night (from people watching Netflix, etc...) |
|
| 
Tonight is horrible yet again! |
As an update:
A FiOS tech *did* call me back today around 4pm to discuss the network issue. He was sympathetic to the issue I was having and eluded to the fact that the rep I had spoken to should have escalated the issue the previous night.
Sadly though, since the issue was not currently occurring he had no way to troubleshoot anything and he told me to call back later if it happened again . I offered to send all the diagnostic information (MTR outputs and SmokePing graphs), but he told me their email was not working at that time (WTF btw that an Internet company has their email broken).
So this evening I called back just a few minutes ago and in a refreshing change, the person I was directly connected with knew who I was since the tech that called me back earlier in the day had passed along my info to her (very awesome!). She was actually one of their higher level folks I think and since the issue was happening right then and since apparently their email was back online, I was able to send graphs and MTR outputs to which she was like "oh wow, that's really awful".
Also, it sounds like lots of people were calling to complain in this area.
So after some more discussion about their escalation policies, it basically came down to the fact that their network folks that they escalate to from the call center don't come in till 5am (perhaps that's 8am Eastern?). If there is a huge outage impacting thousands of people or whatever they can call a manager out of bed, though in discussing this issue with her my basic thought was that it will stop being an issue in about an hour and so by the time they got someone looking at this tonight, it won't matter.
So she is sending all the info I sent her off to the network team, and here is to hoping that someone competent gets the information in the morning and is able to use it to figure out which network links are broken and do something to get them fixed.
I am hoping this is just some ports that need "bounced", or that got mis-configured, etc... If it's hardware that needs replaced at the CO that could mean a longer timeline. :-(
-Eric |
|
 NOYBSt. John 3.16Premium
join:2005-12-15
Forest Grove, OR
kudos:1 | reply to eprosenx
They don't proactively monitor this stuff because they know you the customer will do it for them for free. So they opt for the passive approach, until someone proves it, it's not broke.
Sad way to operate a business, but it's done all the time.
An online business just lost a $600+ order from me last week because of their ordering system confirmation not working and dumping me back to the order payment page instead. Gave them one shot at fixing it. Tried again after they said it was fixed and got the same thing. Good-bye.
--
Be a Good Netizen - Read, Know & Complain About Overly Restrictive Tyrannical ISP ToS & AUP »comcast.net/terms/ »verizon.net/policies/
Say Thanks with a Tool Points Donation |
|
| reply to eprosenx
I'm at 134th and Walnut...this started for me 5 days ago playing BF3. I'm usually 17ms Ping, 25 dl, 25 ul. Now 350 ping, 2 dl, .05 ul. Been on the phone with them...I get nowhere. I am all over Comcast's site, and ready to pull the trigger. Sucks though, left Comcast for Frontier...but for me, this is the equal to having my power or water off. Just can't game, stream or do anything at night. |
|
jpgrWAR
join:2011-08-18
Clyde, OH | reply to eprosenx
DSL for ping is alot better than cable or fiber !If your a hard core gamer like me DSL is better then anything simple fact!
I hated cable gaming in the after noon my ping would always jump up in the afternoon. But know my ping is always 34 no matter what for a year know .! |
|
 darciliciousCyber LibrarianPremium
join:2001-01-02
Forest Grove, OR
kudos:2
 ·Frontier FiOS
| said by jpgr:DSL for ping is alot better than cable or fiber !If your a hard core gamer like me DSL is better then anything simple fact! I don't think you have a clear understanding of FiOS as deployed to ex-VZ / now-Frontier customers...
--
♬ Music is life ♬ |
|
| reply to eprosenx
This is now the fourth night in a row |
So clearly you can see by the graph above that their network team did *NOT* resolve the issue today.
I just called back again this evening and managed to get through to their lead call center folks again.
While both folks I talked to this evening were nice and polite, they really don't seem to be able to do anything about the issue. They did mention that they had several hundred tickets on this and that they know it is a problem with the "184.19.244.36" IP address as the second hop in the path.
I pushed pretty hard to get them to further escalate, but the best they could say is that "folks are working on it".
This is day four of the issue (really three days of it being during prime-time). They need to get a real network engineer to look at this. I am happy to get on the phone with them and run tests in real time from a customer perspective...
-Eric |
|
| reply to jpgr
said by jpgr:DSL for ping is alot better than cable or fiber !If your a hard core gamer like me DSL is better then anything simple fact!
I hated cable gaming in the after noon my ping would always jump up in the afternoon. But know my ping is always 34 no matter what for a year know .!
When FiOS is operating properly this is absolutely an untrue statement. The modulation technique used by FiOS across the fiber results in massively lower latency than DSL for the "last mile". DSL intentionally increases latency in order to deal with bursts of interference without dropping packets.
Now that being said, it sounds like you are on a rock solid DSLAM with no issues with backhaul bandwidth.
The issue with FiOS in this case is that the Frontier backhaul from the FiOS aggregation device at the CO into their network further is having issues.
I would take a properly operated FiOS network any day of the week over a properly managed DSL network.
-Eric |
|
NOYBSt. John 3.16Premium
join:2005-12-15
Forest Grove, OR
kudos:1 | reply to eprosenx
Wonder how long this problem would exist for if they were charging per monthly consumption.
Would there be greater financial incentive to correct usage limiting issues more quickly?
|
|
| reply to eprosenx
They are sending a service tech out to my place...135th/Walnut area today. I told them don't bother, but whatever. Last day for it to be fixed or I'm calling comcast. |
|
| reply to eprosenx
So, I'm no expert, but I swear my internet speeds (namely latency/ping) has been way worse for about 1.5 weeks now. I'm also in Tigard off 161st. I'm on the 35/35 plan and whenever I test on speedtest.net, constantly getting 35 down, 20 up. Ping times in the 110ms+.
Based on a recommendation from another forum member, I ran another speedtest using measurementlab.net. My upload jumped up to the 30 range, but ping speeds still seem much higher than usual (see below). I normally wouldn't care but I'm a gamer and noticing a huge difference is just frustrating, especially because I just upgraded to the 35/35 plan.
Any other solid alternatives out there that's reasonable priced? Maybe with FiOS they ol' saying you get what you pay for is true. I'm serious thinking about switching as well.
1.0E-6 packets lost during test
Round trip time: 30 msec (minimum), 167 msec (maximum), 132.51 msec (average)
Jitter: 137 msec |
|
| reply to eprosenx
It's fixed!!! |
Now I am not sure if all the tickets I put in through the Tech Support channel actually got it fixed, or if some other prodding through "business" channels (i.e. not residential tech support) are what got it fixed today.
For folks that may still be having issues: There are many different scenarios that can be causing trouble for you. Note that if your ONT is "BPON" gear it's likely on one of many TelLabs PON devices back at the CO (or the various CO's we are all attached to). If your ONT is "GPON" gear I suspect it's probably Alcatel/Lucent gear at the CO. From those PON devices they uplink to what I presume is different edge aggregation routers, which then connect back into their "core" backbone devices at some point.
Now please don't think that the Tellabs or the Alcatel gear is "bad" in any way. Both I think can provide rock solid service. I am on Tellabs and I have had zero issues between my house and the CO (i.e. I am on the slightly older stuff, but I don't care because it is rock solid - all the issues have been further back in the network). Your milage may vary just based on how subscribed your specific PON node is.
So the net-net is that you may be right next door to someone that has great performance, while your performance is horrible because your on a different PON device at the CO, or Agg router, etc...
It would seem to me based on my observations (and from talking with some FiOS field tech's while out on service calls) that Frontier has been having a hard time keeping up with the explosive growth of Internet utilization in residences (i.e. driven by streaming video).
What we have been seeing over the months since Frontier took over this market has been them cutting all the Verizon gear (that Verizon stopped upgrading before the sale) over to Frontier's own network, and then adding capacity as fast as possible to try and keep up with everything. What we are seeing is "growing pains".
So what do I think of Frontier?
I am not ditching my FiOS service and switching to Comcast as I have hope that they can pull this together. I do know they are working very hard to add capacity.
From a pure architecture standpoint, Frontier is the horse to bet on long term. NOBODY with Copper twisted pair or COAX (i.e. Comcast) is going to be able to truly compete (from a pure technology standpoint) with Frontier's FiOS service.
Fiber has for all intents and purposes, virtually unlimited bandwidth. 10 gigabit service is simply an exercise in swapping ONT's and CO gear. There is ZERO powered equipment in the outside plant and nothing electrical, so it's impervious to water (which we get a lot of here in Oregon).
Compare this to Comcast who has to have powered midline amps every couple thousand feet and power injectors all over the place that put voltage on the lines to power those amps. Cable in general is a MUCH flakier architecture than fiber PON's.
That being said, I must give Kudo's to Comcast for being really really good at managing their networks. I think they do manage to run an amazing network considering all the crazy factors (like interference from subscribers RF leakage issues) that they have to deal with.
So my recommendation to folks having trouble still:
Be an educated customer. Learn to use tools like MTR aka My Traceroute (an awesome version of traceroute). Setup graphs like I have done with Smokeping from my office (DSL Reports has some of this functionality, though not nearly as aggressive in time interval as mine).
I highly recommend using "real" routers rather than crummy consumer grade ones. I personally use the free version of Vyatta which now comes with MTR installed as of the latest version. If you have an old PC laying around you can put a second network card in, then you have a Vyatta box.
Seek advice from others, and learn as much as possible about reading the output from tools like MTR. Once your sure the problem is with Frontier, then open support cases and press them on them to fix the issue.
There is nothing more powerful than graphs like smokeping over time that allow you to prove that an issue exists and to trend whether it is getting better or worse. This is MUCH more valuable than anecdotal evidence of things like "XBOX live was slow on Monday night". The vast majority of issues are likely out of Frontier's control, which is why it is so difficult to get through tech support's "filters" as they get a lot of false reports.
Good luck!
-Eric |
|
| reply to bros3ph
said by bros3ph:So, I'm no expert, but I swear my internet speeds (namely latency/ping) has been way worse for about 1.5 weeks now. I'm also in Tigard off 161st. I'm on the 35/35 plan and whenever I test on speedtest.net, constantly getting 35 down, 20 up. Ping times in the 110ms+.
Based on a recommendation from another forum member, I ran another speedtest using measurementlab.net. My upload jumped up to the 30 range, but ping speeds still seem much higher than usual (see below). I normally wouldn't care but I'm a gamer and noticing a huge difference is just frustrating, especially because I just upgraded to the 35/35 plan.
Any other solid alternatives out there that's reasonable priced? Maybe with FiOS they ol' saying you get what you pay for is true. I'm serious thinking about switching as well.
I would see if I could get some smokeping graphs setup from somewhere on the Internet back to your house. Graphs are worth a thousand words to see what is going on.
Note that most speed tests still don't provide very accurate results on high speed connections like FiOS, especially not on uploads. I only use speed tests as a very rough estimate of performance. To really test things accurately, I use a command line utility called iPerf that I run from one of my own servers on ISP grade business Internet connections (1 gigabit and up). Of course I realize most folks don't have access to such things.
The reality is that around here, your options for home Internet of any reasonable quality are fundamentally Frontier, and Comcast. Anything wireless is going to suck in comparison (especially latency wise).
-Eric |
|
jpgrWAR
join:2011-08-18
Clyde, OH | reply to eprosenx
what ever makes you feel better |
|
| reply to eprosenx
Mine was fixed last night as well. I did get a hold of the Tech. that was going to come out...he was a great help and actually gave me the confidence to stay away from switching back to Comcast. Foe whatever reason, I was back to a 20ms Ping and 25/25. All good...for now |
|
| It is possible/likely that you were impacted by the exact same issue my friend in Tigard was. An installer tech in Camas, WA yesterday told me that even the stuff there in Camas hubs back through the Beaverton CO, so the issue in Tigard and your issue could all have been one and the same issue. Beaverton is Frontier's biggest Oregon CO and so I think they have a ton of core gear there.
-Eric |
|
