dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
8746
PJL
join:2008-07-24
Long Beach, CA

PJL to SeattleMatt

Member

to SeattleMatt

Re: [Networking] 1st Proof of Verizon / Netflix Direct Peering????

I'm not seeing direct connections in Los Angeles yet, but I am seeing improved streaming performance in general, likely because the loads through the Netflix content provider ISPs is down due to the direct connections. I'm seeing 5800kpbs from ipv4_1.lagg0.c020.lax001.ix.nflxvideo.net via NTT with five simultaneous connections on the web client using HTML5, but only 3000kpbs on the Windows 8 app also via the same servers and NTT but with only two simultaneous connections. It's interesting how the streaming must work.

norm
join:2012-10-18
Pittsburgh, PA

1 recommendation

norm to SeattleMatt

Member

to SeattleMatt
I am seeing Netflix servers in Philly now. Their appearance seems to coincide with Google Global Cache servers in Philly suffering major packet loss (10-20%)...

|---------------------------------------------------------------------------------- --------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|    L100.PITBPA-VFTTP-33.verizon-gni.net -    0 |   15 |   15 |    2 |    5 |   31 |    4 |
|  G0-1-0-5.PITBPA-LCR-21.verizon-gni.net -    0 |   15 |   15 |    3 |    5 |   15 |   15 |
| xe-15-1-1-0.RES-BB-RTR1.verizon-gni.net -    0 |   15 |   15 |    8 |   21 |   52 |    8 |
|                   No response from host -  100 |    2 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |    2 |    0 |    0 |    0 |    0 |    0 |
|                0.ae4.XT1.DCA6.ALTER.NET -    0 |   15 |   15 |   10 |   22 |   92 |   12 |
|               0.ae9.GW12.PHL6.ALTER.NET -    0 |   15 |   15 |   12 |   14 |   16 |   14 |
|                           204.148.22.46 -    0 |   12 |   12 |   12 |   13 |   17 |   13 |
|ipv4_1.lagg0.c007.phl001.ix.nflxvideo.net -    0 |   15 |   15 |   12 |   13 |   15 |   14 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
 
|---------------------------------------------------------------------------------- --------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|    L100.PITBPA-VFTTP-33.verizon-gni.net -    0 |   26 |   26 |    3 |    9 |   72 |    5 |
|  G0-4-1-2.PITBPA-LCR-22.verizon-gni.net -    0 |   26 |   26 |    3 |    5 |   14 |    4 |
| xe-0-1-0-0.PHIL-BB-RTR2.verizon-gni.net -    0 |   26 |   26 |   10 |   23 |  111 |   10 |
|                   No response from host -  100 |    5 |    0 |    0 |    0 |    0 |    0 |
|               0.ae6.GW12.PHL6.ALTER.NET -    0 |   26 |   26 |   10 |   11 |   17 |   12 |
|                            204.148.2.62 -   24 |   13 |   10 |    0 |   11 |   12 |   12 |
|81.32.199.65.philadelphia.google-ggc.verizon.com -   16 |   13 |   11 |   10 |   11 |   12 |   12 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
 

SeattleMatt
Streaming Tech Director
Premium Member
join:2001-12-28
Seattle, WA

SeattleMatt

Premium Member

Norm-
Nice find! Just did another test this AM - fired up to HD in about 10-15 seconds - hitting the direct peer in NYC.

This is now 3 consecutive (random) tests over a 5-day period - and i've now hit the direct peers every time.

Funny what a little bad press will do to a project plan!

norm
join:2012-10-18
Pittsburgh, PA

norm

Member

I just hope we continue to see improvement with both Netflix and Google... and ultimately Verizon's peering with others.

SeattleMatt
Streaming Tech Director
Premium Member
join:2001-12-28
Seattle, WA

1 recommendation

SeattleMatt

Premium Member

Click for full size
Jesus - horrible performance today... took forever to get about 720...

Turns out instead of hitting the direct peers in NYC (8ms) or Philly - i'm routed out to Seattle (of all places) - check out the huge increase in latency at the Telia peer with Netflix. Lends some credence that this IS a Netflix problem...

This is ridiculous... this shit needs to be fixed yesterday.
serge87
join:2009-11-29
New York

serge87

Member

said by SeattleMatt:

Turns out instead of hitting the direct peers in NYC (8ms) or Philly - i'm routed out to Seattle (of all places) - check out the huge increase in latency at the Telia peer with Netflix. Lends some credence that this IS a Netflix problem...

Only Netflix can choose which transit link they will use to send video content to Verizon for any particular region. For me it's usually Level3 or Telia and others it's Cogent or someone else.
PJL
join:2008-07-24
Long Beach, CA

1 edit

PJL

Member

I think the direct connection has made its way to Southern California. I just saw great performance (5800kpbs), with five open connections to the server (using IE in HTML5 mode). The 65.223.13.170 is Verizon (Business) and the next is Netflix. Let's see if it stays.

Tracing route to ipv4_1.lagg0.c002.lax004.ix.nflxvideo.net [198.38.96.131]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 6 ms 4 ms 4 ms L100.LSANCA-VFTTP-128.verizon-gni.net [96.229.59.1]
3 6 ms 7 ms 6 ms G1-6-0-0.LSANCA-LCR-21.verizon-gni.net [100.41.196.42]
4 7 ms 6 ms 23 ms so-6-0-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81.29.124]
5 * * * Request timed out.
6 9 ms 8 ms 7 ms 0.ae4.XT1.LAX9.ALTER.NET [140.222.225.7]
7 10 ms 9 ms 10 ms 0.ae3.GW8.LAX7.ALTER.NET [140.222.230.201]
8 26 ms 10 ms 10 ms 65.223.13.170
9 10 ms 9 ms 10 ms 198.38.96.18
10 8 ms 7 ms 8 ms ipv4_1.lagg0.c002.lax004.ix.nflxvideo.net [198.38.96.131]

I then tried using the Windows 8 app, got the same server with the same routing. However, the stream capped at 3000kpbs with only two connections. So Netflix is limiting stream quality based on the client hitting its servers. But at least it looks like a direct connection for both clients I tested.

SeattleMatt
Streaming Tech Director
Premium Member
join:2001-12-28
Seattle, WA

SeattleMatt

Premium Member

PJ-
Curious to see if it goes away for you as well. I had the direct peers for almost a week consecutively - and now the past 3 days are all downgraded (Telia, L3, Ntt)
PJL
join:2008-07-24
Long Beach, CA

PJL

Member

said by SeattleMatt:

PJ-
Curious to see if it goes away for you as well. I had the direct peers for almost a week consecutively - and now the past 3 days are all downgraded (Telia, L3, Ntt)

Hopefully the direct connections will be back for you soon -- and mine will not go away. I just rechecked and now I'm seeing quick 5800kpbs on IE, the Windows 8 app, Chromecast and a Sony Blu-ray. Chromecast ramped up the fastest -- 5800kpbs in less than 10 seconds.
navyson
join:2011-07-15
Upper Marlboro, MD

1 recommendation

navyson to SeattleMatt

Member

to SeattleMatt
Another night of poor Netflix streaming on my Roku Box last night.

Amazon streamed in perfect HD however.

Chris123NT
join:2001-11-24
Palm Bay, FL

Chris123NT to SeattleMatt

Member

to SeattleMatt
Hmm interesting, so west coast guy has direct peering, and us east coast folks got shoved back on the crappy routes.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

1 recommendation

guppy_fish

Premium Member

Yup, thank NetFlix for that, they have the direct connects to use now, so they are choosing as before to use the low cost path .. some things never change!

norm
join:2012-10-18
Pittsburgh, PA

norm

Member

said by guppy_fish:

Yup, thank NetFlix for that, they have the direct connects to use now, so they are choosing as before to use the low cost path .. some things never change!

Over this past week, routes to Netflix And GGC in the Philly area that were in common with each other showed a lot of packet loss (25%+) starting within Verizon's network. It very well could be that both parties are trying different configurations to see what works best - I hope. Why would Netflix make interconnection deals with Verizon then not use them?
dfwguy
join:2013-10-24

dfwguy

Member

If it costs Netflix $x per GB for data sent directly to Verizon, and $y per GB for data sent over someone else, and x > y, then they're going push as much as they can over someone else to save money, and only use the direct path when congestion requires it.

norm
join:2012-10-18
Pittsburgh, PA

norm

Member

said by dfwguy:

If it costs Netflix $x per GB for data sent directly to Verizon, and $y per GB for data sent over someone else, and x > y, then they're going push as much as they can over someone else to save money, and only use the direct path when congestion requires it.

It's not like Verizon could be charging exceptionally high rates... right?
PJL
join:2008-07-24
Long Beach, CA

PJL

Member

I suspect the routing will settle down over time. The "missing" direct connections may have been problematic and might be being reworked.

Chris123NT
join:2001-11-24
Palm Bay, FL
Ubiquiti EdgeRouter ER-4
Ubiquiti UniFi UAP-nanoHD
Motorola MB8611

Chris123NT to norm

Member

to norm
said by norm:

said by dfwguy:

If it costs Netflix $x per GB for data sent directly to Verizon, and $y per GB for data sent over someone else, and x > y, then they're going push as much as they can over someone else to save money, and only use the direct path when congestion requires it.

It's not like Verizon could be charging exceptionally high rates... right?

This is verizon we're talking about.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

1 recommendation

guppy_fish to norm

Premium Member

to norm
This was never a Verizon issue, from day one this was Netflix choosing the low cost path, regardless if that path could deliver the content the customer, or should one say at a data rate the customer wanted.

Recently, Verizon is getting direct connections area by area working, Verizon isn't going to spend the significant resources to do this wok and then randomly turn them on and off

Netflix is "balancing" its traffic routes, that means the Cogent , L3 and others are preferred routes with direct connections path of last choice. Netflix never said direct connections would be exclusively used.

Most likely, at the combined throughput rates that are approaching 40% of internet backbone traffic, Netflix is finding it very hard to do the dynamic balanced routing. I'm not even sure if this has ever been done before in terms of aggregate data.

norm
join:2012-10-18
Pittsburgh, PA

norm

Member

said by guppy_fish:

This was never a Verizon issue, from day one this was Netflix choosing the low cost path, regardless if that path could deliver the content the customer, or should one say at a data rate the customer wanted.

Recently, Verizon is getting direct connections area by area working, Verizon isn't going to spend the significant resources to do this wok and then randomly turn them on and off

Netflix is "balancing" its traffic routes, that means the Cogent , L3 and others are preferred routes with direct connections path of last choice. Netflix never said direct connections would be exclusively used.

Most likely, at the combined throughput rates that are approaching 40% of internet backbone traffic, Netflix is finding it very hard to do the dynamic balanced routing. I'm not even sure if this has ever been done before in terms of aggregate data.

I don't think that Verizon would turn them off and on for fun. I think that Verizon would try various configurations and see what works best. As I've pointed out in the past and others seem to be noticing now, Google Global Cache servers in the Philadelphia region (and others, I'm sure) are suffering packet loss as these Netflix connections are being lit up. I am wondering if those two are related.

»Anyone else noticing terrible Google performance? (VZ GGC caching)

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

204.148.2.62 ( 204.148.0.0 - 204.151.255.255 Ans Communications, Inc ) is not a Verizon IP block.

looks to just be another transit provider in your link. Direct is .netflix to .alternet, nothing in between

As an example, this is a direct connection, posted by BigDaddy

netflix-gw.customer.alter.net 5 21 20 0 59 62 57
ipv4_1.lagg0.c130.dfw001.ix.nflxvideo.net 0 25 25 128 133 135 135

norm
join:2012-10-18
Pittsburgh, PA

norm

Member

said by guppy_fish:

204.148.2.62 ( 204.148.0.0 - 204.151.255.255 Ans Communications, Inc ) is not a Verizon IP block.

looks to just be another transit provider in your link. Direct is .netflix to .alternet, nothing in between

As an example, this is a direct connection, posted by BigDaddy

netflix-gw.customer.alter.net 5 21 20 0 59 62 57
ipv4_1.lagg0.c130.dfw001.ix.nflxvideo.net 0 25 25 128 133 135 135

Unless I'm mistaken, AN&S closed its doors in late 2008. All of the contacts on ARIN for that block are Verizon specific.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

Could be, I was just going by who is on the IP

norm
join:2012-10-18
Pittsburgh, PA

norm

Member

said by guppy_fish:

Could be, I was just going by who is on the IP

It looks like ANS is being reassigned to VZ but just hasn't been renamed yet.

204.148.1.149 shows as an edge router between VZ and Zayo.
Acct101
Premium Member
join:2011-09-20
Bensalem, PA

Acct101 to SeattleMatt

Premium Member

to SeattleMatt
How do you guys know how to do all that stuff? Where did you pick it up? I haver no clue what your doing. I'm just a hapless ben counter.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

In my case I have my CCNA and have taught networking in college courses
Everywhere15
join:2004-10-20
Virginia Beach, VA

1 recommendation

Everywhere15 to SeattleMatt

Member

to SeattleMatt
Still no love for us in the Hampton Roads region of Virginia :(

Tracing route to ipv4_1.lagg0.c284.ord001.ix.nflxvideo.net [108.175.39.214]
over a maximum of 30 hops:

1 9 ms 9 ms 8 ms Wireless_Broadband_Router.home [192.168.1.1]
2 13 ms 12 ms 15 ms L100.NRFLVA-VFTTP-15.verizon-gni.net [96.238.86.1]
3 29 ms 21 ms 14 ms G0-2-5-3.NRFLVA-LCR-22.verizon-gni.net [130.81.104.220]
4 27 ms 27 ms 25 ms ae2-0.PHIL-BB-RTR2.verizon-gni.net [130.81.199.220]
5 * * * Request timed out.
6 * * * Request timed out.
7 39 ms 39 ms 41 ms 0.ae2.BR2.IAD8.ALTER.NET [140.222.229.169]
8 40 ms 39 ms 40 ms ae17.edge1.washingtondc12.level3.net [4.68.62.137]
9 77 ms 56 ms 56 ms vl-3604-ve-228.ebr2.Washington12.Level3.net [4.69.158.46]
10 54 ms 58 ms 57 ms ae-6-6.ebr2.Chicago2.Level3.net [4.69.148.146]
11 60 ms 57 ms 55 ms ae-2-52.edge3.Chicago3.Level3.net [4.69.138.168]

12 57 ms 59 ms 61 ms 4.53.96.58
13 54 ms 57 ms 55 ms ipv4_1.lagg0.c284.ord001.ix.nflxvideo.net [108.175.39.214]

Trace complete.


A few months ago I switched from Cox to FiOS. At the time I wasn't aware of the issues between Verizon and Netflix. On Cox I had an awesome experience with Netflix. With FiOS so far it's been a total disaster. Had I known I probably wouldn't have switched, at least not until the Verizon-Netflix catfight was over. I will admit though that except for Netflix, FiOS service seems good. I am particularly pleased with MoCA performance, having used powerline (albeit using older equipment) previously.

Verizon says it will take them until the end of the year to get those connections to Netflix set up. Seems to me if Verizon really wanted to, they should be able to get things set up quicker.
qwert9999
join:2001-06-27
Port Richey, FL

qwert9999 to SeattleMatt

Member

to SeattleMatt
i don't see how people can keep blaming netflix. when netflix signed a contract with comcast there speed shot up right away. neither netflix or comcast messed around and the speeds shot right to the top 2.6 mbps

did this happen with verizon? no

if netflix was behind the slow speed then comcast speed wouldn't of shot up like it did.

verizon got slower for the last 3 months since they signed the contract with fios. fios is ranked 54 with 1.5 mbps and there dsl dead last ranked 60. verizon saying it will take up to 8 months? while netflix is paying them? i don't blame netflix for being mad.

expand the list to see fios ranked 54 and there dsl at 60th
»ispspeedindex.netflix.com/usa

anyway someone posted direct connect threw tampa. god i wish but i haven't see it and i'm still watching at 325-750 bit rate. thank god my blu-ray player up's the image

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

said by qwert9999:

i don't see how people can keep blaming netflix

Simple, NetFlix chooses the path for the stream from them to customer. Verizon has no control on this, this is FACT and how the internet works.

Verizon has many new direct connections, Netflix, due to cost is still choosing to use the low bit transit, it also likely struggling to do dynamic on demand routing and is failing on handling when the low cost paths are overloaded.

We will never know, but I believe Comcast not being tier one just matched the transit providers costs, so they get the income, not Cogent L3, NTT ect. Verizon being, well Verizon, is charging NetFlix Business data rates. This is why Netflix is still complaining to anyone whom will listen they are not happy.

There was never ever a capacity issue for NetFlix to send its streams to Verizon customers. Its was NetFlix choice to use the road with a 2 lane bridge as it saved them money where there at 7 other roads with different bridges ( transit provides ) that the tolls were higher than Cogent. Now there is a many more direct paths, Until Nexflix starts using them, nothing will change

norm
join:2012-10-18
Pittsburgh, PA

1 recommendation

norm

Member

said by guppy_fish:

said by qwert9999:

i don't see how people can keep blaming netflix

Simple, NetFlix chooses the path for the stream from them to customer. Verizon has no control on this, this is FACT and how the internet works.

Verizon has many new direct connections, Netflix, due to cost is still choosing to use the low bit transit, it also likely struggling to do dynamic on demand routing and is failing on handling when the low cost paths are overloaded.

We will never know, but I believe Comcast not being tier one just matched the transit providers costs, so they get the income, not Cogent L3, NTT ect. Verizon being, well Verizon, is charging NetFlix Business data rates. This is why Netflix is still complaining to anyone whom will listen they are not happy.

There was never ever a capacity issue for NetFlix to send its streams to Verizon customers. Its was NetFlix choice to use the road with a 2 lane bridge as it saved them money where there at 7 other roads with different bridges ( transit provides ) that the tolls were higher than Cogent. Now there is a many more direct paths, Until Nexflix starts using them, nothing will change

I'm just ready for this to be done. My fear is that Verizon will finally square things away with Netflix yet Verizon's peering with others will still be terrible. It's not just Netflix, it's not just Level 3, it's not just Cogent... It's NTT, Zayo, CenturyLink, Savvis, XO, and many others. At the end of the day, I'm tired of it and I'm tired of suffering as a customer. I'm tired of Verizon pointing its finger at everyone but itself. Verizon isn't always wrong but it's definitely not always right. Maybe if Verizon had actual competitive rates, more companies would use them for transit. With Verizon's peering though, companies buying Verizon transit would be buying A- transit to Verizon customers and C+ transit to non-Verizon customers. The benefit of Verizon specific transit would solely be for connecting to Verizon customers. I'm not sure there are enough Verizon customers to warrant that quite expensive transit...

I monitor a number of servers - a lot of which I have SSH access to - and when they start to spike with latency or packet loss, I'll SSH into them and look at a few other residential ISP connections on ISPs such as Comcast, TWC, and RCN. No big surprise when it's just Verizon that's showing issues.

I'm a Verizon FiOS customer. My employer is a big enough Verizon customer to warrant a x-gw.customer.alter.net address. We have tons of issues with Verizon. Routes to and from Verizon in the DFW area - where most of our VPN servers are - from other parts of the country for Verizon customers are spotty at best. Sometimes they'll just be completely terrible for weeks.

I'm in Pittsburgh and regularly VPN into a DFW host and for over a month just recently, I was having issues that started in the Ashburn area. I never left the Verizon network and could see the issue from both ends being within Verizon's Ashburn area. No timely fixes and it's scary frequent how often that sort of thing happens.

At the end of the day, I don't care who is right or wrong, I just want this crap fixed. You cannot deny the fact that Verizon customers are suffering as a result - and I don't just mean Verizon customers using Netflix. I completely understand that businesses are in the business of making money but if I had the option of choosing an ISP that put its customers first, I would choose that ISP over Verizon every single day of the week.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

1 edit

2 recommendations

guppy_fish

Premium Member

What NetFlix is doing is effecting all the transit providers you mentioned, NetFlix's latest attempts at load balancing are causing havoc with just about everyone. Sure does effect allot of bystanders that until recently just took for granted that the internet just worked, with video approaching 50% of the total traffic things are breaking down, it has nothing to do with Verizon.

You should see what its doing to TWC / Brighthouse in Dallas, Tata is unusable, which is there preferred transit provider. Its an industry wide issue, the fundamental structure of the internet was not designed to handle this type of one way traffic, that's the issue and its not just the peering points so much discussed.