dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
4302
McBane
join:2008-08-22
Wylie, TX

McBane

Member

So, from now on all static subnets on FiOS will remain broken

So a few weeks ago me and a few of us discovered that tracerouting from one IP on our static subnets to another has suddenly started bouncing off the Verizon gateway routers after their maintenance upgrade they rolled out around the same time. After some testing and ticket escalations with support, I finally managed to get a ticket to Verizon's IPNOC. They investigated the issue and here is what they found:
quote:
Ok, so the explanation I got on this probably won't come as good news, but it will give us both an understanding of cause. We recently upgraded the gateway router from a Juniper E320 to an MX960. The profile for static IP on this treats every address on static like it's own /32 without the network or broadcast address. This is what's causing the addresses to arp up to the MX960 and why this change appeared so suddenly. Apparently this configuration wasn't supported on the E320 and was done to avoid the rare scenarios we've run in to where two static IP accounts on the same /24 were unable to communicate with one another.

The bad news part of this is IPNOC advised there is no way to change this, it's a global policy on all MX960s that are being deployed in the FiOS footprint. The advice passed down was to either put everything behind a NAT so all the local traffic stays local or to, ironically, consider Enterprise class service to get classical routing.

I know this wasn't the answer we wanted, but we now have one. On the bright side, the MX960 is a beast of a router and we shouldn't have any capacity issues for years to come.

- (VZ rep name with held, he did all he could to help and I greatly appreciate it)
I appreciate supports help in this but this is a pretty crappy view on customer support. This means there is almost no point in having anything more than 1 static IP if you want your network to behave properly. Verizon seems to be intent on leaving business accounts broken. GG Verizon. Maybe Frontier will be a better option...
McBane

McBane

Member

IMHO for what it's worth, if this is the case going forward they should just bite the bullet and re-do all of us in classical routing instead of this class C /24 for everything voodoo stuff they are trying to pull off for who knows what reason.
serge87
join:2009-11-29
New York

1 recommendation

serge87 to McBane

Member

to McBane
I guess a silver lining is that they're bothering to upgrade their GWRs?
Alphasite
join:2005-07-27
Plano, TX

3 recommendations

Alphasite to McBane

Member

to McBane
said by McBane:

IMHO for what it's worth, if this is the case going forward they should just bite the bullet and re-do all of us in classical routing instead of this class C /24 for everything voodoo stuff they are trying to pull off for who knows what reason.

That's what they should really do, but since this is Verizon I don't think that will happen.

This kind of stuff makes me wonder how broken their IPv6 rollout will be. Which will come first, IPv6 or the completion of the sale to Frontier?

Smith6612
MVM
join:2008-02-01
North Tonawanda, NY
·Charter
Ubee EU2251
Ubiquiti UAP-IW-HD
Ubiquiti UniFi AP-AC-HD

1 recommendation

Smith6612

MVM

said by Alphasite:

said by McBane:

IMHO for what it's worth, if this is the case going forward they should just bite the bullet and re-do all of us in classical routing instead of this class C /24 for everything voodoo stuff they are trying to pull off for who knows what reason.

That's what they should really do, but since this is Verizon I don't think that will happen.

This kind of stuff makes me wonder how broken their IPv6 rollout will be. Which will come first, IPv6 or the completion of the sale to Frontier?

If Verizon Wireline takes a page from Verizon Wireless's IPv6 deployment, the rollout should be rather smooth. Although, if they take too many pages expect CGNAT to also be implemented. This sounds like something yet to come considering the routers they are installing.
dfwguy
join:2013-10-24

1 recommendation

dfwguy

Member

I seriously doubt Verizon would ever go to CGN for wireline service. It would cost them a huge chunk of the market (gamers) and gain them...nothing. They've obviously got enough IP space to handle the current customer load, and with no significant expansion soon (or ever), they aren't going to be adding enough for it to matter. If the blocks currently assigned to CA/TX/FL aren't sold to Frontier as part of the deal, that's at least a few million addresses they'll be able to use elsewhere.

Raphion
join:2000-10-14
Samsara

Raphion

Member

What the hell is Frontier going to do if they don't acquire the blocks currently in use by fios customers? I don't think there's any easy/quick thing for Frontier to do in that case, it would be UGLY.

WalterWhite
@verizon.net

WalterWhite

Anon

Do any of you guys think that my issues that I posted here could be related to them rolling out MX960 around Philly area?? »[Networking] Something Very Weird is Happening as of late....

Because I keep seeing a new MAC ID associated to Juniper Networks in my local network. And also this started happing very recently when they started using new ip range here in Philly.

What do you guys think?

Chris123NT
join:2001-11-24
Palm Bay, FL
Ubiquiti EdgeRouter ER-4
Ubiquiti UniFi UAP-nanoHD
Motorola MB8611

Chris123NT

Member

said by WalterWhite :

Do any of you guys think that my issues that I posted here could be related to them rolling out MX960 around Philly area?? »[Networking] Something Very Weird is Happening as of late....

Because I keep seeing a new MAC ID associated to Juniper Networks in my local network. And also this started happing very recently when they started using new ip range here in Philly.

What do you guys think?

That very well could be the case, if so I would post evidence in the direct forum, that needs to be forwarded to the NOC guys.
McBane
join:2008-08-22
Wylie, TX

4 edits

3 recommendations

McBane to WalterWhite

Member

to WalterWhite
said by WalterWhite :

Do any of you guys think that my issues that I posted here could be related to them rolling out MX960 around Philly area?? »[Networking] Something Very Weird is Happening as of late....

Because I keep seeing a new MAC ID associated to Juniper Networks in my local network. And also this started happing very recently when they started using new ip range here in Philly.

What do you guys think?

Are you a 1 IP residential customer? That honestly is even a more bizarre thing to occur than what I troubleshot and had, but if that's the case and the Verizon border routers are now stealing IPs on customer NAT networks, that would be a whole new level of messed up.

I was honestly thinking of going forward with just 1 static IP and doing some double NATs on my own network, but if they're going to arp for those IPs as well, that would be a deal breaker for me. Goodbye FiOS! (And I've been an hardened FiOS fanboy since day 1)

If you're getting Juniper arps though that's the Verizon routers... Why on earth they would be arping for a customer end NAT network is way beyond me. For my case they were arping as my static IP subnet, which they shouldn't, but for them to steal NAT IPs even not even related to their network...

I still think they should explore re-configuring their old setup (Even though its not supported) or just go the classical routing route. Put the /24 for everything nightmare to bed please Verizon! It isn't working. It has never worked. It will never work. The reason you got this far with it is because you patched it with duct tape and bailing wire and now you managed to even break that.

PS Where are you Karl? You should make the utterly broken state of business FiOS (And even potentially all of FiOS if it's truly arping customer NAT IPs) a frontpage story.
buckweet1980
join:2011-12-31
Saint Petersburg, FL

buckweet1980 to McBane

Member

to McBane
It sounds like the Actiontec is doing proxy-arp along with automatic 1:1 NAT mappings..

Can you replace the actiontec with a different brand of router and retest?

WalterWhite
@verizon.net

WalterWhite to McBane

Anon

to McBane
@Chris123NT I think you might be right. After I read through @McBane's thread here..»So something recently changed on my FiOS routing.... It sounds like that Me and @McBane are victim of the same problem but are experiencing them from a different angle.

@McBane Yes I'm residential customer with 1 DHCP IP. Everything was working good ever since I have had FiOS (6+ months) and it just started doing this during last few weeks. Essentially, that Juniper Network's (MAC ID: 40:B4:F0:08:7F:CD) starts grabbing ip from my local network (192.168.1.xx etc) and so there is a fight between my devices (iPhone, iPad etc) and that Juniper Network device This started happening when Verizon started using new ip range in Philly 100.11.xx.xx but before it used to be in 98.x.x.x. I've tried breaking the lease and hoping to grab a lease from that old 98.xxx range but dozen times or so I did that I just got a new ip from 100.11.x.x range.

@buckweet1980 Unfortunately no because I get my internet coaxial from ONT to Actiontec (MOCA WAN) and therefore I've to use Actiontec unless I pay out of my pocket to buy MOCA LAN device so that my STB's works and then I can use Ethernet from ONT.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish to McBane

Premium Member

to McBane
Why can't you make all your clients /32 with the exit being the .1? , that should resolve the arp issue?

At least that way, they all can hang off a switch.

More Fiber
MVM
join:2005-09-26
Cape Coral, FL

More Fiber to WalterWhite

MVM

to WalterWhite
said by WalterWhite :

Unfortunately no because I get my internet coaxial from ONT to Actiontec

Switch your ONT top an ethernet handoff and put the Actiontec behind a router of your choice. See option 6 here:
»Verizon FiOS FAQ »What are the tradeoffs between the various router configurations

WalterWhite
@verizon.net

WalterWhite to McBane

Anon

to McBane
Also I would like to add that not all the Residential customers who followed this guide are experiencing the same issues as me. »How-to: Make Actiontec MI424WR Revision I (Rev.I) a Network

For instance @purcilas mentions in his thread that everything is working swell for him after following the above guide. Just like it did for me for over 6 months!! »[Networking] Actiontec-MI424WR-Rev.I & Netgear R7000

So I'm guessing they haven't rolled out MX960's in all of the FiOS market yet. @purcilas is from MD whereas I'm from Philly,PA....

@More Fiber I wish I could do that but running Ethernet from my ONT to my router location by myself is going to bit unrealistic. I just bought Rev E off of eBay for 8 bucks and I'm gonna try that as bridge rather than Rev I.
McBane
join:2008-08-22
Wylie, TX

McBane to guppy_fish

Member

to guppy_fish
According to Verizon that's how they're all configured. On my end I have to use a /24 subnet to work with their system though or else their gateway refuses my traffic.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

I'm suggesting you configure YOUR clients all as /32 and set the exit route for *.1, I don't see why that shouldn't work. With the new router configuration, technically its the ONLY way it could work.

There gateway has no knowledge of your network size, all it configures is when the exit route and ARP are used

Have you tried the /32 since the new gateway router install? , I'm betting it works now, otherwise there is no possible way to have your statics work. Adding a router with NAT won't allow more than a single IP to work.
McBane
join:2008-08-22
Wylie, TX

2 edits

McBane

Member

A /32 is a single IP. How can it talk to it's gateway if it's in a different subnet without another IP assigned?

Why don't you try it and tell me if it works?

I don't think you understand how business fios works. The only way their gateway will respond is if you have your subnet configured for a /24, regardless of your true subnet size. It's been like that since day 1. What changed is how their router (my subnet's gateway) is handling it.

The main issue has to do with their router arping for my IPs on my switch. It's creating arp conflicts. It doesn't matter what I configure on my machine, there is nothing I can do to stop Verizon's router from doing that. It's essentially poisoning my mac address table on my switch.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

4 edits

guppy_fish

Premium Member

I am posting this from my PC, set to a static IP, with a /32 and using my gateway which is 192.168.1.9 ( I have the PC set to 192.168.1.120 ).

I'm not a network newbie, I have my CCNA and I'm trying to understand how this is suppose to work from the Verizon perspective. The only time I have used /32 is for loop-back router in Cisco routers

Here is the Juniper explanation of how to use their proxy arp, you set the client to a /32 and default to *.*.*.1

»kb.juniper.net/InfoCente ··· =KB21785

All I was asking is if you did it, sounds like you haven't. If you want to resolve this make all your clients static IP, default gateway of your .1 of your static IP,s it should work and resolve your problem. Reboot your clients and switch and you should be good to go

When the client sends a packet initially, it will have a MAC (L2) of zero for destination, but the IP (L3) will be picked up by the router and handle resolution. The Juniper now is handles everything, this is the only way it could work with the IP network ranges that are not power of 2. Technically everything is WAN, the Juniper will loop back for the IP range assigned to the customer

Its actually pretty common I guess, there is similar info on the Cisco site saying the same thing.
McBane
join:2008-08-22
Wylie, TX

4 edits

McBane

Member

Your article is describing how to configure a /32 static nat on a router, which yeah of course will work. That's the typical deployment for 1to1 NAT, however that is NOT how you configure your PC...

What you're describing to me, at least how I am taking it, is configuring my IP on my PC for let's say 10.0.0.10/255.255.255.255 and setting the gateway to 10.0.0.1, which won't work, because 10.0.0.1 would be outside of the network of 10.0.0.10/32. (You also don't even configure the PC that way for 1to1 NAT, you configure it for the inside subnet that you configured the outside /32 for).

On 1to1 NAT the router/firewall/whatever takes the inside NAT and routes the /32 static IP for it, but the PC end never sees that /32.

Furthermore, all of this has absolutely nothing to do with the original problem at hand in which I started this thread for, so it's a complete side track and waste of time, no offense, because I know you're just trying to help. My problem with Verizon doesn't even involve NAT. It is just straight up static routing using real IPs and how they are implementing it.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

2 edits

guppy_fish

Premium Member

Well I give up, all you have to do is configure /32 as Verizon is doing proxy arp. There is nothing broken on the Verizon side, your network has to be configured for /32, END OF STORY

I know exactly what going on with your setup, your having dueling arps and that is causing the switch to have two MAC's for the same path ( your term was poisoned ).

When your clients are a /32 there is NO ARP, ( all routes travel to the edge router, never from say .3 to .4 ) so your clients WILL use the exit route, just fine,there just won't be a L2 mac address in the header when using the exit route ( which is perfectly fine being outside of the network mask ). Proxy ARP on the edge router takes care of that, this is HOW IT IS DESIGNED TO WORK

Prove me wrong, Make your clients /32 , reboot the switch and clients to clear the mac tables and your done.
said by McBane:

My problem with Verizon doesn't even involve NAT. It is just straight up static routing using real IPs and how they are implementing it.

Nope, your in a 1:1 static NAT on the Juniper, this is how Verizon can make your "network" be oddball IP sizes and the requirement for proxy ARP

mackey
Premium Member
join:2007-08-20

mackey to McBane

Premium Member

to McBane
said by McBane:

What you're describing to me, at least how I am taking it, is configuring my IP on my PC for let's say 10.0.0.10/255.255.255.255 and setting the gateway to 10.0.0.1, which won't work, because 10.0.0.1 would be outside of the network of 10.0.0.10/32.

Actually it works just fine. The OS is smart enough to know the gateway is "directly attached" even though it is technically out of the netmask.

WalterWhite
@verizon.net

WalterWhite to guppy_fish

Anon

to guppy_fish
What about for someone like me though?
said by guppy_fish:

Well I give up, all you have to do is configure /32 as Verizon is doing proxy arp. There is nothing broken on the Verizon side, your network has to be configured for /32, END OF STORY

I know exactly what going on with your setup, your having dueling arps and that is causing the switch to have two MAC's for the same path ( your term was poisoned ).

When your clients are a /32 there is NO ARP, ( all routes travel to the edge router, never from say .3 to .4 ) so your clients WILL use the exit route, just fine,there just won't be a L2 mac address in the header when using the exit route ( which is perfectly fine being outside of the network mask ). Proxy ARP on the edge router takes care of that, this is HOW IT IS DESIGNED TO WORK

Prove me wrong, Make your clients /32 , reboot the switch and clients to clear the mac tables and your done.

said by McBane:

My problem with Verizon doesn't even involve NAT. It is just straight up static routing using real IPs and how they are implementing it.

Nope, your in a 1:1 static NAT on the Juniper, this is how Verizon can make your "network" be oddball IP sizes and the requirement for proxy ARP

McBane
join:2008-08-22
Wylie, TX

McBane to guppy_fish

Member

to guppy_fish
Well if you want to discuss the intricacies and fallacies of 1to1 NAT then go start your own thread. It has nothing to do with the point or the problem of this thread so I don't even know why you keep barking up the wrong tree on this. Even if we did your little experiment results would vary by OS, and do nothing but make my problem worse because my whole problem is computers within the same network/subnet not being able to communicate properly with static IPs. That obviously means having a larger subnet configured on my subnet than a /32.
McBane

4 edits

McBane to WalterWhite

Member

to WalterWhite
said by WalterWhite :

Nope, your in a 1:1 static NAT on the Juniper, this is how Verizon can make your "network" be oddball IP sizes and the requirement for proxy ARP

If that's how it is configured, it is WRONG. Also if they're utilizing proxy arp for this, it is also the WRONG thing for the WRONG place, because they are creating IP/arp conflicts on their own network by having two different machines arp for the same IP. That is why my subnet is dropping traffic because their router is creating these conflicts, when my internal subnet traffic should not even be going to their gateway.

Everyone who says that's no big deal, well guess what, their router isn't returning all the traffic like it should. It's dropping SNMP packets, it's dropping HTTP packets from one of my internal machines to another. No telling what else it could be dropping. I'm not sure if this is because of the arp conflicts (Probably) or their router just plain dropping the traffic from it's own filters protecting itself from the end users.

It's a design problem multiple people have called Verizon out on since day 1 of them rolling out business FiOS. They managed to "fix" this previously by using an unsupported configuration and now they continue to refuse go forward with that configuration because Juniper refuses to let them. This leaves business fios in an utterly broken and unusable state in almost 90% of circumstances normal people would use it.

This method straight up poisons my arp table on my switch. It creates arp conflicts on my network because my PCs are fighting with their router for the arp on their IPs. It's a gaping security hole because I can bring down their router if I really wanted to by simply sending the right kind of traffic from one of my PCs to another, which should never see their router to begin with.

On top of all this, who configures regular old static IP routing as 1:1 static NAT to begin with, if that is really going on? I know it's not standard 1:1 static NAT, because on my PC I'm assigning my static IPs, the same ones they would be configuring on the OTHER side of their router for the NAT to work with. You can't configure a static 1:1 with the same IP on both sides. Well you can, but you'd basically be setting up an overly complicated bridge, which would just be easier to do with a static route to begin with, and might possibly break the routing in general for that IP because the router would be trying to translate a NAT that doesn't exist. It's got to be doing something more than standard 1:1 static NAT for it to work like this.

Also, using NAT for static routing, that's like building a car to cross the Pacific with. It's just the wrong tool for the wrong job. It is just stupid to begin with and because of that we get results like what I'm complaining about in this thread.

Classical static routing is what they need to employ because that is how the internet is designed to handle static IPs and subnets. That is how and why almost every other ISP on the planet configures it like this (Even on the Verizon Business side outside of residential wireline). Either that or they need to employ their old configuration and if people need to talk to one subnet from another they need to be doled out IPs on different /24s, like they used to do on the old config that they're refusing to support now.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

3 edits

guppy_fish to McBane

Premium Member

to McBane
In the time it took to type this, you could have made the change and have been over this. /32 is the correct solution for this configuration, deal with it or move on, its not going to change.

/32 is the industry standard for single IP routing, and there is nothing wrong with a default route being outside the netmask, I suggest you do some reading on the topic.

I'll try one last time to explain:

Maybe where you getting lost is that you have 5 IP's that are all separate WAN IP's, they are not part of a LAN ( same subnet ). If you have your subnet set to anything but /32, then the switch and edge router will cause issues exactly as your seeing.

All traffic has to go out the default gateway, to the edge router and if its target is another one of your static IP's back in to your other client. What you have configured with the /24 is the clients will arp and find a path direct through the switch, which is the problem. /32 no arps, edge router handles ALL routing, including between the 5 IP's you have. Hope this clears up your misunderstanding.

And yes, before, Verizon had a configuration that was different with the /24, but it didn't work and they finally fixed it with this correct /32 schema.
McBane
join:2008-08-22
Wylie, TX

4 edits

McBane

Member

I didn't order 32 /32 addresses and that it is not industry standard. What your asking me to do goes against even how Verizon tells us to configure our networks. I ordered a /27, it's what I pay for, and I expect as much. I've never had a Verizon person tell me to configure my end as a /32, only you.

IMHO I don't think you have any idea what you're talking about, or what Verizon does or how they configure or deploy their static subnets for business customers, so this is my last response to you.

Verizon knows and acknowledges what the problem is. I know what the problem is, and they even acknowledged me on it, but my beef is they basically say yes, it's a problem, but we're not going to fix it anymore.

Thinkdiff
MVM,
join:2001-08-07
Bronx, NY

Thinkdiff to guppy_fish

MVM,

to guppy_fish
said by guppy_fish:

In the time it took to type this, you could have made the change and have been over this. /32 is the correct solution for this configuration, deal with it or move on, its not going to change.

While this will fix the arp issues, it doesn't sound like this is what the OP wants either. I agree the OP's setup isn't ideal (static NAT would be much better, imo), but wouldn't running it in the configuration you suggest cause all traffic to hit Verizon's edge router, even transfers between local computers? Wouldn't this traffic also therefore be subject to the WAN bandwidth limits?

Also, the OP's configuration previously worked and was how Verizon always told users to configure their networks (I've never seen them suggest setting it up as a /32 before. Always /24). The Verizon tech even confirmed that it was a change in the new edge routers that broke the old configuration.
McBane
join:2008-08-22
Wylie, TX

4 edits

McBane

Member

said by Thinkdiff:

Wouldn't this traffic also therefore be subject to the WAN bandwidth limits?

Yes it absolutely would! On top of that, as I mentioned previously, this traffic is somehow filtered and their edge router doesn't return it all. So my own internal network traffic is being filtered by Verizon somehow. That is the broken part of the equation. (On top of the problem if they ever did implement bandwidth caps, all my internal traffic would count against it, which is pretty cheap tactic IMHO).

Whoever says that doing 1to1 NAT as a /32 for every freakin static IP they own is just dumb. Static routing is the answer, and the only way it will work properly. Anything else you're just trying to perform smoke and mirrors voodoo with something you shouldn't, and you'll have problems, just like this. I'm not just going to "accept it" either or accept lies as industry standard. I'm going to let the world know if the product is broken or not. Everyone has a right to know.

I can also say, with firm knowledge that I have, this is the ONLY PLACE in ALL OF VERIZON that static routing is configured in such a way, is on their FiOS deployments.

guppy_fish
Premium Member
join:2003-12-09
Palm Harbor, FL

guppy_fish

Premium Member

said by McBane:

I didn't order 32 /32 addresses and that it is not industry standard.

Yes its is, welcome the the world of IPV4 shortages, they can't piss away unused addresses in a net block anymore

Your on a business account, any company that has someone with a CCNA/CCNP ect wouldn't give this a second thought.
said by McBane:

Yes it absolutely would! On top of that, as I mentioned previously, this traffic is somehow filtered and their edge router doesn't return it all

This is a side effect of you refusing to configure your network /32. Your switch has two routes for the same path, and its causing the packets to be dropped, its YOUR fault for not configuring your network correctly.
said by McBane:

Whoever says that doing 1to1 NAT as a /32 for every freakin static IP they own is just dumb

Pissing away IPV4 address is not an an option any more. This configuration allows ALL IP's in a block to be used

You say you ordered a /27, that's 32 IP address, and you pay for 5 correct?, so that would require the remaining addresses never be used again ... ain't happening anymore.
said by McBane:

Verizon knows and acknowledges what the problem is

nycdave hasn't said this, sounds like whom your talking with has the same level of network knowledge you do, which is a little information and not an in depth understand of routing. Verizon didn't role this out without network ENGINEERS signing off on this.

Your clearly just in denial about this, If your feel your right, reach out to nycdave, he know all about these things, maybe he can teach you network 102.
said by McBane:

So my own internal network traffic is being filtered by Verizon somehow

First its your switch is f*&king that up due to it sees two paths, the /32 fixes that

Also, it is not internal, they are all WAN addresses and as such all going to the edge router as they should. If you want to have the clients directly connected on your end and they all need WAN access, then put in a second NIC and have a private LAN between them.

This is standard operating procedure for anything naked to the internet, you should NEVER have your local network exposed. If you had completed networking classes you would know this, instead you have a basic understanding of one small aspect of networking and think that's the entire world, it is so much more

/32 for all your clients on the PUBLIC WAN IP's

add a second NIC, use a /whatever you want for the second internal LAN

Problem fixed , this is your ONLY option and IS the way it is designed to work, its how the big boys do networking every day.