Business FIOS looses connection in Verizon Fiber Optics http://www.dslreports.com/forum/r20451578 en Sat, 11 Oct 2008 08:33:39 EDT Sat, 11 Oct 2008 08:33:39 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20464081 HaloBox :
said by MadDogMatt :

I had considered attempting to do this, but I really don't feel comfortable taking machines and directly connect them to the internet. That would also take about every machine we have to test.
If you are short on machines, you can always direct connect a single firewalled laptop and cycle through and test each IP address. At least then you would know each address works. While you are doing this, you could inspect the routes and see if you can spot some irregularities. ]]> http://www.dslreports.com/forum/remark,20464081 Sun, 11 May 2008 02:07:14 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20463509 sashwa : Because it's the same poster about the same problem. I did the same in the Cisco thread. :)]]> http://www.dslreports.com/forum/remark,20463509 Sat, 10 May 2008 23:01:57 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20462070 rhard49 : why the cross refrence to service issue with cisco routers?
I use a Cisco 871 series with the FIOS static ip service hosting mail and web servers without any problem]]> http://www.dslreports.com/forum/remark,20462070 Sat, 10 May 2008 15:59:25 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20461482 rmatthewcole : since they are not doing normal subnets with static routes, does anyone know how they are routing the static IP addresses? This would go a long way to resolving the problem.]]> http://www.dslreports.com/forum/remark,20461482 Sat, 10 May 2008 13:02:18 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20457866 birdfeedr :
said by MadDogMatt :

The only thing I can think of is this must have something to do with how Verizon builds the route table for the static IP addresses. A wireshark trace clearly shows traffic leaving the router for the ONT, I find it hard to believe that its an issue with the design... the traffic just never makes it back. I also find it interesting that I don't see ANY arp requests from the Verizon router. its almost like the verizon network is building the route table by arp broadcasts.
You may be onto something there. While I have no proof and I'm not a network engineer, I know enough and have seen enough to know that something is odd in Verizon's network. Look around and see complaints of DNS issues and slow page loads, even no page loads. Sometimes things get fixed with a Retry, sometimes things are fine minutes later. Sometimes it takes a call to FSC, where they say we aren't aware of any issues, but suddenly something gets fixed and it now works.

It's their ad-hoc, flexible routing that seems to stumble to its knees on occasion. It may be their "high-bandwidth so close to the end-point" requirements that you're seeing.]]> http://www.dslreports.com/forum/remark,20457866 Fri, 09 May 2008 15:54:02 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20455403 anon : I had considered attempting to do this, but I really don't feel comfortable taking machines and directly connect them to the internet. That would also take about every machine we have to test.

Keep in mind this design has worked at several other ISP's without issue, this is not new. The config that is being used has been reduced to a textbook NAT config as well. If it was a design issue I doubt we would be able to pass traffic at all but, I can make this config work for 6 hours before the issue re-occurs.

The only thing I can think of is this must have something to do with how Verizon builds the route table for the static IP addresses. A wireshark trace clearly shows traffic leaving the router for the ONT, I find it hard to believe that its an issue with the design... the traffic just never makes it back. I also find it interesting that I don't see ANY arp requests from the Verizon router. its almost like the verizon network is building the route table by arp broadcasts. ]]> http://www.dslreports.com/forum/remark,20455403 Fri, 09 May 2008 07:33:40 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20455147 HaloBox : Did you plug the ethernet run from the ONT into a simple switch and verify each of the IP addresses is working at the same time?

If you have five laptops or regular desktop machines you can test that in short order using external DNS servers.

If they are all working, then the issue isn't FIOS, it's the rest of your design. ]]> http://www.dslreports.com/forum/remark,20455147 Fri, 09 May 2008 03:24:48 EDT Re: Business FIOS looses connection http://www.dslreports.com/forum/remark,20451948 sashwa : Cross-reference:

»Fios Commercial service issue with cisco routers]]> http://www.dslreports.com/forum/remark,20451948 Thu, 08 May 2008 14:19:15 EDT Business FIOS looses connection http://www.dslreports.com/forum/remark,20451578 anon : I also posted this in the Cisco forums to see if anyone had some ideas there.. Hopefully someone here knows the solution for this...

For the past week I have been fighting an issue implementing a new FiOS commercial service implementation and I hope someone here may be able to shed some light on a possible solution. First here is a bit of the background..

Our small business was using comcast as our ISP for the past several years, besides several service outages from comcast we never had a reliability problem. The config looked like this:

The network run from the ISP was wired into a cisco 2924 switch that had an external and internal VLAN defined, and we had a trunk to a 1751 router. This router had a sub interface for outside, and had a sub interface on the inside network. It was configured this way since we did not have 2 Ethernet cards for the normal outside/inside config for NAT. The switch ports essentially became additional router ethernet ports. Essentially the traffic flow looked like this:

isp->cablemodem->switch(external vlan)->router(external subinterface)->NAT rules->internal subinterface->local network

we have 2 types of NAT for the 5 statics we deployed. One IP was an overloaded IP for the general web traffic, the others were static maps created for our in house DNS, mail, and web servers to use for incoming and outgoing traffic.

Once FiOS was available in our area, we were able to get what looked like a more reliable service with more bandwidth for the same cost so we went ahead and ordered it. That's where the problems started.

We took the working comcast config and replaced the public IP addresses with the new addressses provided from verizon. since it is a commercial account there is no PPPoE or anything else we needed to worry about. Verizon provided a CAT5 run from the ONT to plug into our equipment.

The first issue we ran into was the ONT was not configured to allow more than 1 MAC address. Although it took some time to work out, we eventually got to someone who increased the max number of MAC addresses allowed on the ONT and that allowed us to connect to the internet from the hosts using the overloaded NAT address.

This change did not resolve the static addresses not being able to get inbound or outbound traffic. What we did find was that by changing the ip address assigned of the outside sub interface to each of the external ip's we would be NATing, we can bring ALL CONNECTIONS up for a period of 6 hours. Then the statics stop working again. Essentially the PRIMARY ip on the external interface works, the NAT addresses don't work until they have been the primary ip of the Ethernet interface. We even tried to assign some of the static addresses as secondary IP's to see if that would keep the connection up (it didn't).

To try and shortcut the story somewhat.. I have replaced the switch/router with a Cisco 2651 to simplify the troubleshooting. We are now using a more conventional outside (f0/0) inside (f0/1) config with NAT overload and static NAT. We are still having the same problem. Cisco has verified the NAT config (and it was working before) so I do not believe that to be the issue. Debugs clearly show that traffic is leaving the router and NAT is working, we are just not getting a response. Changing the external IP address on the router every 6 hours seems to keep us running, but is not acceptable for the long haul. Verizon's support has not been able to help.

From what I can tell this looks like an ARP issue, that somehow there is a problem with the remote side knowing where to forward the return traffic too. Once it is seen as a primary interface and gets added in cache, it works until the cache expires. I spent a good bit of time attempting to understand why changing the IP address on the interface would make a difference at all. The only thing I can think of that changing the IP would do is an arp broadcast looking for an IP conflict on the new IP address. I don't know why that would allow the verizon equipment to start sending traffic again however.

Any ideas??]]> http://www.dslreports.com/forum/remark,20451578 Thu, 08 May 2008 13:02:24 EDT