Verizon FiOS → [Networking] FiOS MTU size with PPPoE and ZyZEL ZyWALL USG 100

I'm posting this for anyone who encounters the issue I discovered.
Apparently, because Verizon FiOS uses PPPoE on their networks I had to adjust the MTU size from the default 1500 to 1492 on my ZyZEL ZyWALL USG 100 so anyone with FiOS service can get to my website hosted from my location.
I have both FiOS & Comcast ISP's and installed the USG 100 to replace a Cisco RV042G router. After the switch any of our employees who had FiOS service at their homes could not get to our CRM website. Anyone who had Comcast, RCN or other ISP service could get to the website without any problem. I was able to duplicate the problem with a laptop attached to the Verizon router on a separate static IP in our office.
After 2 weeks with FiOS Tech support, who told me nothing, I contacted ZyXel Tech support. We tested for another 4 days and finally found that decreasing the MTU size to 1492 solved the problem.
After digging into the Verizon FiOS Actiontech router, it's set with the MTU size of 1500.
If you use ZyXEL USG products (maybe others) and FiOS users can't get to a website hosted behind that device, try changing the MTU size. I'm guessing there are some that don't even know this problem exists.
PPPoE uses 8 octets for the header which reduces the remaining maximum size to 1492. If you search for "PPPoE verizon fios" you can find the Verizon website that allows you to select various states and shows if you might be on a PPPoE network.
This post could also be applicable to the ZyXEL forum.

Wow that would be a tricky one to figure out. Are you static or DHCP customer.

We're set with 5 Static IP's. Our local Verizon router is also static but still had the same problem. I'm sure my employees were on dynamic IP at their homes.

I am surprised anyone is still on FiOS PPPoE setup. I thought they were retiring (had retired) PPPoE on FiOS.

But anytime you use PPPoE you have to reduce the MTU due to PPPoE header info.

We're not directly on, or at least not supposed to be on, PPPoE. However, Verizon uses a mix of DHCP & PPPoE in their network.

I'm curious if other equipment has to be adjusted as my ZyWALL had to be.

I work in the networking field and have seen different experiences when calculating MTU size across vendors..

Some vendors take the overhead into account automatically based on the transport medium, others don't. Sonet, Ethernet, Ethernet with dot1q tags, ATM, etc.. they all have different overheads. Juniper for one doesn't like to automatically adjust this for you, but vendors like Cisco, HP, etc. do.

Nature of the beast and sounds like this could be the case for your Zyxel box vs the Actiontec.

The only thing it seemed to affect was our hosting of the http website and FiOS users couldn't get the page. All other functions worked except for FiOS customers. Various core routers in the Verizon network are most likely still using the settings required for PPPoE => MTU=1492

I was thinking of upgrading my ASA 5505 to one of those new Zywalls. The price for performance is really nice. I used to run a Zywall 5 back when I first signed up and I really ended up liking it but replaced it after I upgraded my old 35/35 plan.

Not being able to access a website like that is a symptom of broken Path MTU Discovery (PMTUD). As part of the troubleshooting, did anyone ask you to disable ICMP echo request blocking on any filtering devices between your CRM website and affected users' ISP equipment?

The Router Requirements RFC and updated discussions around it do support requiring the public facing interface of a traffic filtering device to respond to ICMP echo requests. Be careful with ICMP filtering. There's lots of subtleties.

ICMP connectivity check was disabled the whole time.
Again, only FiOS customers affected by this. All other ISP customers could get to the website without any problem. It is apparent that Verizon does not operate or route internet traffic like all other ISP's that are used in the Mid-Atlantic area. This could be due to the support they provide DSL customers as they still provide DSL service where they don't have FiOS Fiber service throught our region.

Not sure how to interpret that. Do you mean "The equipment at your site, in front of your CRM website, permits ICMP Echo Request and replies accordingly"?

The proper behavior for your Zyxel is to permit inbound echo requests to it's WAN interface only and send echo matching echo replies. Same for the sending routers.I'm not sure that conclusion is supported by the evidence. IP is IP, you can only shuffle the packets around so many ways.

I've chased PMTUD blackhole problems before. This looked like one. You've found something that works for you at the cost of a small inefficiency, the loss of 8 bytes per packet. It was real popular for years to block all ICMP, especially in the aftermath of the "Ping of Death" attack and others from the late 1990s. Some popular, but misguided practices die slowly. I was just asking to see if maybe your configuration was affected.

Respectfully,
TomS.

ICMP Echo Requests are not blocked. At one point I even completely disabled the firewall to test. In both cases remote users were able to ping my IP and websites and Trace Routes all worked fine.

If any issue were the problem it should affect all ISP users equally.

Throughout my testing anyone using Comcast, RCN, Time Warner & Optimum ISP's could view and access the website I host.
Only users who have Verizon FiOS service at their homes were blocked from viewing my website.

I tried hosting the website on both my Verizon FiOS connection and my Comcast service and there was no difference.

After changing the MTU size from 1500 to 1492 on the ZyWALL USG 100 (not a significant performance loss) Verizon FiOS users were able to access my website.

The only explanation I could find is that Verizon has a mix of fiber & DSL services they provide which include both DHCP & PPPoE connections. Because PPPoE requires a maximum MTU setting of 1492 as the header uses 8 octets for the header Verizon must have core Router equipment in the mix that can't handle a full 1500 Transmission Unit size or their equipment in my area is not functioning properly.

I'm sure that most web surfers wouldn't think twice if they got the "website not available" message and most hosting providers may not know that their website might be blocked to only Verizon FiOS users.

I hope this might help someone who has stumbled across this problem and is trying to find a solution.