Does your router get a public IP address on the WAN side? If not, please explain (your modem is configured as a router, your ISP is doing NAT, etc.)
If the router does get a public IP, then when you tested with the direct-to-modem connection and it failed, there was no NAT involved, so you are likely looking in the wrong area for the trouble.
