If there was a problem at the gateway it would affect all of your connected hosts. The fact that you are having problems with hosts on just one of three routers tells you that there is an issue with that router, or at the limit, the gateway's relationship with that router.
I too would use a multi-LAN router like pfsense or m0n0wall in a setup like this. Even (some versions of) Tomato support vlans and could handle all three of your LANs on a shared internet connection. It's the sensible way to set it up, but if you don't have the hardware to do it, then I guess you make do. There's no reason your 4-router solution shouldn't work, assuming all four routers are in good working condition and functioning correctly.