Here's a diagram - hopefully it's clear enough.
So the issue is when connecting devices to vlan4 at Site B. Ideally, I'd like to have the option of setting a static IP in the 192.168.1.160/28 space and having all internet traffic be routed by Router B.
If you use DHCP on vlan4, it should (and does) grab an IP address from Router A's space through the br0 and bridge0 combination. Then all internet traffic from the device will be routed by Router A using it's own public IP. While in this configuration, if I download a file from the Linux Server at Site A, I get full speed - no packet loss. If I download a file from any internet site, it's less than 1KB/sec and massive packet loss.