First of all, it sounds like it could be an interesting project to play with but a Raspberry Pi is not very well suited to be a router - it only has one 100 megabit/s port, which also shares bandwidth with the USB controller. If this is something you're planning on using long-term, I'd recommend installing Openwrt on an actual router.
Second, which Openwrt build did you use? You may want to try the Barrier Breaker release candidate at »
downloads.openwrt.org/ba ··· generic/ .
And how did you connect both the modem/router and the switch? USB Ethernet card?