1mb is really 120~kb/s split 4 ways is only 30kb/s which may not be enough for your VoIP quality setting.
You are mixing up kilobits (kb) with kilobytes (kB). 1 megabit is 1024 kilobits, or 128 kilobytes. A VOIP call at most should be 64 kilobits or 8 kilobytes. 1 meg should theoretically support 16 simultaneous 64kb phone calls, but in reality wouldn't leave any headroom. Either way, a few simultaneous calls should be fine.