most WISP have 80-90ping which really isn't that much of a delay and should work fine with VoIP.
My guess is you need more upload. 1mb is really 120~kb/s split 4 ways is only 30kb/s which may not be enough for your VoIP quality setting.
*that's assuming nothing else is running and your getting full speed.
/edit I could be wrong I don't use VoIP :/ »
techtips.salon.com/much- ··· 127.html"Depending on the protocol used, each concurrent call requires a bandwidth of between 30Kbps to 80Kbps."
" A low-end 2Mbps download and 512Kbps upload broadband connection should be enough for two or three concurrent calls"