Here are some options:
1. Get a hub if you can find one and connect everything there. If you do this you'll need to add an additional IP to your service (DHCP or static, it's your choice but DHCP will be cheaper. Although they may not hand out statics for residential users.) If you go this route make sure you run a software firewall.
2. Get a switch and can connect everything there. Again, same as the second half of one. Even a simple five port Netgear ProSafe switch from Staples should do the job.
3. Buy a router and connect everything to the appropriate ports. Assuming you're not going to want wireless down the road, might be easier to buy a wired only router. The one IP address you're already paying for will be assigned automatically to the WAN port. Your PCs will connect to one of the four switch ports (LAN ports). They get their own LAN side IP addresses via the DHCP server on the router. With this set up you can share resources internally and give Internet access to both computers.
One more important thing...don't forget to power cycle that modem prior to connecting the new router to it.