Hmmm. Unfortunately, it's doubtful; the fact that you've got multiple computers all knocked offline simultaneously, and all restored simultaneously when the router (only) is rebooted, pretty clearly indicates an issue with the router.
Here's something to test, perhaps: when the connection goes down, are your computers still able to see each other? Eg. could you share files from one computer to another even when the internet is not available?
If you can, then it might be the PPPoE authentication in the router is failing, and we might have some success moving it to the modem. On the other hand, if you cannot share files between the computers while the internet is down, then that really means it's the router itself; time to replace.