Your symptoms usually mean that both of you are suffering from network congestion problems. Your DSLAM may be oversubscribed. The only permanent fix is for Verizon to add/upgrade equipment on their end, which they probably won't do because they have been trying to get out of copper-based broadband for a long time.
You can get an idea of where the problem is by running the line quality test from this site when you have the slowdown and when you don't: »/pingtest You need to sign up to run the test.
If you want to post your results here, be sure to use the results that block out your ip address.
Things that you might also want to do is go into your modem's GUI and find the speed at which you're syncing and find out what your SNR (signal to noise ratio) is and what your attenuation is.