You nailed it spot-on as what I am experiencing as well. It's unfortunate it is the way it is.
So what your saying is at peak times its going slow? Try paying for 50 and get 20 only over night. I think the internet in general is slow at peak times.