Have you looked at the modem stats (usually at »
192.168.0.1/ with the login information printed on the bottom of your modem) to see if it's likely your local line, or something upstream?
Sometimes, this can be a sign of node/dslam/bras congestion, if your line is trained up at its full speed and your signal stats are all good, etc.