posted connect the modem directly to the NID test jack. If stats are different then when connected normally something within your residence is degrading speed.
How are you determining speed? To see if problem is with the DSL circuit check modem stats. Check Sync rate and margin.
If those look good try doing traceroute (tracert in Windows) to stable sites like this one. Latency should slowly increase with distance and hop count. Sudden unexplained jump usually indicates a problem with that or the previous hop. Your ISP may be experiencing congestion somewhere within their network that has nothing to do with the physical DSL circuit.