Interesting, whatever is going on in Nashua does not seem to be affecting Milford. The fact that download is so poor while upload is fine sure sounds like congestion.
Try doing a traceroute (tracert in Windows) and compare it to the one I posted earlier. I assume routing will be the same for both of us except perhaps the FairPoint edge router (10.20.12.1 in my case). I think the first generation FIOS FairPoint inherited from Verizon uses ATM just like DSL.
My guess is you will see high latency to the edge router. If so problem is backhaul congestion over the ATM circuit. If 2nd hop is OK (I'm assuming you have a LAN) then you are being routed differently then me. In that case problem is somewhere within the FairPoint IP network or perhaps peering.
Be interesting to see the result. Not sure how that will help getting it resolved but at least you will know where the problem is.
Perhaps the Nashua Telegraph will be interested in writing about the problem.