Yes, when speeds drop starting in the late afternoon or evening, most likely it's a sign of node congestion.
Another less likely possibility though, is that some streetlight or piece of equipment in the neighborhood turns on at that time of evening, causing/injecting electrical noise into the cable system and causing problems then.
If it only happens on sunny/warm days, it could be that the reduction of heating sunshine or the drop in temperature is affecting a piece of line equipment.
As far as the IPv6 speed test being slower: »
[IPv6] any idea why IPv6 speedtests less than IPv4?Though usually the speed difference isn't normally that much.