Just as a follow up, it started working properly mid-day Sunday, but figured I'd let them come out and check on it anyway. The guy showed up and wound up replacing the modem since it was 4.5 years since I got it, and who knows how old it was since it was new since they always give refurbs. He replaced it with an ARRIS TM822 which appears to be a docsis 3, so maybe that will help with my random peak slowdowns too? He even threw in the "you know, you can always upgrade to Boost for like $11 or $12 extra and get like 55-60 down and like 8 up", which I have thought about, but I already pay $185/mo. Throwing more of my money at CV is not exactly at the top of my to-do list.
Either way, it seems to be working fine now, so you can all sleep easy tonight not worrying about my internet anymore, lol.