The load on the caps is the voltage regulator, and while that's not a perfect isolator, it's a actually a fairly good one.
As per »
www.fairchildsemi.com/ds ··· M317.pdf the typical ripple rejection @120 Hz is between 60 - 75 db, and while that doesn't say anything about other frequencies, I'd expect to see comparable numbers over 100 Hz.
Furthermore, take a look at his preamp at: »
tech.juaneda.com/en/proj ··· mp3.htmlMy analog is a little rusty, but looks to me that thing is operating in class A so there won't be a whole lot of audio going out to the caps on the Vcc line.
He does make some very interesting design choices like bypassing the voltage regulator with a 220 ohm resistor. And the input section (50k pot, 1M, 33.2k, 6.2k) is a rather significant voltage divider adding some extra noise.