I actually found solution that works. I created loopback interfaces, then use aaa group server radius groups, set the radius server and a source interface to one of the loopbacks.
On the radius server, create clients based on the loopback interface addresses, then create policies matching on Client Friendly Name or Client Address (which are the loopbacks addresses)
aaa group server radius vpnradius
server-private 10.0.0.2 auth-port 1645 acct-port 1646 key <deleted>
ip radius source-interface Loopback0
aaa authentication login vpnclientauth group vpnradius local