Ok I see now whats happening. The radius server is seeing all the remote pppoe servers coming in as 10.11.13.1, and not their unique ips as I have configured in in the router setup in usermanager.
For instance, the radius server should see one of my pppoe servers talking to it on 10.12.40.253, but instead it sees it coming in at 10.11.13.1. As soon as I change this ip in radius, they all connect.
So how do I go about this? I am sure its just a route to add somewhere?