Unix and Linux → dually homed routing

I am dually homed for the moment, and am transitioning from one ISP to another. I have not flipped the main routing table's default route out the new ISP yet, but that can be done any time. The problem is, while transitioning services, it'd be desirable to have packet replies to go out the same interface (with the same source address) that they came in. This is because both ISPs are "residential class" accounts, and as such have reverse path filtering on them (e.g., packets going out to ISP "B" will not go further than ISP B's router if they have ISP A's source address). I have worked with a similar scenario on "business class" accounts, but the ISPs in question did not have RPF, presumably because they realized asymmetric routing was not only possible but sometimes necessary, and I think the presumption was that businesses generally know more what they're doing w/r/t network engineering than most home folks.

I already have some policy routing in place for SMTP. I have an iptables rule which looks for the requisite packets, marks them with a firewall mark, a routing rule in place based on fwmark which selects a routing table, and that routing table specifying the device/nexthop IPv4 address. That is implemented with a GRE tunnel to my good friend's Linode, thereby sidestepping a lot of antispam blocklists for being on a residential-class network. So I at least have a general concept on how it might be possible.

I'm thinking something similar should be possible with my addressing problem, but I'm at a loss of what iptables tests/matches to apply. I'm looking at the conntrack match, which would seem to apply. What's not crystal clear is whether the match should be --ctorigsrc or --ctreplsrc, or if one of the other matches would work better.

Any suggestions, or does anyone here have any experience they could lend me?

Here's what I do. $GW1 and $GW2 are the gateways you need for the two ISPs, and $IP1 and $IP2 are your local IP addresses for each respective ISP.


ip route add table 10 default via $GW1
ip route add table 11 default via $GW2
ip route del table main default
ip rule add prio 33000 from $IP1 lookup 10
ip rule add prio 33001 from $IP2 lookup 11
ip rule add prio 33002 lookup 10


This:
* makes two routing tables consisting only of default routes to each ISP
* deletes the default route from the main routing table
* sets default route targets that varies depending on source IPs
* selects ISP #1 as the default, used for undecided sources and NAT events

The issue is that if you are doing NAT then the 'ip rule add' IP matches need to be done based on LAN IP addresses and this system breaks down a bit. Basically port forwarding (DNAT rules) become problematic.