houkouonchi
join:2002-07-22
Corona, CA
clubs:
 ·AT&T U-Verse
 ·DSL EXTREME
·OCN
·Pacific Bell - SBC
 ·Charter Pipeline
| [LARTC] Destination based port routing (multi gateway/wan)
I am having problem setting up destination port routing on a system with 4 connections to the internet (eth0,eth1,br0(eth2),eth3). I am not sure what I am doing wrong. I am trying to make it so whenever the machine tries to connect to something on port 22, 119, or 2710 it goes through a specific interface/IP. Here is how I am setting up the links:
I am then setting up the iptable markings via:
I then add the following rule for ip route to route through 'line1':
I have also tried 'all' instead of 0.0.0.0/0
Unfortunately the iptables rules and the ip rule doesn't seem to be doing anything. SSH'ing from this box to another one on the internet randomly goes through eth0, br0, or eth3 =(
Anyone know what I am missing or what might be causing this?
--
Chugging along on 2x 6016/768k DSL Extreme DSL lines and one 6016/768 ATT DSL DIrect line as well as one 16mb/2mb Charter cable line for a combined total of just over 32 meg download and 3 meg up (after overhead). yay! |
|
Sir Meowmix III
@windstream.net
 from:
houkouonchi 
| Why not setup a DNAT to egress based on the IP you want? I do this with a CIDR 30 allocation where I egress ! TCP 25 on the second IP? |
|
Sir Meowmix III
@windstream.net
from:
houkouonchi
| reply to houkouonchi
I said DNAT, I meant SNAT.
|
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| This is *not* a NAT envrinoment. This box has 4 nics. One of the nics is bridged with a virtual interface which is why one is br0 but all 4 NICs have public WAN ips. There is no NAT subnet or any computers behind this box.
Because of this I don't believe what you posted will work (unless I am mis-understanding what you are suggesting).
--
Chugging along on 2x 6016/768k DSL Extreme DSL lines and one 6016/768 ATT DSL DIrect line as well as one 16mb/2mb Charter cable line for a combined total of just over 32 meg download and 3 meg up (after overhead). yay! |
|
Sir Meowmix III
@windstream.net
from:
houkouonchi
| reply to houkouonchi
It uses the NAT chain, you don't need to have a NAT (in the sense you're thinking), just specify the CIDR 32 source for the ip bound for the interface which you want to route the protocol over the specific WAN one. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| Hmm, well I guess I am a bit confused of how I would set this up then:
Say I have the following interfaces with the following ip's bound
eth0: 1.1.140.21
eth1: 1.1.141.21
br0: 1.1.138.21
eth3: 1.1.143.21
These are all /24's with their gateways being x.x.x.1
Now I want to make it so anytime this machine sends a packet that has a destination port of 22 to go through eth0 (1.1.140.21). How would I do that?
--
Chugging along on 2x 6016/768k DSL Extreme DSL lines and one 6016/768 ATT DSL DIrect line as well as one 16mb/2mb Charter cable line for a combined total of just over 32 meg download and 3 meg up (after overhead). yay! |
|
Sir Meowmix III
@windstream.net
from:
houkouonchi
| reply to houkouonchi
I believe the below will work.
/usr/sbin/iptables -t nat -I POSTROUTING 1 -p tcp --dport 22 -d ! 1.1.0.0/16 -j SNAT --to 1.1.140.1
Not sure if you need to specify the DG or the IP on eth0. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| said by Sir Meowmix III :
I believe the below will work.
/usr/sbin/iptables -t nat -I POSTROUTING 1 -p tcp --dport 22 -d ! 1.1.0.0/16 -j SNAT --to 1.1.140.1
Not sure if you need to specify the DG or the IP on eth0.
I think this did what I wanted! Thanks! It looks like you actually did need the IP so it needed to be like:
/usr/sbin/iptables -t nat -I POSTROUTING 1 -p tcp --dport 22 -d ! 1.1.0.0/16 -j SNAT --to 1.1.140.21
With the actual line you used I couldn't ssh into anything at all =) After changing it to the above version it appears to be always going through eth0 for ssh which is what I wanted. Thanks!
--
Chugging along on 2x 6016/768k DSL Extreme DSL lines and one 6016/768 ATT DSL DIrect line as well as one 16mb/2mb Charter cable line for a combined total of just over 32 meg download and 3 meg up (after overhead). yay! |
|
Sir Meowmix III
@windstream.net | reply to houkouonchi
Glad to help, just modify the rule accordingly for the other services you wanted to forward over those interfaces, such as 119, 2710, etc. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| For some reason after a crash this just kind of broke itself. When I use the same rules that worked before for a port such as ssh I am now no longer able to connet to any remote machine via SSH. I have no clue why. Here is what the rules look like:
# iptables -v -n -L POSTROUTING -t nat
Chain POSTROUTING (policy ACCEPT 4359K packets, 426M bytes)
pkts bytes target prot opt in out source destination
0 0 SNAT udp -- * * 0.0.0.0/0 !1.1.0.0/16 udp dpt:2710 to:1.1.1.21
0 0 SNAT tcp -- * * 0.0.0.0/0 !1.1.0.0/16 tcp dpt:2710 to:1.1.1.21
0 0 SNAT tcp -- * * 0.0.0.0/0 !1.1.0.0/16 tcp dpt:119 to:1.1.2.21
1 60 SNAT tcp -- * * 0.0.0.0/0 !1.1.0.0/16 tcp dpt:22 to:1.1.1.21
Its even showing packets/bytes increasing when I try to ssh so it seems to be correctly getting filtered
About the only thing different about my system I can think of is I now have some ipv6 IPs but I don't even have iptables6 installed so I wouldn't think that would make a difference?
--
100mb/100mb OCN fiber connection for $50/month. YAY! |
|
mich
join:2008-08-30
| reply to houkouonchi
You don't need SNAT here, especially that it may possibly confuse some apps which send packets from one IP but are seen outside as connecting from another IP.
Your first config was good, except for two errors:
1. For locally generated packets you have to use OUTPUT instead of PREROUTING
2. You need to add src x.y.z.v to your routes
This should work as expected:
I've just tested similar configuration on my box and it worked. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| i already had the ip route additions and I do see the packets being marked from iptables -v -L -n -t mangle but it doesnt work:
Chain OUTPUT (policy ACCEPT 961M packets, 3219G bytes)
pkts bytes target prot opt in out source destination
72 9198 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 MARK set 0x42
14 774 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:21 MARK set 0x43
I added:
iptables -t mangle -A OUTPUT -p tcp --dport 21 -j MARK --set-mark 67
iptables -t mangle -A PREROUTING -p tcp --dport 21 -j MARK --set-mark 67
ip rule add fwmark 67 table line2
and here is my ip rule show:
# ip rule show
0: from all lookup local
32754: from all fwmark 0x43 lookup line2
32755: from all fwmark 0x42 lookup line2
32756: from 1.1.143.21 lookup line4
32757: from 1.1.138.21 lookup line3
32758: from 1.1.141.21 lookup line2
32759: from 1.1.140.21 lookup line1
32760: from all fwmark 0x11 lookup line1
32761: from all fwmark 0x4 lookup line1
32762: from 1.1.143.21 lookup line4
32763: from 1.1.138.21 lookup line3
32764: from 1.1.141.21 lookup line2
32765: from 1.1.140.21 lookup line1
32766: from all lookup main
32767: from all lookup default
You might be able to tell that I actually tried something similar earlier from the 0x11 and 0x4 marks. I tried with port 21 this time to test a completely new port.
--
100mb/100mb OCN fiber connection for $50/month. YAY! |
|
mich
join:2008-08-30
1 edit | reply to houkouonchi
Sorry, it seems that you are right, src doesn't seem to work for packets rerouted using fwmarks. They are sent through correct interface (so iptables and ip rule work fine) but with different IP addres. The only fix I can imagine now is adding:
iptables -t nat -A POSTROUTING -o eth0 -s wrong-ip -j SNAT --to-source good-ip
I'll post if I find something better.
/*EDIT*/
Well, I was obviously wrong 
Since kernel has to decide source address before it feeds the packet to netfilter, it is simply impossible to honor the src option in route table for fwmarked packets because before the packet gets there, it already has some address. I think that you can only SNAT those packets. |
|
mich
join:2008-08-30
| reply to houkouonchi
I don't know if you already managed to get it working again, but your routes together with Sir Meowmix' SNAT rules simply must work. Use more tcpdump :)
By the way, I've found something interesting in ssh manual:
Maybe other programs you use also have such configuration option. This would be enough if this box doesn't route external traffic. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| said by mich :I don't know if you already managed to get it working again, but your routes together with Sir Meowmix' SNAT rules simply must work. Use more tcpdump :) By the way, I've found something interesting in ssh manual: Maybe other programs you use also have such configuration option. This would be enough if this box doesn't route external traffic. To be honest SSH is not the main reason I want this for its an example. The problem is if I do load balancing and don't have the default route go through one connection then if I tunnel via my server for usenet it will end up using multiple outgoing IPs which is really bad. Without the load balancing I am limited to only one conneciton speed though so I want to have load balancing enabled and force certain connections through a specific interface.
It seems like it still worked for usenet stuff but took like 10 minutes to finally start working (before it was instant) and ssh and stuff just timeout. I don't know why it doesn't work now as I am using the exact same init script and network settings as before (minus that i have ipv6 now) but I even tried disabling ipv6 and it didn't make a difference.
I was really happy with how it worked for like a month then my server crashed and since the reboot it just doesn't work anymore (it basically kills the connection).
Unfortunately there is no way to bind to an IP/interface (as far as I am aware) for the outgoing connection when utilizing a SSH tunnel.
--
100mb/100mb OCN fiber connection for $50/month. YAY! |
|
mich
join:2008-08-30
| reply to houkouonchi
said by houkouonchi :# ip rule show 0: from all lookup local 32754: from all fwmark 0x43 lookup line2 32755: from all fwmark 0x42 lookup line2 32756: from 1.1.143.21 lookup line4 32757: from 1.1.138.21 lookup line3 32758: from 1.1.141.21 lookup line2 32759: from 1.1.140.21 lookup line1 32760: from all fwmark 0x11 lookup line1 32761: from all fwmark 0x4 lookup line1 32762: from 1.1.143.21 lookup line4 32763: from 1.1.138.21 lookup line3 32764: from 1.1.141.21 lookup line2 32765: from 1.1.140.21 lookup line1 32766: from all lookup main 32767: from all lookup default It shouldn't matter, but maybe try deleting this garbage? This and all iptables rules and routes you don't need.
IMHO this configuration must work:
The last three rules SNAT outgoing packets to correct source IP. If this box is not a router you can replace them with only one rule without "-s ..." check. If it is a router, then skipping source check would SNAT other people's traffic.
If you still won't be able to make it work, try to run tcpdump and see whats going on. Pick some port which nobody uses on this machine, setup routing for this port and launch
Then try to connect to this port on any remote host and see what happens. At least you will know if the packets leave your box at all, where do they leave and how they are addressed. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| said by mich :IMHO this configuration must work: The last three rules SNAT outgoing packets to correct source IP. If this box is not a router you can replace them with only one rule without "-s ..." check. If it is a router, then skipping source check would SNAT other people's traffic. I am confused On the SNAT rules where is the port or even packet marking listed? W How is the marking even used in any of the above. Also you have -o eth0 on all of them but each IP address has its own interface eth0/eth1/br0/eth3
--
100mb/100mb OCN fiber connection for $50/month. YAY! |
|
mich
join:2008-08-30
| It was meant to catch packets sent through eth0 with source IP of eth1, br0 or eth3 and change their source IP to IP of eth0. But you are right that using marks which we have already set is much easier and has no possibility to interfere with other traffic. So, if mark 1 means "route through eth0, which uses IP $IP1" you will need this line:
And remember that new SNAT rules don't work for already established connections so you might have to restart services which you want to manipulate. |
|
houkouonchi
join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline
| Looking at tcpdump it using a rule like this it still seems to end the packet out of the interface it would normally be sent from except the source IP has been changed to the value set to --to-source. I am guessing it did the same thing before and I am curious why it used to work and doesn't anymore =(.
--
100mb/100mb OCN fiber connection for $50/month. YAY! |
|
mich
join:2008-08-30 | reply to houkouonchi
This means that SNAT works, but routing doesn't. Make sure that you are using the same mark numbers in iptables and ip rule, check if the line1 table is correct and run ip route flush cache. |
|
