dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
5370

Mersault
join:2007-10-26
Toronto, ON

Mersault

Member

FreeBSD and Multilink PPP

Anyone else besides me using FreeBSD for Multilink PPP? After posting a link to my own configuration in the 'Everything MLPPP' thread I decided to actually put my connection through it's paces and I found it a little underwhelming. I haven't tuned this connection since I moved and radically altered my configuration (including the move to the new dedicated hardware) so I figure it might be time for another FreeBSD + MLPPP discussion.

My config can be found here.

I'm particularly interested in knowing how the new hardware deals with packet splitting (I remember it used to packet split in one direction only), since that would have an effect on the MTU/MRU values that should be used.
rizlo100
join:2009-05-05
Aurora, ON

rizlo100

Member

I've been running single link MLPPP at home since getting TSI back in May. Works great!

FreeBSD 7.2
mpd5 5.3 (from ports tree)
pf

My network looks something like this...
DSL Modem
|
Switch
|
Vlan Trunk
|
Switch
|
Router
| | |
Rest of network.

The switches are simply due to the modem and router being in physically different locations so a dedicated VLAN bridges the gap. The router does a NAT out the interface facing TSI for a 192.x subnet. It also has a 3rd interface for a /29 from TSI.

My MPD5 conf is as follows

default:
 
        # configure the web server
        set user **** ***** admin
        set web self 192.168.X.1 5006
        set web open
 
      create bundle static B1
      set iface route default
      set ipcp ranges 0.0.0.0/0 0.0.0.0/0
      set ipcp enable req-pri-dns
      set ipcp enable req-sec-dns
      set ipcp disable vjcomp
      set iface enable tcpmssfix
 
      set bundle disable round-robin
      set bundle disable bw-manage
      set bundle links L1 L2
      set iface mtu 1486
      set iface disable on-demand
 
      create link static L1 pppoe
      set auth authname *******@wiredhighspeed.com
      set auth password ************
      set link max-redial 0
      set link keep-alive 10 60
      set pppoe iface fxp0
      set pppoe service "teksavvy"
      set link enable multilink
      set link enable shortseq
      set link disable protocomp
      set link mrru 1492
      set link mru 1486
      set link mtu 1486
      set link bandwidth 5056000
      set link action bundle B1
      open
      
      create link static L2 pppoe
      set auth authname *******@wiredhighspeed.com
      set auth password ****************
      set link max-redial 0
      set link keep-alive 10 60
      set pppoe iface fxp0
      set pppoe service "teksavvy"
      set link enable multilink
      set link enable shortseq
      set link disable protocomp
      set link mrru 1492
      set link mru 1486
      set link mtu 1486
      set link bandwidth 5056000
      set link action bundle B1
      open
 

This does packet splitting in both directions. I've also used the same config (modified a bit) on 2/3/4 line MLPPP setups. I'd be happy to post rc.conf/pf.conf/etc for anyone that'll find it use full.

-Riz
TheInterTubes.ca

JenSuisUn
Premium Member
join:2006-02-23
Chatham, ON

JenSuisUn

Premium Member

said by rizlo100:

I'd be happy to post rc.conf/pf.conf/etc for anyone that'll find it use full.

That would be great.
Thanks
rizlo100
join:2009-05-05
Aurora, ON

rizlo100

Member

Here you go. Ensure when installing bsd you enable pf and relevant modules in the kernel.

fxp0 - modem
fxp1 - /29 from TSI
fxp2 - NAT internal net

For rc.conf (you can ignore the ipv6 stuff)
gateway_enable="YES"
 
#Stuff removed#
 
## Networking Stuff
#defaultrouter="192.168.210.1"
ifconfig_fxp0="inet 192.168.1.1 netmask 255.255.255.0"
ifconfig_fxp2="inet 192.168.xxx.1 netmask 255.255.255.0"
ifconfig_fxp1="inet 75.119.xxx.xxx netmask 255.255.255.248"
 
## IPv6 Stuff
ipv6_enable="YES"
ipv6_gateway_enable="YES"
gif_interfaces="gif0"
gifconfig_gif0="206.248.xxx.xxx 216.66.xxx.xxx"
ipv6_ifconfig_gif0="2001:470:xxx:xxx::2"
ipv6_defaultrouter="2001:470:xxx:xxx::1"
ipv6_ifconfig_fxp1="2001:470:xxx:xxx:1::1/96"
 
mpd_enable="YES"
mpd_flags="-b -s mpd5"
pf_enable="YES"
pf_rules="/etc/pf.conf"
 

I'm listing a fairly full pf.conf mainly so those that are not familiar with it can get an idea of a full rule set. It's not perfect but a good start!

# PF Rules for edge1.theintertubes.ca
#
# PFLOGD is running as well
# To look at the general log:
# tcpdump -n -e -ttt -r /var/log/pflog
#
# To look at specific traffic for a while:
# See man pflogd for details
# pflogd -s 1600 -f /home/*****/suspicious.log port <port#>
#
# To look at the logs realtime:
# See man pflogd for details
# tcpdump -n -e -ttt -i pflog0
#
# Things to remember:
# Required order: options, normalization, queueing, translation, filtering.
# Macros and tables may be defined and used anywhere.
# Note that translation rules are first match while filter rules are last match.
 
#macros:
ext_if="ng0"
ext_ip="206.248.xxx.xxx"
 
modem_if="fxp0"
modem_ip="192.168.1.1"
 
pub_if="fxp1"
pub_ip="75.119.xxx.xxx"
 
int_if="fxp2"
int_ip="192.168.xxx.xxx"
 
ip6_extif="gif0"
ip6_extip="2001:470:xxx:xxx::2"
pub_net="{ 75.119.xxx.xxx/29 }"
int_net="{ 192.168.xxx.xxx/24 }"
 
local_allowports="{ 20, 21, 22, 123, 161, 162 }"
 
# Tables:
# Private IP address blocks - Not allowed (Used on externel interface)
table <private> const { 10.0.0.0/8, 127.0.0.0/8, 128.0.0.0/16, 172.16.0.0/12, 169.254.0.0/16, 191.255.0.0/16, 223.255.255.0/24, 240.0.0.0/4 }
 
# Blacklist Tables
table <blacklist> persist file "/etc/pftables/blacklist"
 
## Options:
##Default values are commented for reference
#set timeout { interval 10, frag 30 }
#set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
 
# Due to how VPN behaves changed established to maxium of 6 hours
set timeout { tcp.first 120, tcp.opening 30, tcp.established 21600 }
 
#set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
# Adjusting the closing and finwait up because of Jabber
set timeout { tcp.closing 1000, tcp.finwait 60, tcp.closed 90 }
 
#set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
#set timeout { icmp.first 20, icmp.error 10 }
#set timeout { other.first 60, other.single 30, other.multiple 60 }
#set timeout { adaptive.start 0, adaptive.end 0 }
#set limit { states 10000, frags 5000 }
 
# Can view packet and byte count stats with pfctl -s info
## THIS BREAKS APPARENTLY
#set loginterface ng0
 
# Just using optimization normal, no need to adjust for my usage
set optimization normal
 
# Setting a default block policy to just silently drop the packet
# Specific block rules below will be nice and send a TCP RST
set block-policy drop
 
# Binding to a interface for keep state rules
set state-policy if-bound
 
# Ensuring the order of statements is followed
set require-order yes
 
# Reference to os fingerprints
set fingerprints "/etc/pf.os"
#
# Prevent PF from processing anything on the loopback interface
set skip on lo0
 
# Normalization:
# reassemble fragments and resolve or reduce traffic ambiguities
scrub in on $ext_if all fragment reassemble
 
## Queueing
 
## NAT
no nat on $ext_if from $pub_if to any
no nat on $pub_if from $ext_if to any
no nat on $pub_if from $int_net to any
no nat on $int_if from $pub_net to any
 
#nat on $ext_if from 192.168.xxx.xxx/32 to any -> $ext_ip
 
# General nat rule for internal nat out 
nat on $ext_if from $int_net to any -> $ext_ip
 
# Nat to allow access to modem from internet net
nat on $modem_if from $int_net to any -> $modem_ip
 
#Redirects
#Torrent port forward
rdr on $ext_if proto { tcp, udp } from any to $ext_ip port 25999:27000 -> 192.168.xxx.xxx port 26000
#
#
# Filtering
# Anti-Spoofing rules
antispoof for lo0
antispoof for fxp0 inet
antispoof for fxp1 inet
antispoof for fxp2 inet
antispoof for ng0 inet
 
#
# External interface ng0 - MLPP connection
#
block return log on $ext_if all
block in log from no-route to any
block in quick on $ext_if from any to 224.0.0.1
block in log quick on $ext_if from any to 255.255.255.255
block in quick on $ext_if from <blacklist> to any label "TBL:Blacklist"
pass on $ext_if inet proto icmp all icmp-type 8 code 0 keep state
pass in on $ext_if proto { tcp, udp, icmp } from any to $pub_net
pass in quick on $ext_if proto tcp from any to $ext_ip port 22 flags S/SAFR keep state label "$if-main:pass $proto $dst port"
pass in quick on $ext_if proto { tcp, udp } from any to 192.168.xxx.xxx port 26000:27000 label "RDR: Torrent"
pass in on $ext_if from 216.66.xxx.xxx to any
 
#IPv6
pass in quick proto icmp6 all
pass out quick proto icmp6 all
pass out on $ext_if inet6 proto { tcp, udp } from any to any
pass in on $ext_if inet6 proto { tcp, udp } from any to any
 
# Trusting all traffic outbound on $ext_if
pass out on $ext_if proto { tcp, udp, icmp } all keep state label "$if-main:pass-out $proto"
 
pass out on $ext_if from any to 216.66.xxx.xxx
 
#
# Public Interface fxp1 - 75.119.xxx.xxx
#
 
block on $pub_if all
pass on $pub_if inet proto icmp all
 
# $pub_if IN
block in log quick on $pub_if from any to 255.255.255.255
block in log quick on $pub_if from any to 224.0.0.1
block in log quick on $pub_if from <blacklist> to any label "TBL: Blacklist"
pass in on $pub_if from $pub_net to any
pass in on $pub_if from $int_net to $pub_net
pass in on $pub_if from any to $pub_net
pass in on $pub_if from $ext_ip to $pub_net keep state
pass in quick on $pub_if from $pub_net to $int_net
 
# $pub_if OUT
block out quick on $pub_if from any to 255.255.255.255
block out quick on $pub_if from any to 224.0.0.1
pass out on $pub_if from any to $pub_net
pass out on $pub_if proto { tcp, udp, icmp } from $pub_ip to any keep state
pass out quick on $pub_if from $int_net to $pub_net keep state
 
# IPv6 Stuff
pass out on $pub_if inet6 all
pass in on $pub_if inet6 all
 
#
# Internal Interface fxp 2 - 192.168.xxx.xxx
#
 
block log on $int_if all
pass on $int_if inet proto icmp all
 
# $int_if IN
block in quick on $int_if from any to 224.0.0.1
pass in quick on $int_if proto { tcp, udp, icmp } from $int_net to $pub_net keep state
pass in quick on $int_if proto { tcp, udp, icmp } from $int_net to $ext_ip keep state
pass in quick on $int_if proto { tcp, udp, icmp } from $int_net to $int_ip keep state
pass in quick on $int_if proto { tcp, udp, icmp } from $pub_net to $int_net keep state
pass in quick on $int_if proto { tcp, udp, icmp } from $int_net to any
pass in quick on $int_if proto { tcp, udp } from any to 192.168.xxx.xxx port 26000 label "RDR: Torrent"
 
# $int_if OUT
pass out quick on $int_if proto { tcp, udp, icmp } from $int_net to any
pass out quick on $int_if proto { tcp, udp, icmp } from $pub_net to $int_net keep state
pass out quick on $int_if proto { tcp, udp } from any to 192.168.xxx.xxx port 26000 label "RDR: Torrent"
 
# IPv6 Stuff
pass out on $int_if inet6 all
pass in on $int_if inet6 all
 
#
# Modem facing interface fxp0 - 192.168.1.1
#
 
pass on $modem_if all
pass on $modem_if inet proto icmp all
 
# $modem_int IN
pass in quick on $modem_if from 192.168.1.254 keep state
block in quick on $modem_if from $pub_ip
pass in quick on $modem_if from $int_net keep state
# $modem_int OUT
pass out quick on $modem_if from $int_net to any keep state
pass out quick on $modem_if from $modem_ip to any keep state
 
#
# IPv6 tunnel interface gif0 - 2001:470:xxx:xxx::xxx
#
 
set skip on gif0
 

That should about cover it!

-Riz
TheInterTubes.ca

Mersault
join:2007-10-26
Toronto, ON

Mersault to rizlo100

Member

to rizlo100
Thanks for posting this mpd.conf. It's the same one I used to use, though I haven't used MPD for a while because I bring up a connection to another DSL provider as well and Userland PPP makes that easy (you just specify more than one connection name in your /etc/rc.conf file). Good to know it still works.

I didn't have time to pay much attention when the switch to the wiredhighspeed realm happened; the new hardware supports packet splitting in both directions, correct? If I'm not mistaken, this should allow an MLPPP connection comprised of two or more tunnels to support 1500 byte packets in both directions, since each tunnel only has to deal with half of the packet (total packet size being 750 byte half packet, 6 byte MLPPP header, and 8 byte regular PPP header for a total packet size of 764 bytes down each link)? Or am I missing something?
Mersault

Mersault

Member

If anyone is interested in another pf.conf configuration example, I've posted an annotated copy of my pf.conf file here.
Hoeser9
join:2004-12-03
Tilbury, ON

Hoeser9 to Mersault

Member

to Mersault
Sorry to resurrect a dead thread, but would you be willing to post your mpd5.conf and other appropriate files for a 3-line setup? Trying to fine tune my 3-line to Teksavvy has been fruitless.

Thanks!

Mersault
join:2007-10-26
Toronto, ON

Mersault

Member

Here's my mpd.conf file. To add another line, just add another stanza for the third link. You can tinker with the MTU values if you like, as I don't know if these are perfectly optimized. Let me know if these settings work for you though, I've been using Userland PPP for the last little while. Specifically, let me know how it does at reconnecting after network issues. I've noticed that occassionally Userland PPP is not particularly good at that.

startup:
        configure mpd users
        set user admin admin admin
        # configure the console
        set console self 127.0.0.1 5005
        set console open
        # configure the web server
        #set web self 0.0.0.0 5006
        #set web open
 
default:
        create bundle static B1
        # set iface route default
        set ipcp ranges 0.0.0.0/0 0.0.0.0/0
        # set ipcp enable req-pri-dns
        # set ipcp enable req-sec-dns
        set ipcp disable vjcomp
 
        set bundle disable round-robin
        set bundle disable bw-manage
        set bundle links L1 L2
        set iface mtu 1486
        set iface disable on-demand
 
        create link static L1 pppoe
        set auth authname username@wiredhighseed.com
        set auth password password
        set link max-redial 0
        set link keep-alive 10 60
        set pppoe iface em1
        set pppoe service "teksavvy"
        set link enable multilink
        set link enable shortseq
        set link disable protocomp
        set link mrru 1592
        set link mru 1486
        set link mtu 1486
        set link bandwidth 5056000
        set link action bundle B1
        open
 
        create link static L2 pppoe
        set auth authname username@wiredhighspeed.com
        set auth password password
        set link max-redial 0
        set link keep-alive 10 60
        set pppoe iface em2
        set pppoe service "teksavvy"
        set link enable multilink
        set link enable shortseq
        set link disable protocomp
        set link mrru 1592
        set link mru 1486
        set link mtu 1486
        set link bandwidth 5056000
        set link action bundle B1
        open
 
        set iface enable tcpmssfix
 
Hoeser9
join:2004-12-03
Tilbury, ON

Hoeser9

Member

Mersault:

That is basically more or less the setup I have now, what I am looking for are the optimal MTU/MRU/MRRU values for the connection. 2 connections work great, but 3 connections are less than optimal it seems (maxing out around 1.35-1.4MB/sec instead of the expected 1.5). What is odd is that at 1.35-1.4MB/sec for the bundle the 3 individual links are going at 515KB/sec which suggests to me the MRRU may not be optimal, or something, what do I know. I'm no networking guru.

However I can tell you how MPD preforms when links go down. In my experience it handles it very well - I have tested it quite a few times. There are some breif "outages" though. Unplugging one modem's phone cable seems to result in an outage for a few minutes - until it brings the link back up or it detects the link as failed, seems to never take longer than 2 or 3 minutes. I've had it running on two links with one link down due to an ethernet cable not being firmly seated. Once it detects the 3rd link is down it seems to work fine. I've even gone so far as to totally power off my modems, disconnect them and move them (reorganizing my desk) and plug them back in and power them up without telling MPD to shut down. They came back up fine. I've never run into a scenario where they have not reconnected and resumed operation even when it is an outage on the peer end, which has happend to me a few times also.

I would suggest mpd over userland ppp any day of the week.

Hope this helps.

derekm
join:2008-02-26

2 edits

derekm to Mersault

Member

to Mersault
If I can throw my $0.02 in:

- it might be worthwhile downing the link mtu/mru to 1482, as MLPPP adds 4 bytes overhead. You could also try 1470 so that the packets are ATM-aligned (I calculated this by floor(1500/48)*48 - 18, maybe that isn't correct)

- I would be tempted to try mrru set to 1500, as well, since the packets will be reassembled, and can (hopefully) avoid IP-fragmentation

- Testing (latency, jitter, bandwidth) while varying these parameters is essential. Posting results would be helpful.

When I was playing around with MPD, I found the jitter/latency to be all over the place, and never really got it resolved. I plan on revisiting this again sometime.

Edit: I'm afraid the numbers in the first point are hopelessly wrong. I'd have to go back and look at the RFCs, or you could do a tcpdump on a physical interface to see what overhead the PPP stuff is causing. It looks like there is a header and footer for PPP, but just a header for MLPPP (RFC1990). I think you could change the formula to: floor(1500/48)*48 - PPPOverhead

Mersault
join:2007-10-26
Toronto, ON

Mersault

Member

DSLRicerDSLRicer[/user] did a lot of work with MTU and similar values when working with the Linux side of things. He worked out a formula that provided optimal values for however many lines were in the setup. Also, and please correct me if I'm wrong, but hasn't bell been busy replacing their ATM infrastructure with GigE infrastructure?

Your calculations actually pretty close, you just got the overhead values wrong. PPP is 8 bytes of overhead, so the MTU for plain DSL has to be 1492. MLPPP is an *additional* 6 bytes of overhead, for a total of 14 bytes. So for MLPPP MTU has to be 1486.

If you want to optimize for an ATM backhail, remember that ATM is a 48 byte payload with a 5 byte header, for a total ATM cell of 53 bytes. Ethernet packets of 1488 or 1440 bytes will align well with ATM. Accounting for MLPPP overhead (which is 14 bytes), that leaves an MTU value of 1474 or 1426.

However, I believe that I am missing something in my calculations. And for that, I'd ask DSLRicer to explain it in detail.
alpovs
join:2009-08-08

alpovs to Mersault

Member

to Mersault
I can suggest to use Netalyzr »netalyzr.icsi.berkeley.edu/ to diagnose any problems and adjust MTU/MRU/MRRU accordingly.

Guspaz
Guspaz
MVM
join:2001-11-05
Montreal, QC

Guspaz to Mersault

MVM

to Mersault
DSL_Ricer's calculations all take ATM into account. The calculations would be far simpler if this weren't all going over ATM and its annoyingly minescule cell size.

The thing you want to avoid is having your MTU/MRRU/etc at a value such that you have an ATM cell that only has one single byte of actual data; you'd be wasting 52 bytes of data per PPPoE packet.