deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| FreeBSD traffic shaping - finding the right scheduler
I'm having a tough time making ALTQ do what I want, so I figured I'd describe the characteristics of what I'd like the traffic queuing/shaping scheduler to do and see if any of the ALTQ schedulers will do, or if ipfw+dummynet will work.
A bit of background on the connection and my circumstances:
- I currently have the FiOS 20/20 symmetric business plan
- The router is a FreeBSD 7.2 box with plenty of horsepower
- I work from home 99.9% of the time and my wife stays at home with the kids (meaning, there's some web/mail traffic throughout the day, though it's relatively light with an occasional large upload to facebook/onetruemedia/etc)
- There are no other routing/firewall devices between the FreeBSD box and the FiOS, it handles all NAT/firewalling/etc for my network
- I have a friend who uses the box to do torrenting on occasion, and I use it for that as well, depending if there are any new Linux/BSD releases I want to seed
All that laid out, what I'd like to do is implement the following (in order of precedence):
- VPN traffic is #1 priority and can consume ALL available bandwidth, starving the rest if necessary, since I work from home
- ACKs are #2 priority (to avoid ACK starvation, though it's not as much of an issue with a symmetrical connection)
- Torrent traffic along with everything else is #3 priority, but if my friend is downloading and seeding via torrent, I'd like to have my packets have priority over his to the extent that I get 3/4 of the bandwidth and he gets 1/4 (or perhaps 2/3 to 1/3, the ratio is something I can tune later)
The 3rd item above is where I'm struggling and where I'm hoping someone can suggest a scheduler and/or more in-depth docs for how to proceed.
I've tried to use ALTQ's priq scheduler, and in simulating both of us seeding at full speeds at the same time (by scp'ing from my account and his at the same time), what ends up happening is his traffic is throttled almost to the point where it's useless and the connection stalls.
The ALTQ cbq scheduler seems to work well if I only define two queues and assign my friend's traffic to the queue with 25% of the bandwidth. The problem comes in when I try to make sure I reserve bandwidth for ACKs, since I then have to "mooch" some bandwidth from either my queue or his and lower the bandwidth value/%.
I haven't looked into too much depth at the hfsc scheduler, so perhaps that's what I want. Ultimately, it seems I need something that provides bandwidth reservation/segregation, but also prioritization to avoid ACK starvation and to allow my VPN tunnel to have #1 priority.
Any thoughts? (sorry, this turned into a long winded question!)
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| Hmm if only hfsc had "borrow", I think it'd do what I want...As it is, it seems I'd be stuck with a similar situation to what I have with cbq in terms of having to "steal" from one of the two queues (main queue vs. the queue versus my friend's queue).
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| reply to deblin
And oops, I forgot to mention that ideally I would want to have the 3/4-1/4 rule for competing traffic between my friend and the rest to also apply for INBOUND traffic.
As I understand it, ALTQ only works egress, right? So I'm not sure I'll be able to accomplish an inbound split with ALTQ, unless I misunderstood its capabilities. If it does support that, I'm all ears, but the simple fact that only one "altq on" rule can be applied per interface implies it's egress only.
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
metrodust
Hey Thats Mine
join:1999-12-10
Seattle, WA
| reply to deblin
I'm using pfSense as my firewall and did some digging in their forums on this. Sadly the interface for pfSense will only let you prioritize one WAN link, but I did run across this:
»forum.pfsense.org/index.php/topi···6.0.html
from the looks of it, you're on the right track with hfsc
--
When you are leaving.. heaven is a distance not a place. --Carissas Weird |
|
koitsu
Premium
join:2002-07-16
Mountain View, CA
| reply to deblin
deblin  , you might consider dropping Max Laier (maintainer of pf(4)) an Email asking for some assistance or clarification. mlaier@freebsd.org
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer. |
|
TakeTheFifth
join:2004-04-20
Anjou, QC
| reply to deblin
said by deblin :As I understand it, ALTQ only works egress, right? So I'm not sure I'll be able to accomplish an inbound split with ALTQ, unless I misunderstood its capabilities. If it does support that, I'm all ears, but the simple fact that only one "altq on" rule can be applied per interface implies it's egress only. Apparently, you can assign traffic incoming to a queue defined on another interface. Is this what you meant ?
Phil |
|
sporkme
drop the crantini and move it, sister
Premium,MVM
join:2000-07-01
Morristown, NJ
 ·Optimum Online
| said by TakeTheFifth :said by deblin :As I understand it, ALTQ only works egress, right? So I'm not sure I'll be able to accomplish an inbound split with ALTQ, unless I misunderstood its capabilities. If it does support that, I'm all ears, but the simple fact that only one "altq on" rule can be applied per interface implies it's egress only. Apparently, you can assign traffic incoming to a queue defined on another interface. Is this what you meant ? Yeah, technically, the inbound traffic should be shaped at the provider side. Obviously most of us can't do that. However, the way TCP works, if you do throttle the inbound, TCP will see the packet drops and adjust it's speed down. For UDP you're basically out of luck, but I can't think of too many multi-megait apps that use UDP for transport.
--
with every mistake we must surely be learning |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| reply to TakeTheFifth
You can use the internal interface to queue for inbound, but that obviously only works for traffic being NAT'd :) In this case, the traffic I'm wanting to queue is originating on the FreeBSD box itself.
So, I read a bit more into hfsc, and it does seem to do what I want, and it's simpler than I had anticipated (for my particular case):
Of course, later I have traffic for my friend's UID assigned to the p2p queue.
What the above does (as I understand it) in english is:
- reserve 5000 Kbit/s for the p2p queue (note: priority 1 is the default, I put it in there for clarity)
- reserve 15480 Kbit/s for the remainder of the traffic and give it a higher priority
- The "rest" queue is guaranteed realtime 15480 Kbit/s
At this point, I think all I need to do is add some sub-queues to "rest" and prioritize things as I see fit (e.g. VPN highest priority, etc).
So far, the above seems to work well. If I scp a file from the limited account, it gets the full 20480 Kbit/s. If I then scp from my account, the limited user's account is queued down to 5000 Kbit/s and my user gets ~15000 Kbit/s or so.
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| Can't fully test this at the moment, as I'm too lazy to boot up the work laptop :) But I'll give it a go tomorrow and see if the responsiveness of my VNC session over the VPN is more responsive.
Here's what I've got now:
I figure I'll try it this way first, and if I need to tweak the vpn queue to use m1/d/m2 for guaranteed bandwidth in a time slice, I'll give that a go. I think this should give me the desired behavior, though. :)
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
TakeTheFifth
join:2004-04-20
Anjou, QC
| said by deblin :I figure I'll try it this way first, and if I need to tweak the vpn queue to use m1/d/m2 for guaranteed bandwidth in a time slice, I'll give that a go. I think this should give me the desired behavior, though.  Nice!
Phil |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| Well, no such luck, but I think it's because of the downstream and the limitations of queuing that.
I tried the above, and if I max out my downstream, pings to a Linux box on the other side of the VPN are up in the ~260ms range. Even with the queuing enabled.
I tried making the vpn queue separate (e.g. child of the root queue) like this:
But the pings are still ~260ms and my VNC responsiveness still goes to hell over the VPN.
I may have to try a solution using pf/altq AND ipfw/dummynet, though it seems like a nasty hack. Basically I can:
- use pf/altq with priq to prioritize VPN traffic over all else
- use ipfw+dummynet to manually throttle my friend's account if/when he's saturating it
Not ideal, but it'd work I think.
I'm still going to play around a bit more with hfsc to see if I can improve the responsiveness over the VPN, but I think it's a simple case of downstream saturation affecting upstream (the opposite of what most run into). :)
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
deblin
Dark Side of the Moon
Premium,MVM
join:2001-09-01
Middletown, DE
| I gave ipfw a shot and while it seems to work well for the upstream side (giving myself 3/4 of the bandwidth), the smooth 2.46 MB/s downloads I normally get from a single fetch/wget jumps all over the place and I average about 2.1 MB/s. I'm not really ok with the 12-13% performance hit, but I realize trying to manage downstream queuing is problematic at best.
Here are the ipfw/dummynet rules I tried:
For the upstream, if I scp from both accounts, I get a nice 3:1 ratio as I'd expect. Downstream, sort of, but again the downstream for a single download is erratic. I guess that's just the price you pay for trying to do queuing on the ingress and not uproute (e.g. @ the ISP level).
--
He who is not contented with what he has, would not be contented with what he would like to have. -Socrates |
|
TakeTheFifth
join:2004-04-20
Anjou, QC
1 edit | said by deblin :I I guess that's just the price you pay for trying to do queuing on the ingress and not uproute (e.g. @ the ISP level). The downstream is a royal pain. Not much control over how much junk is sent your way (even if you control the ACKs you send back). I wonder how much control you would get if you prioritized egress ACKs depending on the account (along with your 3/1 bandwidth settings) ? Also, I was wondering if you dropped packets in your queues (edit: I noticed you set them @ 500).
Phil |
|
Bink
join:2006-05-14
Denver, CO
 ·Qwest.net
| reply to deblin
FWIW, I use pf and altq (on OpenBSD) quite successfully at home and it meets my needs very well. However, my connection is quite asymmetric is yours is not. Originally I used priq very successfully, but when I decided to open a portion of my Internet connection to guest Wi-Fi clients, I needed to move to cbq. As for hfsc, I’ve never been able to wrap my head around it properly—there’s not enough good documentation on it.
The solution works very well for me—I can fully saturate my outbound without, generally, adding more than 15ms of latency. Torrents caused me some minor trouble initially—because I have one public IP. As result, I moved my torrenting to the BSD box itself (tmux + rtorrent is a godsend), created a UID just for torrenting and leveraged pf’s UID capabilities to completely throttle this stuff.
If you chose to move to pf + altq + cbq, I should be able to assist.
Cheers. |
|
