Update
Overall I have been pleased with the shaping solution below, but I’ve made one fairly significant change since first publishing this page. I’ve switched my DSL Modem to one of the new Westell 6000 series with the turbo tcp mode. In a nutshell this mode does the upstream ACK packet shaping for you, and it does it in the right place (the last hop out of your home network.)
I still use the same basic script as before, but I’ve rasied the uplink raw max (UPLINK=510) to well above my actual uplink (UPLINK=760) so my box may well choke the uplink pipe… if it does the westell sorts out the acks just fine. So now, I get maximum download and upload speed and still have control over how much of the uplink is consumed by each of my desired services (ssh/http/smtp.)
Original Shape Info
Since this box is my router for the home network, I can shape all outgoing traffic… this is good because if you don’t do that, a single user (or one of the server processes running on the linux box) can fill the outgoing pipe and kill the download speed for everyone.
This is how I do it. This is based on the many great traffic shaping sites on the web including Linux advanced Routing & Traffic Control HOWTO and Stef Coene’s QOS/Traffic shaping site
I’m using shorewall to set up iptables, so I can mark some packets there, I also added some rules to tag smtp and http traffic to put them in their own buckets. Here’s the script:
#!/bin/bash
#Uplink raw max
UPLINK=510
#reserve 128kb for ack packets
UPLINK_ACK=130
#reserve 12kb for ssh
UPLINK_SSH=34
#web site gets 128kb
UPLINK_WEB=128
#sendmail gets 128kb
UPLINK_MAIL=128
# everything else in 1:50
UPLINKB=90
DEV=eth0
# clean existing uplink qdiscs, hide errors
./tc qdisc del dev $DEV root 2> /dev/null > /dev/null
###### uplink
# install root HTB, point default traffic to 1:50:
./tc qdisc add dev $DEV root handle 1:0 htb default 50
# shape everything at $UPLINK speed - this prevents huge queues in your
./tc class add dev $DEV parent 1:0 classid 1:1 htb rate ${UPLINK}kbit ceil ${UPLINK}kbit burst 2k
# high prio class 1:10:
./tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK_ACK}kbit ceil ${UPLINK}kbit burst 2k prio 0
# high prio class 1:20:
./tc class add dev $DEV parent 1:1 classid 1:20 htb rate ${UPLINK_SSH}kbit ceil ${UPLINK}kbit burst 2k prio 1
# class 1:30 - web server
./tc class add dev $DEV parent 1:1 classid 1:30 htb rate ${UPLINK_WEB}kbit ceil ${UPLINK}kbit burst 2k prio 2
# class 1:40 sendmail
./tc class add dev $DEV parent 1:1 classid 1:40 htb rate ${UPLINK_MAIL}kbit ceil ${UPLINK}kbit burst 2k prio 3
# class 1:50 bulk
./tc class add dev $DEV parent 1:1 classid 1:50 htb rate ${UPLINKB}kbit ceil ${UPLINK}kbit burst 2k
# all get Stochastic Fairness:
./tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
./tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
./tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10
./tc qdisc add dev $DEV parent 1:40 handle 40: sfq perturb 10
./tc qdisc add dev $DEV parent 1:50 handle 50: sfq perturb 10
#maybe ACK already has TOS Mindelay?
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
./tc filter add dev eth0 parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:10
./tc filter add dev $DEV parent 1:0 protocol ip prio 0 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
# TOS Minimum Delay (ssh, NOT scp) in 1:20:
./tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 match ip tos 0x10 0xff flowid 1:20
# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
./tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 match ip protocol 1 0xff flowid 1:10
# put web in 1:30
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip sport 443 0xffff flowid 1:30
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip sport 80 0xffff flowid 1:30
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dport 443 0xffff flowid 1:30
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dport 80 0xffff flowid 1:30
# put smtp in 1:40
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip tos 0x02 0xff flowid 1:40
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip sport 25 0xffff flowid 1:40
./tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dport 25 0xffff flowid 1:40
# rest is 'non-interactive' ie 'bulk' and ends up in 1:50
shorewall "start" script adds:
# mark ack packets iptables -t mangle -A pretos -p tcp --tcp-flags ALL ACK -m state --state ESTABLISHED -m le ngth --length 40:100 -j MARK --set-mark 20
here is my resulting iptables mangle table (most done by shorewall)
# iptables -L -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination pretos all -- anywhere anywhere Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination outtos all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain outtos (1 references) target prot opt source destination TOS tcp -- anywhere anywhere tcp dpt:ssh TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp spt:ssh TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp dpt:ftp TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp spt:ftp TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp spt:ftp-data TOS set Maximize-Throughput TOS tcp -- anywhere anywhere tcp dpt:ftp-data TOS set Maximize-Throughput TOS tcp -- anywhere anywhere tcp spt:smtp TOS set Minimize-Cost TOS tcp -- anywhere anywhere tcp dpt:smtp TOS set Minimize-Cost Chain pretos (1 references) target prot opt source destination TOS tcp -- anywhere anywhere tcp dpt:ssh TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp spt:ssh TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp dpt:ftp TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp spt:ftp TOS set Minimize-Delay TOS tcp -- anywhere anywhere tcp spt:ftp-data TOS set Maximize-Throughput TOS tcp -- anywhere anywhere tcp dpt:ftp-data TOS set Maximize-Throughput TOS tcp -- anywhere anywhere tcp spt:smtp TOS set Minimize-Cost TOS tcp -- anywhere anywhere tcp dpt:smtp TOS set Minimize-Cost MARK tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/ACK state ESTABLISHED length 40:100 MARK set 0x14
Here’s what my traffic output (uptime 8 days) looks like:
# ./tc -s qdisc qdisc sfq 50: dev eth0 quantum 1514b perturb 10sec Sent 2136695 bytes 12976 pkts (dropped 0, overlimits 0) qdisc sfq 40: dev eth0 quantum 1514b perturb 10sec Sent 60809 bytes 484 pkts (dropped 0, overlimits 0) qdisc sfq 30: dev eth0 quantum 1514b perturb 10sec Sent 7410909 bytes 44900 pkts (dropped 0, overlimits 0) qdisc sfq 20: dev eth0 quantum 1514b perturb 10sec Sent 1798586 bytes 4963 pkts (dropped 0, overlimits 0) qdisc sfq 10: dev eth0 quantum 1514b perturb 10sec Sent 276452 bytes 4849 pkts (dropped 0, overlimits 0) qdisc htb 1: dev eth0 r2q 10 default 50 direct_packets_stat 0 Sent 11683451 bytes 68172 pkts (dropped 0, overlimits 2538) qdisc pfifo_fast 0: dev eth1 [Unknown qdisc, optlen=20] Sent 2521226003 bytes 2441881 pkts (dropped 0, overlimits 0)
So looks like stuff is being sorted out pretty well.
testing shaping
Here’s some timing tests, just to verify things Traffic shaping on, flood download & upload (run both of these commands at the same time:)
ftp> put testfile.uncompressed 1048576 bytes sent in 18.1 secs (57 Kbytes/sec) # wget ftp://ftp.sonic.net/pub/testfile.compressed 14:20:00 (402.63 KB/s) - `testfile.compressed.10meg.1' saved [10485760]
Since the upload finishes a second or two before the download, the download number is a little off… I get sustained 380KB down during the upload. A little better than 60% of full download (625KB), but 300KB better than without shaping (see below) Next, just do the upload alone, get full pipe:
ftp> put testfile.uncompressed Transfer complete. 1048576 bytes sent in 15.4 secs (66 Kbytes/sec)
Turn off shaping, rerun tests
./tc qdisc del dev eth0 root 2> /dev/null > /dev/null ftp> put testfile.uncompressed Transfer complete. 1048576 bytes sent in 15.5 secs (66 Kbytes/sec) # wget ... (speeds 80-90k down during transfer, as all bandwidth up sucked up and acks cannot get thru)
upload alone
ftp> put testfile.uncompressed Transfer complete. 1048576 bytes sent in 14.8 secs (69 Kbytes/sec)
NOTE: in the tests above the ‘ftp’ session is me uploading (sending data out) to a sonic.net ftp site. The ‘wget’ session is me downloading (sucking data in) from a sonic.net ftp site. My home network is behind the linux box running the traffic shaping (and being a firewall) – tests were run directly on the linux box but results are identical if you run them on any machine "behind" the firewall… Sorry if that was confused 🙂 A download with shaping on but no upload is
# wget ftp://ftp.sonic.net/pub/testfile.compressed 15:06:48 (626.49 KB/s) - `testfile.compressed.10meg.10' saved [10485760]
So looks like my shaper "steals" 3kb of the uplink – maybe I could tweak that UPLINK=510 line in the script to be a hair higher – say 560… but the 3kb isn’t that big a deal in the grand scheme of things. I’ll try it at 560 now… Nope. I walked it down from 560-510 in steps of 10 running tests each time, 510 yeilds the best results. So, you do have to give up 3% of the uplink to get the best results. I think that’s less than the standard wondershaper script, but who knows… BTW the one command turns off shaping (all the packets are still marked in iptables, but that doesn’t "cost" anything)
./tc qdisc del dev eth0 root 2> /dev/null > /dev/null
My 6M/608k line maxes out at 69kbyte/sec up (69*8*1024 = 565,248 bits/sec, or 552kbits/sec which is 90% of 608kbits – seems right). With shaping on I get 66kbytes/sec up max, or just a hair under 87% of 608k. Not half bad, if you ask me.