Describe Eliezer Croitoru/Drafts/MwanLB here.

Intoduction to MultiWAN LoadBalancing

  • Goal: Understanding linux Load-Balancing routing.

  • Status: 30%

  • State: DRAFT

  • Writer: Eliezer Croitoru

  • More:

  • Side Effects: A Squid Instances LoadBalancer(Tproxy to Proxy Protocol)

- Introduction to Multi-WAN load balancing.

There are couple uses for Multiple WAN connections which differ by nature.
MultiWAN can be there to solve one of two network problems:

  • High network traffic load
  • Network failures

The way MultiWAN solutions are being implemented differ from one network environment to another. It's not only about the technical nature but also the purpose of a solution and it's stability. In a case we are talking about a small house, a small office or a small school they will probably will not have ASN under their hands. And it means that there is no need and probably it is not an option to use BGP as routing protocol.
It's also means that in the mentioned cases the connection will use some kind of NAT to access to the Internet.
There are pros and cons when implementing MultiWAN behind couple NAT routers.

Couple things to know about are:

  • Two connections from the same network can have different SRC-IP at the same second.
  • Network failure on one port can cause weird side effect on speed and network behavior.

An illustration of a MultiWAN connected network:

In the above scenario client will try to access HTTP service and when accessing the service the linux router will use the IP address
When the client will try to access the same service it is not guaranteed that the same IP will be used when accessing the same service on
It means that two clients from the same network can get access to the same service using different SRC ip address.
And for those that knows how to read apache logs you will see the next lines in the access.log of the apache service: - - [31/Oct/2013:09:17:24 +0200] "GET /robots.txt HTTP/1.1" 404 291 "-" "Mozilla/5.0 (compatible)" - - [31/Oct/2013:09:17:25 +0200] "GET /robots.txt HTTP/1.1" 404 291 "-" "Mozilla/5.0 (compatible)"

There are couple other options to run and operate MultiWAN Fail-over and Load-balancing But due to the complexity of these setups I will not cover them for now.

What is MultiWAN and MultiPATH?

The internet which is based on IP networks is a very big and dynamic system.
While it's software can be kind of “subnetted” to allow an administrative way of managing all these networks Static network topology still makes things easy.
Since this network is so big wide and dynamic there is a need to allow couple “options” to make it possible for infrastructure to be relocated or “copied”.
The big need of re-location of an IP network can be real in a case of floods, power-outage, human-error or even a simple temperature drop\rise.
Since humans has a very long history in the world it can be assumed that no matter how hard the cement and steel will be still there is a need to plan migrations.
For this specific option to be effective and efficient there was a way to take the Static IP topology and using software to remove any “hardening” of this network topology structure.
The administrative force of the Internet divided the Internet into AS(Autonomous System) which allows the admin to look at the network in a level above all these weird numbering scheme that Routers use.
It then allows the administrative force of the internet to develop a way to use IP in a less static way that it used to be when it was implemented in one room.

Today these software's allow for an AS, which is basically a human force that has a subnet at hands, to define where through this subnet will be accessed in the IP level.
Most users of the internet are probably just a bunch of nice guys that do not need an AS but they do get access to the Internet through a company or organization that do apply to the basic requirements which certify it to own an AS.
The basic requirements for owning or managing a subnet(or AS) are a well trained and\or certified network operators and engineers.
It can seem like a very simple task for some but since it means becoming a part of a very wide network which is being changed every single moment it is a very complex task.

A MultiWAN is not like any literal understanding of the "multi" word which is "doubles" this or that.
The meaning of a Multi in this area is “Two ways to get access to this same WAN network”.
Once this concept is understood we can clarify that a MultiWAN can also be a MultiPATH which means that the same IP network\subnet can be accessed throw two different ways\roads.
The main terms that are being used by network and system administrators and engineers about this and related subjects are "MultiHomed" and\or "MultiWAN" network in the relation to a specific AS which relates to some subnets.
A MultiPATH is the more literal way of defining a network that can access or can be accessed through two different "cables" to or from the Internet.

Reverse Path Filtering

I have asked myself couple times "What is Reverse Path Filtering?" and there is a literal answer and a technical one.
For us the technical description would fit.
The next details requires some basic networking knowledge:
Assuming that a router should not just pass\forward any packet flows into it reverse path filtering comes to help. The idea is to block traffic that should not be there based on the already known network settings.
The Linux kernel have 3 options using this filter: none, strict, loose.
Related or not it is recommended to filter traffic in any case just for security purpose.

A networking real world example would be a simple router that has two network connections and which doesn't posses a routing daemon. It has two\three or more networks\subnets that are passing throw it and the network interfaces of the device are:

  • eth0 -
  • eth1 -
  • eth2 -
  • lo0 -

A basic logic would be that this router should not pass\forward traffic from networks it doesn't know about such as
If we would add some more routes such as " via" it would be reasonable to receive traffic from network on the interface eth0 but not on eth1. So the filter can be applied on eth1 and any packet flowing into this interface with a source address inside the network will be dropped.

The usage of Reverse Path filtering suits only some cases while in many others it will cause major operations issues.
The basic recommendation is that you better firewall your network or\and in some cases as an alternative to a firewall rules is to throw traffic from a whole subnet into a black-hope.

  • In cases of Internet Exchange Point unauthorized router peering there are places around the world which the only way to handle these bandits is using FIREWALL or ROUTING rules and as much as I and others are good Admins there are out-there some who do not ask for permission to throw packets at a router and see what happens so beware.

Set Reverse Path Filter machine globally script

   1 #!/bin/bash
   2 if [ -z "$1" ];
   3 then
   4         echo "empty value";
   5         exit 1
   6 else
   7         echo "is set to '$1'";
   8 fi
  10 if [ "$1" != "0" ] && [ "$1" != "1" ] && [ "$1" != "2" ];
  11 then
  12    echo "wrong value"
  13    exit 1
  14 fi
  17 echo "setting rp_filter globally for => \"$1\""
  18 for i in `ls /proc/sys/net/ipv4/conf/*/rp_filter`;
  19 do
  20     echo $i
  21     echo "$1" >$i
  22 done

Linux Connection Tracking and LB, Advantages and limitations

On a linux based router which utilizes the connection tracking module it is possible to "track" if a connection from the network clients are already marked or not for both TCP, UDP and couple other protocols.

However on a Linux client machine the only concept of a connection is on the protocol level ie TCP and not UDP.

UDP uses datagrams and doesn't have any "connection" orientation in the kernel level. Due to this in the client side UDP packets that are originated from the client\server machine would not be able to rely only on the linux kernel connection tracking to be able to send UDP packets\datagrams using the same route it was received through.

NATed Environment

LoadBalancing general algorithms

  • GOALS: failover, equal cost per packet, weighted per packet, equal cost per connection, and weighted per connection balancing

Round Robin

Weighted Round Robin

Least Connections

  • csv file with established connections per MARK\PATH

200, 1
100, 2
300, 3
  • A simple selection between multiple marks using least used.

   1 #!/usr/bin/env python
   2 import csv
   3 i = 0
   4 selection_least = -1
   5 selected = -1
   6 with open('marks_stats.csv', 'rb') as csvfile:
   7         statsreader = csv.reader(csvfile, delimiter=',', quotechar='|')
   8         for row in statsreader:
   9                 i = i + 1
  10                 if selection_least == -1:
  11                         selection_least = int(row[0])
  12                         selected = i
  13                         print("Least used is: " + str(selected) )
  14                         continue
  15                 if int(row[0]) < selection_least:
  16                         selection_least = int(row[0])
  17                         selected = i
  18                         print("Least used is: " + str(selected) )
  19 print(selected)
  • A bash script that writes the current established connections into a CSV file(from 3)

   1 #!/usr/bin/env bash
   2 ONE=`conntrack -L 2>/dev/null|grep "mark=1 "|grep ESTABLISHED |wc -l`
   3 TWO=`conntrack -L 2>/dev/null|grep "mark=2 "|grep ESTABLISHED |wc -l`
   4 THREE=`conntrack -L 2>/dev/null|grep "mark=3 "|grep ESTABLISHED |wc -l`
   5 echo "$ONE,1" > marks_stats.csv
   6 echo "$TWO,2" >> marks_stats.csv
   7 echo "$THREE,3" >> marks_stats.csv

Packet By Packet Load Balancing VS Connection based

Route Policy LB vs MARK based LB

Removal of ipv4 routing cache from linux kernel

  • it brings the problem of "packet by packet" routing systems.

  • Quote from the commit

ipv4: Delete routing cache.
The ipv4 routing cache is non-deterministic, performance wise, and is subject to reasonably easy to launch denial of service attacks.

The routing cache works great for well behaved traffic, and the world was a much friendlier place when the tradeoffs that led to the routing cache's design were considered.

What it boils down to is that the performance of the routing cache is a product of the traffic patterns seen by a system rather than being a product of the contents of the routing tables. The former of which is controllable by external entitites.

Even for "well behaved" legitimate traffic, high volume sites can see hit rates in the routing cache of only ~%10.

Signed-off-by: David S. Miller

Efficency of Policy

Efficency of MARK

Combination of both


Linux options for MultiWAN

MultiDSL\Multiother to a HUBPROXY Balancing

NFQUEUE to mark flowing connection


Round Robin mark selection - Python example

  • An example for a RoundRobin LB between 3 iptables marks using NFQUEUE mark_verdict

   1 #!/usr/bin/env python
   2 import time
   3 from daemon import runner
   4 import nfqueue, socket
   5 from scapy.all import *
   7 queue = deque([1, 2, 3])
   9 def get_queue():
  10     global queue
  11     l = queue.popleft()
  12     queue.append(l)
  13     return l
  15 #Set the callback for received packets:
  16 def cb(i,payload):
  17     data = payload.get_data()
  18     p = IP(data)
  19     mark = get_queue()
  20     payload.set_verdict_mark(nfqueue.NF_REPEAT, mark) #4 = nfqueue.NF_REPEAT
  22 class App():
  23     def __init__(self):
  24         self.stdin_path = '/dev/null'
  25         self.stdout_path = '/dev/tty'
  26         self.stderr_path = '/dev/tty'
  27         self.pidfile_path =  '/tmp/'
  28         self.pidfile_timeout = 5
  29     def run(self):
  30                 q = nfqueue.queue()
  31                 q.set_callback(cb)
  33                 q.create_queue(0)
  34                 try:
  35                         q.try_run()
  36                 except KeyboardInterrupt, e:
  37                         print "interruption"
  39                 q.unbind(socket.AF_INET)
  40                 q.close()
  42 app = App()
  43 daemon_runner = runner.DaemonRunner(app)
  44 daemon_runner.do_action()

Round Robin mark selection - GoLang example

   1 package main
   3 /*
   4 license note
   5 Copyright (c) 2016, Eliezer Croitoru
   6 All rights reserved.
   8 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
   9 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  10 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  11 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
  14 */
  16 import (
  17         "encoding/hex"
  18         "flag"
  19         "fmt"
  20         ""
  21         "os"
  22         "os/signal"
  23         "sync/atomic"
  24         "syscall"
  25 )
  27 var marksMax uint64
  28 var logpkt bool
  29 var logmark bool
  30 var queueNum int
  32 var counter = uint64(1)
  34 func real_callback(payload *nfqueue.Payload) int {
  35         if logpkt {
  36                 fmt.Println("Real callback")
  37                 fmt.Printf("  id: %d\n", payload.Id)
  38                 fmt.Printf("  mark: %d\n", payload.GetNFMark())
  39                 fmt.Printf("  in  %d      out  %d\n", payload.GetInDev(), payload.GetOutDev())
  40                 fmt.Printf("  Φin %d      Φout %d\n", payload.GetPhysInDev(), payload.GetPhysOutDev())
  41                 fmt.Println(hex.Dump(payload.Data))
  42                 fmt.Println("-- ")
  43         }
  44         val := (atomic.AddUint64(&counter, 1) % marksMax) + 1
  45         if val == uint64(0) {
  46                 val++
  47         }
  48         if logmark {
  49                 if logpkt {
  50                         fmt.Println("The selected Mark =>", val, "For packet =>", payload)
  51                 } else {
  52                         fmt.Println("The selected Mark =>", val)
  53                 }
  54         }
  55         payload.SetVerdictMark(nfqueue.NF_REPEAT, uint32(val))
  57         return 0
  58 }
  60 func main() {
  61         flag.BoolVar(&logpkt, "log-packet", false, "Log the packet to stdout (works with log-mark option only)")
  62         flag.BoolVar(&logmark, "log-mark", false, "Log the mark selection to stdout")
  64         flag.Uint64Var(&marksMax, "high-mark", uint64(3), "The number of the highest queue number")
  65         flag.IntVar(&queueNum, "queue-num", 0, "The NFQUEQUE number")
  67         flag.Parse()
  69         q := new(nfqueue.Queue)
  71         q.SetCallback(real_callback)
  73         q.Init()
  74         defer q.Close()
  76         q.Unbind(syscall.AF_INET)
  77         q.Bind(syscall.AF_INET)
  79         q.CreateQueue(queueNum)
  80         q.SetMode(nfqueue.NFQNL_COPY_PACKET)
  81         fmt.Println("The queue is active, add an iptables rule to use it, for example: ")
  82         fmt.Println("\tiptables -t mangle -I PREROUTING 1 [-i eth0] -m conntrack --ctstate NEW -j NFQUEUE --queue-num", queueNum)
  84         c := make(chan os.Signal, 1)
  85         signal.Notify(c, os.Interrupt)
  86         go func() {
  87                 for sig := range c {
  88                         // sig is a ^C, handle it
  89                         _ = sig
  90                         q.Close()
  91                         os.Exit(0)
  92                         // XXX we should break gracefully from loop
  93                 }
  94         }()
  96         // XXX Drop privileges here
  98         // XXX this should be the loop
  99         q.TryRun()
 101 }

Least Connections selection algorithm example

   1 #!/usr/bin/env python
   2 import time
   3 import sys
   4 import commands
   5 import os
   6 #from daemon import runner
   7 import nfqueue, socket
   8 from scapy.all import *
  10 queue = deque([1, 2, 3])
  12 def get_queue():
  14     mark = 1
  15     res1 =  commands.getstatusoutput('conntrack -L 2>/dev/null|grep mark=1|grep ESTABLISHED |wc -l')
  16     res2 =  commands.getstatusoutput('conntrack -L 2>/dev/null|grep mark=2|grep ESTABLISHED |wc -l')
  17     res3 =  commands.getstatusoutput('conntrack -L 2>/dev/null|grep mark=3|grep ESTABLISHED |wc -l')
  18     if not int(res1[1]) < int(res2[1]) or not int(res1[1]) < int(res3[1]):
  19         mark = 1
  20     if not int(res2[1]) < int(res1[1]) or not int(res2[1]) < int(res3[1]):
  21         mark = 2
  22     if not int(res3[1]) < int(res2[1]) or not int(res3[1]) < int(res1[1]):
  23         mark = 3
  24     return mark
  26 #Set the callback for received packets:
  27 def cb(i,payload):
  28     data = payload.get_data()
  29     p = IP(data)
  30     mark = get_queue()
  31     payload.set_verdict_mark(nfqueue.NF_REPEAT, mark) #4 = nfqueue.NF_REPEAT
  33 q = nfqueue.queue()
  34 q.set_callback(cb)
  36 q.create_queue(0)
  37 try:
  38   q.try_run()
  39 except KeyboardInterrupt, e:
  40   print "interruption"
  42 q.unbind(socket.AF_INET)
  43 q.close()

iptables rules example

* Example NFQUEUE(0) iptables rules that shows how a connection is being marked by the python helper and then a log target is counting the packets.

   1 #!/usr/bin/env bash
   2 IPTABLES="/sbin/iptables"
   4 $IPTABLES -t mangle -F PREROUTING
   5 $IPTABLES -t mangle -A PREROUTING ! -p tcp -j ACCEPT
   6 $IPTABLES -t mangle -A PREROUTING -p tcp  -m mark --mark 0 -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
   7 $IPTABLES -t mangle -A PREROUTING -p tcp -m state --state NEW -m mark ! --mark 0 -j CONNMARK --save-mark
   9 $IPTABLES -t mangle -A PREROUTING  -m mark --mark 0 -m conntrack --ctstate NEW -j NFQUEUE --queue-num 0
  11 $IPTABLES -t mangle -A PREROUTING  -m connmark --mark 0x1 -j LOG --log-prefix "post, connmark 1: "
  12 $IPTABLES -t mangle -A PREROUTING  -m connmark --mark 0x2 -j LOG --log-prefix "post, connmark 2: "
  13 $IPTABLES -t mangle -A PREROUTING  -m connmark --mark 0x3 -j LOG --log-prefix "post, connmark 3: "
  15 $IPTABLES -t mangle -A PREROUTING -m mark --mark 1 -j LOG --log-prefix "post, mark 1: "
  16 $IPTABLES -t mangle -A PREROUTING -m mark --mark 2 -j LOG --log-prefix "post, mark 2: "
  17 $IPTABLES -t mangle -A PREROUTING -m mark --mark 3 -j LOG --log-prefix "post, mark 3: "

example statistics of iptables with marks

  • An example output of iptables statistics of a running nfqueue marking setup.

$ sudo iptables -t mangle -L PREROUTING -nv
Chain PREROUTING (policy ACCEPT 909 packets, 54107 bytes)
 pkts bytes target     prot opt in     out     source               destination
   68 17255 ACCEPT    !tcp  --  *      *  
  885 52647 CONNMARK   tcp  --  *      *              mark match 0x0 state RELATED,ESTABLISHED CONNMARK restore
   25  1500 CONNMARK   tcp  --  *      *              state NEW mark match ! 0x0 CONNMARK save
   25  1500 NFQUEUE    all  --  *      *              mark match 0x0 ctstate NEW NFQUEUE num 0
   52  2912 LOG        all  --  *      *              connmark match  0x1 LOG flags 0 level 4 prefix "post, connmark 1: "
   48  2695 LOG        all  --  *      *              connmark match  0x2 LOG flags 0 level 4 prefix "post, connmark 2: "
  707 41028 LOG        all  --  *      *              connmark match  0x3 LOG flags 0 level 4 prefix "post, connmark 3: "
   52  2912 LOG        all  --  *      *              mark match 0x1 LOG flags 0 level 4 prefix "post, mark 1: "
   48  2695 LOG        all  --  *      *              mark match 0x2 LOG flags 0 level 4 prefix "post, mark 2: "
  707 41028 LOG        all  --  *      *              mark match 0x3 LOG flags 0 level 4 prefix "post, mark 3: "

Squid and multiWAN LB


MultiWAN StandAlone Host reverse path consistency

For a single host to return traffic the way it came from you need the next iptables and ip rules:

   1 #!/usr/bin/env bash
   3 # Disable Reverse Path filtering
   5 for i in /proc/sys/net/ipv4/conf/*/rp_filter
   6 do
   7   echo $i
   8   cat $i
   9   echo 0 > $i
  10 done
  12 IPTABLES="/sbin/iptables"
  13 IP=/sbin/ip
  15 $IP route flush table 201
  16 $IP route add default via table 201
  18 $IPTABLES -t mangle -F 
  20 $IPTABLES -t mangle -A PREROUTING ! -p tcp -j ACCEPT
  22 $IPTABLES -t mangle -A PREROUTING -j CONNMARK --restore-mark
  23 $IPTABLES -t mangle -A OUTPUT -j CONNMARK --restore-mark
  25 $IPTABLES -t mangle -A PREROUTING -i eth2 -p tcp -m state --state NEW  -j CONNMARK --set-mark 1
  27 $IPTABLES -t mangle -A PREROUTING -m connmark --mark 1 -j MARK --set-mark 1
  29 $IPTABLES -t mangle -A PREROUTING -m connmark ! --mark 0  -j CONNMARK --save-mark
  31 $IP rule|grep  'from all fwmark 0x1 lookup 201' >/dev/null
  32 if [ "$?" -eq "1" ]; then
  33   $IP rule add fwmark 1 table 201
  34 fi

MultiWAN NATed testing environment

After implementing the same lab with different OS I have decided to use Ubuntu instead of TinyCore linux. And the main Reason for that is that TinyCore linux is a great OS but I am feeling like missing some tools with it.(It's not you it's me..)

Indeed Ubuntu gives more tools but TinyCore helped me with the basics of iptables marking and RoundRobin basics.

First LAB - TinyCore

I will use TinyCore linux (CorePlus version) as client and routing OS.

  • Client IP
  • LAN core router IP1:, Wan interface IP2:
  • WAN router-1 IP1: IP2:
  • WAN router-2 IP1: IP2:
  • Internet target Server at:

The scenario is that Client will try to contact through LAN-core router which will load-balance the traffic over 2 WAN connections.

In turn the Load-Balancing rules will be changed and there for the traffic path\flow.

Then I will try to contact couple different Internet hosts through the LAN-core router and we will see what is the different trafic path for each and every one of these IPs.

Second LAB - Ubuntu


  • Client1 IP
  • Client2 IP
  • LAN core router IP1:, Wan interface IP2:
  • WAN router-1 IP1: IP2:
  • WAN router-2 IP1: IP2:
  • WAN router-3 IP1: IP2:
  • WebServer IP:

Third LAB - OpenSUSE+ZeroShell


  • Windows Client1 IP
  • Linux Client2 IP
  • LAN core router(ZeroShell) IP1:, Wan interface IP2:

  • WAN router-1(OpenSUSE) IP1: dsl connection)
  • WAN router-2(CentOS) IP1: vpn over the to the Internet)
  • WebServer

Fourth LAB - OpenSUSE+Alpine


  • Windows Client1 IP
  • Linux Client2 IP
  • LAN core router(Alpine) IP1:, Wan interface IP2:
  • WAN router-1(OpenSUSE) IP1: dsl connection)
  • WAN router-2(CentOS) IP1: vpn over the to the Internet)
  • WebServer


  • Linux\Windows client
  • Ubuntu LB +
  • 3 VYOS NAT routers with IPIP tunnel towards the HUB server(next..)
  • Remote Ubuntu LB HUB with IPIP tunnels towards the nat routers public IP address(and back from them), and NATTING the incomming traffic to the Internet.

This setup is similar to PEPLINK product which offers Load Balancing over low cost lines and termination of the connection on the other side in a DATA-CENTER. In the lab on the local LB we run the golang helper on the way to the internet and route connections based on their mark, one TCP connection will stay over the same NAT router and the same PATH in one direction. Every side(LB, HUB) has it's own LB service and there for it can happen very often that while the traffic to the Internet will use one PATH ie IPIP tunnel, and when the data will be balanced back to the client it will pass in antother PATH.

Load Balancing - out of the box

As a Computer Science novice one of the important tasks in the real world would be to maintain balance between many worlds.
From one hand the Computer Science is tempting and gives lots of power while on the other side of these machines there are Billions of lives around the clock in the past, present and future.
From my side of the picture I know that the machine is simple but this was granted to me as a gift from my parents and ancestors.
However it's very easy for my generation to operate the "Thinking Machine" while for former generations which had no electricity and pumped water the issue was to get\understand a sentence right.
Thankfully we were embedded with all this wisdom to help us operate our "Thinking Machine" better then the old generation could. With this in mind it is very important to understand that Load Balancing is an art.
These words are here to help but sometimes these are forgotten with the stream of life:
We have the power of Thousands and Millions on our shoulders!!
We can embrace the power of Millions and utilize them for one of the couple hats: White, Black or Red.
The White is the hat which reflects only good Intention and while the Black and the Red are mixed and the Black is sourced probably from bad Intention but it's root source is good.
Specifically the Red one would use both powers for good and this is the preferred one.
Another hat which deserves mentioning is the Blue which is mainly granted for these who have done bad deeds but with good Intention while assuming that this is the right thing to do.

  • The above mentioned hats are not connected to the RedHat corporation in any way and is merely a reflection of these colors features by some spiritual concepts.

My suggestion is to Load the Balance with any of the hats you get in life.
For example: Don't do\use drugs unless you have help in analyzing their influence from a licensed personnel.
The above was designed to help you or others to recognize the complexity and nature of the Literal subject and to give an example on the subject for these who talks the "Computer Language".


dia icons -

EliezerCroitoru/Drafts/MwanLB (last edited 2019-08-07 04:07:42 by Eliezer Croitoru)