Equal Cost Multipath

Leave a reply

Equal Cost Multipath (ECMP) is a network load-balancing method that enables the coexistence of multiple network paths form one source node to a destination node. The two or more paths between the nodes have the same routing cost, thus the traffic will be split evenly across, avoiding congestion and increasing the bandwidth.

ECMP is also a network redundancy method. In case one ECMP link fails, the traffic will move on the remaining links with minimal interruption in service.

For ECMP to work, a router will need special support in the forwarding plane and in the routing protocols deployed in the network. For experimentation I will use two Linux virtual machines on my host computer. The Linux kernel has an excellent ECMP implementation, as for the routing protocol I will use OSPF.

The virtual machines are set using virtenv. It is a very light virtualization solution based on Linux containers (LCX) implementation in Linux kernel. Each machine owns a slice of the Linux kernel running on the host computer, with full network and process separation. In each virtual machine I run one instance of RCP100. RCP100 is a router control plane for Linux platforms, supporting among other things OSPF and ECMP. RCP100 also features a CISCO-like command line interface (CLI) which simplifies router operation for people already skilled in configuring commercial routers.

The network diagram is as follows:

ECMP test network

The two virtual machines are plugged into two Linux bridges, br0 and br1. On router connections I use network addresses in the range 10.20.x.0/24 range. Router IDs are defined in 192.168.10.x range and are implemented as loopback interfaces on the router. The loopback interfaces are redistributed into OSPF and should be advertised by OSPF throughout the network. The relevant configuration for the two routers looks like this:

Router r0:

r0#show running-config | no-more 
hostname r0
[snip]
!
router ospf
  router-id 192.168.10.0
  network 10.20.0.0/16 area 0
  redistribute connected loopback
!
interface ethernet eth0
  ip address 10.20.0.2/24
  ip mtu 1500
  no shutdown
!
interface ethernet eth1
  ip address 10.20.1.2/24
  ip mtu 1500
  no shutdown
!
interface loopback 0
  ip address 192.168.10.0/32
  ip mtu 16436
!
[snip]
r0#

Router r1:

r1#show running-config | no-more 
hostname r1
[snip]
router ospf
  router-id 192.168.10.1
  network 10.20.0.0/16 area 0
  redistribute connected loopback
!
interface ethernet eth0
  ip address 10.20.0.3/24
  ip mtu 1500
  no shutdown
!
interface ethernet eth1
  ip address 10.20.1.3/24
  ip mtu 1500
  no shutdown
!
interface loopback 0
  ip address 192.168.10.1/32
  ip mtu 16436
!
r1#

We login into the routers by telnet 0 in the virtual machine control terminals. The routing tables will show the equal cost multipath routes installed for the neighbor’s router ID. As we configured them earlier as loopbacks, router IDs are real routes, we can ping them for example.

Router r0:

r0#show ip route
Codes: C - connected, S - static, R - RIP, B - blackhole, O - OSPF
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2

C    10.20.0.0/24 is directly connected, eth0
C    10.20.1.0/24 is directly connected, eth1
O E2 192.168.10.1/32[110/20] via 10.20.0.3, eth0
                             via 10.20.1.3, eth1
r0#

Router r1:

r1#show ip route
Codes: C - connected, S - static, R - RIP, B - blackhole, O - OSPF
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2

C    10.20.0.0/24 is directly connected, eth0
C    10.20.1.0/24 is directly connected, eth1
O E2 192.168.10.0/32[110/20] via 10.20.0.2, eth0
                             via 10.20.1.2, eth1
r1#

With the ECMP routes installed, we can try now to see how the traffic is distributed across the two links. We login into router 0 and from there we telnet several times into r1 at 192.168.10.1.

r0#telnet 192.168.10.1
Trying 192.168.10.1...
Connected to 192.168.10.1.
Escape character is '^]'.
User: rcp
Password: 
r1>exit
Connection closed by foreign host.
r0#

Each telnet connection goes trough one path or the other alternatively. Once the telnet session is established, the traffic stays on the specific path until the telnet connection is closed. This is accomplished in the Linux kernel by hashing source/destination address and port numbers, and associating specific hash values with specific connections.

The following picture shows two Wireshark windows side by side, while pinging r1 from r0. The ICMP requests are spread over the two links, r1 will respond on the link it received the request.

ECMP traffic in two WIreshark windows

Leave a comment