Equal-Cost Multi-Path Routing (ECMP)

Introduction

Equal-Cost Multi-Path routing (ECMP), is a routing behavior that allows for multiple routes for the same destination to be active at the same time, if they share the same cost. This means that flows of packets, bound for any particular destination, could be routed using different paths to reach the destination, if such paths are available.

Overview

ECMP is active on the system by default, it is not enabled or disabled by any active action by the operator. If two different routes specify the same destination, it is possible that they will be subject to ECMP, if the following is also true:

The routes in question have the same cost. As the name implies, Equal-Cost Multi-Path routing, will only operate on routes that have the same cost. If the cost is different, only the best of the routes will be inserted into the routing table, as usual.
The routes need to originate from the same routing protocol in order to be eligible for ECMP routing.

                   Path 1 ------------->
                             .---------.
             192.168.1.0/24  |         |
                    .--------+ RouterA +--------.
                  .´      .1 |         |         `.
                .´           '---------'           `.
              .´                                     `.
            .´     Path 2 ------------->               `.
.---------+´                 .---------.                 `+---------.
|         |               .1 |         |                  |         |
| Router1 +------------------+ RouterB +------------------+ Router2 +-----------------
|         |  192.168.2.0/24  |         |                  |         |  172.16.0.0/24
'---------+.                 '---------'                 .+---------'
            `.                                         .´
              `.                                     .´
                `.           .---------.           .´
                  `.      .1 |         |         .´
                    `--------+ RouterC +--------´
             192.168.3.0/24  |         |
                             '---------'
                   Path 3 ------------->

Figure 1: ECMP Routing being employed in order to distribute traffic towards the network 172.16.0.0/24 across three different next hop routers, producing three different paths for the traffic to traverse.

This is an example how it may look if we add a few static routes for the same destination, with different next hops, based on the topology in Figure 1 above:

Router1:/#> configure ip
Router1:/config/ip/#> route 172.16.0.0/24 192.168.1.1
Router1:/config/ip/#> route 172.16.0.0/24 192.168.2.1
Router1:/config/ip/#> route 172.16.0.0/24 192.168.3.1
Router1:/config/ip/#> leave
Applying configuration.
Configuration activated.  Remember "copy run start" to save to flash (NVRAM).
Router1:/#>
Router1:/#> show ip route
S - Static | C - Connected | K - Kernel route  | > - Selected route
O - OSPF   | R - RIP       | [Distance/Metric] | * - FIB route

S>* 172.16.0.0/24 [1/0] via 192.168.1.1, vlan1, weight 1, 00:00:03
  *                     via 192.168.2.1, vlan2, weight 1, 00:00:03
  *                     via 192.168.3.1, vlan3, weight 1, 00:00:03
C>* 192.168.1.0/24 is directly connected, vlan1, 00:02:01
C>* 192.168.2.0/24 is directly connected, vlan2, 00:00:46
C>* 192.168.3.0/24 is directly connected, vlan3, 00:00:43

As we can see from the output of the show ip route command, all of the three added routes are active at the same time, for the 172.16.0.0/24 destination. This should now allow traffic bound for the 172.16.0.0/24 network to be distributed on the three different paths, on a per traffic flow basis.

Further, if we for some reason want to exclude one potential route as part of the ECMP routing, we can adjust the distance of the route. This will lead to an increase in the cost of the specific route and cause it to no longer be of equal cost to the others. In this situation that route would not be active, as long as the routes with better cost are. As an example, we change the configuration for one of the routes in the following manner:

Router1:/#> configure ip
Router1:/config/ip/#> route 172.16.0.0/24 192.168.3.1 50
Router1:/config/ip/#> leave
Applying configuration.
Configuration activated.  Remember "copy run start" to save to flash (NVRAM).
Router1:/#>

Now when we check the active routes, it should look something like this:

Router1:/#> show ip route
S - Static | C - Connected | K - Kernel route  | > - Selected route
O - OSPF   | R - RIP       | [Distance/Metric] | * - FIB route

S   172.16.0.0/24 [50/0] via 192.168.3.1, vlan3, weight 1, 00:02:56
S>* 172.16.0.0/24 [1/0] via 192.168.1.1, vlan1, weight 1, 00:02:56
  *                     via 192.168.2.1, vlan2, weight 1, 00:02:56
C>* 192.168.1.0/24 is directly connected, vlan1, 00:11:07
C>* 192.168.2.0/24 is directly connected, vlan2, 00:11:07
C>* 192.168.3.0/24 is directly connected, vlan3, 00:11:07

Now we can see that the route we adjusted, with next hop 192.168.3.1, is not selectable and active in the routing table. Therefore, that specific path will no longer be considered when a specific flow of traffic is directed to a potential next hop.

Path Selection

When a specific destination have multiple valid next hops active in the routing table, a hash will be calculated in order to decide which of the next hops should be selected. The calculation of this hash is performed in such a manner that flows of packets should be distributed towards a specific next hop, and not just individual frames. If this would not be taken into account, it could cause issues with out of order packets.

The hash is calculated based on the header information present in a packet bound for routing. Currently WeOS will look on the following fields in a frame to determine the hash:

Destination IP address
Source IP address

When the hash calculation is performed on these values, it will guarantee that frames originating from the same source bound for the same destination, will be selecting the same next hop every time, since the hash calculation will be the same. Therefore, flows of packets should be routed over the same path.

Note

If the number of available next hops for a specific destination changes, the path a given packet flow currently selects, based on the hash calculation, could change. This also includes traffic that were not routed through a next hop that may no longer be available.

              ROUTER
             .-----------------------------------------------------.
             |    Hash calc        Hash Mapping    Next Hops       |  192.168.1.0/24
.----.       |                                                     |
| H1 +------>+--->=========--.                                     |       .---------.
'----'       |               |                                     |    .1 |         |
             |               +---> Hash Map 1 ---> 192.168.1.1 --->+------>+   R1    |
.----.       |               |                                     |       |         |
| H2 +------>+--->=========--'                                     |       '---------'
'----'       |                                                     |
             |                                                     |
.----.       |                                                     |
| H3 +------>+--->=========--.                                     |       .---------.
'----'       |               |                                     |       |         |
             |               +---> Hash Map 2 ---> 192.168.2.1 --->+------>+   R2    |
.----.       |               |                                     |    .1 |         |
| H4 +------>+--->=========--'                                     |       '---------'
'----'       |                                                     |
             |                                                     |  192.168.2.0/24
             '-----------------------------------------------------'

Figure 1: Simplified overview how flows that enter a router with ECMP, could be mapped towards different next hop addresses.

In Figure 1 above, we can see a representation how this selection is performed. For some given destination we have two different next hops configured on the device ROUTER. The four different hosts have a flow of traffic that they are sending towards the destination.

When the frames enter the ROUTER device, they will first undergo a hash calculation, based on the previous mentioned values. The resulting has value is mapped to either of the two different next hop addresses. Hence, in this example the flows from hosts H1 and H2 will be mapped towards R1 on next hop 192.168.1.1. Similarly, the flows from H3 and H4 will be mapped towards R2 using the next hop 192.168.2.1.

Weights

ECMP routes also have an additional concept that is referred to as Weighted Equal-Cost Multi-Path routing. Each individual route will be assigned a specific weight value. This value indicates how large share of possible traffic flows should utilize that particular route. Routes with a higher weight value will receive a larger portion of the possible traffic flows.

Currently, WeOS will have all ECMP routes assigned a weight value of 1, meaning that they routes will attempt and distribute the flows evenly. This weight value can be seen for each individual route when using the show ip route command.

Warning

At the moment no support is present for changing the weights that are assigned to any ECMP routes, they will always have a weight of 1. This includes static as well as dynamic routes.

Route monitor

ECMP static routes, can be augmented using a feature called route monitor. Route monitor track a route using ping trigger and policy route, if the route becomes unreachable then the admin distance of the route will be set to max, this makes the route inactive. All traffic will be directed to the remaining routes.

And if the route becomes reachable again then the admin distance is reset to the configured value.