WeOSSplit ECN over ETB HowTo (advanced)
This document describes a specific use case, where two ECNs and a non-ECN network are overlaid on top of the ETB, thus not needing dedicated ECN cabling between the ETBNs.
This can be useful in certain cases where adding another physical cable (in addition to the regular ETB cabling) is difficult between the ETBNs, or in cases where a direct connection between the ETBNs is prevented by the ECN architecture.
In the case presented below, there are two ETBNs (ETBN1, ETBN2) connected to the same ECN, without dedicated ECN cabling. There is also a redundant pair of externally managed firewall devices (GW1, GW2), splitting the ECN into two subnets. As the firewall devices are considered third-party devices, only configuration hints for them will be provided. There are also two end devices, or hosts (H1, H2) added for illustration.
The VLAN used for the ECN in this scenario will be vlan10. When using the “ECN over ETB” functionality, it is important that the same VID be used for each ECN across all participating ETBNs in the consist, since the ECN traffic is sent across the ETB using VLAN tags.
This use case utilizes the “ECN subnetting” functionality. While not strictly needed to use the “ECN over ETB” function, the two can be used together for greater effect. In effect, this feature lets the ECN be split into two distinct subnets locally, while still presented as one ECN to the rest of the train.
There are three VRRP virtual routers used in this example, the ETBNs being one of them (this is the usual setup for redundant ECNs when using TTDP: see the redundant ETBN HowTo), with the VIP address 10.0.0.10. This address is used by every device on the combined ECN as the default route, in order to facilitate traffic via the ETB, as in a normal TTDP scenario. The other two VRRP routers are handled by the firewall devices, one instance on each half of the ECN. The VIP addresses of each of these is then used by devices to reach the other half of the ECN. These are 10.0.0.100 for the upper part of the ECN, and 10.0.32.100 for the lower part.
Schematically the network topology looks as follows.
.--------------------.
/ cst1 \
'----------------------'
.-------. .-------.
--+ +----+ +-- dir1 = (ethX5, ethX6)
<-dir1| ETBN1 | | ETBN2 |dir2-> dir2 = (ethX7, ethX8)
--+ +----+ +--
'---+---' '---+---'
ethX3|.1 ethX3|.2
| |
.----. | | .----.
| H1 +-+ .11 .12 +-+ H2 | ^
'----' | | '----' |
| | |
eth0|.21 eth0|.22 "Upper" part of ECN
.---+---. .---+---. (disjoint)
| | | | 10.0.0.0/19
| GW1 | | GW2 | -------------------
| | | | "Lower" part of ECN
'---+---' '---+---' (joint)
eth1|.21 eth1|.22 10.0.32.0/19
| | |
----+-----+------+---- |
| .99 V
.--+---.
| ECSC |
'------'
Figure 1: Consist network layout.
Note
As far as the IEC61375 stack is concerned, there is just one ECN in the consist. This is what TTDP communicates to any other coupled consists. This entire ECN has the address space 10.0.0.0/18, and it will be mapped to one /18 subnet in the train scope, as usual. Internally, however, this ECN is split up into two /19 networks, on either side of the FW devices. This requires some special configuration on the ETBNs, and is made possible by the “ECN subnetting” feature.
Note
Note also that the upper part of the ECN is disjoint - there is no direct path between ED1 and ED2 on the 10.0.0.0/19 network. For the two to communicate directly, the network needs to be overlaid on top of the ETB cabling. This is made possible by the “ECN over ETB” feature.
Configuration
Non-WeOS configuration
The network shown in Figure 1 contains some non-WeOS devices, for illustration
as well as context. It is assumed that they are configured as follows. Note that
link-local routes are not explicitly specified (e.g., it is assumed that a
device with an address of a subnet has a link-local route to that subnet). Note
also that ED1 and ED2 can be configured using DHCP, including static routes -
see the DHCP Server HowTo and the DHCP Server
documentation, in particular the route subnet
setting.
ED1: address 10.0.0.11/19
static route to 10.0.32.0/19 via 10.0.0.100
default gateway 10.0.0.10
ED2: address 10.0.0.12/19
static route to 10.0.32.0/19 via 10.0.0.100
default gateway 10.0.0.10
GW1: eth0 (upper) address 10.0.0.21/19
eth1 (lower) address 10.0.32.21/19
IP forwarding enabled between eth0 and eth1
default gateway 10.0.0.10
VRRP configuration:
VRRP instance 1:
base interface: eth0 (upper)
address 10.0.0.100
VRRP instance 2:
base interface: eth1 (lower)
address 10.0.32.100
GW2: eth0 (upper) address 10.0.0.22/19
eth1 (lower) address 10.0.32.22/19
IP forwarding enabled between eth0 and eth1
default gateway 10.0.0.10
VRRP configuration:
VRRP instance 1:
base interface: eth0 (upper)
address 10.0.0.100
VRRP instance 2:
base interface: eth1 (lower)
address 10.0.32.100
ECSC: address 10.0.32.99/19
default gateway 10.0.32.100
ECSP address 10.0.0.10
ETBN1 configuration
Start by configuring the required VLAN and VRRP settings on ETBN1. Also, add a route to 10.0.32.0/19 (the lower part of the ECN), using the upper VIP address of the GW devices as the router.
etbn1:/#> config etbn1:/config/#> vlan 10 Creating new VLAN vid:10 with name: vlan10 etbn1:/config/vlan-10/#> untagged ethX3 Moving untagged port eth4 from vid 1 to vid 10. etbn1:/config/vlan-10/#> end etbn1:/config/#> iface vlan10 etbn1:/config/iface-vlan10/#> no inet etbn1:/config/iface-vlan10/#> inet static 10.0.0.1/19 etbn1:/config/iface-vlan10/inet-static-10.0.0.1/#> end etbn1:/config/iface-vlan10/#> end etbn1:/config/#> vlan 2 Creating new VLAN vid:2 with name: vlan2 etbn1:/config/vlan-2/#> no multicast-snooping etbn1:/config/vlan-2/#> end etbn1:/config/#> router etbn1:/config/router/#> vrrp ecn1 Creating new VRRP instance: ecn1 Invalid settings: Interface not set. etbn1:/config/router/vrrp-ecn1/#> iface vlan10 Invalid settings: Invalid IPv4 address. etbn1:/config/router/vrrp-ecn1/#> address 10.0.0.10 Invalid settings: VRID invalid. Valid values: 1 to 255. etbn1:/config/router/vrrp-ecn1/#> vrid 1 etbn1:/config/router/vrrp-ecn1/#> priority 100 etbn1:/config/router/vrrp-ecn1/#> end etbn1:/config/router/#> end etbn1:/config/#> ip etbn1:/config/ip/#> route 10.0.32.0/19 10.0.0.100 etbn1:/config/ip/#> end etbn1:/config/#>
Then, set up the necessary ETB port settings.
etbn1:/config/#> etbn1:/config/#> port ethX5 etbn1:/config/port-ethX5/#> speed-duplex 100-full etbn1:/config/port-ethX5/#> no auto-negotiate etbn1:/config/port-ethX5/#> fastlink etbn1:/config/port-ethX5/#> mdix-mode mdi etbn1:/config/port-ethX5/#> end etbn1:/config/#> port ethX6 etbn1:/config/port-ethX6/#> speed-duplex 100-full etbn1:/config/port-ethX6/#> no auto-negotiate etbn1:/config/port-ethX6/#> fastlink etbn1:/config/port-ethX6/#> mdix-mode mdi etbn1:/config/port-ethX6/#> end etbn1:/config/#> port ethX7 etbn1:/config/port-ethX7/#> speed-duplex 100-full etbn1:/config/port-ethX7/#> no auto-negotiate etbn1:/config/port-ethX7/#> fastlink etbn1:/config/port-ethX7/#> mdix-mode mdi etbn1:/config/port-ethX7/#> end etbn1:/config/#> port ethX8 etbn1:/config/port-ethX8/#> speed-duplex 100-full etbn1:/config/port-ethX8/#> no auto-negotiate etbn1:/config/port-ethX8/#> fastlink etbn1:/config/port-ethX8/#> mdix-mode mdi etbn1:/config/port-ethX8/#> end etbn1:/config/#>
Note
In this HowTo, it is assumed that the ETB uses crossover cabling. For this reason, all ETB ports on both ETBNs use the same MDI-X mode. If non-crossover cabling is used, these settings must be changed for two of the inward-facing ETB ports.
Finally, set up the TTDP configuration and add the ETB link aggregates to the
ETB VLAN. Note specifically the use of the ecn-over-etb setting, where we
indicate that VID 10 is to be forwarded over the ETB cabling.
Note
There is an extra parameter, 18, given to the ecn command. This is normally
omitted in the TTDP configuration, but is required here for the “ECN subnetting”
functionality. What this does is to let TTDP know that even though the device
uses a /19 address on vlan10, the ECN, as presented in TTDP, should use the
entire /18 space (the other /19 net is reachable via the GW devices).
etbn1:/config/#> etbn1:/config/#> ttdp Activating TTDP with default settings. etbn1:/config/ttdp/#> uuid 11111111-1111-1111-1111-111111111111 etbn1:/config/ttdp/#> node 1 ecn 1 etbn1:/config/ttdp/#> node 2 ecn 1 etbn1:/config/ttdp/#> ecn 1 vlan10 18 etbn1:/config/ttdp/#> dir1 ethX5 ethX6 etbn1:/config/ttdp/#> dir2 ethX7 ethX8 etbn1:/config/ttdp/#> local-id 1 etbn1:/config/ttdp/#> ecsp-iface vlan10 etbn1:/config/ttdp/#> ecsp-leader-addr 10.0.0.3 etbn1:/config/ttdp/#> ecsc-addr 10.0.32.99 etbn1:/config/ttdp/#> ecn-over-etb 10 etbn1:/config/ttdp/#> end TTDP: Created VLAN 492. Note that this VLAN will not be deleted automatically if TTDP is deactivated etbn1:/config/#> vlan 2 etbn1:/config/vlan-2/#> untagged lag-dir1 lag-dir2 etbn1:/config/vlan-2/#> leave Applying configuration. Configuration activated. Remember "copy run start" to save to flash (NVRAM). etbn1:/#> cp ru st etbn1:/#>
ETBN2 configuration
The configuration for ETBN2 is similar. The changed values have been marked in orange text. The static interface address and VRRP priority are different.
etbn2:/#> config etbn2:/config/#> vlan 10 Creating new VLAN vid:10 with name: vlan10 etbn2:/config/vlan-10/#> untagged ethX3 Moving untagged port eth4 from vid 1 to vid 10. etbn2:/config/vlan-10/#> end etbn2:/config/#> iface vlan10 etbn2:/config/iface-vlan10/#> no inet etbn2:/config/iface-vlan10/#> inet static 10.0.0.2/19 etbn2:/config/iface-vlan10/inet-static-10.0.0.2/#> end etbn2:/config/iface-vlan10/#> end etbn2:/config/#> vlan 2 Creating new VLAN vid:2 with name: vlan2 etbn2:/config/vlan-2/#> no multicast-snooping etbn2:/config/vlan-2/#> end etbn2:/config/#> router etbn2:/config/router/#> vrrp ecn1 Creating new VRRP instance: ecn1 Invalid settings: Interface not set. etbn2:/config/router/vrrp-ecn1/#> iface vlan10 Invalid settings: Invalid IPv4 address. etbn2:/config/router/vrrp-ecn1/#> address 10.0.0.10 Invalid settings: VRID invalid. Valid values: 1 to 255. etbn2:/config/router/vrrp-ecn1/#> vrid 1 etbn2:/config/router/vrrp-ecn1/#> priority 80 etbn2:/config/router/vrrp-ecn1/#> end etbn2:/config/router/#> end etbn2:/config/#> ip etbn2:/config/ip/#> route 10.0.32.0/19 10.0.0.100 etbn2:/config/ip/#> end etbn2:/config/#>
The port configuration for ETBN2 is identical to that of ETBN1.
etbn1:/config/#> etbn1:/config/#> port ethX5 etbn1:/config/port-ethX5/#> speed-duplex 100-full etbn1:/config/port-ethX5/#> no auto-negotiate etbn1:/config/port-ethX5/#> fastlink etbn1:/config/port-ethX5/#> mdix-mode mdi etbn1:/config/port-ethX5/#> end etbn1:/config/#> port ethX6 etbn1:/config/port-ethX6/#> speed-duplex 100-full etbn1:/config/port-ethX6/#> no auto-negotiate etbn1:/config/port-ethX6/#> fastlink etbn1:/config/port-ethX6/#> mdix-mode mdi etbn1:/config/port-ethX6/#> end etbn1:/config/#> port ethX7 etbn1:/config/port-ethX7/#> speed-duplex 100-full etbn1:/config/port-ethX7/#> no auto-negotiate etbn1:/config/port-ethX7/#> fastlink etbn1:/config/port-ethX7/#> mdix-mode mdi etbn1:/config/port-ethX7/#> end etbn1:/config/#> port ethX8 etbn1:/config/port-ethX8/#> speed-duplex 100-full etbn1:/config/port-ethX8/#> no auto-negotiate etbn1:/config/port-ethX8/#> fastlink etbn1:/config/port-ethX8/#> mdix-mode mdi etbn1:/config/port-ethX8/#> end etbn1:/config/#>
For the TTDP configuration, only the local ETBN id must be changed. Save and commit the configuration.
etbn2:/config/#>
etbn2:/config/#> ttdp
Activating TTDP with default settings.
etbn2:/config/ttdp/#> uuid 11111111-1111-1111-1111-111111111111
etbn2:/config/ttdp/#> node 1 ecn 1
etbn2:/config/ttdp/#> node 2 ecn 1
etbn2:/config/ttdp/#> ecn 1 vlan10 18
etbn2:/config/ttdp/#> dir1 ethX5 ethX6
etbn2:/config/ttdp/#> dir2 ethX7 ethX8
etbn2:/config/ttdp/#> local-id 2
etbn2:/config/ttdp/#> ecsp-iface vlan10
etbn2:/config/ttdp/#> ecsp-leader-addr 10.0.0.3
etbn2:/config/ttdp/#> ecsc-addr 10.0.32.99
etbn2:/config/ttdp/#> ecn-over-etb 10
etbn2:/config/ttdp/#> end
TTDP: Created VLAN 492. Note that this VLAN will not be deleted automatically if TTDP is deactivated
etbn2:/config/#> vlan 2
etbn2:/config/vlan-2/#> untagged lag-dir1 lag-dir2
etbn2:/config/vlan-2/#> leave
Applying configuration.
Configuration activated.
Remember "copy run start" to save to flash (NVRAM).
etbn2:/#> cp ru st
etbn2:/#>
Result
Once the configuration is applied and saved, the TTDP stack should inaugurate,
and once that happens the “ECN over ETB” function will overlay ECN traffic
between ETBN1 and ETBN2 on top of the ETB. To ensure that this is the case,
verify connectivity between H1 and H2 (e.g., via ping).
Ensure that VRRP is operating correctly - exactly one of the ETBNs should be in
MASTER state (use show vrrp on the ETBNs to verify this), and the VRRP
instances on the GW devices should also behave correctly.
Also, verify connectivity between the ECSC and the ETBNs, using ping and/or
the ECSP interfaces.