WeOS
5.29.0

# Split ECN over ETB HowTo (advanced)

This document describes a specific use case, where two ECNs and a non-ECN network are overlaid on top of the ETB, thus not needing dedicated ECN cabling between the ETBNs.

This can be useful in certain cases where adding another physical cable (in addition to the regular ETB cabling) is difficult between the ETBNs, or in cases where a direct connection between the ETBNs is prevented by the ECN architecture.

In the case presented below, there are two ETBNs (ETBN1, ETBN2) connected to the same ECN, without dedicated ECN cabling. There is also a redundant pair of externally managed firewall devices (GW1, GW2), splitting the ECN into two subnets. As the firewall devices are considered third-party devices, only configuration hints for them will be provided. There are also two end devices, or hosts (H1, H2) added for illustration.

The VLAN used for the ECN in this scenario will be vlan10. When using the “ECN over ETB” functionality, it is important that the same VID be used for each ECN across all participating ETBNs in the consist, since the ECN traffic is sent across the ETB using VLAN tags.

This use case utilises the “ECN subnetting” functionality. While not strictly needed to use the “ECN over ETB” function, the two can be used together for greater effect. In effect, this feature lets the ECN be split into two distinct subnets locally, while still presented as one ECN to the rest of the train.

There are three VRRP virtual routers used in this example, the ETBNs being one of them (this is the usual setup for redundant ECNs when using TTDP: see the redundant ETBN HowTo), with the VIP address 10.0.0.10. This address is used by every device on the combined ECN as the default route, in order to facilitate traffic via the ETB, as in a normal TTDP scenario. The other two VRRP routers are handled by the firewall devices, one instance on each half of the ECN. The VIP addresses of each of these is then used by devices to reach the other half of the ECN. These are 10.0.0.100 for the upper part of the ECN, and 10.0.32.100 for the lower part.

Schematically the network topology looks as follows.

            .--------------------.
           /         cst1         \
           '----------------------'
            .-------.    .-------.
          --+       +----+       +--     dir1 = (ethX5, ethX6)
      <-dir1| ETBN1 |    | ETBN2 |dir2-> dir2 = (ethX7, ethX8)
          --+       +----+       +--
            '---+---'    '---+---'
           ethX3|.1     ethX3|.2
                |            |
         .----. |            | .----.
         | H1 +-+ .11    .12 +-+ H2 |             ^
         '----' |            | '----'             |
                |            |                    |
            eth0|.21     eth0|.22          "Upper" part of ECN
            .---+---.    .---+---.         (disjoint)
            |       |    |       |         10.0.0.0/19
            |  GW1  |    |  GW2  |         -------------------
            |       |    |       |         "Lower" part of ECN
            '---+---'    '---+---'         (joint)
            eth1|.21     eth1|.22          10.0.32.0/19
                |            |                    |
            ----+-----+------+----                |
                      | .99                       V
                   .--+---.
                   | ECSC |
                   '------'

Figure 1: Consist network layout.

Note

As far as the IEC61375 stack is concerned, there is just one ECN in the consist. This is what TTDP communicates to any other coupled consists. This entire ECN has the address space 10.0.0.0/18, and it will be mapped to one /18 subnet in the train scope, as usual. Internally, however, this ECN is split up into two /19 networks, on either side of the FW devices. This requires some special configuration on the ETBNs, and is made possible by the “ECN subnetting” feature.

Note

Note also that the upper part of the ECN is disjoint - there is no direct path between ED1 and ED2 on the 10.0.0.0/19 network. For the two to communicate directly, the network needs to be overlaid on top of the ETB cabling. This is made possible by the “ECN over ETB” feature.

## Configuration

### Non-WeOS configuration

The network shown in Figure 1 contains some non-WeOS devices, for illustration as well as context. It is assumed that they are configured as follows. Note that link-local routes are not explicitly specified (e.g., it is assumed that a device with an address of a subnet has a link-local route to that subnet). Note also that ED1 and ED2 can be configured using DHCP, including static routes - see the DHCP Server HowTo and the DHCP Server documentation, in particular the `route` subnet setting.

    ED1: address 10.0.0.11/19
         static route to 10.0.32.0/19 via 10.0.0.100
         default gateway 10.0.0.10

    ED2: address 10.0.0.12/19
         static route to 10.0.32.0/19 via 10.0.0.100
         default gateway 10.0.0.10

    GW1: eth0 (upper) address 10.0.0.21/19
         eth1 (lower) address 10.0.32.21/19
         IP forwarding enabled between eth0 and eth1
         default gateway 10.0.0.10
         VRRP configuration:
           VRRP instance 1:
             base interface: eth0 (upper)
             address 10.0.0.100
           VRRP instance 2:
             base interface: eth1 (lower)
             address 10.0.32.100

    GW2: eth0 (upper) address 10.0.0.22/19
         eth1 (lower) address 10.0.32.22/19
         IP forwarding enabled between eth0 and eth1
         default gateway 10.0.0.10
         VRRP configuration:
           VRRP instance 1:
             base interface: eth0 (upper)
             address 10.0.0.100
           VRRP instance 2:
             base interface: eth1 (lower)
             address 10.0.32.100

    ECSC: address 10.0.32.99/19
          default gateway 10.0.32.100
          ECSP address 10.0.0.10

### ETBN1 configuration

Start by configuring the required VLAN and VRRP settings on ETBN1. Also, add a route to 10.0.32.0/19 (the lower part of the ECN), using the upper VIP address of the GW devices as the router.

    etbn1:/#> config
    etbn1:/config/#> vlan 10
    Creating new VLAN vid:10 with name: vlan10
    etbn1:/config/vlan-10/#> untagged ethX3
    Moving untagged port eth4 from vid 1 to vid 10.
    etbn1:/config/vlan-10/#> end
    etbn1:/config/#> iface vlan10
    etbn1:/config/iface-vlan10/#> no inet
    etbn1:/config/iface-vlan10/#> inet static 10.0.0.1/19
    etbn1:/config/iface-vlan10/inet-static-10.0.0.1/#> end
    etbn1:/config/iface-vlan10/#> end
    etbn1:/config/#> vlan 2
    Creating new VLAN vid:2 with name: vlan2
    etbn1:/config/vlan-2/#> no multicast-snooping
    etbn1:/config/vlan-2/#> end
    etbn1:/config/#> router
    etbn1:/config/router/#> vrrp ecn1
    Creating new VRRP instance: ecn1
    Invalid settings: Interface not set.
    etbn1:/config/router/vrrp-ecn1/#> iface vlan10
    Invalid settings: Invalid IPv4 address.
    etbn1:/config/router/vrrp-ecn1/#> address 10.0.0.10
    Invalid settings: VRID invalid. Valid values: 1 to 255.
    etbn1:/config/router/vrrp-ecn1/#> vrid 1
    etbn1:/config/router/vrrp-ecn1/#> priority 100
    etbn1:/config/router/vrrp-ecn1/#> end
    etbn1:/config/router/#> end
    etbn1:/config/#> ip
    etbn1:/config/ip/#> route 10.0.32.0/19 10.0.0.100
    etbn1:/config/ip/#> end
    etbn1:/config/#>

Then, set up the necessary ETB port settings.

    etbn1:/config/#>
    etbn1:/config/#> port ethX5
    etbn1:/config/port-ethX5/#> speed-duplex 100-full
    etbn1:/config/port-ethX5/#> no auto-negotiate
    etbn1:/config/port-ethX5/#> fastlink
    etbn1:/config/port-ethX5/#> mdix-mode mdi
    etbn1:/config/port-ethX5/#> end
    etbn1:/config/#> port ethX6
    etbn1:/config/port-ethX6/#> speed-duplex 100-full
    etbn1:/config/port-ethX6/#> no auto-negotiate
    etbn1:/config/port-ethX6/#> fastlink
    etbn1:/config/port-ethX6/#> mdix-mode mdi
    etbn1:/config/port-ethX6/#> end
    etbn1:/config/#> port ethX7
    etbn1:/config/port-ethX7/#> speed-duplex 100-full
    etbn1:/config/port-ethX7/#> no auto-negotiate
    etbn1:/config/port-ethX7/#> fastlink
    etbn1:/config/port-ethX7/#> mdix-mode mdi
    etbn1:/config/port-ethX7/#> end
    etbn1:/config/#> port ethX8
    etbn1:/config/port-ethX8/#> speed-duplex 100-full
    etbn1:/config/port-ethX8/#> no auto-negotiate
    etbn1:/config/port-ethX8/#> fastlink
    etbn1:/config/port-ethX8/#> mdix-mode mdi
    etbn1:/config/port-ethX8/#> end
    etbn1:/config/#>

Note

In this HowTo, it is assumed that the ETB uses crossover cabling. For this reason, all ETB ports on both ETBNs use the same MDI-X mode. If non-crossover cabling is used, these settings must be changed for two of the inward-facing ETB ports.

Finally, set up the TTDP configuration and add the ETB link aggregates to the ETB VLAN. Note specifically the use of the `ecn-over-etb` setting, where we indicate that VID 10 is to be forwarded over the ETB cabling.

Note

There is an extra parameter, `18`, given to the `ecn` command. This is normally omitted in the TTDP configuration, but is required here for the “ECN subnetting” functionality. What this does is to let TTDP know that even though the device uses a /19 address on `vlan10`, the ECN, as presented in TTDP, should use the entire /18 space (the other /19 net is reachable via the GW devices).

    etbn1:/config/#>
    etbn1:/config/#> ttdp
    Activating TTDP with default settings.
    etbn1:/config/ttdp/#> uuid 11111111-1111-1111-1111-111111111111
    etbn1:/config/ttdp/#> node 1 ecn 1
    etbn1:/config/ttdp/#> node 2 ecn 1
    etbn1:/config/ttdp/#> ecn 1 vlan10 18
    etbn1:/config/ttdp/#> dir1 ethX5 ethX6
    etbn1:/config/ttdp/#> dir2 ethX7 ethX8
    etbn1:/config/ttdp/#> local-id 1
    etbn1:/config/ttdp/#> ecsp-iface vlan10
    etbn1:/config/ttdp/#> ecsp-leader-addr 10.0.0.3
    etbn1:/config/ttdp/#> ecsc-addr 10.0.32.99
    etbn1:/config/ttdp/#> ecn-over-etb 10
    etbn1:/config/ttdp/#> end
    TTDP: Created VLAN 492. Note that this VLAN will not be deleted automatically if TTDP is deactivated
    etbn1:/config/#> vlan 2
    etbn1:/config/vlan-2/#> untagged lag-dir1 lag-dir2
    etbn1:/config/vlan-2/#> leave
    Applying configuration.
    Configuration activated.
    Remember "copy run start" to save to flash (NVRAM).
    etbn1:/#> cp ru st
    etbn1:/#>

### ETBN2 configuration

The configuration for ETBN2 is similar. The changed values have been marked in orange text. The static interface address and VRRP priority are different.

    etbn2:/#> config
    etbn2:/config/#> vlan 10
    Creating new VLAN vid:10 with name: vlan10
    etbn2:/config/vlan-10/#> untagged ethX3
    Moving untagged port eth4 from vid 1 to vid 10.
    etbn2:/config/vlan-10/#> end
    etbn2:/config/#> iface vlan10
    etbn2:/config/iface-vlan10/#> no inet
    etbn2:/config/iface-vlan10/#> inet static 10.0.0.2/19
    etbn2:/config/iface-vlan10/inet-static-10.0.0.2/#> end
    etbn2:/config/iface-vlan10/#> end
    etbn2:/config/#> vlan 2
    Creating new VLAN vid:2 with name: vlan2
    etbn2:/config/vlan-2/#> no multicast-snooping
    etbn2:/config/vlan-2/#> end
    etbn2:/config/#> router
    etbn2:/config/router/#> vrrp ecn1
    Creating new VRRP instance: ecn1
    Invalid settings: Interface not set.
    etbn2:/config/router/vrrp-ecn1/#> iface vlan10
    Invalid settings: Invalid IPv4 address.
    etbn2:/config/router/vrrp-ecn1/#> address 10.0.0.10
    Invalid settings: VRID invalid. Valid values: 1 to 255.
    etbn2:/config/router/vrrp-ecn1/#> vrid 1
    etbn2:/config/router/vrrp-ecn1/#> priority 80
    etbn2:/config/router/vrrp-ecn1/#> end
    etbn2:/config/router/#> end
    etbn2:/config/#> ip
    etbn2:/config/ip/#> route 10.0.32.0/19 10.0.0.100
    etbn2:/config/ip/#> end
    etbn2:/config/#>

The port configuration for ETBN2 is identical to that of ETBN1.

    etbn1:/config/#>
    etbn1:/config/#> port ethX5
    etbn1:/config/port-ethX5/#> speed-duplex 100-full
    etbn1:/config/port-ethX5/#> no auto-negotiate
    etbn1:/config/port-ethX5/#> fastlink
    etbn1:/config/port-ethX5/#> mdix-mode mdi
    etbn1:/config/port-ethX5/#> end
    etbn1:/config/#> port ethX6
    etbn1:/config/port-ethX6/#> speed-duplex 100-full
    etbn1:/config/port-ethX6/#> no auto-negotiate
    etbn1:/config/port-ethX6/#> fastlink
    etbn1:/config/port-ethX6/#> mdix-mode mdi
    etbn1:/config/port-ethX6/#> end
    etbn1:/config/#> port ethX7
    etbn1:/config/port-ethX7/#> speed-duplex 100-full
    etbn1:/config/port-ethX7/#> no auto-negotiate
    etbn1:/config/port-ethX7/#> fastlink
    etbn1:/config/port-ethX7/#> mdix-mode mdi
    etbn1:/config/port-ethX7/#> end
    etbn1:/config/#> port ethX8
    etbn1:/config/port-ethX8/#> speed-duplex 100-full
    etbn1:/config/port-ethX8/#> no auto-negotiate
    etbn1:/config/port-ethX8/#> fastlink
    etbn1:/config/port-ethX8/#> mdix-mode mdi
    etbn1:/config/port-ethX8/#> end
    etbn1:/config/#>

For the TTDP configuration, only the local ETBN id must be changed. Save and commit the configuration.

    etbn2:/config/#>
    etbn2:/config/#> ttdp
    Activating TTDP with default settings.
    etbn2:/config/ttdp/#> uuid 11111111-1111-1111-1111-111111111111
    etbn2:/config/ttdp/#> node 1 ecn 1
    etbn2:/config/ttdp/#> node 2 ecn 1
    etbn2:/config/ttdp/#> ecn 1 vlan10 18
    etbn2:/config/ttdp/#> dir1 ethX5 ethX6
    etbn2:/config/ttdp/#> dir2 ethX7 ethX8
    etbn2:/config/ttdp/#> local-id 2
    etbn2:/config/ttdp/#> ecsp-iface vlan10
    etbn2:/config/ttdp/#> ecsp-leader-addr 10.0.0.3
    etbn2:/config/ttdp/#> ecsc-addr 10.0.32.99
    etbn2:/config/ttdp/#> ecn-over-etb 10
    etbn2:/config/ttdp/#> end
    TTDP: Created VLAN 492. Note that this VLAN will not be deleted automatically if TTDP is deactivated
    etbn2:/config/#> vlan 2
    etbn2:/config/vlan-2/#> untagged lag-dir1 lag-dir2
    etbn2:/config/vlan-2/#> leave
    Applying configuration.
    Configuration activated.
    Remember "copy run start" to save to flash (NVRAM).
    etbn2:/#> cp ru st
    etbn2:/#>

## Result

Once the configuration is applied and saved, the TTDP stack should inaugurate, and once that happens the “ECN over ETB” function will overlay ECN traffic between ETBN1 and ETBN2 on top of the ETB. To ensure that this is the case, verify connectivity between H1 and H2 (e.g., via `ping`).

Ensure that VRRP is operating correctly - exactly one of the ETBNs should be in MASTER state (use `show vrrp` on the ETBNs to verify this), and the VRRP instances on the GW devices should also behave correctly.

Also, verify connectivity between the ECSC and the ETBNs, using `ping` and/or the ECSP interfaces.

« Previous Next »