Close Menu
    Tuesday, August 25
    Virtualization

    Routing Down on Tier-0 Service Router

    DaniloBy Updated:No Comments12 Mins Read
    Tier-0 Service Router routing down

    Routing Down on Tier-0 Service Router is an article that explains what happens if a Tier-0 Service Router (SR) loses its routing adjacencies.

    As we know, in the NSX world we have two gateway types:

    • T0 or Tier-0
    • T1 or Tier-1

    Both gateways are hosted in the Edge Transport Node device!
    Each one is responsible for some activities in the NSX topology and has different characteristics.
    We wrote an article that shows some examples of NSX topologies and the meaning of each gateway type and its interfaces. You can access this article by clicking on the following link:
    Some Examples of NSX Topologies – DPC Virtual Tips

    T0 Service Router (SR)

    The Tier-0 Service Router (SR) is responsible for handling the north-south traffic. To do that, uplink interfaces are created and configured in this service router. Commonly, we use dynamic routing protocols to establish adjacencies within the physical routers to provide north-south communication.

    In the following picture, we have a lab topology just to explain the concept to you:

    • We have both physical routers (router01 and router02). Both are placed outside the NSX environment (outside the virtual “world”);
    • We have an edge cluster with four (4) edge transport nodes (edge01, edge02, edge03, and edge04);
    • We have a vSAN cluster with three (3) ESXi hosts, already prepared to work as a host transport node.

    As we can see in the above picture, the “transit segment” is used by the transport node devices (for their internal communication) and an address in the network range 169.254.X.X is assigned for each one. All host transport nodes shared the same address (169.254.0.1), as we can see in the following output. This interface is the LIF (Logical Interface):

    HOST-01:

    vhost-comp-01.lab.local# nsxcli
vhost-comp-01.lab.local> get gateways
Wed Jan 31 2024 UTC 13:42:21.023
                                      Gateways Summary
 ------------------------------------------------------------------------------------------
               VDR UUID                LIF num   IPv4 Route num   IPv6 Route num  Max Neighbors  Current Neighbors
 c08ecb08-e21e-4d4b-9163-f66d2553bec7     4             7                5            50000              16


vhost-comp-01.lab.local> get gateway c08ecb08-e21e-4d4b-9163-f66d2553bec7 interface
Wed Jan 31 2024 UTC 13:42:23.007
                            Gateway Interfaces
---------------------------------------------------------------------------
IPv6 DAD Status Legend:  [A: DAD_Sucess], [F: DAD_Duplicate], [T: DAD_Tentative], [U: DAD_Unavailable]

LIF UUID                 : ba201756-d373-469c-a309-ee313a4eb9bc
Mode                     : [b'Routing-Backplane']
Overlay VNI              : 73728
IP/Mask                  : 169.254.0.1/28;  fe80::50:56ff:fe56:4452/128(U)
Mac                      : 02:50:56:56:44:52
Connected DVS            : VDS-Computing
Control plane enable     : True
Replication Mode         : 0.0.0.1
Multicast Routing        : [b'Enabled', b'Oper Down']
State                    : [b'Enabled']
Flags                    : 0x90308
DHCP relay               : Not enable
DAD-mode                 : ['LOOSE']
RA-mode                  : ['SLAAC_DNS_THROUGH_RA(M=0, O=0)']
...
...

    HOST-02:

    vhost-comp-02.lab.local# nsxcli
vhost-comp-02.lab.local> get gateways
Wed Jan 31 2024 UTC 13:44:56.056
                                      Gateways Summary
 ------------------------------------------------------------------------------------------
               VDR UUID                LIF num   IPv4 Route num   IPv6 Route num  Max Neighbors  Current Neighbors
 c08ecb08-e21e-4d4b-9163-f66d2553bec7     4             7                5            50000              11

vhost-comp-02.lab.local> get gateway c08ecb08-e21e-4d4b-9163-f66d2553bec7 interface
Wed Jan 31 2024 UTC 13:44:58.424
                            Gateway Interfaces
---------------------------------------------------------------------------
IPv6 DAD Status Legend:  [A: DAD_Sucess], [F: DAD_Duplicate], [T: DAD_Tentative], [U: DAD_Unavailable]

LIF UUID                 : ba201756-d373-469c-a309-ee313a4eb9bc
Mode                     : [b'Routing-Backplane']
Overlay VNI              : 73728
IP/Mask                  : 169.254.0.1/28;  fe80::50:56ff:fe56:4452/128(U)
Mac                      : 02:50:56:56:44:52
Connected DVS            : VDS-Computing
Control plane enable     : True
Replication Mode         : 0.0.0.1
Multicast Routing        : [b'Enabled', b'Oper Down']
State                    : [b'Enabled']
Flags                    : 0x90308
DHCP relay               : Not enable
DAD-mode                 : ['LOOSE']
RA-mode                  : ['SLAAC_DNS_THROUGH_RA(M=0, O=0)']
...
...

    HOST-03:

    vhost-comp-03.lab.local# nsxcli
vhost-comp-03.lab.local> get gateways
Wed Jan 31 2024 UTC 13:46:14.946
                                      Gateways Summary
 ------------------------------------------------------------------------------------------
               VDR UUID                LIF num   IPv4 Route num   IPv6 Route num  Max Neighbors  Current Neighbors
 c08ecb08-e21e-4d4b-9163-f66d2553bec7     4             7                5            50000              11


vhost-comp-03.lab.local> get gateway c08ecb08-e21e-4d4b-9163-f66d2553bec7 interface
Wed Jan 31 2024 UTC 13:46:20.098
                            Gateway Interfaces
---------------------------------------------------------------------------
IPv6 DAD Status Legend:  [A: DAD_Sucess], [F: DAD_Duplicate], [T: DAD_Tentative], [U: DAD_Unavailable]

LIF UUID                 : ba201756-d373-469c-a309-ee313a4eb9bc
Mode                     : [b'Routing-Backplane']
Overlay VNI              : 73728
IP/Mask                  : 169.254.0.1/28;  fe80::50:56ff:fe56:4452/128(U)
Mac                      : 02:50:56:56:44:52
Connected DVS            : VDS-Computing
Control plane enable     : True
Replication Mode         : 0.0.0.1
Multicast Routing        : [b'Enabled', b'Oper Down']
State                    : [b'Enabled']
Flags                    : 0x90308
DHCP relay               : Not enable
DAD-mode                 : ['LOOSE']
RA-mode                  : ['SLAAC_DNS_THROUGH_RA(M=0, O=0)']
...
...

    Each edge node has an interface using the address on this network (backplane port), as we can see in the following outputs:

    EDGE-01:

        Interface     : 6289a5dc-65c2-40a2-b6bc-53f3bd7b4770
    Ifuid         : 275
    Name          : bp-sr0-port
    Fwd-mode      : IPV4_ONLY
    Internal name : downlink-275
    Mode          : lif
    Port-type     : backplane
    IP/Mask       : 169.254.0.2/25;fe80::50:56ff:fe56:5300/64(NA)
    MAC           : 02:50:56:56:53:00
    VNI           : 66561
    Access-VLAN   : untagged
    Segment port  : 1fb264c0-ecd6-4dcc-9856-0f759ec65e62
    Urpf-mode     : NONE
    DAD-mode      : LOOSE
    RA-mode       : RA_INVALID
    Admin         : up
    Op_state      : up
    MTU           : 1500
    arp_proxy     :

    EDGE-02:

        Interface     : 1207a97d-691a-4db7-b1e0-c9097efb30c8
    Ifuid         : 275
    Name          : bp-sr1-port
    Fwd-mode      : IPV4_ONLY
    Internal name : downlink-275
    Mode          : lif
    Port-type     : backplane
    IP/Mask       : 169.254.0.3/25;fe80::50:56ff:fe56:5301/64(NA)
    MAC           : 02:50:56:56:53:01
    VNI           : 66561
    Access-VLAN   : untagged
    Segment port  : df2e0cc7-606c-4100-91e3-ed32ba081d78
    Urpf-mode     : NONE
    DAD-mode      : LOOSE
    RA-mode       : RA_INVALID
    Admin         : up
    Op_state      : up
    MTU           : 1500
    arp_proxy     :

    EDGE-03:

        Interface     : 3aad399a-bb37-4d82-81d5-6111b4802863
    Ifuid         : 285
    Name          : bp-sr2-port
    Fwd-mode      : IPV4_ONLY
    Internal name : downlink-285
    Mode          : lif
    Port-type     : backplane
    IP/Mask       : 169.254.0.4/25;fe80::50:56ff:fe56:5302/64(NA)
    MAC           : 02:50:56:56:53:02
    VNI           : 66561
    Access-VLAN   : untagged
    Segment port  : 61a1d1b8-3d43-4e01-80e2-1918213b055f
    Urpf-mode     : NONE
    DAD-mode      : LOOSE
    RA-mode       : RA_INVALID
    Admin         : up
    Op_state      : up
    MTU           : 1500
    arp_proxy     :

    EDGE-04:

        Interface     : 08b7faef-3aab-421d-9018-a011c7ed1bad
    Ifuid         : 299
    Name          : bp-sr3-port
    Fwd-mode      : IPV4_ONLY
    Internal name : downlink-299
    Mode          : lif
    Port-type     : backplane
    IP/Mask       : 169.254.0.5/25;fe80::50:56ff:fe56:5303/64(NA)
    MAC           : 02:50:56:56:53:03
    VNI           : 66561
    Access-VLAN   : untagged
    Segment port  : fda7ed2e-dcd1-4e04-8950-dfab9b7f3500
    Urpf-mode     : NONE
    DAD-mode      : LOOSE
    RA-mode       : RA_INVALID
    Admin         : up
    Op_state      : up
    MTU           : 1500
    arp_proxy     :

    Additionally, we can access the service router and confirm its BGP neighbors.
    For edge-01, for instance, we have both physical routers as neighbors (IPs 10.0.22.1 and 10.0.23.1):

    edge-01> get logical-routers
Wed Jan 31 2024 UTC 14:03:32.358
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                        Ports   Neighbors
736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                        TUNNEL                      3       3/5000
20169f10-8099-4d27-8915-8e1e0d47e68e   1      2050   DR-T0-GW                          DISTRIBUTED_ROUTER_TIER0    5       1/50000
c9e7c979-0d69-4aab-bd6c-9ee863c96070   2      3074   SR-T0-GW                          SERVICE_ROUTER_TIER0        7       5/50000

edge-01> vrf 2

edge-01(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.0.22.11  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.0.22.1                           65100       Estab 02:16:15     UP  1594    2004    6      6
10.0.23.1                           65100       Estab 02:14:50     UP  1594    2002    6      12

    The same occurs for other edge nodes.
    Look that we are using the BFD (Bidirectional Forwarding Detection) on both BGP peers. The protocol BFD helps us to identify if the peer is alive or not.

    Simulating a Routing Failure on Edge-01

    We will simulate a routing failure on the service router hosted on the edge transport node 01 (edge-01).
    To simulate a routing failure, we will do:

    • On each physical router, we will shut down (administratively) the BGP peer to the edge-01;
    • And we will disable the BFD configuration on both physical routers as well.

    What is the expected behavior?
    The LIF (Logical Interface) on edge-01 will be moved temporarily to another edge transport node!

    The picture below shows the failure scenario (in this example, the LIF interface from edge-01 is moving to edge-02. But, other edge nodes can be selected to receive this interface – this is an automatic process):

    Checking the BGP peers on each physical router. Both peers 10.0.22.11 and 10.0.23.11 are to the edge-01 (from VLAN 22 and VLAN 23):

    PHYSICAL ROUTER-01:

    vyos@vyos-a:~$ show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.255.8, local AS number 65100 vrf-id 0
BGP table version 35
RIB entries 21, using 4032 bytes of memory
Peers 7, using 143 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.0.20.2       4      65100      1706      1706       35    0    0 1d04h13m           10       10 N/A
10.0.21.2       4      65100      1706      1706       35    0    0 1d04h13m           10       10 N/A
10.0.22.11      4      65000      1591      1605       35    0    0 02:27:53            6       12 N/A
10.0.22.12      4      65000      1654      1665       35    0    0 1d03h22m            6       12 N/A
10.0.22.13      4      65000      1655      1662       35    0    0 1d03h21m            6       12 N/A
10.0.22.14      4      65000      1651      1662       35    0    0 1d03h21m            6       12 N/A
10.0.200.1      4      65100      1698      1707       35    0    0 1d04h14m            1       10 N/A

    PHYSICAL ROUTER-02:

    vyos@vyos-b:~$ show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.255.9, local AS number 65100 vrf-id 0
BGP table version 35
RIB entries 21, using 4032 bytes of memory
Peers 7, using 143 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.0.20.1       4      65100      1707      1707       35    0    0 1d04h14m           10       10 N/A
10.0.21.1       4      65100      1707      1707       35    0    0 1d04h14m           10       10 N/A
10.0.23.11      4      65000      1606      1608       35    0    0 02:27:47            6       12 N/A
10.0.23.12      4      65000      1659      1666       35    0    0 1d03h23m            6       12 N/A
10.0.23.13      4      65000      1653      1665       35    0    0 1d03h23m            6       12 N/A
10.0.23.14      4      65000      1657      1665       35    0    0 1d03h23m            6       12 N/A
10.0.201.1      4      65100      1698      1707       35    0    0 1d04h14m            1       10 N/A

    Disabling the BGP peers for the edge-01 and removing the BFD configuration as well.
    Look the BGP peer is “Idle (Admin)”. It means that we disabled administratively this BGP peer:

    PHYSICAL ROUTER-01:

    set protocols bgp neighbor 10.0.22.11 shutdown
delete protocols bgp neighbor 10.0.22.11 bfd

vyos@vyos-a:~$ show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.255.8, local AS number 65100 vrf-id 0
BGP table version 35
RIB entries 21, using 4032 bytes of memory
Peers 7, using 143 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.0.20.2       4      65100      1711      1711       35    0    0 1d04h18m           10       10 N/A
10.0.21.2       4      65100      1711      1711       35    0    0 1d04h18m           10       10 N/A
10.0.22.11      4      65000      1596      1610        0    0    0 00:00:10 Idle (Admin)        0 N/A
10.0.22.12      4      65000      1659      1670       35    0    0 1d03h27m            6       12 N/A
10.0.22.13      4      65000      1661      1668       35    0    0 1d03h27m            6       12 N/A
10.0.22.14      4      65000      1657      1668       35    0    0 1d03h27m            6       12 N/A
10.0.200.1      4      65100      1703      1712       35    0    0 1d04h19m            1       10 N/A

    PHYSICAL ROUTER-02:

    set protocols bgp neighbor 10.0.23.11 shutdown
delete protocols bgp neighbor 10.0.23.11 bfd

vyos@vyos-b:~$ show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.255.9, local AS number 65100 vrf-id 0
BGP table version 35
RIB entries 21, using 4032 bytes of memory
Peers 7, using 143 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.0.20.1       4      65100      1713      1713       35    0    0 1d04h20m           10       10 N/A
10.0.21.1       4      65100      1713      1713       35    0    0 1d04h20m           10       10 N/A
10.0.23.11      4      65000      1613      1614        0    0    0 00:00:08 Idle (Admin)        0 N/A
10.0.23.12      4      65000      1665      1672       35    0    0 1d03h29m            6       12 N/A
10.0.23.13      4      65000      1659      1671       35    0    0 1d03h29m            6       12 N/A
10.0.23.14      4      65000      1663      1671       35    0    0 1d03h29m            6       12 N/A
10.0.201.1      4      65100      1704      1713       35    0    0 1d04h20m            1       10 N/A

    Checking the BGP peer status on edge-01. Both peers are “Active” and not “Established”:

    edge-01(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.0.22.11  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.0.22.1                           65100       Activ 00:03:46     DC  1610    2043    0      0
10.0.23.1                           65100       Activ 00:01:39     DC  1613    2032    0      0

    Now, accessing each edge transport node and seeing the backplane interfaces, we can see (in this case) that the LIF IP address has been assigned to the edge-03, as we can confirm in the following output:

        Interface     : 3aad399a-bb37-4d82-81d5-6111b4802863
    Ifuid         : 285
    Name          : bp-sr2-port
    Fwd-mode      : IPV4_ONLY
    Internal name : downlink-285
    Mode          : lif
    Port-type     : backplane
    IP/Mask       : 169.254.0.2/25;169.254.0.4/25;fe80::50:56ff:fe56:5300/64(NA);fe80::50:56ff:fe56:5302/64(NA)
    MAC           : 02:50:56:56:53:02
    VNI           : 66561
    Access-VLAN   : untagged
    Segment port  : 61a1d1b8-3d43-4e01-80e2-1918213b055f
    Urpf-mode     : NONE
    DAD-mode      : LOOSE
    RA-mode       : RA_INVALID
    Admin         : up
    Op_state      : up
    MTU           : 1500
    arp_proxy     :

    With the above output, we confirmed that the LIF was moved to another edge transport node and the forwarding and routing processes can continue through this IP 169.254.0.2!

    Removing the Routing Failure on Edge-01

    As the routing peers on edge-01 can be recovered, the LIF IP address will move back to edge-01 immediately:

    edge-01> get gateways
Wed Jan 31 2024 UTC 14:36:51.000
Gateway
UUID                                   VRF    Gateway-ID   Name                              Type                        Ports   Neighbors
736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                              TUNNEL                      3       3/5000
20169f10-8099-4d27-8915-8e1e0d47e68e   1      2050         DR-T0-GW                          DISTRIBUTED_ROUTER_TIER0    5       1/50000
c9e7c979-0d69-4aab-bd6c-9ee863c96070   2      3074         SR-T0-GW                          SERVICE_ROUTER_TIER0        7       5/50000


edge-01> get gateway 20169f10-8099-4d27-8915-8e1e0d47e68e interfaces

...
...
Interface     : 6289a5dc-65c2-40a2-b6bc-53f3bd7b4770
    Ifuid         : 275
    Name          : bp-sr0-port
    Fwd-mode      : IPV4_ONLY
    Internal name : downlink-275
    Mode          : lif
    Port-type     : backplane
    IP/Mask       : 169.254.0.2/25;fe80::50:56ff:fe56:5300/64(NA)
    MAC           : 02:50:56:56:53:00
    VNI           : 66561
    Access-VLAN   : untagged
    Segment port  : 1fb264c0-ecd6-4dcc-9856-0f759ec65e62
    Urpf-mode     : NONE
    DAD-mode      : LOOSE
    RA-mode       : RA_INVALID
    Admin         : up
    Op_state      : up
    MTU           : 1500
    arp_proxy     :

...
...

    To Wrapping this Up

    In this simple example, we can see what happens with the TO DR LIF interface when we have a routing issue. The LIF IP address is moved to another edge transport node just to avoid a massive update on all devices’ routing tables and it is an intelligent way to maintain the traffic flowing by this interface without interruption (or with minimal interruption).

    Share.

    Infrastructure Engineer with experience in Virtualization, Linux, Windows Server and learning automation using Python. DPC Virtual Tips was created to share practical tutorials, lab experiences and troubleshooting guides focused on enterprise infrastructure technologies.

    Related Posts

    Comments are closed.