Recover ESXi Host Network

Recover ESXi Host Network shows all the necessary steps to recover an ESXi management network when connected to a vSphere Distributed Switch (VDS).

First things first: Describing the Scenario

So, in this example, we have a four-node vSAN cluster, and the first ESXi host, for some reason (wrong configuration such as VLAN tagging, etc), is with “not responding” status, as we can see:

All nodes are using a vSphere Distributed Switch (VDS), and all VMkernels adapters communicate through Port Groups attached to this VDS:

Each ESXi host has four physical adapters:

Each Port group uses two physical adapters: one Active and another Standby:

To check out what physical interface is using the “Uplink 1” – In this example, the “Uplink 1” is mapped to “vmnic0”:

We can access a working node by SSH, executing the “esxtop” command, and typing the “n” to see network details. As we can see, the VMkernel “vmk0” is using the physical network adapter “vmnic0”:

To wrap this up, when we use a vSphere Distributed Switch (VDS), the management plan is performed by the vCenter Server. All configurations are made on the vCenter Server, and a copy is sent to registered ESXi hosts. So, based on it, all VDS configurations are performed on a vCenter Server, and afterward, they are sent to each ESXi host. It’s essential to understand it before continuing!

Accessing the Console of the Unavailable Host

As I said before, the first ESXi host was not responding from the vCenter Server perspective for some reason, and we need to fix it ASAP (from the network perspective, the ESXi host is unavailable as well):

So, in this example, we can access the unavailable ESXi host through its physical console:

Hit “F2” to enable the ESXi Shell:

Go to “Troubleshooting Options”:

Look that ESXi Shell is not enable. Hit “Enter” under it to enable:

Afterward, hit “Alt” + “F1” to access the ESXi Shell:

Steps to Recover the Host Management Network

1. Get the “DVPort ID” used by the management VMkernel:

esxcfg-vswitch -l

As we can see, the vmk0 uses the “DVPort ID” 1 – This number can differ in your environment:

2. Get the Teaming Policy configuration applied for this VMkernel:

net-dvs -l | grep -A20 -B1 -i "port 1:"

Change the “port 1” with your “DVPort ID” value!
In this example, this DVPort ID has “Uplink 1” as Active and “Uplink 2” as Standby:

We have seen that “Uplink 1” is mapped to “vmnic 0”. However, you can access a working node by SSH and execute the following command to map each uplink to its vmnic:

esxcfg-vswitch -l | grep -i vmnic[0-9] | sed 1d | awk -F' ' '{print $1"-"$3}' | while read line; do net-dvs -l | grep -A1 -i "port ${line%%-*}" | \
awk -v line="$line" 'BEGIN{split(line,a,"-")} /alias/{print a[1], a[2], $3$4}'; done

3. Before creating a vSphere Standard Switch (VSS), we must remove the physical interface vmnic0 from the vSphere Distributed Switch (DVS). To get the vmnic0 DVPort ID:

esxcfg-vswitch -l

In this example, the vmnic0 has the DVPort ID 32:

Removing the vmnic0 from VDS:

esxcfg-vswitch -Q vmnic0 -V 32 VDS-VSAN-OSA

Explaining the command:

-Q vmnic0 = physical interface to be removed from VDS
-V 32 = DVPort ID for vmnic0
VDS-VSAN-OSA = VDS name

4. Create a vSphere Standard Switch (VSS):

esxcli network vswitch standard add --vswitch-name=Temp-VSS

5. Create a Port Group into the previously created vSphere Standard Switch (VSS):

esxcli network vswitch standard portgroup add --portgroup-name=Temp-MGMT --vswitch-name=Temp-VSS

6. Add a physical network interface (vmnic0) to the previously created vSphere Standard Switch (VSS):

esxcli network vswitch standard uplink add --uplink-name=vmnic0 --vswitch-name=Temp-VSS

7. Remove the vmk0 that was using into VDS and create a new one using the new standard switch:

esxcli network ip interface remove --interface-name=vmk0

8. Create a new VMkernel interface and associate it with the previously created Port Group:

esxcli network ip interface add --interface-name=vmk0 --portgroup-name="Temp-MGMT"

At this point, we have a vSphere Standard Switch (Temp-VSS) with one uplink (vmnic0), and with one Port Group (Temp-MGMT):

The vmk0 interface was created, but it does not have any IP address:

9. Set up a static IP address on the vmk0:

esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=192.168.255.11 --netmask=255.255.255.0 --type=static

10. Set up a default route through vmk0:

esxcli network ip route ipv4 add --gateway=192.168.255.1 --network=default

Note: If you need to specify a VLAN ID under the new Port Group, you can use the following – In this example, the Port Group name is “Temp-MGMT” and the VLAN ID is 10:

esxcli network vswitch standard portgroup set -p "Temp-MGMT" -v 10

After executing all these steps, the first ESXi host went back to the vCenter Server, and we can manage it through vCenter usually:

Migrating the Management VMkernel from VSS to VDS

We can migrate the management VMkernel from VSS to VDS following the steps below:

Select the ESXi host –> Configure –> Networking –> Virtual Switches –> Expand the VDS –> Click on the three dots –> Migrate Networking:

Under the “Manage VMkernel adapters” page, we can see the vmk0 using the VSS “Temp-VSS” and the Port Group “Temp-MGMT”:

Click on “ASSIGN PORT GROUP”:

Under the “Assign port group” tab, we can see all VDS Port Groups. In our example, the “DPG-MGMT” is a VDS Port Group used for management traffic. Click “ASSIGN” and then click “NEXT”: