Virtual machines lose network connectivity

We had a weird issue where we randomly had virtual machines lose network connectivity from a Windows 2012 R2 Hyper-V host server. The VM’s would just lose connectivity from the network. The virtual servers were up, we could connect to the console but we couldn’t ping them and they couldn’t ping out. It’s rare to find an article that specifically outlines the EXACT issue that you are experiencing. Luckily for us, we found the following technet article which helped us resolve this issue!
 
According to the technet article below, when you have Hyper-V running on Windows Server 2012 R2 together with Broadcom NetXtreme 1-gigabit network adapters, you might experience VM’s randomly losing network connectivity. This was the exact scenario in our case.In our case, we had the latest Broadcom drivers already installed but still experienced the issue.
 
This is a known issue with Broadcom NetXtreme 1-gigabit network adapters that use the b57nd60a.sys driver, when VMQ (virtual machine queue) is enabled on the network adapter. (By default, VMQ is enabled by the Broadcom network driver).  VMQ uses hardware packet filtering to deliver packet data from an external virtual machine network directly to virtual machines, which reduces the overhead of routing packets and copying them from the management operating system to the virtual machine

Using this article as a guide, we verified that the Hyper-V network team has VMQ enabled.  Resolution was to DISABLE VMQ on the Hyper-V team.  We did this on both Hyper-V hosts as they were setup identical. 

https://support.microsoft.com/en-us/help/2986895/virtual-machines-lose-network-connectivity-when-you-use-broadcom-netxtreme-1-gigabit-network-adapters

Run the following powershell command (as administrator) to view the VMQ settings for all your adapters on the Hyper-V host. You will receive output similar to below.

> Get-NetAdapterVmq

Name Interface Description Enabled BaseVmq
Processor
Max
Processors
NumberOf
ReceiveQueues
Network Team Network Adapter… FALSE 0:00   24
FlexibleLOM Port 4 Ethernet 1Gb port… FALSE 0:00 16 12
FlexibleLOM Port 3 Ethernet 1Gb port… FALSE 0:00 16 12
FlexibleLOM Port 2 Ethernet 1Gb port… FALSE 0:00 16 12
FlexibleLOM Port 1 Ethernet 1Gb port … FALSE 0:00 16 12
HVLAN2 Ethernet 1Gb port 1 TRUE 0:00 16 12
HVLAN1 Ethernet 1Gb port 2 TRUE 0:00 16 12
LAN2 Ethernet 1Gb  port… FALSE 0:00 16 12
LAN1 Ethernet 1Gb port… FALSE 0:00 16 12
Hyper-V Team Network Adapter… TRUE 0:00   24

Notice the highlighted network interfaces have VMQ enabled (TRUE). Run the following PowerShell command to get the network adapter’s VMQ names. You will receive output similar to below.

> Get-NetAdapterVmqQueue

Name                      QueueID MacAddress   VlanID Processor VmFriendlyName
—-                           ——- ———-        —— ——— ————–
HVLAN2                         0                                0:0                     HostServer01
HVLAN1                         0                                0:0                     HostServer01
Hyper-V Team              0                                0:0                     HostServer01

Once you have the VMQ names, run the following PowerShell commands as administrator to disable VMQ on each adapter.

> Set-NetAdapterVmq -Name “HVLAN1” -Enabled $False

> Set-NetAdapterVmq -Name “HVLAN2” -Enabled $False

> Set-NetAdapterVmq -Name “Hyper-V Team” -Enabled $False

Verify the VMQ settings on those interfaces by running the following PowerShell command as administrator.

> Get-NetAdapterVmq

Name Interface Description Enabled BaseVmq
Processor
Max
Processors
NumberOf
ReceiveQueues
Network Team Network Adapter… FALSE 0:00   24
FlexibleLOM Port 4 Ethernet 1Gb port… FALSE 0:00 16 12
FlexibleLOM Port 3 Ethernet 1Gb port… FALSE 0:00 16 12
FlexibleLOM Port 2 Ethernet 1Gb port… FALSE 0:00 16 12
FlexibleLOM Port 1 Ethernet 1Gb port … FALSE 0:00 16 12
HVLAN2 Ethernet 1Gb port 1 FALSE 0:00 16 12
HVLAN1 Ethernet 1Gb port 2 FALSE 0:00 16 12
LAN2 Ethernet 1Gb  port… FALSE 0:00 16 12
LAN1 Ethernet 1Gb port… FALSE 0:00 16 12
Hyper-V Team Network Adapter… FALSE 0:00   24

All the interface should now read FALSE.

George Almeida

Welcome to my little corner of the blogosphere. I'm an Information Technology manager for a Fortune 500 company. I specialize in Windows operating systems, applications, servers, storage, networks and also have a technical background on the IBM iSeries platform. My only purpose for this blog is the hope that it helps someone, someday, somewhere. Any meager proceeds derived from our sponsors will be donated to charity.

You may also like...

Leave a Reply

Be the First to Comment!

avatar