I have recently been working on upgrading some two node Hyper-V Clusters running Server 2008 R2 to the latest and greatest Hyper-V Server 2012 R2. I have now completed the first complete cluster upgrade, and will summarize some findings here.
I wouldn't call this a technical reference, but more of a, "Here's what I had to do, and here's some stuff I ran into while doing it" kind of thing.
My scenario starts with basic 2 node Hyper-V Clusters running Server 2008 R2. Existing config looks like this:
- The clusters are built on HP Hardware, and use HP MSA storage arrays for shared storage (SAS attached).
- Networking is configured with 3 teams of 2 NICs each, using HP's network utility to manage the teaming. Teams are set for Live Migration, VM traffic, and Storage/Cluster traffic.
The upgrade is performed one node at a time. The process lays out like so:
- All VMs are migrated to one host.
- The empty host is evicted from the cluster.
- The host then gets Bios and iLo updates.
- The host is rebuilt using Microsoft's core-only Hyper-V Server 2012 R2 media (Free, and NO GUI!)
- Since the networking no longer requires a third party utility, the six NICs I was using before are configured in one big port channel on the switch. A converged team is created in windows, and three virtual switches are created in Hyper-V for Vms, Live Migrations, and Cluster/Storage.
- Disk is added locally to the host and configured.
- Here is the outage point. Each VM is shut down, copied to the local storage, and imported to the new host.
- Interestingly, Server 2012 R2 no longer recognizes exported VMs from 2008 r2. Instead, you copy the existing VM files as-is and import them directly. I actually consider this an improvement.
- Once all VMs are moved, the other host is built and the Cluster is re-formed.
- The MSA Controller and disk firmware is updated. (this step applies to my situation, but if your SAN or other shared storage needs work, now's the time!)
- We now have the ability to live migrate (storage and all) all of the VMs back to our shared storage, and add them to the cluster (also live).
Issues I ran into, and how I worked around them:
SAS PCI card prevented install media for Hyper-V Server 2012 R2 from loading.
I had to disable this card in the BIOS to allow the install media to completely load. Re-Enabling it after the OS installed caused no issues. An annoyance.
Creating the virtual Nics. This one is weird.
- A powershell script was used create the converged team, Vswitches, etc.
- I had no issues on host 1, script executed as planned.
- When using the script to configure host 2, it again worked, but it created the vEthernet ports with identical MAC addresses to the first host. MAC-flapping like crazy. Connections unstable. Bad things.
I worked around this one by using powershell to change the Mac Addresses on the Nics on Host 2.
- Command used, for the curious: set-netadapter -name "ExampleNic1" -MacAddress "SomeMacAddress"
- I will need to look into why Microsoft's own commands would use identical MAC addresses on two different servers. Especially when these commands are most commonly used to create clustered environments.
In researching this, I am finding that this exact issue is not being reported, but a similar issue where Hyper-V 2012 R2 is duplicating Mac Addresses is being talked about quite a bit.
- Link: https://social.technet.microsoft.com/Forums/office/en-US/3513c16c-99ff-4ac0-8c36-e098ef6716f6/windows-server-2012-r2-hyperv-host-virtual-switch-mac-conflict?forum=winserverhyperv
Shared Storage hijacking?
I will need to dig further into this also, but I found that when I rebuilt the cluster, Hyper-V decided to grab and use anything that could be considered shared storage, and add it to the cluster. Even if you specify only a certain volume, it may still see other ones, and use them. Even if they are already in use, and mounted in the OS….
This behavior offlined about 17 VMs for me inadvertently. Luckily it was a quick fix to recover them (remove cluster storage, and re-add volume locally to host), but it was annoying.
2012 R2 removes support for VMs exported from 2008 R2.
This one is sticky. I guess the biggest takeaway here is, whatever you do, do NOT export your VMs, and plan on importing them later. You're going to have to find, or build, a 2012 or earlier Hyper-V host, import them, and then copy them. Messy. No fun.