Blog Archives
Hyper-V Live Migration is not working | The disk structure is corrupted and unreadable | Cluster resource ‘Virtual Machine VM in clustered service or application VM failed | Virtual Machine VM failed to start.
Author: Zahir Hussain Shah | MVP Exchange Server
Applies to: Hyper-V Live Migration is not working | CSV is not accessible from passive cluster nodes| Validate Disk Failover failed | Failed to write file data on cluster disk 0 partition 1, failure reason: The disk structure is corrupted and unreadable | Virtual Machine VM live migration did not succeed at the destination | Cluster resource ‘Virtual Machine VM in clustered service or application VM failed | Virtual Machine VM failed to start.
Microsoft Hyper-V Clustering Recommendation and Best Practices for Live Migration and Cluster Shared Volumes (CSV) – Quick Solutions Guide
This is my second blog post related to Microsoft Windows Server 2008 R2 Clustering for Microsoft Hyper-V 2008 R2 Clustering and High Availability, in my previous blog article for building Hyper-V 2008 R2 High Availability, in which I presented to you the links to a wonderful video series for building Hyper-V HA for creating Highly Available Virtual Machine work-loads with Hyper-V Live Migration and Cluster Shared Volumes.
In this article, I will walk you through the several post-implementation related issues and their resolution, which you may get encounter with your configuration and some of the best practices, for helping you to fix your problem, and get your highly available VMs running on Hyper-V Cluster.
Okay, now lets start here with each post-issue, I have seen occurs after Implementing Hyper-V Cluster with CSV and Live Migration, I will explain one by one.
1) Failed to write file data on cluster disk 0 partition 1, failure reason: The disk structure is corrupted and unreadable
By the way, it is not a post-implementation problem, but I would like to share this with you, this happened to me, while I was building my five nodes Windows Failover Cluster on Microsoft Windows Server 2008 R2 for Hyper-V, this issue is related to Cluster Validation failure for Failover Disks, it happened to my newly attached CLUTER DISKs (LUNs), which I formatted and made offline before I ran Cluster Validation, but I got the failure that “The disk structure is corrupted and unreadable.”
Problem:
Failed to write file data on cluster disk 0 partition 1, failure reason: The disk structure is corrupted and unreadable corrupted and unreadable.
Resolution:
To fix this problem, I formatted all the disks, made them offline again, and re-run the Cluster Validation Test, and this time it turned out as passed!
2) Live Migration Best Practices and Post-Implementation Issues and their Resolutions:
In this section, I will guide you the best practices for fixing issues related to the Live Migration, after the post-implementation of Windows Failover Clustering for Hyper-V with CSV:
I. Recommended Cluster Network Microsoft Failover Cluster Settings:
Lets assume you got two networks physical cards in your Server, as follows:
LAN Connected Server VLAN
Cluster Heartbeat Network Cluster Network for Live Migration
In the Failover Cluster Manager, you have to set the below settings for both of them:
-
- image001
-
- image002
II. Recommended Cluster Network File and Printing and NetBIOS Settings:
As you know that disabling File and Printing Services and NetBIOS, for Cluster Internal Network, have always been a recommended practice, but with Live Migration and Hyper-V Clustering, it is required to turn on these both settings on the Cluster Network, which we will be using for Live Migration.
III. Refreshing Virtual Machine Configuration:
With my experience, while I saw that the Quick Migration was working but Live Migration was not working, so I gave a Refresh Virtual Machine Configuration from the Microsoft Failover Manager, and after that tried to give a live migration, and it worked, so you can say that after making any changes to VM (Highly Available) in Clustering, you should give Refresh to VM Configuration Cluster, so which tells the cluster that these are the changes made to the VM, and which directly interact with Clustering behavior of the VM.
IV. CSV- Cluster Shared Volumes behavior:
When you add CSV in the Microsoft Cluster Failover Manager for the support and funtionlity of Live Migration, you see a different behavior of these disk, as compare to the other type of storage on the Cluster Nodes, and over time, I have seen some misconception and misguided assumptions about these CSV LUNs, so let me explain the expected behavior of CSV LUNs on the Cluster nodes as follows:
a. Online / Offline: CSV LUNs will only be online on the Current Owner of the LUNs in the Cluster, e.g. as shown in the below image that only disk three is online and disk 1, disk 2 are both offline, is because that the Cluster Node 1 is the current owner of the DISK 3, and thats why it is showing here online, and both DISK 1, and DISK 2 are offline.
b. Reserved: When you add at LUN to Cluster as CSV, the Disk as become Reserved, means that it will only work for Windows Server 2008 R2 CSV functionality, where it is allowed for file-level locking, not the disk level.
c. Accessing CSV LUNs from Windows Explorer and over NetBIOS SMB session:
After adding the CSV LUNs to the Failover Cluster Manager in the CSV Tab, you will start seeing the below behavior in the C: drive of the Cluster Nodes, and when you will double click on the ClusterStorage, it will get buys, but will let you go inside the folder after sometime, depending on your Network speed and server performance, but as per expected functionality, you should be able to open this folder on all cluster nodes.
Additionally, while it is taking time to opening the folder, if you try to open the same location over SMB NetBIOS session, it will open it bit fast, but make sure that you enabled the File and Printer Sharing and NetBios settings on the Cluster Network, and others. E.g. \172.16.55.1c$ClusterStorageVolume1.
V. Same Hyper-V Virtual Network Settings on all Hyper-V Nodes in the Cluster:
Lets assume that we got five cluster node setup for Hyper-V clustering, and before building the cluster, as recommended we installed the Hyper-V role on these five cluster nodes, and as part of configuration, we will be creating Virtual Network in Hyper-V for VMs to communicate to the reset of the Network, and therefore we created a Network called Server-VLAN, and for allowing the any type of Migration, whether its a quick migration, live migration or VM move, it is required that you keep the same Virtual Network name on all Cluster (Hyper-V) nodes, because when the Virtual Machine moves in case of Live Migration with VM Configuration file, when the second node where you try to make the VM online, the migration process tries to open and make online the VM with the pre-configured VM configuration parameters, where if the Network Name would be different or the same name of Virtual Network (Server-VLAN example) is not there, VM migration and making the VM online will fail.
I hope with the all above explained as a best practices, will help you to make your Hyper-V High Availability Implementation a success project.
Cheers!
Install Windows Server 2008 R2 Cluster for Hyper-V | How to build Hyper-V Clustering | Hyper-V Live Migration | Hyper-V Cluster Shared Volumes | Create Hyper-V VM from Failover Cluster Manager | Hyper-V Quick and Live Migration – Hyper-V Training Video
Author: Zahir Hussain Shah | MVP – Exchange Server
Step by Step Video Training for Building Microsoft Windows Server 2008 R2 Failover Cluster for Microsoft Hyper-V High Availability with Cluster Shared Volumes, Pass-Through Disk and Managing Hyper-V VMs from Windows Server Failover Cluster Manager snap-in
Being a Microsoft Exchange Server Solution Architect and MVP, we all can see the Hyper-V as product getting mature day by day, and its fast growing usage at every level of business sector, and as initially Hyper-V introduced himself as a Hypervisor for Testing / RnD VM based Servers, and a Virtualization Layer for its competitors, but with the recent fast growth of the product, and changes the people perception, now Microsoft Hyper-V is being used as a Production Hypervisor, and many Application and Systems Developers started making their products to run over Hyper-V, which is indeed a success of Hyper-V in the todays Information Technology world.
As long as Microsoft Exchange Server is concerned, so I’m pleased to inform you that Microsoft Exchange Server 2010 is officially supported over Microsoft Hyper-V, and for information, please read my below posts:
http://zahirshahblog.com/2011/12/19/microsoft-exchange-server-2010-virtualization-support-compatibility-with-different-hypervisors-microsoft-hardware-and-application-virtualization-support-for-microsoft-windows-server-and-microsoft-e/
http://zahirshahblog.com/2011/12/25/exchange-server-2010-virtualization-virtualize-microsoft-exchange-2010-with-hyper-v-installing-exchange-server-2010-on-hyper-v-how-to-virtualize-microsoft-exchange-server-2010-with-hyper-v/
Let’s back to the Hyper-V, in this blog post, as it sounds with the blog post name, I will be provide you training video for “Step by Step Video Training for Building Microsoft Windows Server 2008 R2 Failover Cluster for Microsoft Hyper-V High Availability with Cluster Shared Volumes, Pass-Through Disk and Managing Hyper-V VMs from Windows Server Failover Cluster Manager snap-in”, created by Matt McSpirit as part of his screencast at Microsoft TechNet – TechEdge.
Matt divided this screencast into four video, which are linked as follows:
Ø Hyper-V R2: Building a Hyper-V R2 Cluster : Part I
Ø Hyper-V R2: Making Highly Available VMs : Part II
Ø Hyper-V R2: Introducing Cluster Shared Volumes : Part III
Ø Hyper-V R2: Failover & Live Migration : Part IV
I hope this blog post and Matt’s screencast will help all of you, who wants to start with Hyper-V HA and Live Migration, it really works…
Cheers!
How Live Migration Works in Hyper-V 2008 R2
The live migration process moves a running VM from the source physical host to a destination physical host as quickly as possible. A live migration is initiated by an administrator through one of the methods listed below. The speed of the process is partially dependent on the hardware used for the source and destination physical computers, as well as the network capacity.
Three methods can initiate a live migration:
1. Using the Failover Cluster Management console, an administrator can initiate a live migration.
2. If Virtual Machine Manager is managing physical hosts that are configured to support live migration, the Virtual Machine Manager administration-console can be used to initiate a live migration.
3. A WMI or PowerShell script can be used to initiate a live migration.
Any guest operating system supported by Hyper-V will work with the live migration process.
After initiating a live migration, the following process occurs:
1. Live migration setup
During the live migration setup stage , the source physical host creates a TCP connection with the destination physical host. This connection transfers the VM configuration data to the destination physical host. A skeleton VM is set up on the destination physical host and memory is allocated to the destination VM.
Stage 1: Live Migration Setup
2. Memory pages are transferred from the source node to the destination node
In the second stage of a live migration, the memory assigned to the migrating VM is copied over the network to the destination physical host. This memory is referred to as the working set of the migrating VM. A page of memory is 4 kilobytes.
For example, suppose that a VM named SERVER2 configured with 1024MB of RAM is migrating to another Hyper-V physical host. The entire 1024MB of RAM assigned to this VM is the working set of SERVER2. The utilized pages within the SERVER2 working set are copied to the destination Hyper-V physical computer.
In addition to copying the working set of SERVER2 to the destination physical host, Hyper-V on the source physical host monitors the pages in the working set for SERVER2. As memory pages are modified by SERVER2, they are tracked and marked as being modified. The list of modified pages is simply the list of memory pages SERVER2 has modified after the copy of its working set has begun.
During this phase of the migration, the migrating VM continues to run. Hyper-V iterates the memory copy process several times, each time a smaller number of modified pages will need to be copied to the destination physical computer.
After the working set is copied to the destination physical host, the next stage of the live migration begins.
Stage 2: Memory pages are transferred from the source node to the destination node
3. Memory pages transferred
Stage three is a memory copy process that duplicates the remaining modified memory pages for SERVER2 to the destination physical host. The source physical host transfers the register and device state of the VM to the destination physical host.
During this stage, the network bandwidth available between the source and destination physical hosts is critical to the speed of the live migration and using a 1 Gigabit Ethernet or faster is important. The faster the source physical host transfers the modified pages from the migrating VMs working set, the more quickly the live migration will complete.
The number of pages transferred in this stage is dictated by how actively the VM is accessing and modifying memory pages. The more modified pages, the longer the VM migration process takes for all pages to be transferred to the destination physical host.
After the modified memory pages are copied completely to the destination physical host, the destination physical host has an up-to-date working set for SERVER2. The working set for SERVER2 is present on the destination physical host in the exact state it was in when SERVER2 began the migration process.
Note: You can cancel the live migration process at any point before this stage of the migration.
Stage 3: Memory pages transferred
4. Move the storage handle from source to destination
During the fourth stage of a live migration, control of the storage associated with SERVER2, such as any VHD files or pass-through disks, is transferred to the destination physical host.
Stage 4: Storage Handle Moved
5. The VM is brought online on the destination server
In stage five of a live migration, the destination server now has the up-to-date working set for SERVER2 as well as access to any storage used by SERVER2. At this point SERVER2 is resumed.
Stage 5: VM Resumed
6. Network cleanup occurs
The migrated VM is running on the destination server in the final stage of a live migration. At this point a message is sent to the physical network switch causes it to re-learn the MAC addresses of the migrated VM so that network traffic to and from SERVER2 can use the correct switch port.
