Hyper-V Live Migration is not working | The disk structure is corrupted and unreadable | Cluster resource ‘Virtual Machine VM in clustered service or application VM failed | Virtual Machine VM failed to start.
Author: Zahir Hussain Shah | MVP Exchange Server
Applies to: Hyper-V Live Migration is not working | CSV is not accessible from passive cluster nodes| Validate Disk Failover failed | Failed to write file data on cluster disk 0 partition 1, failure reason: The disk structure is corrupted and unreadable | Virtual Machine VM live migration did not succeed at the destination | Cluster resource ‘Virtual Machine VM in clustered service or application VM failed | Virtual Machine VM failed to start.
Microsoft Hyper-V Clustering Recommendation and Best Practices for Live Migration and Cluster Shared Volumes (CSV) – Quick Solutions Guide
This is my second blog post related to Microsoft Windows Server 2008 R2 Clustering for Microsoft Hyper-V 2008 R2 Clustering and High Availability, in my previous blog article for building Hyper-V 2008 R2 High Availability, in which I presented to you the links to a wonderful video series for building Hyper-V HA for creating Highly Available Virtual Machine work-loads with Hyper-V Live Migration and Cluster Shared Volumes.
In this article, I will walk you through the several post-implementation related issues and their resolution, which you may get encounter with your configuration and some of the best practices, for helping you to fix your problem, and get your highly available VMs running on Hyper-V Cluster.
Okay, now lets start here with each post-issue, I have seen occurs after Implementing Hyper-V Cluster with CSV and Live Migration, I will explain one by one.
1) Failed to write file data on cluster disk 0 partition 1, failure reason: The disk structure is corrupted and unreadable
By the way, it is not a post-implementation problem, but I would like to share this with you, this happened to me, while I was building my five nodes Windows Failover Cluster on Microsoft Windows Server 2008 R2 for Hyper-V, this issue is related to Cluster Validation failure for Failover Disks, it happened to my newly attached CLUTER DISKs (LUNs), which I formatted and made offline before I ran Cluster Validation, but I got the failure that “The disk structure is corrupted and unreadable.”
Problem:
Failed to write file data on cluster disk 0 partition 1, failure reason: The disk structure is corrupted and unreadable corrupted and unreadable.
Resolution:
To fix this problem, I formatted all the disks, made them offline again, and re-run the Cluster Validation Test, and this time it turned out as passed!
2) Live Migration Best Practices and Post-Implementation Issues and their Resolutions:
In this section, I will guide you the best practices for fixing issues related to the Live Migration, after the post-implementation of Windows Failover Clustering for Hyper-V with CSV:
I. Recommended Cluster Network Microsoft Failover Cluster Settings:
Lets assume you got two networks physical cards in your Server, as follows:
LAN Connected Server VLAN
Cluster Heartbeat Network Cluster Network for Live Migration
In the Failover Cluster Manager, you have to set the below settings for both of them:
-
- image001
-
- image002
II. Recommended Cluster Network File and Printing and NetBIOS Settings:
As you know that disabling File and Printing Services and NetBIOS, for Cluster Internal Network, have always been a recommended practice, but with Live Migration and Hyper-V Clustering, it is required to turn on these both settings on the Cluster Network, which we will be using for Live Migration.
III. Refreshing Virtual Machine Configuration:
With my experience, while I saw that the Quick Migration was working but Live Migration was not working, so I gave a Refresh Virtual Machine Configuration from the Microsoft Failover Manager, and after that tried to give a live migration, and it worked, so you can say that after making any changes to VM (Highly Available) in Clustering, you should give Refresh to VM Configuration Cluster, so which tells the cluster that these are the changes made to the VM, and which directly interact with Clustering behavior of the VM.
IV. CSV- Cluster Shared Volumes behavior:
When you add CSV in the Microsoft Cluster Failover Manager for the support and funtionlity of Live Migration, you see a different behavior of these disk, as compare to the other type of storage on the Cluster Nodes, and over time, I have seen some misconception and misguided assumptions about these CSV LUNs, so let me explain the expected behavior of CSV LUNs on the Cluster nodes as follows:
a. Online / Offline: CSV LUNs will only be online on the Current Owner of the LUNs in the Cluster, e.g. as shown in the below image that only disk three is online and disk 1, disk 2 are both offline, is because that the Cluster Node 1 is the current owner of the DISK 3, and thats why it is showing here online, and both DISK 1, and DISK 2 are offline.
b. Reserved: When you add at LUN to Cluster as CSV, the Disk as become Reserved, means that it will only work for Windows Server 2008 R2 CSV functionality, where it is allowed for file-level locking, not the disk level.
c. Accessing CSV LUNs from Windows Explorer and over NetBIOS SMB session:
After adding the CSV LUNs to the Failover Cluster Manager in the CSV Tab, you will start seeing the below behavior in the C: drive of the Cluster Nodes, and when you will double click on the ClusterStorage, it will get buys, but will let you go inside the folder after sometime, depending on your Network speed and server performance, but as per expected functionality, you should be able to open this folder on all cluster nodes.
Additionally, while it is taking time to opening the folder, if you try to open the same location over SMB NetBIOS session, it will open it bit fast, but make sure that you enabled the File and Printer Sharing and NetBios settings on the Cluster Network, and others. E.g. \172.16.55.1c$ClusterStorageVolume1.
V. Same Hyper-V Virtual Network Settings on all Hyper-V Nodes in the Cluster:
Lets assume that we got five cluster node setup for Hyper-V clustering, and before building the cluster, as recommended we installed the Hyper-V role on these five cluster nodes, and as part of configuration, we will be creating Virtual Network in Hyper-V for VMs to communicate to the reset of the Network, and therefore we created a Network called Server-VLAN, and for allowing the any type of Migration, whether its a quick migration, live migration or VM move, it is required that you keep the same Virtual Network name on all Cluster (Hyper-V) nodes, because when the Virtual Machine moves in case of Live Migration with VM Configuration file, when the second node where you try to make the VM online, the migration process tries to open and make online the VM with the pre-configured VM configuration parameters, where if the Network Name would be different or the same name of Virtual Network (Server-VLAN example) is not there, VM migration and making the VM online will fail.
I hope with the all above explained as a best practices, will help you to make your Hyper-V High Availability Implementation a success project.
Cheers!
Earned Certified Information Systems Security Professional (CISSP) Certification | Information Systems Security Blog Series
It is my great pleasure to publish this blog post that recently that I have successfully passed and earned the Certified Information Systems Security Professional (CISSP) Certification exam in the First Attempt!
Moreover, Im also planning to write an article series of how to prepare for CISSP Exam, and where I will be sharing some of the CISSP exam study material and exam tips.
The good-news is that now, I will be starting publishing blog articles related to the below mentioned CISSP Security Domains, including Cloud Computing, Microsoft Hyper-V, and especially Microsoft Exchange Server.
At last, I really want to thank you all, and especially those ones, who have been providing their feedback, and comments in particular on my blog post, and helped me to contribute more efficiently. I have been hosting this blog since last two years, and I really enjoyed the experience of sharing knowledge, and helping the community to learn and grow.
CISSP convers the below Information Systems Security Domains:
Access Control
Telecommunications and Network Security
Information Security Governance and Risk Management
Software Development Security
Cryptography
Security Architecture and Design
Operations Security
Business Continuity and Disaster Recovery Planning
Legal, Regulations, Investigations and Compliance
Physical (Environmental) Security
Cheers!
Install Windows Server 2008 R2 Cluster for Hyper-V | How to build Hyper-V Clustering | Hyper-V Live Migration | Hyper-V Cluster Shared Volumes | Create Hyper-V VM from Failover Cluster Manager | Hyper-V Quick and Live Migration – Hyper-V Training Video
Author: Zahir Hussain Shah | MVP – Exchange Server
Step by Step Video Training for Building Microsoft Windows Server 2008 R2 Failover Cluster for Microsoft Hyper-V High Availability with Cluster Shared Volumes, Pass-Through Disk and Managing Hyper-V VMs from Windows Server Failover Cluster Manager snap-in
Being a Microsoft Exchange Server Solution Architect and MVP, we all can see the Hyper-V as product getting mature day by day, and its fast growing usage at every level of business sector, and as initially Hyper-V introduced himself as a Hypervisor for Testing / RnD VM based Servers, and a Virtualization Layer for its competitors, but with the recent fast growth of the product, and changes the people perception, now Microsoft Hyper-V is being used as a Production Hypervisor, and many Application and Systems Developers started making their products to run over Hyper-V, which is indeed a success of Hyper-V in the todays Information Technology world.
As long as Microsoft Exchange Server is concerned, so I’m pleased to inform you that Microsoft Exchange Server 2010 is officially supported over Microsoft Hyper-V, and for information, please read my below posts:
http://zahirshahblog.com/2011/12/19/microsoft-exchange-server-2010-virtualization-support-compatibility-with-different-hypervisors-microsoft-hardware-and-application-virtualization-support-for-microsoft-windows-server-and-microsoft-e/
http://zahirshahblog.com/2011/12/25/exchange-server-2010-virtualization-virtualize-microsoft-exchange-2010-with-hyper-v-installing-exchange-server-2010-on-hyper-v-how-to-virtualize-microsoft-exchange-server-2010-with-hyper-v/
Let’s back to the Hyper-V, in this blog post, as it sounds with the blog post name, I will be provide you training video for “Step by Step Video Training for Building Microsoft Windows Server 2008 R2 Failover Cluster for Microsoft Hyper-V High Availability with Cluster Shared Volumes, Pass-Through Disk and Managing Hyper-V VMs from Windows Server Failover Cluster Manager snap-in”, created by Matt McSpirit as part of his screencast at Microsoft TechNet – TechEdge.
Matt divided this screencast into four video, which are linked as follows:
Ø Hyper-V R2: Building a Hyper-V R2 Cluster : Part I
Ø Hyper-V R2: Making Highly Available VMs : Part II
Ø Hyper-V R2: Introducing Cluster Shared Volumes : Part III
Ø Hyper-V R2: Failover & Live Migration : Part IV
I hope this blog post and Matt’s screencast will help all of you, who wants to start with Hyper-V HA and Live Migration, it really works…
Cheers!
VM could not initialize – Hyper-V Virtual Machine is not starting | Windows Server 2008 R2 Hyper-V VM is not initializing
Author: Zahir Hussain Shah | MVP Exchange Server
<VM> could not initialize Hyper-V Virtual Machine could not be initialize | Windows Server 2008 R2 Hyper-V VM is not starting
Today while I was setting up one of my machine for Windows Server 2008 R2 Hyper-V, and after I enabled the DPE and Intel VT Virtualization Technology, and gave a try to start the VM, it simply said that VM NAME could not initialize, so once again I revisited the BIOS settings to verify, but found that everything is seem okay, so why the VM is not starting?
Problem:
<VM> could not initialize Hyper-V Virtual Machine could not be initialize
Cause:
After doing some research, I found that there is a bug in Windows Server 2008 R2, which has a CPU is installed that supports the AVX feature, and due to this bug, it prevents the VM created on Windows Server 2008 R2 to start.
Resolution:
For fixing this problem, either you can download the Windows Server 2008 R2 Service Pack 1, and install on the effected Hyper-V instance, or you can just download a single hotfix, and install on the server to fix this problem.
For reference:
check out this link
I hope this article will help you to get Hyper-V Virtualization journey in a smooth and pleasant way.
Cheers!
Optimize Microsoft Hyper-V Server Networking Performance | Improve Microsoft Hyper-V Server Networking Performance for Microsoft Exchange Server Virtualization
Author: Zahir Hussain Shah MVP Exchange Server
Improve Microsoft Hyper-V Server Networking Performance for Microsoft Exchange Server Virtualization, and in general
Recently, while I kicked off the Installation and Configuration of our HP c7000 Blade Systems for building Microsoft Hyper-V Servers, as the first layer of Virtualized Infrastructure for our Internal Cloud (aka Private Cloud) implementation, so I tried to search for How to optimizing / improving the network performance of a Microsoft Hyper-V Server, and then I came to know the series of article from Cristian Edwards, where he documented the various elements of a Windows Server, which can be tweaked to get the better performance from a Microsoft Windows Server, and Microsoft Hyper-V Server in particular.
You can visit the below links to see the each part of his series of article for Hyper-V Network Optimization:
Hyper-V Networking Optimizations Part 1 of 6 (TCP Chimney Offload)
Hyper-V Networking Optimizations Part 2 of 6 (VMQ)
Hyper-V Networking Optimizations Part 3 of 6 (RSS)
Hyper-V Networking Optimizations Part 4 of 6 (Jumbo Frames)
Hyper-V Networking Optimizations Part 5 of 6 (Features compatibility matrix)
Hyper-V Networking Optimizations Part 6 of 6 (Monitoring Hyper-V Network consumption)
Note: There is a link to a Microsoft PowerShell Script Advanced_NetFeature.ps1, he wrote, which provides a nifty overview of the different TCP /Network level settings on your Server, you can download it and run in on the server before, you perform any of modification for the Network level settings.
Below are the links to my past blog post, in which I explained the best practices for Virtualizing Microsoft Exchange Server 2010 on Hyper-V in particular, and if you want, you can also apply these recommendation on any of the Hypervisor you use to virtualize the Exchange Server.
I hope this post would be help for you, and will provide a nifty way to improve the network performance of our Hyper-V Servers.
Cheers!
Best Practices for Migrating Hyper-V Storage (SAN LUNs) for Microsoft Exchange Server 2010 | Recommendations for Migrating SAN LUNs for Hyper-V Server, where Exchange Server 2010 VMs are using Pass-Through Disk for Mailbox Databases
Author: Zahir Hussain Shah, MVP Exchange Server
Best Practices for Migrating Hyper-V Storage and SAN LUNs for Microsoft Exchange Server 2010 VMs and Hyper-V Pass-Through Disks for Microsoft Exchange Server 2010 Mailbox Servers Mailbox Databases
This article is in correlation with my earlier article for Expected issues after upgrading EMC Power-Path software for Hyper-V Server to access SAN Storage for hosting Microsoft Exchange Server 2010 VMs, for VHDs and Pass-Through Disks, in which I documented some of the problems and issues, we faced while migrating the SAN Fabric for Hyper-V Server, and with this article I will continue and will conclude this series of blog post with some of the best practices, which I learned for migrating our SAN Fabric from EMC CLARiiON to EMC VNX, where we moved the SAN storage for couple of Hyper-V Servers, including few of them, where we are running Microsoft Exchange Server 2010 Hyper-V Virtual Machines.
Problems and issues assicated with SAN Migration for Hyper- Server VMs:
Either you have simply assigned a storage LUN to a Hyper-V Server, where you are keeping the VHDs for the VMs, or you are using the SAN LUNs to be taken inside the Hyper-V Virtual Machine, by making them offline on the Host (Hyper-V) and add Physical Disk into a SCSI Controller, to have the physical SAN LUN assigned to the VM, when you perform any kind of upgrade activity on the SAN, I mean either you upgrade the EMC or any other software on the Server (Hyper-V), which provides multipath functionality, like EMC PowerPath, where EMC PowerPath provides the Server to see the LUN as a single DISK on the Server, instead of seeing it as a multiple disk, because usually the Servers are zoned from the SAN for multiple SAN Switches using multiple cables (in case of FC), so without using such MultiPathing software, OS does not know that whether its a same assignment for me or there are actually two different disks.
So as being said, when you perform the EMC PowerPath upgrade for firmware, either you perform in-place upgrade or uninstall and reinstall, or while migrating the SAN System, lets see from EMC CLARiiON (OLD VERSION) to the EMC latest VNX SAN, if you perform SAN COPY or use EMC OpenMigrator, when you will provide the new LUNs with the copied data, then Microsoft Windows Server recognizes these LUNs as new and different LUNs, and what Windows does here, it changes the UUID / GUIDs of the DISK, and which turns as a disaster for the Hyper-V, because when after migrating the SAN LUNs, and your storage administrator provides the new LUNs to your server, and you go to either turn on the VM in Hyper-V or you go to the Hyper-V Virtual Machine Properties, you will see that the SCSI Controller Hard Disk for Physical Disk (Pass-Through) will show you as NO DISK, and the order of the Pass-Through Disk will forgotten by the VM.
Solution / Recommendations for Migrating SAN LUNs for Hyper-V Server, where Exchange Server 2010 VMs are using Pass-Through Disk for Mailbox Databases:
Recommendation I: Note down the order and the disk names of the SAN LUNs on the Hyper-V Host Server, because it will help you a lot, once you will finish the SAN migration and reconfigure the Pass-Through Disks in the Hyper-V Virtual Machine for the same old order.
Example:
| Disk Name on Hyper-V Server | Disk Capacity (Size) | |
| 1 | DISK 3 | 245 GB |
| 2 | DISK 4 | 120 GB |
| 3 | DISK 5 | 50 GB |
| 4 | DISK 6 | 245 GB |
| 5 | DISK 7 | 130 GB |
| 6 | DISK 8 | 100 GB |
| 7 | DISK 9 | 100 GB |
| 8 | DISK 10 | 200 GB |
Recommendation II: It is also advisable to either take the snapshot or note down the Hyper-V VM Disk Configuration, as follows:
Example:
|
SCSI Controllers (Total SCISI Controller 4) |
DISK CAPACITY (SIZE) |
DRIVE LETTER INSIDE VM |
DISK VOLUME NAME INSIDE VM |
| SCSI Controller 1 (MEMBER DISK): | |||
| DISK 3 | 245 GB | O:\ | 1_DB |
| DISK 6 | 245 GB | P:\ | 2_DB |
| SCSI Controller 2 (MEMBER DISK): | |||
| DISK 4 | 120 GB | Q:\ | REMOTE_SITES_DB |
| DISK 7 | 130 GB | R:\ | MANAGERS_DB |
| SCSI Controller 3 (MEMBER DISK): | |||
| DISK 5 | 50 GB | S:\ | MGMT_DB |
| DISK 8 | 100 GB | T:\ | IT_DB |
| SCSI Controller 1 (MEMBER DISK): | |||
| DISK 9 | 100 GB | V:\ | USERS_DB |
| DISK 10 | 200 GB | Y:\ | REOCVERY_DB |
Tip: Before your shutdown the Hyper-V Server for letting your SAN Admin to cut-over to the new SAN, make sure that you DISABLED THE HYPER-V Windows Services on the Hyper-V Server, which will prevent the Hyper-V Node to boot-up, while the disk are in offline mode.So lets say you have noted down all the VM Pass-Through Disk Configuration and Hyper-V Disk LUNs disk associations, now your SAN Admin will ask you to shutdown the Hyper-V Server, so I can swtich over to the new SAN.
And once the SAN Admin (I call them penny-pinching people) done with his work, and will ask you to power-up the server, make sure that you ENABLED and START the Hyper-V Services, if you stopped and disabled them before shutting down the Hyper-V Server, okay, so now you booted up the Server, and will go to the Disk Management, you will see all those SAN LUNs, where you were keeping the VMs VHDs, are offline, and all others DISKs, which supposed to be offline on the Hyper-V Host, because we wanted them to stay offline, because we have taken them inside the VM as Pass-Through Disk, they will also be offline once the server boots-up, ok so whats the problem, problem happens, when your Hyper-V Server boots-up with the boot-up of the Server, it will try to start all the VMs on the Server, and since the Hard Disk on which the VHDs are stored, is still offline, so the Hyper-V will not show you that machine in the available VM list on the Hyper-V Node (dont freak out here :$, that where the hell your VM gone J), bring up your VHD SAN LUN disk from the Windows Disk Management, and refresh the Hyper-V Console, and you will see the VM back (By the way, it happened with me that to some how when the Hyper-V started and the disk on which I was having VHD, was offline, it deleted the VH Configuration Files).
So as best practice, before you started the Hyper-V Services or Hyper-V VMs, you ensured that all the Physical Disk on the Hyper-V Node are in their perfect fashion, that all the disk on which you are keeping VHDs are online and all others which are configured as Pass-Through for VMs are offline on the Hyper-V Node, here you can verify the Disk Numbers and Disk Capacity, with the tables mentioned above.
Recommendation III: Once we verified all the disk and their orders, now its time to visit the Hyper-V VM Configuration Settings, and as explained above that you will see that the Pass-Through Physical Disk Configuration inside the VM will be configured, means SCSI Controllers would be there, along with the DISK added into them, but the Disk Configuration means, which VM Disk is mapped to the which Physical Disk.
Since we do have everything on paper or in snapshot (image mspaint you rock), so we can just go to the Exchange Server Mailbox VM, and we can select each Disk and select the corresponding Physical Disk.
Once the all the SCSI Disk are corrected, you can turn on the Exchange Server Mailbox Node, and you will see it will turn on without any problem.
Remember, once you will do the same thing, all you have to do is to go to the Disk Management in the Exchange Server Mailbox Node (or any server), and bring the Disk Online, and you will see all the Microsoft Exchange Server 2010 Mailbox Database Disk will be back, and mailbox replication can be resumed from the Microsoft Exchange Server 2010 Management Console or shell.
Cheers!
My blog review for 2011
The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.
Here’s an excerpt:
The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 59,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 22 sold-out performances for that many people to see it.
Exchange Server 2010 Virtualization | Virtualize Microsoft Exchange 2010 with Hyper-V | Installing Exchange Server 2010 on Hyper-V | How to virtualize Microsoft Exchange Server 2010 with Hyper-V
Best Practices for Virtualize Microsoft Exchange Server 2010 with Microsoft Windows Server 2008 R2
This is in reference to my earlier article, Microsoft Exchange Server 2010 Virtualization Support Compatibility with different Hypervisors | Microsoft Hardware and Application Virtualization Support for Microsoft Windows Server and Microsoft Exchange Server 2010, where I tried to clear out some of the misconception for Virtualizing Microsoft Exchange, by taking all of you to the Microsoft Virtualization Support Compatiblity Tester site, which provides Microsoft Windows Server platform and Microsoft different products virtualization support compatibility support testing with different Hypervisor available in the industry todays.
Lets make our association with Exchange, well when it comes to the Microsoft Exchange Server Best Practices for Virtualizing, so therefore, I would like to share a document, which covers all the ins and outs for Virtualizing Microsoft Exchange Server 2010 on Microsoft Windows Server 2008 R2.
Cheers!
Public Speaking at Microsoft Virtualization Boot-Camp (14-15,Dec,11), Dubai, UAE – Event Photos
Im truly honored and thankful to Microsoft Gulf-Dubai, and especially Ms. Natasha Zaza and Ms. Naveen Nasif, for giving me this great opportunity to talked and represented Microsoft Most Valuable Professional (MVP) Program and Microsoft Exchange Server 2010 at Microsoft Virtualization Boot-Camp, held at Le Royal Mridien Beach Resort, Dubai, UAE from 14-Dec-2011 to 15-Dec-2011.
It was an amazing experience for me to meet all of you, who visit the event and gave me your precious time, while discussing some of your Microsoft Exchange Server related issues, and among these discussions, related to some of the Microsoft Exchange Server Deployment and Migration scenarios, they are already documented and available on my blog, and for some which I havent documented, soon you would be able to find them on my blog.
On-behalf of Microsoft Most Valuable Professional Program, I would like to thank you all of you, those who have trust in ourselves to support you and to be at your doorstep for making your experience with Microsoft products more interesting and meaning-full.
-
- image003
-
- image004
-
- image005
-
- image006
-
- image007
-
- image008
-
- image009
-
- image0011
Cheers!
