Zahir Hussain Shah's | Exchange Ideas Blog

Author: Zahir Hussain Shah, MVP Exchange Server, CISSP

Building Exchange Server Disaster Recovery and Site Resilient setup requires proper planning and choosing the right option to select the underlying infrastructure. Recently I came across one Exchange DR solution architecting where customer was replicating its primary Exchange Server DAG LUNs to its secondary or DR EMC VNX SAN using EMC RecoverPoint. At this point of time customer didn’t had Exchange Servers installed in the secondary / DR Datacenter. Now when customer started planning phase to build the Exchange disaster recovery and site resilient setup in its secondary datacenter then, they started looking into various option of replicating Exchange Mailbox Databases to DR Exchange Servers.

Since customer was using EMC RecoverPoint for replicating all its data and primary SAN LUNs to DR storage, they first preferred to use the same underlying infrastructure. EMC RecoverPoint provides data loss by keeping replicated data in the journals and restore of the replicated data in any point in time back. In addition to this it also does data compression and data dedup. With this entire feature EMC RecoverPoint saves a lot of network bandwidth for WAN connection between two sites.

Now the question is can we use EMC RecoverPoint to replicate Exchange Mailbox Databases for setting up Exchange Disaster Recovery and Site Resilient setup?

According to EMC documentation (Exchange 2010 Disaster Recovery options with Cross-site Exchange DAG and EMC RecoverPoint) they provide two possible options, where in the first option they highlight few of the shortcomings of Exchange DAG native replication over EMC RecoverPoint, and in the second option they want you to use EMC RecoverPoint instead of Exchange DAG native replication.

So which option shall we select for Exchange Mailbox Database replication? To answer this question let’s see the below bullet points:
EMC RecoverPoint Replication:

• According to above provided EMC documentation link,  EMC provides option to setup Exchange Cold DR, where one has to keep its DR Exchange Servers shutdown and primary SAN/RecoverPoint will keep replicating the Exchange Mailbox LUNs to the DR SAN.
• This is going to be your Exchange COLD DR means it’s not straight forward there are possibility that you might see unknown issues while recovering from disaster.
• In this option upon perform Exchange DR drill or trying to activate Exchange Server from the DR there penalty of steps need to be carry out in order to bring the databases up. These steps might include creating new Exchange Databases on DR Exchange instances and re-homing user mailboxes.
• This Exchange cold DR setup with EMC RecoverPoint would require you to have more administrative overhead by attaching the LUNs to the node, ensuing that correct LUNs order and their drive letters are in use, and etc…
• There are always possibilities that during Exchange DR drills or datacenter failover you might run into split-brain syndrome, because there is no method available to check if databases are being mounted in both the datacenters during or after recovering from the disaster or drill.
• Not supported by Microsoft.

Microsoft Exchange DAG Native Replication:

• Going with Exchange DAG native replication option over EMC RecoverPoint is an absolute wise choice because you will be using Exchange Database Availability Group and all other associated Exchange Disaster Recovery and Site Resilient components such as DAC, Log shipping and log replay which ensures that your Exchange DR Mailbox database and services are healthy.
• Choosing Exchange DAG option would allow you to get support from Microsoft in case any thing breaks in the middle while going to EMC cold Exchange DR Option will not be supported.
• Less possibility of data corruption.
• One of the bad side of Exchange DAG replication is that it would ask for little bit more WAN network bandwidth.
• More control on Exchange DR drills and data integrity, where mounting one database from the Exchange DR servers and performing the data integrity while having everything running from the primary datacenter. Flexibility is available with Exchange DAG.

To conclude this it is recommended that one should choose Exchange DAG native replication over any other sort of appliance based technology for replicating Exchange mailbox database LUNs data. Exchange DAG provides proper safety against data corruption and split brain scenarios.

I will further update this article soon with more facts and reasons to share with you guys.

Cheers!

Recently I happened to look into an Exchange Server 2010 Management Console connectivity problem. Upon opening of Exchange 2010 Management Console on this server I was getting error “The following error occurred while attempting to connect to the specified Exchange Server “ExchCASHUB1.Domain.Com”.

The error message also includes further explanation that “The attempt to connect to
http://ExchCASHUB1.Domain.Com/PowerShell
using “Kerberos” authentication failed: Connecting to remote server failed with the following message: Log on failure: unknown user name or bad password.

After reading this last phrase about unknown user name or bad password, an idea clicked in my mind that lets check the local users accounts in the control panel for any saved password.

Solution:

When I opened the User Accounts tab within Control Panel, and clicked on “Manage your credentials”.

I found that within Windows Credentials there was an account with the user name and password saved. I removed this saved credential from this server by clicking on “Remove from vault”.

After removing this saved credential when I opened the Exchange Management Console on the same server, I could successfully connect to the PowerShell directory and my Exchange Management Console got connected. So actually what was happening in the background that Exchange Management Console was trying to use this saved credential to connect to PowerShell directory and since this account password is no longer valid so that’s why it was failing to authenticate.
Cheers!

Author: Zahir Hussain Shah, MVP Exchange Server, CISSP

Setting up Exchange Disaster Recovery and Site Resilient setup by stretching Exchange Database Availability Group (DAG) across different Active Directory sites would require proper network infrastructure in place. This network infrastructure has two sides to it, in the first phase you have to configure layer 1 connectivity between primary Exchange mailbox servers and secondary or DR site Exchange Servers. Once you have the network connectivity established then the second phase starts, in which we have to configure the routing of Exchange DAG replication network between Exchange Mailbox Servers.

For setting up routing between Exchange DAG Mailbox Servers, I recently published a blog post “Configuring Exchange 2010 Replication Network for Remote DR Exchange DAG Mailbox Servers”. In this article I explained step by step procedure you have to take to configure routing between primary and secondary site Exchange Mailbox Servers.

In the third and last phase of configuring Exchange DAG networks for your Exchange Disaster Recovery and Site Resilient setup, we have to adjust the Exchange DAG network settings within Exchange Server. Before we hit and understand the steps required to adjust Exchange DAG networks for DR Exchange replication network, we will take below replication network example for Contoso Corporation:

Primary Exchange DAG Mailbox Server
Mailbox Server Name	Interface Name (NIC)	Replication Network / IP Address	Default Gateway for replication network
MBX1	Replication	172.XXX.1.100	172.XXX.1.1
MBX1	Replication	172.XXX.1.101	172.XXX.1.1

 

Secondary / DR Exchange DAG Mailbox Server
Mailbox Server Name	Interface Name (NIC)	Replication Network / IP Address	Default Gateway for replication network
MBXDR1	Replication	10.XXX.84.102	10.XXX.84.1
MBXDR2	Replication	10.XXX.84.103	10.XXX.84.1

Now when you will add the additional MBXDR1 and MBXDR2 in the existing Exchange DAG (Database Availability Group) as DAG members, so you would see two new DAG networks will be created with the name of DAG Network 1 and DAG Network 2.

These two new DAG network would include the networks from MBXDR1 and MBXDR2 for their replication network interfaces.

As a recommended and best practice to configure DAG networks for primary and secondary DAG Exchange Mailbox Server, we should go to existing MAPI network and add DR Exchange Server Replication Network, for example as per below DAG network image we would add the “10.XXX.35.0/24” network into the MAPI network, which would automatically will add the required interfaces into the MAPI network interfaces area. Then we will also add the “10.XXX.84.0/24” network into the Replication Network which is enabled for DAG continues replication.

After performing the above outlined steps of adding / merging the MAPI and Replication network into the existing created MAPI and Replication sets, your DAG network should look like the below image:

 

I hope this example would help you to configure your DAG network in a best of possible manner.

Cheers!

Author: Zahir Hussain Shah, MVP Exchange Server, CISSP

Recently I happened to audit and health-check one of large Exchange 2010 customer, where we found couple of Exchange 2010 configuration items which were not configured at par with best practices. Among these configuration items we found that Transport Dumpster settings and its size for per database was not set as per best practice.

What is Exchange Transport Dumpster Feature?

Okay first of all for those who are not familiar with Exchange Transport Dumpster feature lets first get to know more about this guy. Exchange Transport Dumpster feature was first introduced in Exchange 2007 and is also available in newer version of Exchange. Exchange 2007 and on-wards Exchange Server version routes all e-mail flow via Exchange Hub Transport server, where Transport Dumpster features provides compliance and data loss prevention safety for the messages while in-transit.

Recommended configuration for Transport Dumpster

Upon auditing of Exchange messaging environments it has been seen that many Exchange Server administrator forgets to give required attention to configure correct settings for Exchange Transport Dumpster size for organization wide message transportation. As most of us know that default message size limit in Exchange 2010 comes as 10 MB, for which transport dumpster settings for dumpster size for each database is set by default to 18 MB. But when customer changes their message attachment size couple of times it has been seen that they don’t change the required settings on dumpster level.

This result of having dumpster not working at par and thus their message compliance and safety requirements goes bad.

It is recommend that your MaxDumpsterSizePerDatabase value should be set to 1.5 times of your message attachment size or allowed message size sent via Exchange Hub Transport Server.

Cheers!

Author: Zahir Hussain Shah, MVP Exchange Server, CISSP

Sometime back Exchange Server 2010 introduced a new feature called Shadow Redundancy Promotion. This feature enables Exchange Hub Transport Server maintains a local copy of the message before the next destination server acknowledge the receipt of the message. If the next hop server fails to provide the successful receipt acknowledgement then Exchange Hub Transport Server tries to re-send the communication again.

Let me try to elaborate it, Exchange  2010 SP1 and newer version uses Shadow Redundancy Promotion feature, which ensures that each message which submitted for transportation stays on two or more of Exchange Hub Transport Server in the AD site if next hope destination server is not reachable. By default shadow redundancy feature is disabled and it is upto Exchange administrator to enable it or not.

How to enable Shadow Redundancy Feature for your Exchange Hub Transport Servers?

For enabling Exchange Shadow Redundancy Feature for your Exchange 2010 Hub Transport servers, we have to change the EdgeTransport.exe.config file located within your Exchange server installation directory at “Program Files\Microsoft\Exchange Server\V14\bin”. Within EdgeTransport.exe.config file change the “ShadowRedundancyPromotionEnabled” settings to “TRUE” which is set to false by default.

Note: Make sure that you change the EdgeTransport.exe.config file for all your Exchange Hub Transport Servers available in the environment.

Cheers!

Author: Zahir Hussain Shah, MVP Exchange Server

Configuring Exchange DAG replication network can be tricky when it comes to allowing primary Exchange DAG replication network to chat with secondary / DR Exchange DAG replication network. If you try to use default gateway on the replication network either on primary or secondary site you might find that Exchange DAG replication at primary site goes bad and sometime I witnessed that Exchange DAG mailbox member servers loose the access to File Share Witness. Or possibly you might find your replication network would go down from the failover clustering prospective.

Having two default gateway on any server would make things go wrong and it is not recommended. Okay so you would be wondering what about adding Static Route for the replication network?

Well upon checking for the above explained issue with adding default gateway, I found that static route would not be able to fix your problem and even if they work for you configuring them with persistence switch, sometime it is also seen that they get disappeared.

Adding IP Routes with NETSH:

So what is the solution? The best solution for allowing Exchange DAG replication network to talk to remote DAG member is to use the NETSH IP routes for adding replication interface. In the below steps we will see steps you need to do on both side for adding NETSH routes for replication networks.

Let’s assume that we have following replication networks configured on the Exchange Servers on the both sides:

Primary Exchange DAG Mailbox Server
Mailbox Server Name	Interface Name (NIC)	Replication Network / IP Address	Default Gateway for replication network
MBX1	Replication	172.16.1.100	172.16.1.1
MBX1	Replication	172.16.1.101	172.16.1.1

Secondary / DR Exchange DAG Mailbox Server
Mailbox Server Name	Interface Name (NIC)	Replication Network / IP Address	Default Gateway for replication network
MBXDR1	Replication	172.16.2.100	172.16.2.1
MBXDR2	Replication	172.16.2.101	172.16.2.1

Now follow the below steps to configure the NETSH routes first on the primary DAG members:

1. Open Command Prompt on the server with Run As Administrator rights.
2. Type “netsh” and hit enter to go into netsh command shell.
3. Then type “interface” and hit enter to go into interfaces section.
4. Type “ipv4” and press enter to go inside IPv4 interfaces.
5. If you don’t know the name of your interface you can also see all the IPV4 interfaces, by executing “show interfaces” command.
6. Now we will add route by executing the command “add route 172.16.2.0/24 “Replication” 172.16.1.1” while remain in the NetSH command shell.
7. Perform the same above steps on the second primary DAG Exchange Mailbox Server.
8. Now we have to perform the same steps outlined above on secondary Exchange DAG Mailbox Servers for reverse routes.
   
   Perform the reserve route adding process explained in the above image on the both secondary Exchange DAG Mailbox Servers.Verifying NETSH Interface Routes:
   After adding the routes we can also verify for the successful addition of the routes for the replication network, by showing IP routes for interfaces by executing the below command:
   
9. We can also verify the successful addition of NETsh routes by checking the registry (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\PersistentRoutes) for the available routes.

I hope it would be helpful to fix Exchange DAG replication network routing issues.

Cheers!

I’m big fan of having Microsoft product architecture poster pasted on my desk back wall, so on and on I can revisit it and make things clear whenever I architect the solutions.

Exchange Server 2013 PDF architecture poster is now finally available download. Cheers!

Im pleased to blog about Windows Server 2012 Hyper-V Replica capacity planner, which is recently released and made available for public.

It is available for download on Microsoft download center. You can download it from here.

To know more about Windows Server Hyper-V Replica feature, see my book (Windows Server 2012 Hyper-V: Deploying Hyper-V Enterprise Server Virtualization Platform) for Chapter 3 – Setting up Hyper-V Replication.

Cheers

Author: Zahir Hussain Shah, MVP, CISSP

Embracing the cloud is everywhere, either in one or another way you will find the cloud computing discussion or its usage hitting door next to you. Last year Microsoft released plenty of products including Windows Server 2012 and System Center 2012 family products. Windows Server 2012 is first Windows Server operating system which is built from the cloud up, means while building this operating system all its features and functionalities were designed in such way that it can provide cloud ready based platform.  While combing Windows Server 2012 with System Center family product it is now possible to build our own provide cloud with hell lot of automation, optimization and multi-tenancy.

Okay we had an enough intro about these two products, and now we will discuss the topics related to this blog post. Recently I happened to downgrade a running Windows Server 2012 Hyper-V Failover Cluster, during we were doing this activity we found few errors on the way to downgrade the cluster nodes, so I thought it would be nice to share with you folks on my blog.

In this blog article, we will be covering following listed topics:

• Choosing correct Windows Server 2012 license for your Hypervisor
• Downgrading Windows Server 2012 Failover Cluster from Datacenter to Standard Edition
• Troubleshooting Windows Server 2012 Cluster Validation issues

Choosing correct Windows Server 2012 license for your Hypervisor (Standard V/S Datacenter)

With Windows Server 2012 Microsoft changes its licensing schema where now Windows Server Enterprise edition is no longer available. And Windows Server is now licensed for Standard and Datacenter Edition. By means of feature wise comparison both license editions are pretty much as same. Unlike old licensing model where MCS (Microsoft Clustering Services) was only available in Enterprise edition of Windows Server is not the case with Windows Server 2012. As we said both edition standard and datacenter provides same features.

The differentiating factor is the licensing rights, where Windows Server Standard Edition gives you only two Virtual Machine free licenses. While Windows Server Datacenter edition license provides unlimited number of Virtual Machine licenses running the same server. These virtualization rights are not different from Hyper-V, VMware or Citrix Xen Server, the same applies to all types of hypervisor application vendors.

So the question is: Which Windows Server license is good economically and technically for your server operating system needs?

To summarize the recommendation for both edition, look at the below table:

 

	Windows Server Standard Edition	Windows Server Datacenter Edition
Hypervisor Use	Limited number of virtual machine use, where number of running virtual machines are from 1 to 8 or are less than 10 then Windows Server Standard edition is a good choice for your server.	If your company is highly mature for virtualization, where majority of all applications are virtualized, and number of virtual machine running on each hypervisor server are 10 or more than 10, then Windows Server Datacenter Edition license is should be your choice. This recommendation is for all hypervisors regardless of which Hypervisor product your company uses.
Member Server and Application Use	Any Microsoft or third-party application servers. If any of your application needs Microsoft Clustering Service running on it, you can still use Windows Server Standard Edition license.	Windows Server Datacenter is not at all good choice for any application other than hypervisor products. Because feature wise Standard edition has everything to provide supported platform for your application.

Downgrading Windows Server 2012 Failover Cluster from Datacenter to Standard Edition

Recently I downgraded a production Windows Server 2012 Hyper-V Failover cluster, where all five cluster nodes were installed as Windows Server 2012 Datacenter Edition. And customer initially after installing datacenter edition realized that they don’t need Windows Server datacenter edition at this moment. And since there is no downgrade possible where we can use go back to standard edition, so the only way left for us is to first remove each cluster node from the cluster one by one and reinstall the operating system as standard edition.

Since it was a Windows Server 2012 failover cluster, which provides new Dynamic Quorum Mode, by having odd number of cluster nodes doesn’t need disk majority and configures as node majority. By using this dynamic quorum model cluster allows you to bring the cluster work load on to a single cluster node in case all other cluster node fails. So in order to continue our downgrade activity, we performed the following steps in sequence:

For this example we will consider that we have five node Windows Server 2012 failover cluster on which we have also installed Hyper-V role.

1. First we will migrate all the highly available virtual machines from all four cluster nodes to single cluster node.
2. Go to Failover Cluster Manager and right-click cluster node go to more actions menu and click on Evict. This will remove the cluster node from the cluster gracefully. Follow the same procedure with all four nodes and remove them from the cluster.
3. Now re-install the operating system on all the formatted nodes in the cluster and install their respective drivers. If you want you can keep the node names same or different it doesn’t matter on the cluster.
4. Once your machines are ready and you mapped the CSV and all other LUNs back to the node and get them verified by seeing them in the disk management, then you can proceed with the next step of adding these newly formatted nodes back to the cluster.
5. For adding the nodes back to the cluster we have to go the cluster node which is still part of the cluster and we will expand the cluster and from there in the nodes container we will right click and add the node.
6. Once you add all the remaining four nodes back to the cluster perform the same procedure of evicting the node from the cluster then format OS and re-add the node back to the cluster.

Troubleshooting Windows Server 2012 Cluster Validation issues (Validate Microsoft MPIO-based disks failed)

While performing the above downgrading activity I came across cluster validation issue, which I thought would worth to discuss here. This problem I faced with Windows Server 2012 RTM edition with most up-to-date windows updates installed. During cluster validation we received problem related to Microsoft MPIO-based disks failed.

While closely looking at this problem I found that Microsoft DSM QFE number for all four nodes were same but only for one node as you can see in the below screen-shot is different among all:

 

In the above screen-shot you can see that node 1 QFE number is 16384 while node to has 16465, which is causing of mismatch QFE numbers. And why these numbers are different is because of the different patch level of the these two nodes. Node 1 QFE number is lower because it doesn’t have the all same patches installed as same as node 1. While finding the required patch which is making this issue, we found that “Windows8-RT-KB2785094-x64” didn’t get installed and due to this there was a mismatch patch update status on all the nodes.

Solution:
As a solution to fix this problem we tried a lot to install the “Windows8-RT-KB2785094-x64” on node 1, but due to some patching issue we were not able to install it on node 1. So finally we had to remove the same patch on all the nodes, and then validated the cluster and this time it successfully get validated without any problem. After the monthly patches released in the month of April, 2013 we are now seeing that “Windows8-RT-KB2785094-x64” is now got installed on all the nodes in addition to the all other patches released in the month of April, 2013.

I hope you will find this blog post helpful for downgrading your Windows Server 2012 cluster.
Cheers!