Blog Archives
Clear Submission Queue on Microsoft Exchange Server 2007 / 2010 | Remove message from queue on Hub Transport Server | Delete ambigous messages from submission queue | Emails are stuck in the Exchange Server queue | Troubleshooting Exchange Queue
Author: Zahir Hussain Shah | MVP Exchange Server, CISSP
Troubleshooting Microsoft Exchange HUB Transport Server (Transport Queues)
Often due to loss of Internet connectivity and DNS name resolution issues, we run into a situation, where we see a big count of e-mails get stuck at Exchange HUB Transport Server queue, usually these e-mails get stuck at your smart-host connector, which is the next-hop for your e-mails, but sometimes, due to some conflicts and other issues, we see e-mails get stuck in the Exchange Hub Transport Server for different queues, instead of the one, which is bound with your smart-host, for an example, lets say if you are using FOPE (Microsoft Forefront Online Protection for Exchange Online Service), then in this case, you will see emails getting stuck at mail.global.frontbridge.com (SmartHostConnectorDelivery).
Okay, we said something about the queues, now let me drive your attention towards the goal of this blog post, where I will guide you the basic troubleshooting for your Exchange Hub Transport Server and its queue.
Here are few of the common issues with Microsoft Exchange Hub Transport Server and its queues:
Problem:
1) E-mails get stuck at the SmartHostConnectorDelivery for out-bound e-mail delivery
2) Last Error: A matching connector cannot be found to route the external recipient
3) Last Error: 420 4.2.0 RESOLVER.ADR.Ambiguous; ambiguous address / Large number of messages get stuck at submission queue
Resolution:
E-mails get stuck at the SmartHostConnectorDelivery for out-bound e-mail delivery Issue:
For this type of problem, usually the cause is the loss of connectivity with Internet or with the HUB Site, which is responsible for routing e-mails to the appropriate destination (commonly to the Internet), and sometimes, it happens due to the bad DNS cache, which does not allow you to connect with the Smart-Host for the email delivery, and also the authentication which plays a vital part for allowing your HUB Transport Server to connect with remote-smart-host for delivering e-mails.
You can use the below solutions to fix this type of issues with e-mail delivery:
–>> Fix the IP connectivity issues for Internet / remote smart-host.
–>> Clear the DNS cache on DNS Servers and Exchange Hub Transport Servers (IPconfig /flushdns).
–>> Based on your remote-smart-host authentication requirements, set the appropriate authentication providers on your Send-Connector.
Last Error: A matching connector cannot be found to route the external recipient:
As it sounds with its name that the corresponding or matching send-connector is missing from Exchange HUB Transport Configuration, and for fixing this problem, you can create a send connector for the similar SMTP Address Space, but if you want to create a single SMTP connector for all type of domains, then make sure that you create *, a wild-card based SMTP connector, so all emails for any destination will be routed through this send-connector:
Last Error: 420 4.2.0 RESOLVER.ADR.Ambiguous; ambiguous address / Large number of messages get stuck at Submission Queue
This type of situation happens, when you perform some type of activity, which in result conflict the LegacyExchangeDN attribute on Exchange Server, (for reading more about legacyExchangeDN, read the below links), in this case, Exchange Server cannot decide where to send email for the particular legacyExchagneDN value, where the same value being used for either two different Mail-Enabled Users (Mailboxes) or between Mail-Enabled User (Mailbox) and Mail-Enabled Contact, in this situation, this emails will always stay at the Exchange Submission Queue, and you will constantly will receive the below errors in the Exchange Hub Transport Server event viewer:
Messages stuck at Submission Queue on Exchange Hub Transport Server:
Question: Okay, since these are ambiguous messages, which got created when there is a conflict of LegacyExchangeDN, now I have fixed it, so how can I remove these messages, because they are logging the above shown error log in me Exchange Hub Transport Servers event viewer?, well unfortunately, there is no way to delete messages from Exchange Management Console in both Exchange 2007 / 2010, and recently, while working for one case with Microsoft Support for Exchange Team, where I asked them can I delete these messages, so they told me the ONLY way to delete these ambiguous message from Exchange HUB Submission Queue, you have to Stop the Microsoft Exchange Transport Service, on Exchange Hub Transport Server, where the messages are stuck in the submission queue, and then rename the C:Program FilesMicrosoftExchange ServerV14TransportRolesdata” data folder to Data_OLD, and then start the “Transport Service”, which will recreate the “Data” folder, with fresh database for queue and other required files and folders, so this way you can delete these messages, and which sounded quite annoyed to me, that renaming these type of folders, is not a good idea, so I started looking for a solution, where I can delete these messages from Exchange Management Shell.
And a good thing that when I looked at the possibility for deleting these messages from EMS, so I found that it is pretty straight forward and easy, so lets see how you can delete them from Exchange Management Shell in both Exchange 2007 / Exchange 2010:
| [PS] C:Windowssystem32>Get-Queue -id Submission | Get-Message | Remove-Message -WithNDR $false |
Where in the above CMDLet WithNDR $false is telling the Exchange Server that dont send NDR to the original senders of the email, so you can scilently remove these messages from your Exchange without letting your end-users know.
In addition to the above CMDLet, you can use the same CMDLet for other queues, which are getting generate in your Exchange Hub Transport Server, for knowing about the Exchange Hub Transport Queue, you can run the below CMDLet:
| [PS] C:Windowssystem32>Get-Queue
Identity DeliveryType Status MessageCount NextHopDomain——– ———— —— ———— ————-Server_Name\Submission Undefined Ready 0 Submission Server_Name \Unreachable Unreachable Ready 1 Unreachable Domain |
To know more about LegacyExchangeDN, see the below links:
Update legacyExchagneDN attribute for Mail-Enabled Contacts
GAL Sync issues for LegacyExchangeDN
This blog post will enable you to manage and troubleshoot your Exchange Hub Transport Server and it’s queue efficiently.
Cheers!
Cluster Management with PowerShell
PowerShell for Failover Clustering: Getting Started
Hi Cluster Fans,
Amongst the long list of features that is being added in Windows Server R2, Clustering team is also providing a set of PowerShell cmdlets (“command-lets”) to manage your Failover Cluster. Hopefully this is not news for most cluster administrators as we have publishing information on the blog and in our webcasts. We have written a guide for using PowerShell in the Beta release and will add additional information shortly for the RC release: PowerShell for Failover Clustering in Windows Server 2008 R2.
Cluster PowerShell cmdlets have come a long way from Beta to RC. There have been quite a few changes in some of the cmdlets, cmdlet names and parameter names. These changes have been made to improve stability, consistency among cmdlets and to ensure that they are self-discoverable.
Naming Convention
All Cluster cmdlets adhere to the naming convention guide line set by the PowerShell Team. Every cmdlet can be broke in two parts, a verb and a noun.
While designing the names of these cmdlets, we took action oriented approach. What I mean is, all our cmdlets corresponds to an action that can be performed on a cluster object. This should give administrators a jump start with using clustering cmdlets to reduce the learning curve.
Verbs
Below is a table that lists all the verbs used by cluster cmdlets with an explanation. The table does not detail the cmdlets. The Idea is to help familiarize yourself with the verbs and that should help a user discover the right cmdlets for the operation.
| Verb | Description |
| Add | The verb is self explanatory. All the cmdlets that start with the verb Add would add the noun to the Cluster. For example, Add-ClusterDisk, would add Disks to the cluster, Add-ClusterGroup would add a group to the cluster. |
| Block | We have only one cmdlet that has the Block verb and that is Block-ClusterAccess. This cmdlet is used to manage user permissions and block would deny permissions on the cluster. |
| Clear | Cmdlets starting with Clear should help an administrator recover or re-claim a lost node or disk due to incomplete cleanup or information still lingering on them due to incomplete cleanup. |
| Get | This should be the simplest and easily discoverable. The Get-Cluster* cmdlets are to query either individual or all cluster object. For example, Get-ClusterNode is equivalent to Cluster.exe Node, Get-ClusterGroup is equivalent to Cluster.exe Group. |
| Grant | Cmdlets starting with Grant are among the set of cmdlets available to manage user permissions on Cluster. This can be paired up with Block. |
| Move | If you need to perform a Move operation on a cluster object, you need to be looking at cmdlets that start with Move to update the location of a cluster resource or group. |
| New | There is only one cmdlet that start with New and that is New-Cluster used to create a cluster. |
| Remove | Cmdlets starting with Remove should help you delete\destroy a cluster object that you do not need. For example, Remove-ClusterGroup, will delete a group. Remove-Cluster should destroy the cluster. |
| Repair | We have a cmdlet Repair-ClusterShareVolume that would enable us to Run ChkDisk and Defrag on a CSV Volume. This cmdlet needs to be run on a cluster node. |
| Resume | Any cluster object that can be suspended (paused) can be resumed using Resume-Cluster* cmdlets. |
| Set | This verb is not as obvious and a little harder to understand, but not really if you continue reading  .Set-Cluster* cmdlets were introduced to make some non-trivial tasks trivial, for example, setting values of cluster object parameter ( private properties), setting Group preferred owner list, or resource owner list. |
| Start | Cmdlets starting with Start are equivalent to Online or bringing a cluster resource up. For example, Start-ClusterResource, Start-ClusterGroup, Start-Cluster, Start-ClusterNode. |
| Stop | Stop is equivalent to Offline. |
| Suspend | After knowing about Resume, you can guess that Suspend is used to pause a resource. |
| Test | Some interesting and more common cmdlets start with Test verb. There are two cmdlets that start with Test, namely Test-Cluster (to run the “Validate a Configuration…” wizard) and Test-ClusterResourceFailover (to simulate a resource failure). |
| Update | Again two cmdlets start with Update. We have Update-ClusterIPResource (used to renew\release the DHCP IP Address leases) and Update-ClusterVirtualMachineConfiguration (used to refresh a VM’s information after a change) |
I understand that there is change in terminology and it could take a while for us to get used to it, but it is a necessary design decision to align with the changes in technology.
I would also agree that some of you that the selection of verbs is somewhat inconsistent with our UI, but this is to align with the larger Windows Server goal of consistent naming conventions, which gives us a limited set of verbs to use.
Nouns
Now let’s move on to the nouns…
The interesting part of the cmdlets is the noun, the second part of the cmdlet. The first word is usually Cluster followed by a cluster object. For example, resource is a cluster object, so the appropriate noun would be ClusterResource, group is a cluster object, so it’s ClusterGroup.
We do have a smaller set of cmdlets that have only one word (as noun) and that is cluster. Some examples are New-Cluster, Get-Cluster.
With some knowledge of verbs and nouns, you should have some guidance towards discovering the cluster cmdlet you need.
Discovering Parameters
One of the objectives of PowerShell is self-discovery. If you type ‘-’ and press <tab> you should get the name of parameters which the cmdlet accepts. If you press <tab> again you should get the next parameters.
Another way to learn about the cmdlet is to view the built-in help topic. I know as of today there is no MSDN documentation available for the cmdlets, but do you really need it? Have you tried Get-Help on any of Cluster cmdlet? If not, please try it and send us any feedback.
This should not be new for any PowerShell user and don’t worry, I do not intend to talk about Get-Help and using <-><tab> in this section.
Some simple things first…
All our cmdlet’s can be run from a cluster node or from a remote client. The cmdlets ship as part of admin pack and works with PowerShell V2.
For these cmdlet’s to work against a cluster, we do need to provide cluster name. There are multiple ways to provide cluster information to a cmdlet.
1. Every cmdlet has a -Cluster parameter, where you can specify the name of the cluster, or an IP address of the cluster, or the name of a Network Name resource type hosted on the cluster. If I am confusing you, sorry, but for now just keep in mind the -Cluster parameter.
2. Most of our cmdlet do accept Cluster Object as piped in input. And a cluster object can be obtained by using Get-Cluster -Name <Cluster Name>. Get-Cluster is the only cmdlet that does not have a -Cluster parameter.
3. If you are executing a command on a cluster node, -Cluster need not be specified. The cmdlets are usually intelligent enough to detect that you are working on cluster node. But if you are managing multiple clusters from a cluster node, please use the -Cluster parameter to address appropriate cluster, if not specified, the cmdlet would assume that you are working with the local cluster.
4. The -Name parameter allows you to specify an object name representing the noun in the cmdlet. For example, Get-ClusterResource -Name <resource name>. If you provide a group name or any other name that is not a resource, the cmdlet would return an error.
5. Positional parameters! Positional parameters are parameter values that are not paired up with the parameter name. The Idea behind positional parameters is that a user does not always type the parameter name, if the values can be identified based on position. Most of our cmdlets are expected to work with positional parameters. In most cases the value for -Name is positional parameter, but there are some cases where the noun can be passed in as a piped in object and in that cases the position shifts.
For example, you could do the below to move a group to node n1
Get-Clustergroup <group name> | Move-ClusterGroup n1
In the example above <group name> is taking the advantage of positional parameter as I have not linked the value with any parameter, but then Move-ClusterGroup accepts -name <group name> -node <destination node>. In the example above, the cmdlet is getting the group object through piped input, thus it assumes that the parameter passed to it is a node name.
The Idea here is to keep things simple and make them work in most obvious cases.
Cluster.exe to PowerShell Comparison
To help you get started with the cmdlets, I built this table which shows some common cluster.exe command and their matching PowerShell cmdlets.
| Cluster.Exe Command | PowerShell Command |
| Cluster.exe <Cluster Name> /prop | Get-Cluster <Cluster Name> | fl * |
| Cluster.exe <Cluster Name> /priv | Get-Cluster <Cluster Name> | Get-ClusterParameter |
| Cluster.exe <Cluster Name> group | Get-ClusterGroup -Cluster <Cluster Name> |
| Cluster.exe <cluster Name> node | Get-ClusterNode -Cluster <Cluster Name> |
| Cluster.exe <cluster name> res | Get-ClusterResource -Cluster <Cluster name> |
| Cluster.exe <Cluster name> net | Get-ClusterNetwork -Cluster <cluster name> |
| Cluster.exe <cluster name> netint | Get-ClusterNetworkInterface -Cluster <cluster name> |
| Cluster.exe <cluster name> group /move:<Node name> | Move-ClusterGroup -Name <group name> -Node <Node name> -Cluster <cluster name> |
| Cluster.exe log | Get-ClusterLog |
| Cluster.exe <cluster name> /create … | New-Cluster |
| Cluster.exe <cluster name> group /add | Add-ClusterGroup |
| Cluster.exe <cluster name> group <group name> /on | Start-ClusterGroup -Name <group name> -Cluster <cluster name> |
That is all for now, please check back later for more PowerShell information
