250 Hello

Random Musings on Exchange and Virtualization

Slow Response To Exchange Virtual Directory Cmdlets

Slow Response To Exchange Virtual Directory Cmdlets

  • Comments 15
  • Likes

Some folks in the field may have seen this before, but it’s worth bubbling up to make sure everyone is aware of it! 

I was sitting with one of my esteemed consulting colleagues today and he remarked that it was talking a long time to run one of his Exchange PowerShell scripts.  The customer in question is a global organisation with hundreds of Exchange servers in all corners of the globe.  My colleague was ensuring that the customer had correctly implemented the design and in essence was auditing the configuration.  The script was taking several hours to run.

This customer actually has very good WAN links across the globe and the lines are dedicated for their usage, else the script could have taken even longer! 

Whilst discussing this over coffee, and how to wear a shirt correctly (don’t ask), I asked how he was accessing the Virtual Directory information.  He was using the default mechanism, which unfortunately is the slowest.  So when pulling in the data from each server’s virtual directory it was taking a long, long time.  You will see these symptoms with any of the following cmdlets:

  • Get-WebServicesVirtualDirectory
  • Get-OwaVirtualDirectory
  • Get-ActiveSyncVirtualDirectory
  • Get-AutodiscoverVirtualDirectory
  • Get-EcpVirtualDirectory
  • Get-PowerShellVirtualDirectory
  • Get-OABvirtualDirectory

 

For the purposes of this article, we shall use the Get-OwaVirtualDirectory cmdlet, but the behaviour is mirrored for all of the above.  The core parameters are the same for all of these cmdlets.   Here is the Exchange 2013 link for Get-ActiveSyncVirtualDirectory for example. 

 

What Causes This

When Get-OwaVirtualDirectory cmdlet is executed against a server the default mechanism is to go over the wire and make RPC calls to the IIS Metabase on that sever.  This is fine if the server making the request and the target  are in the same datacentre.  It is not so fine if they are on different continents! 

 

How To Workaround This

The solution to this is frustratingly simple.  We can add a parameter to the cmdlet which instructs it not to go to the remote server to get the answer, rather it will query AD to get the data.  Since this data is stored within the Configuration naming context in AD, and those Global Catalog servers are conveniently spread across the enterprise we can make a query to a local GC obviating the need to make a remote query. 

As an example, you may be running this:

Get-OwaVirtualDirectory -Identity "Contoso\owa (default Web site)"

To query AD, simply add –ADPropertiesOnly switch

Get-OwaVirtualDirectory -Identity "Contoso\owa (default Web site)" –AdpropertiesOnly

 

One thing to note the properties stored in the Internet Information Services (IIS) metabase aren't returned when ADPropertiesOnly is used.  Only the properties stored in AD are returned.  Funny, eh?

Update 12-9-2014: Please also see this post if you are looking for authentication attributes as you may be misled by the results.

 

Real World Example

Sitting at a machine in Canada running Get-OwaVirtualDirectory against a Singapore machine took just over 5 minutes to return data.  With the AdpropertiesOnly parameter I could see the internal and external URLs in less than 2 seconds!  WIN !!

 

 

Bonus Tip

I personally hate having to specify the "Contoso\owa (default Web site)" text when running such cmdlets as it takes more time and is error prone.   To avoid this I just specify the server using the –Server parameter.  For the vast majority of installations a server has a single OWA VDir so this works well.  Should an Exchange 2007/2010 server have multiple OWA VDirs then you may have to be more specific. 

Get-OwaVirtualDirectory –Server Contoso-CAS01  –AdpropertiesOnly


One final thing!  Don’t go looking for ADPropertiesOnly in an Exchange 2007 Management Shell.  It is not there.   The below is from an x64 Exchange 2007 SP3 RU10 machine.

Looking In Vain For ADPropertiesOnly In Exchange 2007 SP3 RU10

 

Cheers,

Rhoderick

Can You Help Us? - YES!

If you would like to have Microsoft Premier Field Engineering (PFE) visit your company and assist with the topic(s) presented in this blog post, then please contact your Microsoft Premier Technical Account Manager (TAM) for more information on scheduling and our varied offerings!

If you are not currently benefiting from Microsoft Premier support and you’d like more information about Premier, please email the appropriate contact below, and tell them you how you got introduced!

US

Canada

For all other areas please use the US contact point.

Comments
  • Super!

  • Thanks always interesting the truth is I noticed this switch somewhere on web but wasn't sure what it saves me exactly:) now I know:)

  • Is there any workaround if the properties that you need are the ones that aren't returned, or if you are using the Set-*virtualdirectory cmdlets?

  • Not that I've seen David - if you need a property that requires a trip to the metabase, then get a coffee....

  • LOL. Thanks Rhoderick! I'm actually querying hundreds of servers, so it'll be more like take a day off. I'm looking into querying IIS directly via the webadministration module and invoke-command, so I'll let you know if I come up with a solution. Cmdlet should do this already. :-)

  • My sense of humour can be a bit blunt sometimes -- :) I suppose you could do that for read operations, but the product group will not have tested that for write tasks, thus we will not support that for regular Exchange tasks. Out of curiosity, what are you looking for? Cheers, Rhoderick

  • Thanks for the helpful article Rhoderick. Here's my scenario though, which is a little different. I'm running the Get commands locally and it takes forever to run. This is the second and third exchange server. The first exchange server is in a different site/datacenter and running commands locally is fast. Not sure how to speed up RPC calls to the IIS Metabase on the additional two exchange severs. Is this authentication related, even though the commands eventually successfully return the results....after minutes of waiting.

  • Hi Rhoderick,

    I was trying to find this information everywhere, I thought maybe I was being firewalled between sites perhaps, and that it timed out from trying a secured connection, dropped back to port 80 remote powershell, and finally went through.... but that wasn't it. I then started thinking along the lines of MTU limitations and your article pointing out that it actually deviates from port 80 remote powershell and actually sends RPCs finally clarified it for me.

    Now I am wondering this: I have come across Kerberos auth issues via VPN before that can be resolved by forcing those packets to use TCP which can be fragmented and aren't limited by MTU the same way as UDP is, all via a registry key. So I wonder if the same logic can be applied here, and these RPC packets be forced to use TCP. It might then speed up the transmission between sites, if the gateways are the bottlenecks? I haven't tried it yet, wondering if anyone has?

  • Hi Hados,

    I have also seen the fun of truncated packets due to the VPN overhead, not so happy memories with that one!

    RPC between boxes should be TCP, I'd netmon to prove it though.

    I do need to revisit this as its popping up with 365 installs at the moment.
    When I get time........

    Cheers,
    Rhoderick

  • Has anything else been found with this? I'm trying to install Exchange 2013 in a two AD site organization and I'm getting bit hard by this slow RPC issue.

    My mailbox is hosted on a server in Site1, but I can't for the life of me configure all the vDirs for the servers in Site2 (I'm sitting at 10+ minutes waiting for a single vdir setting to be changed with no end in sight). I'm doing this through the EAC GUI. I see no way to get connected to the local site machines, with the GUI anyway. I'd prefer not to resort to using powershell directly for things like this.

    Hoping you've made some headway...

  • Hi Matthew,

    Do you see any performance difference if you go to https://serverInSecondSite/ECP and make changes from that URL?

    I've been looking at automating this via powerShell, and have not focussed on the GUI.

    Cheers,
    Rhoderick

  • Hi Rhoderick. Thanks for responding. Sorry for the delay, but I'm not getting notified of responses to my comments for some reason.

    At any rate, I haven't tried making anymore changes as my project got derailed due to RPC/HTTP proxy issues with my 2013 deployment. Will revisit that when I can.

    On a side note, do we know why RPC over the WAN links are so slow? I'm not in a predicament where I'm trying to document the entire Exchange environment running a litany of get-XYZvirtualdirectory commands in a single script against all of my servers, and I don't think this thing is EVER going to complete!

    I keep searching for an answer to this RPC slowness issue but I just keep coming up with this blog post as the most common result. I'm on 2007 so I can't rely on the AD parameter. Is my only recourse to create datacenter-specific scripts and target servers in each location individually? Wish I didn't have to do that. I would like to just understand the underlying issue in hopes of fixing it.

  • Hi Matthew,

    RPC over latent WANs is never a great situation, and combined with the data volume transferred here leads to the slowness that you see. There is not much that you can do to fix this with Exchange 2007.

    For 2007 datacentre specific scripts (executed locally in that DC) will probably be the best way to go about this since PowerShell is local and we cannot do PowerShell remoting.

    sorry!

    Cheers,
    Rhoderick

  • so, I have always had this problem, even more so in 2013. I can verify that using the -adpropertiesonly, and then piping that to a set-owavirtualdirectory, or whichover one, works beautifully. Thanks Rhoderick.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
Post Comment Fixer