250 Hello

Random Musings on Exchange and Virtualization

May, 2013

  • Hyper-V Did Not Find Virtual Machine To Import

    Since I have this lovely Lenovo W530 laptop with a couple of disks and 32GB of RAM, it is great to run VMs on!  As you can imagine, I have multiple different Exchange labs to reflect all the different versions and builds that are in use.  Generally I have been able to import all of my older VMs into Hyper-V on Windows 8 and 2012 with no issues.  But as you probably have guessed by now, if I had not ran into something this blog post would not be needed………

    Update 22-10-2013:  If trying to import a Windows Server 2008 R2 VM into Windows Server 2012 R2 or Windows 8.1 please also see this post

    Update 10-1-2014:    Updated error message is present in Windows 8.1.  Please see this post

     

    What’s The Story, Morning Glory?

    <Courtesy link to Oasis>

    The machine we will look at is an Exchange ActiveSync emulator running Windows Mobile 6.5, but that’s not too relevant.  I copied the files over to the C:\VMs folder, and then went to import the VM using the wizard:

     

    Windows 8 Hyper-V Virtual Machine Import

     

       We can see the files have been copied over:

    Hyper-V VM Copied to Destination Folder

     

    Now lets import the bad boy into Hyper-V:

     

    Windows 8 Hyper-V Virtual Machine Import

              

    As the Eurovision judges from Belarus say, “nil point” Sad smile 

    Hyper-V Did Not Find Virtual Machines To Import From Location

    In the Microsoft-Windows-Hyper-V-VMMS/Admin event log we can see that an error was logged.

    • Event Source: Hyper-V-VMMS
    • Event ID: 18364
    • Event Body: The virtual machine configuration could not be read: Unspecified error (0x80004005).

    The error is stating that Hyper-V is a smidge unhappy with the configuration it is trying to import.

    Hmmm.  That’s a trifle annoying, isn’t it?  So let’s go get this fixed!

     

    Don't Look Back in Anger

    <Courtesy link to Oasis>

    Taking a look at the directory structure, we see the following:

               

    Hyper-V Virtual Machine Directory Structure

     

    Looks like we have all the files, no?  There are VHD, export and some config type files.

     

    Cracking open the World’s #1 troubleshooting tool, Process Monitor from Sysinternals, let’s trace to see what is going on at the file system level.  Maybe there is a deny NTFS permission ??

     

    When firing up Process Monitor, it will capture a lot of data by default.  A lot.  So much so that we will need to tweak the capture filter to make it easier to parse the results.  When Process monitor is opened up, either click the magnifying glass icon or press Ctrl + E to stop the capture.  The magnifying glass icon will then have a red cross to indicate it is not capturing.

    Process Monitor Showing Capture Option

    Then we can click onto the filter icon or press Ctrl + L to specify our custom filter.

     

    Process Monitor Showing Filter Option

     

    In this case we know that the VM files are located in “C:\VMs\EAS VM” so let’s filter on that:

    Process Monitor Filter Set to Hyper-V VM Directory

     

    Click Include then add to include the path filter, then click OK.  You will see a new row with a green icon at the top of the filter view (not shown above).  Now that our filter has been saved let’s clear the screen with Ctrl + X.

    We will then click the magnifying glass to start the capture, and then re-do the VM import that originally failed.  When the import fails, stop the capture with Ctrl + E.

              

     

    Process Monitor Showing VM Import Failing

     

    As you can see, there are no access denied messages.  You can search for that easily in larger captures with Ctrl + F and searching.

     

    What is interesting is that Hyper-V is looking at the .exp file and then moving on to the .xml file.  Hmm, wasn’t that one of the neat new features?  In Windows 2012 Hyper-V -- we do not have to specifically export a VM before it can be imported.  So why are both files here? Since I know that this VM was originally exported from a Windows 2008 R2 Hyper-V machine the .exp file is what is expected.

     

    So lets backup and then remove the .xml file from the directory, and then re-do the import.  Note I said backup and not nuke, just in case!!  What happens now?

    Directory Structure Showing .xml File Removed

     

    With the .xml file removed the import wizard is able to parse the VM and moves on to the next screen:

    Hyper-V VVirtual Machine Import

     

    A couple more clicks to finish the wizard and then success!  The VM was successfully imported!  In the Process Monitor capture the .exp is processed and no errors are reported.

     

    Process Monitor Showing Successful VM Import

     

     

    Champagne Supernova

    <Courtesy link to Oasis>

    After removing the spurious .xml file I was able to import the VM.   Note that there are other causes for the same error being raised.  The exact fix will depend on the version of Hyper-V.

    As mentioned above each new release of Hyper-V has added new capabilities, and if we focus on just the VM import/export aspect:

    Windows 2008 R2 Hyper-V added additional features when importing and exporting VMs, namely:

    • Copy On Import – Can choose a different directory to import to.  Windows 2008 required admins to copy the VM package to the final destination and import from there.
    • Export Snapshot – Can export just a single snapshot from the snapshot history as a complete VM.
    • Fine grained Control – improved APIs to provide control over import & export operations.  Note that Configuration Only Exports are now done only via the API in Windows 2008 R2.

    Windows 2012 Hyper-V removed the dependency on the .exp file.  In Windows 2008 and 2008 R2, a VM must have been exported before it could be imported.  This is a great feature for admins, and the following was a common scenario.  Why do we want this you ask?

    Say that your lab Windows 2008 R2 Hyper-V host lost the RAID1 OS array, and you were left with only the RAID 5 array containing the VMs.  Because the VMs were not manually exported before the OS mirror failed, you could not just import them.  Backups of the VMs would be great, but not all lab VMs are backed up for example.  Sure you could create new VMs and attach the existing .vhd files to the new configurations but that means re-doing the IP addresses and depending on the OS having to clear out the old IP and NIC configuration.  How many times have I had to use KB 241257 with the command devmgr_show_nonpresent_devices=1 to purge stale NICs from a VM.

    In Windows 2012 Hyper-V the VMs can be imported as-is – how neat is that?

     

    One more thing to add on the Sysinternals topic.  You can go to live.sysinternals.com and directly download the tools that you want.  This is very handy if you know exactly what you want.

     

    Cheers,

    Rhoderick

    Technorati Tags: ,
  • Exchange 2010 SP3 RU1 Released

    image

    The Exchange Team released the first Update Rollup for Exchange 2010 SP3.  Exchange 2010 SP3 RU1 is available for download from the download centre as update 2803727.   

     

    This is build 14.03.0146.000 of Exchange 2010.  Article 2803727 has the full details for this release. 

     

    Probably the two biggest changes that we have been waiting on are:

    2814847 Rapid growth in transaction logs, CPU use, and memory consumption in Exchange Server 2010 when a user syncs a mailbox by using an iOS 6.1 or 6.1.1-based device

    2822208 Unable to soft delete some messages after installing Exchange 2010 SP2 RU6 or SP3

     

    You can find a previous blog about the soft delete issue here.

     

    In addition to this, there is a fix for the move mailbox issue in Exchange 2010 SP2 RU3+

    “Post-move cleanup failed. The operation will try again in 30 seconds”

    2763065 Move request log is logged when you move a mailbox in an Exchange Server 2010 SP2 environment

     

    Issues Resolved

    Exchange 2010 DAG AllowCrossSiteRPCClientAccess Reverts to False is resolved in SP3 RU1.

     

    • 2561346 Mailbox storage limit error when a delegate uses the manager's mailbox to send an email message in an Exchange Server 2010 environment

    • 2729954 Can't send voice message to a selected non-primary email address in an Exchange Server 2010 environment

    • 2750846 Information Store service crashes when you mount public folder databases on an Exchange Server 2010 server

    • 2751628 Event ID 9682 does not record the folder name when you delete a public folder in an Exchange Server 2010 environment

    • 2756460 You cannot open a mailbox that is located in a different site by using Outlook Anywhere in an Exchange Server 2010 environment

    • 2763065 Move request log is logged when you move a mailbox in an Exchange Server 2010 SP2 environment

    • 2777742 Address Book service crashes on an Exchange Server 2010 Client Access server when a server has been running for 25 days or more

    • 2781488 RPC_S_SERVER_UNAVAILABLE (0x6BA) error code when you use a MAPI or CDO-based application in an Exchange Server 2010 environment

    • 2782683 Email message that a user sends by using the "Send As" or "Send On Behalf" permission is saved only in the Sent Items folder of the sender in an Exchange Server 2010 environment

    • 2784210 Ethical wall does not function as expected when the ReportToOriginatorEnabled property is disabled in an Exchange Server 2003 and Exchange Server 2010 coexistence environment

    • 2793348 Read receipt is sent unexpectedly when you view an email message by using Outlook Web App

    • 2796490 Microsoft Exchange Information Store service crashes on an Exchange Server 2010 Mailbox server

    • 2802569 Mailbox synchronization fails on an Exchange ActiveSync device in an Exchange Server 2010 environment

    • 2806602 EdgeTransport.exe process crashes on an Exchange Server 2010 Hub Transport server

    • 2814723 Server loses network connectivity and UDP ports are used up on an Exchange Server 2010 server

    • 2814847 Rapid growth in transaction logs, CPU use, and memory consumption in Exchange Server 2010 when a user syncs a mailbox by using an iOS 6.1 or 6.1.1-based device

    • 2816934 Error code 0X800CCC13 when an additional POP3 or IMAP account is used to send an email message and Outlook online mode is used to connect to an Exchange Server 2010 environment

    • 2817140 Exchange Replication service crashes intermittently in an Exchange Server 2010 environment

    • 2817852 Cyrillic characters are displayed as question marks in the "To" field of message items in the Sent Items folder in an Exchange 2010 environment

    • 2818456 Attachments are missing from an embedded message in an Exchange Server 2010 SP2 environment

    • 2822208 Unable to soft delete some messages after installing Exchange 2010 SP2 RU6 or SP3

    • 2826066 VSAPI-based antivirus software causes delayed response in an Exchange Server 2010 environment

    • 2827037 Copy of an item is created in the Version subfolder in an Exchange Server 2010 environment

    • 2833888 No items are displayed in Outlook after you install Exchange Server 2010 SP3 or Update Rollup 6 for Exchange Server 2010 SP2

    • 2840099 ArgumentOutOfRangeException exception when an EWS application creates a new MIME email in an Exchange Server 2010 environment

     

     

    Please Note – Language Issue

    This may, or may not be an issue depending on the language used for your Windows Server 2012 installation. 

     

    You cannot install or uninstall Update Rollup 1 for Exchange Server 2010 SP3 on a computer that is running the double-byte character set (DBCS) version of Windows Server 2012 if the language preference for non-Unicode programs is set to the default language. To work around this issue, you must first change this setting. To do this, follow these steps:

    1. In Control Panel, open the Clock, Region and Language item, and then click Region.
    2. Click the Administrative tab.
    3. In the Language for non-Unicode programs area, click Change system locale.
    4. On the Current system locale list, click English (United States), and then click OK.

    After you successfully install or uninstall Update Rollup 1, revert this language setting, as appropriate.

    The Exchange Team are aware of this and will look to correct this in a future update.

     

    Important Notes

    Now, before we rush off to download and install this there are a couple of items to mention!

    • Test the update in your lab before installing in production.  If in doubt test…
    • If the Exchange server does not have Internet connectivity then this introduces significant delay in building the Native images for the .Net assemblies as the server is unable to get to http://crl.microsoft.com.  To resolve this issue, follow these steps:
      1. On the Tools menu in Windows Internet Explorer, click Internet Options, and then click the Advanced tab.
      2. In the Security section, click to clear the Check for publisher's certificate revocation check box, and then click OK.
      We recommend that you clear this security option in Internet Explorer only if the computer is in a tightly controlled environment. When setup is complete, click to select the Check for publisher’s certificate revocation check box again.
    • Update Internet facing CAS servers first
    • Backup any OWA customisations as they will be removed
    • Test (yes technically this is in here for a second time but it is important!)

     

    Cheers,

    Rhoderick

    Technorati Tags: ,
    >>>
  • Exchange 2010 DAG AllowCrossSiteRPCClientAccess Reverts to False

    Update 28-7-2013:  This issue is now resolved in Exchange 2010 SP3 RU1.  See end for details.

     

    When browsing a customer’s Disaster Recovery document I noticed that the Exchange admins had added a step at the end of the document when recovering the DAG back to the primary datacentre. This was to enable AllowCrossSiteRPCClientAccess in the DAG.  This was curious as it had already been enabled when the DAG was first created.  This environment has Exchange 2010 SP2 RU3 and higher installed so AllowCrossSiteRPCClientAccess is a supported capability.

    The admins were running:

     

    Set-DatabaseAvailabilityGroup –Identity DAG -AllowCrossSiteRPCClientAccess:$True

     

    This was being done as they had noticed in testing (full marks for testing !!) that the AllowCrossSiteRPCClientAccess  setting was being changed from $True to $False.  To ensure that the DAG was configured as per the design they were running the Set-DAG command to again reset the value for AllowCrossSiteRPCClientAccess.

    This was a concern as we assumed that the setting should not be getting changed.

     

    Sometimes we want to use Set-DatabaseAvailabilityGroup with no additional parameters to:

    • Remediate incorrect cluster quorum types.  Running Set-DatabaseAvailabilityGroup with no other parameters will set the correct quorum type based on the number of nodes in the underlying cluster.
    • Correct issues with File Share Witness (FSW) folders and permissions.  For example if the FSW share is manually removed this will re-create it.

     

    What happens if Set-DatabaseAvailabilityGroup is run to change another aspect of the DAG – would the AllowCrossSiteRPCClientAccess be changed in that scenario? Let’s see…..

     

    Quick Batman, to the Lab…..

    So Robin, here we are in an Exchange 2010 SP3 lab.  We have a simple 3 node DAG, called “DAG-01”.  All Exchange servers have the same build running on Windows 2008 R2 SP1 Enterprise Edition.

    Exchange 2010 DAG Starting Configuration - Exchange Build Is SP3

    This is the DAG we shall look at:

     

    Exchange 2010 3 Node DAG Starting Configuration

     

    Initially DAG-01 does not have AllowCrossSiteRPCClientAccess set to $True

     

    Exchange 2010 DAG - AllowCrossSiteRPCClientAccess Is Not Enabled

     

    Then we set the AllowCrossSiteRPCClientAccess value to $True, and then verify

    Exchange 2010 DAG - AllowCrossSiteRPCClientAccess Enabled & Verified

     

    So far so good!  But let’s now run the following command and see what happens to the AllowCrossSiteRPCClientAccess value:

    Set-DatabaseAvailabilityGroup –Identity DAG-01

     

    Exchange 2010 DAG - AllowCrossSiteRPCClientAccess Has Changed From Enabled to Disabled

     

    You can see that as a result of running the Set-DatabaseAvailabilityGroup cmdlet the value for AllowCrossSiteRPCClientAccess has changed from $True to $False.

     

    The same behaviour was also noted when running other parameters in Set-DatabaseAvailabilityGroup and again without specifying the AllowCrossSiteRPCClientAccess  parameter.

    Exchange 2010 DAG - AllowCrossSiteRPCClientAccess  Has Changed From Enabled to Disabled

     

     

    Conclusion

    The behaviour that we saw was the AllowCrossSiteRPCClientAccess was reverting back to it’s default value ($False) unless it was explicitly specified in the command.

     

    If you have set AllowCrossSiteRPCClientAccess to $True for a DAG in your environment, please ensure that when running Set-DatabaseAvailabilityGroup cmdlet you specify the AllowCrossSiteRPCClientAccess.  You can also run Get-DatabaseAvailabilityGroup to validate the configuration on a scheduled basis.

     

    BONUS TIP:  Some of the Exchange cmdlets do not retrieve all of their data by default as they are costly operations.  To instruct the cmdlet to retrieve this additional data add the –Status parameter.   Set-DatabaseAvailabilityGroup is a good example of this!  Try it and see the difference when piping to Format-List.

     

    Update 28-7-2013

    This issue is resolved in Exchange 2010 SP3 RU1.  To illustrate, the example below shows a DAG with AllowCrossSiteRPCClientAccess set to $True. When Set-DAG this then run, note that the $True does not revert to $False.

    Exchange 2010 - SP3 RU1 AllowCrossSiteRPCClientAccess  Preserved When Running Set-DatabaseAvailabilityGroup

     

     

    Cheers,

    Rhoderick

    Technorati Tags: ,
  • Dude, Where’s My Set-DAG Command

    Have you read a blog or web site that asked you to run the Set-DAG command?  Were you told by someone to run the Get-DAG command?  You then went to your lab Exchange server to test and validate the command only to find that it does not work.  At this point it’s a case of: Dude, where’s my command**

    The Exchange Management Shell will look like the below:

     

    <Soup Nazi> No Set-DAG Command for you!  <Soup Nazi>

     

    The reason is as mentioned in the red output text, the command is not recognised.  It’s simply just not there…

     

    This is because Set-DatabaseAvailabilityGroup is a little wordy to type, and is often abbreviated to Set-DAG.  The same is true for the other DAG commands as well.  I’ll put them into a table so that search engines will pick them up and people can easily find them.  And for the record, I’m pretty guilty of shortening them which is why I started to write this in the first place……

     

    Database Availability Group

     

    Full Command Abbreviated
    Restore-DatabaseAvailabilityGroup Restore-DAG
    Start-DatabaseAvailabilityGroup Start-DAG
    Stop-DatabaseAvailabilityGroup Stop-DAG
    New-DatabaseAvailabilityGroup New-DAG
    Remove-DatabaseAvailabilityGroup Remove-DAG
    Set-DatabaseAvailabilityGroup Set-DAG
    Get-DatabaseAvailabilityGroup Get-DAG
       

    Database Availability Group Server

     

    Full Command Abbreviated
    Add-DatabaseAvailabilityGroupServer Add-DAGServer
    Remove-DatabaseAvailabilityGroupServer Remove-DAGServer

     

    Database Availability Group Network

     

    Full Command Abbreviated
    Get-DatabaseAvailabilityGroupNetwork Get-DAGNetwork
    New-DatabaseAvailabilityGroupNetwork New-DAGNetwork
    Remove-DatabaseAvailabilityGroupNetwork Remove-DAGNetwork
    Set-DatabaseAvailabilityGroupNetwork Set-DAGNetwork
      

    Cheers,

    Rhoderick

    ** – Ashton Kutcher films never really became more cerebral, did they?

     

    Technorati Tags: ,
  • Exchange 2013 SCOM Management Pack Download

    Update: Management Pack was updated on 12th July 2013

    Update: Management Pack was updated on 28th October 2014 to version 15.0.652.19

     

     

    imageThe Exchange 2013 System Center Operations Manager (SCOM) Management Pack has been released to the download centre.

    This Management Pack is designed to work with the new features in Exchange 2013, namely Managed Availability.  It is supported on SCOM 2012 RTM or later, and on SCOM 2007 R2.  More on these big changes later on in this post!

     

     

    The Management Pack (MP) is build 15.00.0620.018 can be obtained here.    Before installing the MP, please be sure to review all of the documentation and prerequisites.  A great starting place is the MP Guide.

     

    One of great benefits of running Office 365 is that the Exchange product group can share solutions to scenarios that they see in the service which will benefit on premise installations.  The Exchange Server 2013 Management Pack works with the Managed Availability feature in Exchange 2013. The alerts within the System Centre Operations Manager (SCOM) portal indicate unhealthy states as reported by the Managed Availability components in Exchange 2013.

    What’s New

    The following are some of the new features in the Exchange 2013 Management Pack:

    • Simplified dashboard The dashboard of the Exchange 2013 Management Pack has been simplified and refined into the following three categories:
      • Active Alerts Provides a list of all outstanding alerts in your organization.
      • Organization Health Provides an overview of the overall service health in your organization.
      • Server Health Provides an overview of the health of individual servers in your organization.
    • User focused monitoring Exchange 2013 introduces a monitoring and recovery infrastructure called Managed Availability. Managed Availability focuses on the user experience. All Exchange 2013 components have built-in monitors that detect problems and attempt to recover the service availability. Any issues that can't be recovered automatically are escalated to the Exchange 2013 Management Pack as an alert.

     

    What’s The Big Deal With That?

    What does that really mean?  Well you will see a radically simpler, smaller and leaner management pack!  Why is this?

    Exchange 2013 with Managed Availability is now able to monitor itself and determine if it is healthy or not.  Who better to gauge the condition of Exchange than Exchange itself!    Another benefit is that this can be controlled by Exchange itself in two distinct ways:

    • Admins can tune Managed Availability themselves without having to engage SCOM team
    • Exchange Product Group can ship Managed Availability enhancements without having to ship a new Exchange Management Pack.

    Both of these mechanisms will allow for a greater degree of flexibility and agility when monitoring Exchange 2013.  Additionally you may have seen previous issues around the Exchange 2010 Management Pack when it was updated for Exchange 2010 SP2 and then re-released.

     

    One other change that will bake people’s noodles is that changes to monitoring thresholds are to be done in Exchange directly and not in SCOM.  Again this is because Exchange is monitoring itself, and SCOM is responsible for the escalations that are produced by Managed Availability.

     

    Management Pack Troubleshooters

    The Exchange Server 2013 Management Pack relies on the Managed Availability feature in Microsoft Exchange Server 2013. In Managed Availability, each component in Exchange Server 2013 monitors itself using probes, monitors, and responders. Any component that implements Managed Availability is referred to as a health set.   For example the OWA Health Set has documented troubleshooting steps.

    This includes getting details about the server and Health Set in question

    Get-ServerHealth <server name> | ?{$_.HealthSetName -eq "<health set name>"}

    Get-ServerHealth Exch-1.Tailspintoys.com | ?{$_.HealthSetName -eq "OWA"}

    Check to see that none of the Alert Values are displayed as unhealthy.  If one is shown in an unhealthy state, then rerun the associated monitoring probe:

    Invoke-MonitoringProbe <health set name>\<probe name> -Server <server name> | Format-List

    All of the Exchange 2013 Management Pack Troubleshooters are documented here.

    Other Changes From the Exchange 2010 MP

    We already covered that Exchange is now responsible for monitoring itself and SCOM will do the escalation portion of the alerting system.  In addition there are a couple of other changes that are worth highlighting:

    • Correlation engine – there is no correlation engine in this MP.  The correlation engine was introduced in the Exchange 2010 MP to link alerts which likely had the same root cause.  In large environments this could be a significant resource impact on the SCOM RMS.
    • There are no additional reports included with this MP
    • No additional performance monitor counters are included.  Exchange 2013 does monitor a lot of performance data by default.  Please look for an upcoming post on this topic!

     

    Additional Reading

    Exchange Team blog

    SCOM Team blog

    Lessons from the Datacenter: Managed Availability

     

    Cheers,,

    Rhoderick

    Technorati Tags: ,
  • Office 2013 Click-To-Run Not Updating

     

    When the office updates were released last patch Tuesday, there were a couple of issues of mine that were resolved.   Great, I thought, let’s download and install them!    After checking that my installed build of Outlook etc. was older than the released updates I downloaded the updates from the Microsoft Download Centre and then went to install them only for it to barf:

    image

    After muttering briefly, I realised that I was not actually using a traditional .msi install of Office 2013. I’m actually using a Click-To-Run installation on the MSIT Windows 8 image.  DOH!

     

    Stop, Hammer Time!

    But, wait.  Wasn’t my office build out of date?  Part of the advantage of Click-To-Run is that it will automatically update.  Why was this not happening?  One quick Bing search later I noted that in the article Microsoft Office 2013 Click-to-Run virtualization there were steps to retry the Office update :

     

    1. Open any Office application
    2. Click on the File tab
    3. Click on Account (Office Account in Outlook)
    4. Click on Update Options
    5. Click on Disable Updates
    6. Click on Update Options again
    7. Finally, click on Enable Updates

     

    Update 8-4-2013

    After updating to Office 2013 SP1, the interface appears different.  Note that there is not a button to check for updates

     

    Office 2013 Click To Run Update Now

     

     

       As soon as the updates were re-enabled – boom!  They started to download!

     

    image

     

     

    The updates then installed in the background as I continued to use the applications, and only when all the updates had been installed was I prompted to restart them.

     

    image

    Note that a restart of the OS was not needed here, just the applications had to be restarted.  Again another neat feature of the Click-To-Run package!

     

    Untitled

     

     

    Those of you familiar with App-V will be thinking this is kind of familiar, and it is Smile.  Click-to-Run is based on App-V, which itself came from the Soft Grid acquisition).

    Click-To-Run was launched with Office 2010, and with more and more people moving to Office 365 it is destined to become even more popular!

     

    One thing that you may experience when running Click-to-Run is that some of the traditional troubleshooting tools that expect Outlook to be installed as a .msi will have issues.  Mr MAPI (AKA Stephen Griffin) mentions a workaround on his blog for MFCMAPI to get it to work with Click-to-Run Outlook.

     

    Cheers,

    Rhoderick

    Technorati Tags: