This post was promoted from the draft bin to production after a customer visit a couple of weeks ago. When onsite we were looking at how the environment was configured. The admins had written a series of scripts to determine the environment state which was excellent to see! One thing that they had assumed though was that the ServerName attribute on a mailbox was where the database was currently mounted. In this customer’s case they had 12 mailbox servers in the DAG and this data led them to believe that mailboxes were evenly balanced across all of the servers.
Let’s see what is going on, and what caused the issue.
In this lab we have Exchange 2010 SP3 RU5 servers. A single Database Availability Group (DAG) exists which has three members CONSEA-MB1, CONSEA-MB2 and CONDAL-MB1
No database copies are mounted on mailbox server CONSEA-MB1 apart from DB01:
Just to ensure that there are no MMC refresh issues, PowerShell shows the same:
Since only DB01 is mounted and active on server CONSEA-MB1, then we should expect to see the same number of mailboxes returned if we check both – no?
Get-Mailbox –Server CONSEA-MB1 Get-Mailbox –Database DB01
Get-Mailbox –Server CONSEA-MB1
Get-Mailbox –Database DB01
Let’s run both commands, and pipe to Measure-Object since that makes it easy to count.
Are the numbers the same? Well, not so much…
In this case we are only off by one, but this is a tiny test lab and not representative of reasonable production environment.
What is causing this?
The ServerName attribute is written to AD when the mailbox is created or moved into that database. Which name is used? The server which was hosting the active mailbox copy at that time. The ServerName attribute is held in AD and is stamped on the user object. We can see this on the below test mailbox:
The ServerName value is not updated when the database is activated on another server. If it were, then that would add considerable overhead to AD replication. That would make the grumpy triangle people even more grumpy, and we don’t want that!
Even *IF* this value was updated and replicated by AD, the other issue is replication latency. AD may take hours to replicate between AD sites. This is far too slow for certain Exchange database tasks such as updating log generation values which is why we use the cluster database to ensure fast guaranteed updates for critical database information.
To illustrate, let’s activate DB01 on a different server, in this case CONSEA-MB2. In the below screen shot you will see that DB01 was moved from CONSEA-MB1 to CONSEA-MB2. Then we check to make sure that there are no other active databases on server CONSEA-MB1. Finally we re-run the Get-Mailbox –Server cmdlet to see how many mailboxes are stamped with a ServerName attribute of CONSEA-MB1, and if that value has changed from the initial result of 30.
Has the count changed from the initial value of 30?
No it has not. This shows that the attribute is not updated when a *-over event occurs in a DAG.
In the below example we shall move mailbox Test-100 from database DB01 to DB02. The initial ServerName value is CONSEA-MB1. DB02 is currently mounted on server CONDAl-MB1. This is indicated in the red box below. Note that once the move request completes, the ServerName value is updated with the name of the mailbox server which hosted the active copy of the database at that point –> CONDAL-MB1.
Again, we see in the below example that activating another copy of the mailbox does not change the ServerName value. Initially it was mounted on CONDAL-MB1, then moved to CONSEA-MB2.
In a DAG environment, the ServerName attribute becomes less useful as there are typically multiple copies of a given mailbox database which can seamlessly transition between multiple servers. The ServerName attribute is not updated in AD when the *-over event occurs.
The ServerName value is stamped based on where the database was mounted when the mailbox was created or last moved. It is possible to get it to update by running:
Set-Mailbox <user> –Database samedatabasename
Note that we are setting the same database to the user. In the lower line the ServerName field has now been updated.
When determining how many databases are actually running off a given mailbox server in a DAG, it is necessary to see what databases are currently mounted on each mailbox server and then enumerate the mailboxes from there. This could look something like the below one-liner:
$(Get-MailboxServer | Get-MailboxDatabaseCopyStatus | Where-Object {$_.Status –eq “Mounted”} | Sort-Object) | ForEach-Object { Write-Host $_.DatabaseName (Get-Mailbox –Database $_.Databasename –ResultSize Unlimited).Count }
Please note that the above is one line, and it may wrap.
Cheers,
Rhoderick
This is a link throw-down for the items that we discussed during a recent Office 365 workshop that I delivered to customers in sunny Calgary.
I’m posting the links here since they will be available to all of the attendees, and thought that others may also find them useful/interesting.
Exchange Online Service Description – required reading! Especially the limits section. Read this now. Do not be surprised…..
MXToolbox – useful site to test DNS records, SMTP blacklists etc.
Remote Desktop Connection Manager (RDCMAN) 2.7 is now available. Downloadable from here.
New Outlook for MAC - New version of Outlook for MAC (MacLook).
The new Outlook for Mac includes:
Planning an Exchange hybrid deployment. This page has the support statement that shared mailboxes and mailboxes accessing them must reside in the same premises.
Mailbox permissions On-premises mailbox permissions such as Send As, Receive As, and Full Access that are explicitly applied on the mailbox are migrated to Exchange Online if the tenant in Exchange Online has been fully synchronized using Dirsync or AAD Sync. Inherited (non-explicit) mailbox permissions such as permissions applied to the mailbox database and any permissions on non-mailbox objects (such as distribution lists or a mail-enabled user) are not migrated. Therefore, you should recreate these permissions in Exchange Online using the Add-MailboxPermission or Add-RecipientPermission cmdlets.
Cross-premises permissions Mailbox permissions such as Send As, Receive As, and Full Access are not supported if the user trying to access the mailbox is in Exchange Online but the target mailbox is on-premises, or vice versa. Typically, when migrating a user mailbox from on-premises to Exchange Online, in order to overcome this limitation, mailboxes belonging to users who have access to the first mailbox should also be migrated at the same time to ensure the delegate scenarios continue to work.
Authentication changes to Office 2013. This was first announced at MEC 2014 and earlier this year on the Office blog. The November update is here.
Microsoft Virtual Academy – multiple training videos
Office Technical Blog
Garage Series
DirSync release announcement of Password Sync.
List of Attributes that are Synced by the Azure Active Directory Sync Tool
How To Run Manual DirSync / Azure Active Directory Sync Updates
DirSync: How To Switch From Single Sign-On To Password Sync
Exchange Innovation lab – if only it were real….
Do fish drown? – yes they do….
Cheese phobia is called Turophobia. It’s a long story…………….
Purpose of this script is to report on particular performance monitor counters for Outlook RPC Client Access, OWA and Exchange ActiveSync on multiple servers. This was required as during the course of the year there have been a few issues where the third party load balancer device was not really doing equitable load balancing. In some cases 60% of the user load of 50,000 mailboxes was directed onto a single server.
Using the script we can easily see the number of Outlook RPC connections, the number of OWA users and how many ActiveSync requests are issued per second. These are all standard performance monitor counters, the script simply pulls them all into one place.
The script will build a collection of all the Exchange 2010 CA servers in the organisation. if you need to restrict the list to a single site the query can be easily modified. For more PowerShell filtering examples please see this post. This is the line that would require editing if you wished to restrict the collection to a single AD site, or subset of servers:
$ExchangeServers = Get-ExchangeServer | Where-Object {$_.AdminDisplayVersion -match "^Version 14" -and $_.ServerRole -Match "ClientAccess" } | Sort-Object Name
The script has a handy dandy progress bar to show completion status:
And the finished results should look something like the below, except that your numbers will likely not be zero. Mine are zero as this is a lab…
Please download the script from the TechNet Gallery:
Please also provide feedback here or on the TechNet Gallery site!
PS Note to self: Script is clearly lacking as there is no magenta text………..
The Exchange team today announced the availability of Update Rollup 8 for Exchange Server 2010 Service Pack 3. RU8 is the latest rollup of customer fixes available for Exchange Server 2010. The release contains fixes for customer reported issues and previously released security bulletins. In addition to addressing previous security issues, Exchange 2010 SP3 RU8 also corrects the security issue MS14-075. For Exchange 2010 this is also discussed in Outlook Web App Token Spoofing Vulnerability - CVE-2014-6319.
Update 12-12-2014: Exchange Server 2010 SP3 Update Rollup 8 has been re-released to the Microsoft download centre resolving a regression discovered in the initial release. The updated RU8 package corrects the issue which impacted users connecting to Exchange from Outlook. The issue was isolated to the MAPI RPC layer and was quickly remediated to deliver the updated RU8 package. The updated RU8 v2 package is version number 14.03.0224.002.
Update 10-12-2014: Please see comments at the end of this post with an issue relating to RPC Client Access. There is a TechNet forum where the community is discussing this issue. In the RPC Client Access the forum post notes that the following can be observed: “Log Watson: [IndexOutOfRangeException] Index was outside the bounds of the array”.
Update 10-12-2014: Exchange 2010 SP3 RU8 has been removed from the download centre until the above issue has been resolved.
This is build 14.03.0224.002 of Exchange 2010 (14.03.0224.001 was the initial SP3 RU8 build) , and KB2986475 has the full details for the release. The update file name is Exchange2010-KB2986475-x64-en.msp.
Note that this is only for the Service Pack 3 branch of Exchange 2010. Why? Exchange 2010 SP2 exited out of support on the 8th of April 2014and will no longer receive updates.
3004235 Exchange Server meetings in Russian time zones as well as names of time zones are incorrect after October 26, 2014
3009132 Hybrid mailbox moves to on-premises environment but finishes with CompletedWithWarnings status
3008999 IRM restrictions are applied to incorrectly formatted .docx, .pptx, or .xlsx files in an Exchange Server 2010 environment
3008370 Group members are not sorted by display name when HAB is used with OAB in Exchange Server 2010
3008308 Public folder database migration issue in a mixed Exchange Server environment
3007794 Hub Transport server cannot deliver messages when a database fails over to a cross-site DAG in Exchange Server 2010
3004521 An Exchange server loses its connection to domain controllers if a public folder server is down in Exchange Server 2010
2999016 Unreadable characters when you import ANSI .pst files of Russian language by using the New-MailboxImportRequest cmdlet
2995148 Changing distribution group takes a long time in an Exchange Server 2010 environment
2992692 Retention policy is not applied to Information Rights Management protected voice mail messages in Exchange Server 2010
2987982 Issues caused by ANSI mode in Exchange Server 2010
2987104 Email message is sent by using the "Send As" instead of "Send on Behalf" permission in Exchange Server 2010
2982017 Incorrect voice mail message duration in Exchange Server 2013 and Exchange Server 2010
2977279 You cannot disable journaling for protected voice mail in Exchange Server 2013 and Exchange Server 2010
Now, before we rush off to download and install this there are a couple of items to mention!
Test the update in your lab before installing in production. If in doubt test…
If the Exchange server does not have Internet connectivity then this introduces significant delay in building the Native images for the .Net assemblies as the server is unable to get to http://crl.microsoft.com. To resolve this issue, follow these steps:
On the Tools menu in Windows Internet Explorer, click Internet Options, and then click the Advanced tab.
In the Security section, click to clear the Check for publisher's certificate revocation check box, and then click OK.
We recommend that you clear this security option in Internet Explorer only if the computer is in a tightly controlled environment. When setup is complete, click to select the Check for publisher’s certificate revocation check box again.
Update Internet facing CAS servers first
Backup any OWA customisations as they will be removed
Test (yes technically this is in here for a second time but it is important!)
Exchange 2013 CU7 has been released to the Microsoft download centre! Exchange 2013 has a different servicing strategy than Exchange 2007/2010 and utilises Cumulative Updates (CUs) rather than the Rollup Updates (RU/UR) which were used previously. CUs are a complete installation of Exchange 2013 and can be used to install a fresh server or to update a previously installed one. Exchange 2013 SP1 was in effect CU4, and CU7 is the third post SP1 release. CU7 contains AD DS schema changes so please test and plan accordingly!
One aspect to note is that CU7 does contain the security fix for the issues described in security bulletin MS14-075. To address these security issues in pre CU7 builds of Exchange 2013, there are separate updates available. While not directly applicable to CU7, it is worth noting should you wish to implement the security fix prior to upgrading to CU7.
Update 10-12-2014: Corrected CU7 security update wording.
CU7 provides support for Public Folder Hierarchies in Exchange Server 2013 which contain 250,000 public folders. Yay!!!! CU7 also resolves backup and restore issues. The Exchange product group recommend upgrading to Exchange 2013 CU7 and then taking a full backup.
Please take the time to review these additional posts:
For those co-existing with Exchange 2007 there were some issues with CU6, that should all be resolved in CU7. For example if you are deploying into a mixed environment with Exchange 2007, you need to review KB2997209 Exchange Server 2013 databases unexpectedly fail over in a co-existence environment with Exchange Server 2007. Customer with Exchange 2007 and 2013 also had to review KB 2997847 You cannot route ActiveSync traffic to Exchange 2007 mailboxes after you upgrade to Exchange 2013 CU6. There were some issues with the post CU6 IU that was released to correct these issues as it did not always copy all the OWA files. CU6 also had an issue which affected Hybrid mailboxes.
This is build 15.00.1044.025 of Exchange 2013 and the update is helpfully named Exchange2013-x64-cu7.exe. Which is a great improvement over the initial CUs that all had the same file name! Details for the release are contained in KB2986485.
3008438 User who is trying to Log on to Exchange Admin Console is logged in to OWA instead
3006672 Move request fails if the IsExcludedFromProvisioning option is true in Exchange Server 2013
3005391 Exchange Server 2013 Cumulative Update 5 breaks free/busy lookup from Exchange Online to Exchange Server 2007
3001217 TLS 1.0 is hardcoded for SMTP traffic encryption in Exchange Server 2013
3003580 Event ID 4999 and 4401 when the Microsoft Exchange Replication service crashes in Exchange Server 2013
3012655 New-MailboxImportRequest causes unreadable characters when you import an ANSI format .pst file of Russian language
3012652 CalendarProcessing cmdlet does not generate delegate permissions to universal security groups in Exchange Server 2013
3009631 Advanced Find against the Sent Items folder in Outlook returns no result in Exchange Server 2013
3009612 Outlook Web App shows organization details on the contact card beyond the scope of user ABP in Exchange Server 2013
3009291 Shared mailbox cannot be opened in Outlook in an Exchange Server 2013 environment that has multiple domains
3008453 Cannot edit or delete forms from the organizational forms library in Exchange Server 2013
3003986 RejectMessageReasonText in transport rule appears in the user section of a DSN in Exchange Server 2013
3001037 Distribution group cannot send email messages to a mail enabled public folder in an Exchange Server 2013 environment
2999031 A cross-forest mailbox move from Exchange Server 2007 to Exchange Server 2013 finishes with CompletedWithWarnings status
2998144 New-MoveRequest cmdlet with RemoteLegacy parameter cannot perform a cross-forest mailbox move
2988553 Add-ADPermission and Remove-ADPermission can be run outside the management scope in Exchange Server 2013
2981538 Exchange Control Panel crashes when you proxy from Exchange 2013 to Exchange 2010
3014051 Cannot migrate mailboxes in a multiple domains environment in Exchange Server 2013
3012986 ContentIndexRetryQueueSize value for a passive node never drops to zero in Exchange Server 2013 Cumulative Update 6
3004011 Sound alerts do not work in Outlook Web App when new email or calendar notification is received in Exchange Server 2013
3003518 "550 5.7.1" NDR when you send messages to external recipients in an Exchange Server 2013 hybrid environment
3003068 Cannot see online archive mailbox after you upgrade to Exchange Server 2013 Cumulative Update 6
3000944 Subfolders under the Deleted Items folder are not visible in Outlook in an Exchange Server 2013 environment
2997847 You cannot route ActiveSync traffic to Exchange 2007 mailboxes after you upgrade to Exchange 2013 CU6
2997355 Exchange Online mailboxes cannot be managed by using EAC after you deploy Exchange Server 2013 CU6
2997209 Exchange Server 2013 databases unexpectedly fail over in a co-existence environment with Exchange Server 2007
2995263 OAB cannot be rebuilt if the .flt file is larger than two GB in Exchange Server 2013
2994216 PublicFolderMoveRequest deletes all read or unread state in target mailbox for each user in Exchange Server 2013
2993871 Resource Booking Assistant crashes after you upgrade to Exchange Server 2013 Cumulative Update 5
2983216 Category setting on an item in Outlook jumps the selection to the top of the list in an Exchange Server 2013 environment
2931223 MAPI virtual directory is missing from Default Web Site node
As with previous CUs, CU7 follows the new servicing paradigmthat was previously discussed on the blog. The CU7 package can be used to perform a new installation, or to upgrade an existing Exchange Server 2013 installation to CU7. You do not need to install Cumulative Update 1 or 2 for Exchange Server 2013 when you are installing CU. Cumulative Updates are well, cumulative. What else can I say…
After you install this cumulative update package, you cannot uninstall the cumulative update package to revert to an earlier version of Exchange 2013. If you uninstall this cumulative update package, Exchange 2013 is removed from the server.
Note that customised configuration files are overwritten on installation. Make sure you have any changes fully documented!
CU7 contains AD Schema updates – please test and plan accordingly!
What do I mean by that? Well, you need to ensure that you are fully informed about the caveats with the CU and are aware of all of the changes that it will make within your environment. Additionally you will need to test the CU your lab which is representative of your production environment.
This is the case of one of those bizarre and very annoying issues to tack down. The server in question is one of my lab servers where an automated build was used to create it. It is a Windows 2008 R2 SP1 based server which has TMG 2010 SP2 installed as a reverse proxy. Said server has been a little “squirrely”, and some “interesting things have happened where a restart would flush the gremlins out. It managed to soldier on and worked.
After the last patch Tuesday I could not get the server to scan for updates. There were a multitude of errors noted by the Windows Update client. The data below is from the WindowsUpdate.log and you can see just how varied the errors are. To keep one of my older friends happy, yes that is you Charles Of the Desert, let’s use findstr to parse the log looking for the specific phrase:
Findstr.exe /i /c:"Warning: exit code =" WindowsUpdate.log
Using Excel, the de-duplicated list of errors were:
WARNING: Exit code = 0x80244023 WARNING: Exit code = 0x80072EE2 WARNING: Exit code = 0x8024D011 WARNING: Exit code = 0x8024001B WARNING: Exit code = 0x800401FD WARNING: Exit code = 0x8007000E WARNING: Exit code = 0xC80003F3 WARNING: Exit code = 0x8024402F WARNING: Exit code = 0x80080005
WARNING: Exit code = 0x80244023 WARNING: Exit code = 0x80072EE2
The error codes are all over the place so there is not single issue in the WU client that explains all of them. Time to expand the net….
Looking at the event logs there were some items of note:
Interesting, but nothing there screamed at a root cause. Though why is Windows complaining about low resources, and then the Windows Modules Installer terminated due to insufficient resources?
The server in question is a VM with 3GB of RAM installed, and task manager shows plenty of available memory. Yes that is not a perfect check but it’s generally sufficient for a quick peek. So if there is enough physical memory installed, but did the automated build set page file to match the RAM?
<Borat>
Not so much….
</Borat>
For some reason, the automated build had set the server with a single static page file of only 256 MB. If this was a NT4 server and the year was 1999 then that would be good. In the year 2014, it was far less than awesome….
Increasing the page file to match physical RAM immediately corrected the issues and Windows Update then installed all updates!
Since 2014 is drawing to a close, I thought it would be interesting to see what were the 10 most popular articles on the blog. For reference purposes, the top 10 from 2013 were also included for comparison.
I also wanted to wish everyone a great holiday. If you are going skiing, staying home with family or slipping an extra shrimp on the barbie, take the time to enjoy it with your friends and family and see you in 2015!
For the year 2014 the following were the ten most popular articles on this blog:
And here is the top 10 for 2013:
Comparing 2014's with the 2013 top 10 posts we can see that there are certainly a few similarities, though there are also differences. Office 365 adoption has increased dramatically and this is reflected in what people are searching for. This is also reflected in what I am posting here too!
On-premises Exchange still commands the lions share of the traffic in 2014 with the Windows 2012 R2 & Exchange support post remaining rather popular. Autodiscover is always a contested area, and continues to bubble to the top. Exchange 2007/2010 version checking and calculating database whitespace still remain surprisingly active.
In addition to the core Exchange and Office 365 articles there are also posts which were never envisioned as highly desired, yet find themselves in the top 10. Downloading RDCMan and installing the Desktop Experience feature are prime examples.
Maybe next year the RDCMan 2.7 Download post will be in the top 10….