For a while now I’ve been using the Netsh SkipAsSource Flag to allow multiple IPs on a server and only the primary IP registers in DNS.  The previous series of articles in this series are here:

1. Fine Grained Control When Registering Multiple IP Addresses On a Network Card
2. Fine Grained Control When Registering Multiple IP Addresses–Part Deux
3. Fine Grained Control When Registering Multiple IP Addresses–Part Trois

Update 19-9-2013;  Tried this out on the RTM build of Windows Server 2012 R2.  Same issue.

Life was good until Shawn Martin added a comment asking about a hotfix for Windows Server 2012 to fix a previous issue from Windows Server 2008 R2.  Hotfix 2554859 was noted in the original article, but I had not experienced the behaviour on Server 2012 as I was not using the GUI.

This is an excerpt from KB 2554859 describing the issue:

• You install hotfix 2386184 on the computer to enable the skipassource flag of the netsh command.
• You assign many IP addresses to a network adapter on the computer by using the netsh command together with the skipassource flag.
• You update some IP settings for the network adapter in the Network and Sharing Center graphical user interface (GUI). For example, you edit the subnet mask of an IP address that has the skipassource flag set to true.
• In this scenario, the skipassource flag of the IP address and of all IP addresses that are listed under that address in the GUI are cleared incorrectly.

Unfortunately, this issue also manifests itself in Windows Server 2012, though the last bullet is slightly different. 

Let’s run through the behaviour you will see and then what to do about it

Starting Configuration

This is our starting configuration.  A simple setup with a single IPv4 address bound to one NIC. 

Get-NetAdapter | Get-NetIPAddress | Select IPAddress, SkipAsSource

Let’s use PowerShell’s New-NetIPAddress cmdlet to add an additional IP 192.168.10.121 and specify the –SkipAsSource parameter. 

New-NetIPAddress –IPAddress 192.168.10.121 –InterfaceAlias “Ethernet 2” –SkipAsSource $True

Looks OK, but we shall verify…

Get-NetAdapter | Get-NetIPAddress | Select IPAddress, SkipAsSource 

So far so good!  Time to change it up!

Enter the GUI

To replicate the issue, let’s now make a change to the IP we just added through the GUI, and change the subnet mask:

And when we check the SkipAsSource settings after saving the change in the GUI, we can see that the SkipAsSource flag has been lost for the IP address of 192.168.10.121

Interestingly enough if we add multiple additional IPs, each of which is set to SkipAsSource $True, and we only edit IP 192.168.10.121 in the GUI; only that specific IP loses it’s SkipAsSource flag. 

We can see this below, IPs 192.168.10.121 to .125 were added all of which have SkipAsSource set to $True. 

GUI is then used to edit just 192.168.10.121, and afterwards only that IP has lost its SkipAsSource flag.

Workaround

At this time to please do not edit the IP configuration of a machine where SkipAsSource is used via the GUI.  PowerShell can be used to configure IPs where this feature is used.  If the GUI is used the SkipAsSource will be set changed for the IP(s) that were modified.

This is an example of using Set-NetIPAddress to change the PrefixLength (Subnet Mask) to /24.  Note that even through SkipAsSource was not explicitly used in the Set-NetIPAddress cmdlet, the original setting was honoured

Cheers,

Rhoderick

>>>

Even though some of the very old KBs, which  now refer to unsupported products, state that taking databases offline to run periodic offline defragmentation with ESEUTIL is not recommended some folks in the field still want to do this.

Previously when there was only a single copy of a database, running offline defragmentation would cause minimal impact, apart from the time required to do the defragmentation process which could be several hours or longer depending on database size and disk throughput. This changes when we consider having multiple copies of a database in a Database Availability Group (DAG).

So you may be wondering how best to defragment Exchange 2010 databases that are in a DAG as people often look at the white space in a database and seek to immediately reclaim it.

In short, this is not a good idea for a couple of reasons:

• Defragmenting DAG databases leads to more work
• Mailboxes are offline while the defragmentation completes
• This is generally a short sighted view as white space will be re-used

Please note that we are discussing offline defragmentation via ESEUTIL /D, and not online maintenance routines that now run 24 * 7 in newer versions of Exchange and in online maintenance windows in previous versions.

Background

What happens when an Exchange database is defragmented using ESEUTIL /D?  The defragmentation process will copy out valid pages of ESE data from the old database file to a new database.  This process leaves white space behind as it does not contain data.  You will note that I specifically said new database.  This has a different GUID than the original database.  Creating a database with the same name, but different GUID, means that Exchange sees them as different databases not as multiple copies of the same database.

This will result in errors like the following since the databases are not copies of one another.  Errors that may be seen include, but are not limited to:

• An Active Manager operation failed. Error Operation failed with message: MapiExceptionJetErrorAttachedDatabaseMismatch: Unable to mount database. (hr=0x80004005, ec=-1216)
• The Exchange store database <databasename> copy on this server appears to be inconsistent with the active database copy or is corrupted. For more details about the failure, consult the Event log on the server
• Event ID 494:  Database recovery failed with error -1216 because it encountered references to a database, 'database path', which is no longer present
• Event ID 454: Information Store (PID) <databasename>: Database recovery/restore failed with unexpected error –1216
• Event ID 9519: The following error occurred while starting database <databasename>: 0xfffffb40. Failed to configure MDB.

Let’s look at an example of the impact caused by running offline defrag against a database that is replicated in a DAG.

Defragmenting Exchange 2010 DAG Database

We shall defragment database, DB01.  Our starting configuration has two copies of this database and all is currently running well.

So let’s dismount DB01, and then validate that the two mailbox servers have the same GUID for DB01.  We are using ESEUTIL /MH to dump out the header from the database.

On the first mailbox server we see the Rand of 2733649.  The GUID is displayed in the ‘DB Signature’ line and is the 'Rand’ value.  Be sure to look at the correct signature as there is a signature for both logs and databases.  It is expected that the Rand in these two lines will be different.

On the second mailbox server we see the same Rand of 2733649, you can see the server name in the title bar of the PowerShell window.

We have shown that the same database is present on both servers, i.e. both copies have the same Rand of 2733649.

Let’s now defragment DB01 on the first server, then see what happens……

Then let’s check the Rand to see if the old value of 2733649 is still present:

Nope, It’s not.  The Rand is now 143007541.  That shows that this is a different database.  Same name, but this is a different database.

Trying to activate the database copy on another server will create a sea of red in the application event log.  You will receive the errors listed above, and the most descriptive is Event ID 4807:

Recovering From Defragmenting DAG Database

At this point since the databases are no longer copies of one another we will have to re-seed the copy of the database.  Depending upon database size, disk throughput and network capacity this can take an extended period of time.  Let’s use PowerShell to re-seed the database copy:

Update-MailboxDatabaseCopy –DeleteExistingFiles –Identity DB01\Consea-MB2

This will have to be repeated for all database copies of the database in question.  If there are multiple copies over a WAN link then it would be a good idea to manually specify the seeding source using the –SourceServer switch.  That way one copy can be seeded over the WAN, and other copies can then use that as a  local source, thereby minimising WAN traffic and decreasing time.

Note that there are multiple options worth checking out with Update-MailboxDatabaseCopy.  They include options to explicitly choose a network, encryption and compression.  Chances are if you used Exchange 2010 RTM then you are quite adroit at using the –CatalogOnly switch!

When the seeding task completes, we can check that the database copies are OK

Checking the Rand on the updated copy of the database, we can see that it has been updated and now has the same Rand which was generated by the defrag, 143007541. 

Having to take a database offline for hours to defragment, and then manually reseeding all of its database copies is pretty painful.  Is there a better way to do this?

There certainly is!

A New Hope

Since Exchange 2010 introduced the online mailbox move feature, it is now pretty seamless to perform mailbox moves to a new mailbox database and when the old database is empty, simply delete it!  This process can be made even better with use of the SuspendWhenReadyToComplete parameter.  As an example:

New-MoveRequest -Identity 'User-21' -TargetDatabase DB01   –SuspendWhenReadyToComplete

This copies the vast majority of the mailbox content and then pauses.  The administrator will manually resume the move request using  Resume-MoveRequest.  So this means we can copy mailbox content through the day with no user impact.  After hours the suspended move can then be rapidly completed.  This has to be one of my favourite Exchange 2010/2013 features!

The same logic can also be applied to a mailbox database that must be evacuated for other reasons.  This may be necessary if file system AV has scanned the database as it will be in an unknown and thus unsupported state.

Note that the Mailbox Replication Service (MRS) is throttled, and if you wish to apply a little accelerando to the move process then you will need to take a look at the throttling configuration.

Cheers,

Rhoderick

>>>

From time to time we need to see how our Exchange databases are doing so that they are being managed proactively.  One aspect is tracking size and the utilization of each database.  At the simplest level we want to ensure that the database and transaction log LUNs do not run out of disk space.  That would be bad.

Sometimes we want to take a peek at the database, and check that the size of the database is within design expectations.  We can look at the amount of user data stored in the database, system usage for recoverable items in addition to looking at the size of the database on disk.  Some folks also want to see the blank space that is within a database.  This blank space is sometimes called:

• Available Mailbox Space
• White space
• Free database pages
• Free space

Historical Approach To Checking White Space

In ye olden days when Exchange admins went to work on horses (alright, iron horses), we would look for the venerable application event log entry 1221.  This event would tell us the amount of white space within the database.  This event was introduced in Exchange 5.5 SP1 back in August 1998.  That’s more than 15 years ago, eeek!!   

In those simple Exchange days we would have one Public Folder (pub.edb) and one Mailbox database (priv.edb) per server.  Standard Edition could have a database of up to 16GB, and Enterprise had the “unlimited” database.  This was before the advent of the .stm file so all content lived in the .edb file.  Event ID 1221 would then report on the white space contained within the database file.  Life was indeed simple and good! Additionally Exchange 5.5 SP1 also added a second enhancement for determining white space which was the ESEUTIL /MS switch to dump out the space consumed by tables in the database.  More on this later….

As mentioned in the KB, the free space that is reported by Event 1221 is a conservative estimate. If you perform offline defragmentation, you will recover at least the amount of space that is reported as free. All space in an Exchange database is owned either by the database root or by particular tables in the database. Event 1221 estimates free space by calculating the number of empty pages owned by the messages table, the attachments table, and the database root. Free pages that are owned by other tables in the database are not taken into account.

Things advanced, the dotcom bubble popped and with the advent of Exchange 2000 the streaming file was introduced.  The intent was to store native RFC content in the streaming file as the Internet was the future and content would be converted between the .stm and .edb as needed.  Event 1221 does not review this shiny streaming thingymabob, and did not report what free space may have been available within the .stm file.  For a trip down memory lane there is an excellent read in this document Determining the True Amount of Space in an Exchange Database

If ESEUTIL /D was executed against a database the .stm would have been defragmented in addition to the .edb by default.  Though this could be changed by specifying the /I switch.     Details on defragmenting databases are can be found in this KB.

Exchange 2007 dropped the .stm file and just like Atomic Kitten, the .edb became whole again.  Event 1221 was still with us and reporting on the database white space!

Then along came Exchange 2010, sans Event ID 1221….

Getting White Space In Exchange 2010

Exchange 2010 introduced numerous improvements to the Mailbox role.  These improvements included things like

• Enhanced ESE physical store to improve performance
• Larger 32 Kb ESE page size
• Dumpster 2.0
• No database level attachment table
• Updated Online Maintenance routines

As part of the changes to the mailbox role, the venerable event ID 1221 was removed, and output was added to Get-MailboxDatabase called AvailableNewMailboxSpace.  This data is returned when the –Status parameter is also specified to execute the more expensive work items.  We can see the difference below, note the second command has –Status added:

So NewAvailableMailboxSpace looks good?  Well not so much.  This parameter only looks at the root portion of the database.  This is clearly stated in the Exchange team blog post.  For reference the relevant text is shown below.

Remember we would come back to ESEUTIL /MS? 

We need to use ESEUTIL /MS to get an accurate picture of white space in an Exchange 2010 database. 

Running ESEUTIL /MS against a lab database shows the following:

From this we can see the breakdown in the database structure and where space is being consumed.  Note that this can only be executed against a dismounted database, and you will receive the following error if running against an active mailbox:

Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use)

And if you are thinking that I’ll be sneaky and run it on a passive database copy in a DAG, then you will receive this error instead, after stopping the replication service else you will get the same error as listed above:

Operation terminated with error -550 (JET_errDatabaseDirtyShutdown, Database was not shutdown cleanly. Recovery must first be run to properly complete database operations for the previous shutdown.)

Conclusion

To get an accurate representation of the amount of white space in an Exchange 2010 database we need to use ESEUTIL /MS.    Note that the database must be dismounted, else you will receive JET errors and no data will be returned. 

Dumping out the table information is a fairly quick operation, unless the verbose /V option is used and then the time taken will greatly increase. 

How often do I expect this to be used?  Rarely and typically as part of troubleshooting.

The days of online maintenance killing itself trying to defragment partially filled 4K pages of data to be on a single ESE page are gone.  The same goes for offline defragmentation.  We now live in a world where large mailboxes are the norm and any white space will be quickly re-used.  Also consider that many organisations are now running 24 * 7, and it is hard to justify the impact caused by taking a database offline. 

Defragmenting databases to reclaim whitespace should be a rare event nowadays.  Exchange 2010/2013 are designed to use larger and cheaper storage which means you can get more storage capacity for the same price point.  Rather than defragmenting databases we should look to leverage the online mailbox move experience in Exchange 2010/2013 and simply move mailboxes to a new database and then discard the original one.  This is critical when we are discussing DAG replicated databases. 

Cheers,

Rhoderick

>

Update 14-8-2013: The security update for Exchange 2013 has been temporarily removed due to an issue with the search infrastructure.

If you have installed it please refer to article “Update 2874216 breaks the content index in Exchange Server 2013”

Update 14-8-2013: Please see an update on the Exchange team blog.

Update 27-8-2013: Updated release of the security update for Exchange 2013 CU1 and CU2 is now available.

The first security update for Exchange 2013 was release to the download centre earlier today to address the security issues that are described in Microsoft Security Bulletin MS013-061.  Exchange 2007 SP3, Exchange 2010 SP2 and Exchange 2010 SP3 also received updates today to resolve the security issues described in the bulletin.  While all supported Exchange 2007 and 2010 builds were updated to resolve CVE-2013-2393 and CVE-2013-3776, there is an additional security vulnerability in Exchange 2013.  This is covered in CVE-2013-3781 and discussed in the Oracle Critical Patch Update Advisory - July 2013.    Microsoft has classified all three of these issues as critical for Exchange 2013.

Please note that there are separate updates available for Exchange 2013 RTM CU1 and CU2.  They have the same file names, so make sure to save to properly named folders. 

Security Update for Exchange Server 2013 RTM CU1

Security Update for Exchange Server 2013 RTM CU2

This is the first Exchange 2013 update that resolves only security issues.  As previously mentioned with the new servicing model security updates will be release separately from the other Cumulative Updates servicing updates. 

In order to install this security update, you must have installed the updated build of Exchange 2013 RTM CU2 which is described here.  This is the 712.24 build of Exchange.

Please test before installing this update into production!  This is not a surprise, as every update to the messaging infrastructure should be carefully tested and reviewed prior to installing in production. 

Cheers,

Rhoderick

.

Exchange 2010 SP3 RU2 has been released to resolve the security issues described in Microsoft Security Bulletin MS13-061.  Additionally Exchange 2007, Exchange 2010 SP2 and Exchange 2013 also received updates to resolve the security issues as mentioned in the MS13-061 bulletin and other product specific fixes. 

Click the above image to download, or the direct download link is here. 

Exchange 2010 SP3 RU2 is also released as part of Patch Tuesday to resolve the security issues that are addressed in CVE-2013-2393 and CVE-2013-3776. 

They are also discussed in the Oracle Critical Patch Update Advisory - July 2013.    Microsoft has classified both of them as critical for Exchange 2010. 

This is build 14.03.0158.001 of Exchange 2010, and KB2866475 has the full details for the release. 

Some updates of particular note:

• 2863473 Users cannot access Outlook mailboxes that connect to a Client Access server array in an Exchange Server 2010 environment
• 2841150 Cannot change a distribution group that contains more than 1,800 members by using ECP in OWA in an Exchange Server 2010 environment

And it also resolves two transport crash issues that were introduced in Exchange 2010 SP3 RU1

1. 2870028 EdgeTransport.exe crashes when an email message without a sender address is sent to an Exchange Server 2010 Hub Transport server

2. 2871758 EdgeTransport.exe process consumes excessive CPU resources on an Exchange Server 2010 Edge Transport server

Note that you may be prompted for the Exchange 2010 SP3 installation files when installing Exchange 2010 SP3 RU2.  Specifically it prompts for "exchangeserver.msi.      This does not happen for all installations.  If you are prompted, simply point to the SP3 installation files.   

Issues Resolved

• 2837926 Error message when you try to activate a passive copy of an Exchange Server 2010 SP3 database: "File check failed"

• 2841150 Cannot change a distribution group that contains more than 1,800 members by using ECP in OWA in an Exchange Server 2010 environment

• 2851419 Slow performance in some databases after Exchange Server 2010 is running continuously for at least 23 days

• 2853899 Only the first page of an S/MIME signed or encrypted message is printed by using OWA in an Exchange Server 2010 environment

• 2854564 Messaging Records Management 2.0 policy can't be applied in an Exchange Server 2010 environment

• 2855083 Public Folder contents are not replicated successfully from Exchange Server 2003 or Exchange Server 2007 to Exchange Server 2010

• 2859596 Event ID 4999 when you use a disclaimer transport rule in an environment that has Update Rollup 1 for Exchange Server 2010 SP3 installed

• 2860037 iOS devices cannot synchronize mailboxes in an Exchange Server 2010 environment

• 2861118 W3wp.exe process for the MSExchangeSyncAppPool application pool crashes in an Exchange Server 2010 SP2 or SP3 environment

• 2863310 You cannot send an RTF email message that contains an embedded picture to an external recipient in an Exchange Server 2010 SP3 environment

• 2863473 Users cannot access Outlook mailboxes that connect to a Client Access server array in an Exchange Server 2010 environment

• 2866913 Outlook prompts to send a response to an additional update even though the response request is disabled in an Exchange Server 2010 environment

• 2870028 EdgeTransport.exe crashes when an email message without a sender address is sent to an Exchange Server 2010 Hub Transport server

• 2871758 EdgeTransport.exe process consumes excessive CPU resources on an Exchange Server 2010 Edge Transport server

• 2873477 All messages are stamped by MRM if a deletion tag in a retention policy is configured in an Exchange Server 2010 environment

Important Notes

Now, before we rush off to download and install this there are a couple of items to mention!

• Test the update in your lab before installing in production.  If in doubt test…
• If the Exchange server does not have Internet connectivity then this introduces significant delay in building the Native images for the .Net assemblies as the server is unable to get to http://crl.microsoft.com.  To resolve this issue, follow these steps:
  1. On the Tools menu in Windows Internet Explorer, click Internet Options, and then click the Advanced tab.
  2. In the Security section, click to clear the Check for publisher's certificate revocation check box, and then click OK.
  We recommend that you clear this security option in Internet Explorer only if the computer is in a tightly controlled environment. When setup is complete, click to select the Check for publisher’s certificate revocation check box again.
• Update Internet facing CAS servers first
• Backup any OWA customisations as they will be removed
• Test (yes technically this is in here for a second time but it is important!)

Note that you may be prompted for the Exchange 2010 SP3 installation files when installing Exchange 2010 SP3 RU2.  Specifically it prompts for "exchangeserver.msi.      This does not happen for all installations.  If you are prompted, simply point to the SP3 installation files

Cheers,

Rhoderick

>