A Collection of Random Thoughts : Exchange

Move Mailbox from a CSV Import file

benw — Mon, 02 Jun 2008 16:53:00 GMT

A while back, I had promised to write another post detailing how to use Import-CSV and then run Move-Mailbox against the users in that CSV file. Well, quite a bit of time has passed since then, and getting this figured out proved a little more difficult than I thought it would be. I finally got it worked out, though.

Here are the details of the setup. You have users that you want to split up between several target databases. In my CSV file, I have 3 columns, as outlined below (sample file)

Name, Username, TargetDB

A user, auser, "Server\Storage Group\Mailbox Database"
B user, buser, "Server\Storage Group\Second Database"
C user, cuser, "Server\Storage Group\Third Database"

The TargetDB column means that you will have to do a little bit of legwork in determining where you want to put the users, and populate that column, but it shouldn't take too much effort.

Next, we'll start writing the script itself.

##################################################################
#                                                                                                                                                                #
#                                   Script to move mailboxes multithreaded from CSV import                                    #
#                                                                                                                                                                #
##################################################################

write-output "Moving mailboxes from CSV import"

#First we are going to import from your csv file
#where the CSV file contains 3 columns. Name, Username and TargetDB

$users = import-csv c:\userimport.csv

#Now we are going to filter the list down and get a
#list of just the users on the first database

$db1 = $users| where {$_.targetdb -like "*mailbox database"} |
foreach { $_.name } | get-mailbox

#getting a list of the users on the second target database

$db2 = $users | where {$_.targetdb -like "*Second Database"} |
foreach { $_.name } | get-mailbox

#Repeat the above line for each additional database that you have.

#Now we move each group of users to their respective databases

$db1 | move-mailbox -targetdatabase "E2K7-2\First Storage Group\Mailbox Database" -confirm:$false -maxthreads:10

$db2 | move-mailbox -targetdatabase "E2K7-2\Second Storage Group\Second Database" -confirm:$false -maxthreads:10

#Repeat the above line for each additional mailbox database that you have

write-output "Completed Mailbox moves from CSV file"

I first tried to get this working with just one command, but kept failing on the target database portion. When I was creating my arrays, I had a separate array for the target database, but the problem I kept running into was that the value for the array was only being recognized as the last value in the CSV file. So if Third Database was the last value for TargetDB, that was where all users ended up being moved. After multiple failed attempts to correct that, I decided that the best way to approach this would be to filter the CSV and move all users with a common target database, then move on to the next group. It still achieves the desired result of being able to take advantage of multi-threading, and it also does everything in one shot.

If you have any suggestions for improvement, please feel free to pass on your suggestions.

MOM Exchange Management Pack alerts

benw — Mon, 31 Mar 2008 17:12:40 GMT

Part of my job involves supporting the Exchange Management Pack for Microsoft Operations Manager (MOM). In doing so, we sometimes come across things that don't really make a lot of sense (or at least they don't to me). Take the following scenario.

With MOM, you can create Mailboxes that MOM then uses to make sure it can log on to at specified intervals. This is performed via a simple MAPI logon. The Exchange Management Pack has a script that performs this action, and it triggers by default every 15 minutes. When you deploy and configure the Exchange Management Pack, you can choose to create a mailbox for the entire server (per-server monitoring), or you can create a separate mailbox for every single mailbox store.

When the MAPI logon script runs, it logs an event based on the result. The expected result is obviously a Success, but if it fails, there are several different reasons this could be occurring. Many of these errors are easy to interpret, and have prescribed actions that can be done to troubleshoot/fix the issue.

One event that may get logged is a MAPI logon failure because the logon timed out. At first glance, that may not really make sense - if a logon is going to work, it is going to work. Keep in mind, however, that MOM uses scripts to perform the tests, and we could simply be dealing with the script taking a long time to run. This results in a sort of false-positive. The MAPI logon doesn't actually fail, and no clients are reporting issues, yet you have this glaring error in your MOM console.

The default timeout for the script itself is 30 seconds. While this is sufficient in most cases, there may be circumstances where 30 seconds is insufficient, and is resulting in the script timing out (even if it is almost done...), which then results in the error being logged to MOM. The solution is to modify the MAPI logon script itself and increase the timeout value. In the script, you will find a line

var TIME_OUT_THRESHOLD = 30000

If you change the threshold to a higher value - 90000, the errors will go away because you are giving the script more time to run.

Exchange 2007 Cluster Setup - Did it complete?

benw — Tue, 07 Aug 2007 06:30:00 GMT

As you no doubt know by now, Exchange 2007 introduces different types of clustering. It still retains the traditional Single Copy cluster where all data is stored in a central location (i.e. SAN). It also introduces the concept of Cluster Continuous Replication, or CCR, where centralized storage is not required, rather each server contains a separate copy of the data. This is achieved by an initial "seeding", and is then maintained by replication of log files. Service Pack 1 for Exchange 2007 includes ~~yet a third type of cluster~~ a new technology for managing server failures, called Standby Continuous Replication, or SCR, but we'll leave that for another post.

[edit] as was pointed out to me, SCR isn't actually a clustered solution, as it does not use Windows clustering. I've thus modified the original post to reflect that. However, it can be used as the target for an Exchange 2007 cluster. More details to come when I post about SCR.

For today, I've installed a new CCR cluster, but I didn't allow it to complete. Let's have a look at the setup log. This is located in the c:\ExchangeSetupLogs directory. The one that contains the most information is ExchangeSetup.log. Since this log file contains a lot of information, I'll only post part of it.

[8/3/2007 4:46:12 PM] [0] Starting Microsoft Exchange 2007 Setup
[8/3/2007 4:46:12 PM] [0] **********************************************
[8/3/2007 4:46:12 PM] [0] Setup version: 8.0.685.24.
[8/3/2007 4:46:12 PM] [0] Logged on user: E12CCR\Administrator.
[8/3/2007 4:46:12 PM] [0] Command Line Parameter Name='mode', Value='Install'.
[8/3/2007 4:46:12 PM] [0] Command Line Parameter Name='sourcedir', Value='D:\i386'.
[8/3/2007 4:46:12 PM] [0] Command Line Parameter Name='fromsetup', Value=''.
[8/3/2007 4:46:12 PM] [0] ExSetupUI was started with the following command: '-mode:install -sourcedir:D:\i386 /FromSetup'.
[8/3/2007 4:46:17 PM] [0] Setup is choosing the domain controller to use
[8/3/2007 4:46:18 PM] [0] Setup is choosing a local domain controller...
[8/3/2007 4:46:20 PM] [0] Setup has chosen the local domain controller e12dc.E12CCR.com for initial queries
[8/3/2007 4:46:21 PM] [0] PrepareAD has been run, and has replicated to this domain controller; so setup will use e12dc.E12CCR.com
[8/3/2007 4:46:21 PM] [0] Setup is choosing a global catalog...
[8/3/2007 4:46:21 PM] [0] Setup has chosen the global catalog server e12dc.E12CCR.com.
[8/3/2007 4:46:21 PM] [0] Setup will use the domain controller 'e12dc.E12CCR.com'.
[8/3/2007 4:46:21 PM] [0] Setup will use the global catalog 'e12dc.E12CCR.com'.
[8/3/2007 4:46:21 PM] [0] Exchange configuration container for the organization is 'CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=E12CCR,DC=com'.
[8/3/2007 4:46:21 PM] [0] Exchange organization container for the organization is 'CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=E12CCR,DC=com'.
[8/3/2007 4:46:21 PM] [0] This machine is part of a Windows failover cluster.
[8/3/2007 4:46:21 PM] [0] This server is not an active node for any Clustered Mailbox servers.
[8/3/2007 4:46:21 PM] [0] Setup will search for an Exchange Server object for the local machine with name 'E12CCR2-N1'.
[8/3/2007 4:46:21 PM] [0] No Exchange Server with identity 'E12CCR2-N1' was found.
[8/3/2007 4:46:36 PM] [0] The following roles are unpacked:
[8/3/2007 4:46:36 PM] [0] The following roles are installed:
[8/3/2007 4:46:36 PM] [0] The local server does not have any Exchange files installed.
[8/3/2007 4:46:36 PM] [0] Setup will use the path 'D:\i386' for installing Exchange.
[8/3/2007 4:46:36 PM] [0] The server is cluster type: 'None'.
[8/3/2007 4:46:36 PM] [0] The requested cluster type: 'None'.
[8/3/2007 4:46:36 PM] [0] The installation mode is set to: 'Install'.
[8/3/2007 4:46:36 PM] [0] An Exchange organization with name 'First Organization' was found in this forest.
[8/3/2007 4:46:36 PM] [0] Active Directory Initialization status : 'True'.
[8/3/2007 4:46:36 PM] [0] Schema Update Required Status : 'False'.
[8/3/2007 4:46:36 PM] [0] Organization Configuration Update Required Status : 'False'.
[8/3/2007 4:46:36 PM] [0] Domain Configuration Update Required Status : 'False'.
[8/3/2007 4:46:37 PM] [0] Applying default role selection state

This simply shows an overview of the setup choices that have been made.

[8/3/2007 4:52:39 PM] [1] Setup launched task 'test-setuphealth -DomainController 'e12dc.E12CCR.com' -DownloadConfigurationUpdates $false -ExchangeVersion '8.0.685.24' -Roles 'ClusterMailbox' -ScanType 'PrecheckInstall' -SetupRoles 'AdminTools','Mailbox','ClusterMailbox' -CmsName 'CCR-EVS2' -CmsDataPath 'E:\Exchsrvr\MDBData' -CmsIPAddress '10.10.201.230' -CmsSharedStorage $false -CreatePublicDB $false'

This is showing the pre-requisite check tests that are being performed, and shows that the setup roles that have been selected are AdminTools, Mailbox, ClusterMailbox. Most of the rest of the log file just shows the progress of each role, so I won't bore you with that. Let's get on to the interesting part. Once the mailbox role finishes installing, setup runs the new-ClusteredMailboxServer to create the actual cluster. One nice thing is that you don't have to worry about creating the Exchange group in Cluster Admin as you had to do with Exchange 2003. Setup now creates it for you automatically. Ok - here is the failure. Note the section in Bold.

[8/6/2007 5:36:17 PM] [1] Processing component 'Mailbox System Attendant Dependent Tasks' (Configuring tasks dependent on System Attendant service).
[8/6/2007 5:36:17 PM] [1] Executing 'start-ClusteredMailboxServer -Identity:$RoleName', handleError = False
[8/6/2007 5:36:17 PM] [2] Launching sub-task '$error.Clear(); start-ClusteredMailboxServer -Identity:$RoleName'.
[8/6/2007 5:36:18 PM] [2] Beginning processing.
[8/6/2007 5:36:18 PM] [2] Administrator Active Directory session settings are: View Entire Forest: 'True', Configuration Domain Controller: 'e12dc.E12CCR.com', Preferred Global Catalog: 'e12dc.E12CCR.com', Preferred Domain Controllers: '{ e12dc.E12CCR.com }'
[8/6/2007 5:36:18 PM] [2] Searching objects "ccr-evs2" of type "Server" under the root "$null".
[8/6/2007 5:36:18 PM] [2] Previous operation run on domain controller 'e12dc.E12CCR.com'.
[8/6/2007 5:36:18 PM] [2] Start-ClusteredMailboxServer is trying to start clustered mailbox server ccr-evs2.
[8/6/2007 5:37:05 PM] [2] Start-ClusteredMailboxServer finished starting (bringing online) clustered mailbox server ccr-evs2.
[8/6/2007 5:37:05 PM] [2] [ERROR] Unexpected Error
[8/6/2007 5:37:05 PM] [2] [ERROR] Clustered mailbox server 'ccr-evs2' is not in a started (online) state (Failed). The cluster resource 'First Storage Group/Mailbox Database (ccr-evs2)' is in state (Failed).
[8/6/2007 5:37:05 PM] [2] Ending processing.
[8/6/2007 5:37:05 PM] [1] The following 1 error(s) occurred during task execution:
[8/6/2007 5:37:05 PM] [1] 0. ErrorRecord: Clustered mailbox server 'ccr-evs2' is not in a started (online) state (Failed). The cluster resource 'First Storage Group/Mailbox Database (ccr-evs2)' is in state (Failed).
[8/6/2007 5:37:05 PM] [1] 0. ErrorRecord: Microsoft.Exchange.Management.Tasks.NewCmsNotOnline: Clustered mailbox server 'ccr-evs2' is not in a started (online) state (Failed). The cluster resource 'First Storage Group/Mailbox Database (ccr-evs2)' is in state (Failed).
[8/6/2007 5:37:05 PM] [1] [ERROR] Clustered mailbox server 'ccr-evs2' is not in a started (online) state (Failed). The cluster resource 'First Storage Group/Mailbox Database (ccr-evs2)' is in state (Failed).
[8/6/2007 5:37:05 PM] [1] Setup is halting task execution because of one or more errors in a critical task.
[8/6/2007 5:37:05 PM] [1] Finished executing component tasks.
[8/6/2007 5:37:05 PM] [1] Ending processing.
[8/6/2007 5:40:48 PM] [0] End of Setup

The Setup UI will now show that setup has completed with an error. Ok, so let's re-run setup to allow it to complete. What's that? The Setup UI only allows you to Add or Remove roles. Oh yeah - since this was set up as a Cluster, you can't have any other roles on it, and you can't de-select the Active Cluster role. Now at this point, I can actually go in to Cluster Admin, and verify that all the resources are online. In the above case where my First Storage Group/Mailbox Databases (ccr-evs2) was in a failed state, I simply brought it online, and it came online just fine. Since all the resources are online, you might think that everything is hunky dory. Not so, my friend. We've got a BIG problem. The System Attendant Object was never fully provisioned. If you leave your cluster running in the above state, you'll see lots of issues related to many tasks that the System Attendant Mailbox is used for. In my above setup, if we look at the System Attendant object using ADSIEdit, we will find that there are no proxy addresses (BAD BAD BAD), and msExchPoliciesIncluded is not set. This will cause massive problems with Free/Busy, and you'll see NDR's being generated for any messages sent to your System Attendant mailbox (duh - it doesn't have an e-mail address)

OK - so how do you fix it? Well, before we get there, there is one more thing you need to look at that will tell you if setup encountered any problems that need to be fixed. Open up the registry, and navigate to

HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\v8.0

and see which subkeys exist. You should see a separate key for Each role that is installed. For the above cluster, if setup had completed successfully, you would only see the following keys.

v8.0
   AdminTools
   MailboxRole
   Setup

In my case, however, I saw one more (bolded)

v8.0
   AdminTools
   ClusteredMailboxServer
   MailboxRole
   Setup

Viewing that key showed a Value with a name of "WaterMark" that contained a hex string. This Watermark is very important! It is how setup notes where it left off, and what still needs to be completed.

Ok - now on to the resolution. If you experience a failure during setup (clustered or not), the best way to resolve it in almost all cases is to run setup from the command prompt. The command-line version of setup is called from setup.com. For my cluster issue, since the Mailbox role completed, I don't have to actually run setup and select a role to install, rather I need to re-setup the cluster. The command I used was

setup.com /newcms /cmsname:ccr-evs2 /cmsipaddress:10.10.21.230 /cmsdatapath:e:\exchsrvr\mdbdata

When this command is run on an existing cluster with this same name, all setup will do is determine if there are any actions that have not been completed. In other words, it checks for a Watermark. In my case, it picked up at about 75% or so, and ran through to completion successfully. Inspecting the setup log after completion shows the following tasks (which I'd consider fairly important!)

[8/6/2007 5:51:55 PM] [1] Executing 'enable-SystemAttendantMailbox -Identity:$RoleFqdnOrName -DomainController $RoleDomainController', handleError = True
[8/6/2007 5:51:55 PM] [2] Launching sub-task '$error.Clear(); enable-SystemAttendantMailbox -Identity:$RoleFqdnOrName -DomainController $RoleDomainController'.
[8/6/2007 5:51:55 PM] [2] Beginning processing.
[8/6/2007 5:51:55 PM] [2] Searching objects "ccr-evs2" of type "ADSystemAttendantMailbox" under the root "First Organization".
[8/6/2007 5:51:55 PM] [2] Previous operation run on domain controller 'e12dc.E12CCR.com'.
[8/6/2007 5:51:55 PM] [2] Processing object "Microsoft System Attendant".
[8/6/2007 5:51:55 PM] [2] Applying RUS policy to the given recipient "Microsoft System Attendant" with the home domain controller "e12dc.E12CCR.com".
[8/6/2007 5:52:17 PM] [2] The RUS server that will apply policies on the specified recipient is "CCR-EVS2.E12CCR.com".
[8/6/2007 5:52:17 PM] [2] Saving object "Microsoft System Attendant" of type "ADSystemAttendantMailbox" and state "Changed".
[8/6/2007 5:52:17 PM] [2] Previous operation run on domain controller 'e12dc.E12CCR.com'.
[8/6/2007 5:52:17 PM] [2] Ending processing.

If you aren't installing a cluster, you may still see a Watermark value for one of the other roles (ClientAccess, HubTransport, Mailbox, UnifiedMessaging, etc.

If you have a watermark for one of those roles, you need to note which role has a watermark, then run the following

setup.com /roles:mb (or insert the name of the role that shows the watermark)

This will allow setup to process that role, and based on the watermark in the registry, it will continue where it left off. Once setup completes successfully, the watermark value will be removed.

I'm hoping and expecting that Service Pack 1 for Exchange 2007 will be able to handle issues like this one much better, as it should include a "Reinstall" option. If not, you can always fall back to the trusty command-line options.

Exchange 2003, ISA and attachment size

benw — Thu, 03 May 2007 21:33:00 GMT

I've seen a few of these reports recently and worked on one personally, so I thought I would mention this in case anyone else has run into this issue and wasn't sure how to fix it.

The environment:
ISA server 200x using a Web Publishing Rule to publish OWA on an Exchange 2003 server.

The problem:
When using OWA externally, files larger than 50kb cannot be attached. The symptom is that the attachment screen will simply go blank. If you wait long enough, you may eventually see a timeout error, but that could be a long time (30 minutes or more). Attaching the same file from inside the network works fine.

The apparent solution:
On the Exchange 2003 server, there is a setting within the IIS metabase that needs to be modified. Specifically, the parameter UploadReadAheadSize needs to be set.

Why does this need to be changed?

I don't know. I'm not an ISA expert, nor am I am IIS expert. The above-mentioned parameter is not set by default, and IIS will use the default schema value of 49152 bytes (48kb). It is worth noting that this parameter apparently does not *always* need to be modified. You should thus only modify this attribute if you are encountering this issue.

This parameter can be modified using adsutil.vbs, which is included by default with IIS, and exists in the inetpub\adminscripts directory. It can also be modified by editing the metabase.xml file in the \winnt\system32\inetsrv directory.

First, verify that no value is set.

C:\inetpub\adminscripts>adsutil get w3svc/UploadReadAheadSize

You should get back a response

The parameter "UploadReadAheadSize" is not set at this node.

To modify this parameter, run the following command

C:\inetpub\adminscripts>adsutil set w3svc/UploadReadAheadSize "<Size in bytes>"

For example, if you want to change this value to 20mb, you would put in "20000000" as the size.

The response should be

UploadReadAheadSize : (INTEGER) 20000000

Once you make this change, it should be effective immediately.