Richard's Exchange Ramblings

Meetings Disappearing From Exchange 2003 Resource/Room Mailboxes

Richard Roddy [MSFT] — Mon, 08 Aug 2011 22:11:37 GMT

In this case, I was presented with the symptom of meetings disappearing from a Resource mailbox in Exchange 2003 that had been set up for automatic booking with the Exchange 2003 Auto Accept Agent.

After going round and round with the customer to try and get good tracing information from the server, we finally got a good store trace of the behavior.

What was found was that there was a Delegate defined on the Resource Mailbox. Mixing delegates with the Auto Accept Agent can potentially end up causing issues if we're not sure to take precautions. This was the case here.

Note this excerpt from the Auto Accept Agent Deployment and Administration Guide, Managing Resource Mailboxes topic (http://technet.microsoft.com/en-us/library/bb124314(EXCHG.65).aspx):

Modifying Delegate Settings:

You can add delegates to a resource mailbox who can then access the mailbox directly to check existing appointments, check mail in the Inbox, or processed requests in the Sent Items folder. However, if you allow delegates to receive copies of meeting requests sent to a resource mailbox, it is possible that the delegate may accept a meeting request even though the agent has declined the meeting. To avoid this situation, it is recommended that you do not allow delegates to receive copies of meeting requests. You must open the resource mailbox as the primary mailbox to modify delegate settings. You cannot modify delegate settings if the resource mailbox is a secondary mailbox.

To prevent delegates from receiving copies of meeting requests:
1. In Outlook 2003, click Tools and then click Options.
2. On the Delegates tab, select a delegate, and then click Permissions.
3. Clear the check box for Delegate receives copies of meeting-related messages sent to me.

What I saw in our store traces, is that for the problems, the delegates were causing modifications to the calendar. What was likely happening is that the delegate was deleting the copy of the meeting invitation they received, and this in turn was deleting the meeting from the calendar.

We then found that the resource mailbox was indeed set to forward the meeting request copies to the delegates and turned that option off.

After this there was a thought by one user that a meeting disappeared, but additional logging and investigation showed that meeting to have been one that was removed prior to the change in the setting.

Problems accessing Free/Busy information and logging into OWA in Exchange 2007

Richard Roddy [MSFT] — Tue, 02 Aug 2011 23:31:29 GMT

In this case, the customer had an extended Exchange 2003 outage, and so they brought some important users up with new Exchange 2007 mailboxes. However, some of these users were encountering problems with people accessing their Free/Busy (availability) information. Further investigation also found problems with logging into OWA ‘Premium’ for the users.

The Free/Busy issue showed with hashes in the availability, and when we hovered over the information in Outlook, we could see that it was giving us something to the effect of ‘the mailbox could not be contacted’.

The OWA logon issue exhibited with the error "The item that you attempted to access appears to be corrupted and cannot be accessed."

Searching for information on this OWA error, I found some other cases where the solution was that the Reminders information in the mailbox had become corrupt. In those cases, running Set-CASMailbox <user> -OWARemindersAndNotificationsEnabled:$False would stop the issue, but in our case, OWA still showed the problem, so it did not appear to be the reminders issue.

We then looked further at the OWA exception received and saw a reference to userconfigurationtype.

Digging around with MFCMAPI, we finally found that the problems with both availability and OWA being encountered were due to corruption in a couple of options that are stored in the Associated Contents table of the Calendar folder in the user's mailbox.

We used the following process to fix them:

Create a profile for that user
Launch MFCMAPI (downloaded from http://mfcmapi.codeplex.com)
Click Session, Logon and Display Store Table
Choose the user profile
Double click on the user's mailbox from the table
Expand the Root Container, and then Top of Information Store
Right click on the Calendar folder and choose Open Associated Contents Table
We figured out from the OWA exception information that the problems were the IPM.Configuration.CategoryList and IPM.Configuration.WorkHours items, so we deleted them both.

Once those items were deleted, we could access everything in OWA again and our availability information showed up for the users.

Running ESEUTIL against a database from another server on a different OS version

Richard Roddy [MSFT] — Tue, 02 Aug 2011 23:11:51 GMT

In this case, the customer was running snapshot backups against their Exchange Server 2007 running on Windows Server 2008 (SP2). When trying to run a integrity check (eseutil /g) against the snapshot, eseutil was always reporting that the database was corrupt.

A quick check of servers found that the machine we were running the eseutil /g on was running Windows Server 2008 R2. While the ‘uneducated’ might think that Windows Server 2008 (with SP2) and Windows Server 2008 R2 would basically be the same OS version, the truth is much different; contrary to the naming (I still don’t know why someone made the “brilliant” decision in naming Windows Server 2008 R2), they are very different OS’es (Server 2008 is ‘Vista Server’ and 2008 R2 is ‘Windows 7 Server’).

Anyway, the issue here is that when we cross OS versions, the secondary indexes of the database have to be rebuilt to be compatible with the new OS. This is something that begins upon mounting the database on the new server with the higher level OS. Moving up in OS level (2003 to 2008 or R2, 2008 to R2) is supported, but moving back is not. What it looks like we're seeing here is that ESEUTIL is reporting corruption due to changing OS versions and the database needing the secondary indexes rebuilt.

To specifically test this, I built an Exchange 2007 SP3 server on Windows 2008 R2. I then took a database from one of my Exchange 2007 SP2/Windows Server 2008 SP2 servers and copied it to the SP3/R2 server. I attempted an eseutil /g and got back the -1206 error.

Then I mounted the database on the Exchange 2007 SP3/Win 2008 R2 server, dismounted and ran eseutil /g successfully.

Out of curiosity, I decided to test if I would see the same behavior if I took an Exchange 2007 database from a server running Windows Server 2003 (SP2) and ran eseutil /g against it on a Windows Server 2008 SP2 machine. I got the same -1206 error.

So, the conclusion here is that you cannot run eseutil /g without error on a different OS platform against a database, unless that database has been mounted on that OS to trigger the deletion/update of the secondary indexes first.

This information can be found in the Exchange 2007 Database Portability documentation:
http://technet.microsoft.com/en-us/library/bb123954(EXCHG.80).aspx

Database Portability Across Windows Server Versions:

As with previous versions of Microsoft Exchange, an upgrade of the operating system for an Exchange server results in the updating of the value for OS Version in the database header. This update triggers the rebuilding of internal database indexes.

When using database portability to move a database from a Mailbox server running Windows Server 2003 to a Mailbox server running Windows Server 2008, the Extensible Storage Engine (ESE) detects the operating system upgrade and takes the following actions:
• During the first database mount operation, all secondary indexes are discarded. A secondary index is used to provide a specific view of the mailbox data (for example, when messages in a mail folder are sorted using Outlook in Online mode). The database will not be mounted and available to clients until this initial operation is complete. The amount of time to complete the operation is largely dependent on the size of the database. The larger the database is, the longer the mount operation will take.

The customer then asked if there is a way to determine the OS version a database came from, and indeed we can. All we need to do is run eseutil /mh to dump the database header; in the output is an OS version line that shows the OS it was last mounted on.

Exchange 2007 - Legacy Free/Busy Info not Publishing to Public Folders

Richard Roddy [MSFT] — Thu, 03 Mar 2011 15:36:30 GMT

Problem:
=====================================

Free/Busy information was not being published to Public Folders for room/resource mailboxes.

This was causing the Free/Busy information for the rooms, as seen by Outlook 2003 clients, to not be accurate, causing users to send meeting requests for times they though the room was free, and would then be returned denied because the room was actually already booked. The issue was affecting only one mailbox server. Outlook 2007 or higher clients, or users accessing via OWA saw correct Free/Busy information.

Summary of Events:
======================================

When I saw the problem, I began troubleshooting the process by which "Legacy" Free/Busy is published from a mailbox when a meeting is scheduled by a non-Outlook client. This process applies to meetings scheduled through the AutoAccept functionality of Room mailboxes, as well as meetings scheduled through OWA by regular users. This Legacy Free/Busy information is only used by Outlook 2003 or earlier clients, or for mailboxes on Exchange 2003 (or earlier). This is why the problem was limited to Outlook 2003 clients viewing the information.

The process by which the Legacy F/B is published breaks down as follows:
- Meeting is scheduled in the mailbox by non-Outlook client (OWA or AutoAccept)
- F/B information is then sent to the System Attendant (SA) mailbox
- The F/B publishing process (MSExchangeFBPublish) that runs in the System Attendant regularly checks the SA for new F/B messages on a regular basis (roughly 5 minutes) and then publishes the information to the F/B public folders

Most of the time, it is the actual publishing process that breaks, and the F/B messages are found sitting in the SA mailbox.

So, the first thing I starting looking at was if we had messages sitting in the SA mailbox. We did not. So the troubleshooting concentrated there with the question, "why aren't the messages getting to the SA mailbox?"

Because there is very little tracing or logging around the process of the F/B message being sent to the SA mailbox, and how it is sent is not well documented, we needed to get debug tracing analyzed to see where the process was breaking down.

Once the tracing was captured, it was analyzed here, and it was found that when the meeting was being scheduled in the room mailbox, the F/B message was indeed being submitted for the SA mailbox. One piece of information that was revealed in looking at the code responsible for this, was that we do indeed use standard message transport to deliver the message. The message is sent to transport, initially with the LegacyExchangeDN value of the SA mailbox, which is then resolved by transport for delivery. This is done by the Hub Transport server(s).

Once this was known, we now knew to look in transport for what happened to the message. When we went to use the Message Tracking center to track messages bound for the SA mailbox, putting the LegacyExchangeDN value for it in the Recipient box and clicking "Resolve Recipient" caused the box to blank out. What normally occurs is that the recipient box is filled in with the SMTP address of the System Attendant.

We then used ADSIEdit to check the properties of the System Attendant for the problem server, and found that the proxyAddresses attribute on the System Attendant was blank.

We then used ADSIEdit to populate this attribute with a valid SMTP address. Once we did this, we found that the System Attendant mailbox was receiving messages and we also saw that the Free/Busy information started appearing for some of the test meetings we had previously created.

Then then scheduled a meeting for a large batch of the rooms, and after 5-10 minutes they were all displaying the Free/Busy information for the meeting created.

Findings / Root Cause:
==========================

The problem here was that the System Attendant was missing values for the proxyAddresses attribute, thus the Free/Busy messages could not be delivered to it.

This would not have just "happened"; there must have been something done with this server at a previous time that caused the addresses to be cleared from the System Attendant object. When Exchange is installed, the default SMTP address is set in the proxyAddresses attribute as SMTP:SERVERNAME-SA@domain.com , where SERVERNAME is the server name and domain.com is the default SMTP domain.

Also, the DisplayName of the System Attendant mailbox is normally "Microsoft System Attendant". For this server, it was "System Attendant". This provides additional suspicion that something previously occurred with the System Attendant mailbox and when it was "fixed", it was not properly fixed, and we ended up with missing proxyAddresses.

More Information:
============================

In some additional research I did after knowing what our root cause was, I went back to see if there was something that would have alerted us to the issue sooner, and found that indeed there was. We could have looked at the output of an Exchange Best Practices Analyzer report run against the server, and it would have alerted to the System Attendant missing the proxyAddresses attribute values.

However, prior to the further research regarding the publishing process, we likely would not have immediately made the connection between the missing value and the F/B publishing failure. So, we would have followed up on the ExBPA output and fixed the SA proxyAddresses attribute, and then found out that ended up fixing our problem.

Exchange 2010 (SP1) problems exporting LARGE mailbox to PST

Richard Roddy [MSFT] — Wed, 16 Feb 2011 20:45:59 GMT

In this case, the customer was having to export mailboxes to PST regularly. With a couple of large mailboxes (both over 5 GB+) the exports were failing.

When attempting to export larger mailboxes to PST using New-MailboxExportRequest, the process would appear to hang and then later fail. The problem was occurring on an intermittent basis, but with two very large mailboxes (5 GB+), the process was having trouble completing at all.

The error code being seen was 0x80040115, which translates to MAPI_E_NETWORK_ERROR. When we see this error, it normally indicates a problem with connection to one of the stores involved in the process.

In this case, the suspicion was that the connection to the PST was the issue, since the PST was being accessed across the network. The PST was being put in a share on the Windows 7 workstation on which the Exchange Management Shell (EMS) was being used to run the command.

The main question that needed to be answered in the process was, "What do we need to do for the PST to be on the same machine that the export process is running on so that it does not need to go over the network?"

To answer that, we needed to look at how the export process runs:

1) In Exchange 2010, your PowerShell operations are remote and are always being executed on an Exchange server; they are not executed on the machine running the Shell, unless you are on the Exchange Server and the shell has connected to the machine you are on. So, even if you are running the EMS on a workstation, it actually connects to one of your Exchange servers, and all commands are executed on that server.

2) The MailboxExportRequest uses the Mailbox Replication Service (MRS), just like a Mailbox Move does. This means that any CAS server running the MRS service, which by default is all CAS servers, could be the CAS server to run the export. We can specify a specific CAS to use with the -mrsserver switch in our New-MailboxExportRequest command.

So, we did 2 things:

1) We created a folder/share on CAS server 1, and gave the Exchange Trusted Subsystem Read/Write access to the share as directed in http://technet.microsoft.com/en-us/library/ff607299.aspx.

2) In our command we specified the new share as our file path, and also specified CAS 1 in the -MRSServer setting in our New-MailboxExportRequest command (i.e. New-MailboxExportRequest -Mailbox user1 -FilePath “\\CAS1\PSTFileShare\user1.pst” -MRSServer CAS1.domain.com).

The Mailbox export kicked off and we saw a much higher transfer rate than the customer had seen with any prior exports, and we expected to have a much better chance of success, since the PST was now on the same machine as the MRS and was not going across the network.

The customer later informed me that the export of both very large mailboxes was successful, and was done within hours, whereas it had previously run for more than a day before finally failing.

Exchange 2010 Management Tools are very slow to open and respond

Richard Roddy [MSFT] — Wed, 16 Feb 2011 20:10:42 GMT

When this case came to me, the situation was that the Exchange Management tools (both the Shell and Console) were very slow to open.

Given that most of what we are doing when the tools first open is talking to Active Directory, I checked to see if other AD tools, like Active Directory Users & Computers, were also slow. In this case, we found that when any of the tools went against certain Domain Controllers, we did indeed see major slowdowns.

We gathered Network Monitor traces during the opening of the various tools, and the notable thing that we saw was that there were 5 second delays between many of our LDAP Requests and the corresponding responses from the Domain Controllers.

After much troubleshooting by our Directory Services team, including debug tracing of the AD processes, etc., that showed that AD performance was just fine, it was finally found that the problem was due to the EnableTCPA setting under HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters. The value was set to 1, enabling the feature, while the other Scalable Networking Pack (SNP) features (EnableTCPChimney, EnableRSS) were disabled.

According to the Windows Networking team this combination can cause the TCP driver on that machine to think that the sender has reduced its sending capacity. The TCP driver then begins to perform regular jobs in response to the low sending capacity, rather than just immediately responding to the requests. This behavior causes the slow response/pauses that we could see in the network traces. However, the fact that the TCP driver is waiting to send the outgoing packets is something that cannot be seen.

The solution: disable the feature by setting EnableTCPA to 0.

“This email address already exists in the Organization”

Richard Roddy [MSFT] — Thu, 13 Jan 2011 21:20:14 GMT

This was a case where whenever we would try to give a specific Email Address to an existing user using "Exchange tasks, Establish Email Address" , it errors out with the below message :

"This email address already exists in this Organization"
ID no: c10312e7

Article 280765 is the solution:

Error message: This e-mail address already exists in this organization
http://support.microsoft.com/default.aspx?scid=kb;EN-US;280765

Method 1 did not find it, so next was to use a Network Monitor capture.

We used the following instructions to disable LDAP Encryption on the server:
http://blogs.technet.com/b/benw/archive/2008/12/15/disabling-ldap-encryption-and-signing-for-netmon-on-an-exchange-server.aspx

We then got our capture, but the question was how to find the string in the capture?

NMSimpleSearch to the rescue! (http://code.msdn.microsoft.com/NmSimpleSearch)

I loaded this expert into Network Monitor, and then used it (from the Experts menu) to search for the email address we were trying to define, and found the search request for the address and then what object was returned.

From these, I could see both the object returned and could see the IP address of the GC we were finding it on.

We used ADSIEdit to connect to the GC port (3268) of the DC shown by the IP address in the capture, which was a child domain DC/GC in the same site as the Exchange server, and we found the object causing the problem, which was a Contact in the same OU as the user we were trying to enable.

This contact did not exist on the DC from the domain the object was from.

So, that meant our problem was replication to this specific GC.

We decided the easiest way to resolve the problem was to remove the GC role from the problem DC, then put it back.

We removed GC role, waited for events in the Directory Services event log indicating it was removed, then added the GC role back. We then verified with ADSIEdit the bogus contact object was gone, and then we successfully mail-enabled the user. using the desired email address

Exchange 2010 SP1, Outlook 2010 Cached Mode, and Retention Policies - Why do my items show expired in Outlook, but are not being removed by Exchange?

Richard Roddy [MSFT] — Tue, 14 Dec 2010 15:38:56 GMT

The symptoms of this issue were pretty simple: The customer had applied a Retention Policy to the Deleted Items folder to delete anything older than 14 days, and when looking at the items in the folder using Outlook, everything over 14 days old showed as expired, but the items weren’t getting removed by Exchange.

After quite a bit of troubleshooting around whether Exchange was running it’s retention process properly, I finally found the answer, which was that Outlook’s view of the items’ expiration, and Exchange’s actually expiration of the items, did not match.

Outlook 2010, which is where we were looking at the items, was running in cached mode.

We then looked at some items in OWA, and then looked at them with Outlook in online mode, and there we saw that the old items that were showing as expired with Outlook in cached mode, were showing that they would expire in 4 days. I then asked the customer when they first applied the policy, and it was early the previous week - 10 days prior.

So, what is occurring is that Outlook in cached mode pulls the Policy Tag information for the folder and then does the calculation for expiration itself. If we're in the initial 'x' number of days after the policy was originally applied to the items (they were just moved into the folder, etc.), we may have items that Outlook in cached mode calculates as expired, however, since they were just tagged within the expiration period, they are not expired or deleted on the server.

I did indeed find documentation that says that in cached mode, Outlook calculates the expiration, where in Online mode, it is read from the server.

The property involved is the RetentionExpirationDate property (http://msdn.microsoft.com/en-us/library/microsoft.office.interop.outlook._mailitem.retentionexpirationdate.aspx).

The conclusion here is that if you’ve applied Retention Policies to any of your folders and the items are being shown by Outlook as expired, but are still there days later, take a quick look at them in OWA or an online mode Outlook profile and see if they really are expired.

Exceptions Received After Running Successful Modification Operations with Exchange 2010 SP1

Richard Roddy [MSFT] — Mon, 13 Dec 2010 18:14:51 GMT

In this case, what we were running into, is that when commands are run that make modifications to objects, either in the Exchange Management Console (EMC) or Exchange Management Shell (EMS), an exception is returned:

Error : The cmdlet extension agent with the index 0 has thrown an exception in OnComplete(). The exception is: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

This was occurring on servers in 2 of the 3 sites the customer's environment was divided into. In one site, the same modifications could be made without error.

Discussing the behavior with one of my colleagues brought up the fact that this might be related to Administrator Audit Logging, which is enabled by default in Exchange 2010 SP1. We ran Set-AdminAuditLogConfig -AdminAuditLogEnabled $False to disable the logging and the exceptions ceased, so it was indeed the logging causing them.

After a bit of searching, I found the information regarding the mailbox we put the audit logging in and how we save it:

The logs will be saved to a dedicated folder under “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}”. To reduce latency, Exchange Web Services (EWS) is used to send the asynchronous call to CreateItem in the mailbox.

What we had in this case, and I determined it by investigating the IIS logs from the Mailbox Server, is that our calls to EWS were going to the Default Web Site, and the EWS virtual directory was not under the Default Web Site (DWS). There were 2 web sites set up in this customer’s case, and EWS was under their alternate site.

The IIS logs showed:

2010-11-19 13:35:42 172.26.4.21 POST /EWS/Exchange.asmx - 443 - 172.24.152.80 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.4952) 404 0 0 109

So, the question was, why were our EWS calls going to the DWS, instead of the alternate site? After all, standard EWS calls from client access were working OK.

The reason was because we were going between sites, and thus CAS proxying was occurring. When this happens, we use the InternalNLBBypassUrl path specified for the EWS virtual directory. To see the configuration for the virtual directory we run:

Get-WebServicesVirtualDirectory -Server 'servername'.

And what we saw was:

InternalNLBBypassUrl : https://servername.domain.int/EWS/Exchange.asmx
(note, names have been changed )

This resolved to the IP used for the Default Web Site, and so we would try to go there and find no EWS folder.

There are a couple of possible solutions. The simplest is just to set InternalNLBBypassUrl to the same value being used for InternalUrl, which was https://mail.company.com/ews/exchange.asmx. Although this address goes through NLB, it still works fine.

The InternalNLBBypassURL was changed, and after that, no more exceptions.

--------------------

I had another occurrence of the same exceptions, and in that case, the solution was as follows:

We had a Host Name set in the https binding in IIS, and it was configured with the load-balanced/external name being used by the customer.

This meant that IIS wasn't responding to the internal name, and this is what is used by the auditing extension for the EWS calls to the System Mailbox used for it auditing.

Once we removed the host name from the bindings (leaving the bindings as *), everything worked without exceptions.

After Migrating Resource Mailbox to Exchange 2010 Automatic Booking Doesn’t Work

Richard Roddy [MSFT] — Fri, 05 Nov 2010 21:03:07 GMT

When migrating resource mailboxes from Exchange 2003 to Exchange 2007 or 2010, if you have been using Direct Booking for the resources, you need to make sure to follow this article to the letter:

http://technet.microsoft.com/en-us/library/bb232195(EXCHG.80).aspx

In this case, we verified the customer had done exactly this. We verified that Direct Booking was off with an Outlook client logged into the mailbox and also determine that the CalendarProcessing settings were correct to the automatic booking.

At that point, the next thing to do was to look at the mailbox and see if the messages were sitting in the mailbox and not being processed. This is where it got interesting. None of our test meeting requests showed up anywhere in our room mailbox.

Next we looked to see if the messages were being delivered. Using the GUI message tracking interface in the Exchange 2010 ECP, we could see that the messages were indeed being delivered to the store. Using Get-MessageTrackingLog as indicated below gave us more interesting output:

Get-MessageTrackingLog -Start "03/13/2010 9:00AM" -End "03/15/2010 5:00PM" -Sender john@contoso.com

Looking at that output, we saw that not only was the message being delivered to the store, but then a message was being sent to another internal mail address. However, looking at Rules, etc. indicated no rules in effect.

That’s when it finally struck me - THE MAILBOX HAD A DELEGATE SET ON IT, and the "Deliver meeting requests addressed to me and responses to meeting requests where I am the organizer to:" was set to "My delegates only".

We went into the Delegate Access settings:

Outlook 2010 (on the File Tab):

Outlook 2007 (Tools, Options):

We removed the delegate and then sent a new Meeting Request to the resource - VOILA! It worked!

Mystery solved!

So, if you have a mailbox (resource or otherwise) and you find Meeting Requests aren’t getting delivered to the mailbox, check the delegate settings.

Even More Exchange 2010 Management Tools Fun

Richard Roddy [MSFT] — Wed, 06 Oct 2010 14:47:24 GMT

Edited 1/17/2011 to add second error message listed below.
Edited 3/7/2011 with additional netsh winhttp set proxy information

This time we were getting the following errors on 3 new Exchange 2010 servers (2 MBX, 1 Hub/CAS):

When opening the Exchange Management Shell (EMS):

Connecting to remote server failed with the following error message : Access is denied.

OR:

Connecting to remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests on the correct port and HTTP URL.

Opening the Exchange Management Console (EMC) returned:

The following error occurred when searching for On-Premises Exchange Server:
Connecting to remote server failed with the following error message: Access is Denied. For more information, see the about_Remote_Troubleshooting Help topic. It was running the command "discover-exchangeserver -usewia $true -suppresserror $true".

The strange thing is that we had 3 older 2010 servers that we could open the tools fine on. And, we could use those servers to connect to and manage the servers we were unable to connect from.

After doing a bunch of troubleshooting of WinRM to make sure it was all well, and finding nothing wrong, we finally ran a network capture with Network Monitor 3.4 while opening the EMS. One very nice feature of Netmon 3.4 is the ability to view the network traffic per application. This allowed us to select Powershell.exe and see all traffic from/to it. Here is what we saw:

The odd thing here is that when EMS was opening and trying to connect, it was indicating that it was trying to connect to all 6 servers in the organization and getting back the Access Denied, so why do we only see 2 addresses here? Looking at the addresses, we determined that the first was a DC, which would be expected, since we go to a DC first to determine our Exchange server names so we can then attempt to connect to them through Remote Powershell. The 2nd address however, turned out to be a PROXY SERVER!

In hindsight, this all makes sense, as Remote Powershell/WinRM ends up going through HTTP (or HTTPS) to IIS on whatever server, including the local one, that we are trying to connect to and run our Exchange management on. Going through the proxy server breaks the passing of our credentials using Kerberos. In the netmon capture, I could see this for each server we tried to access:

HTTP:Request, POST http://servername.domain.com/powershell, Query:serializationLevel=Full;ExchClientVer=14.0.639.21;PSVersion=2.0, Using Kerberos Authorization

HTTP:Response, HTTP/1.1, Status: Ok, URL: http://servername.domain.com/powershell, Using Kerberos YIGYBgkqhkiG9xIBAgICAG+BiDCBhaADAgEFoQMCAQ+ieTB3oAMCARKicARu/6C25Su4nr7mR4uEyaESeLJ1XVll6ptl9NPo3zocEHXezAJ/NT5ofY3N2kp8yD/FSJmqFKt7GGLMQV4IA2EUvL0tQux3wWRxr91I

HTTP:Request, POST http://servername.domain.com/powershell, Query:serializationLevel=Full;ExchClientVer=14.0.639.21;PSVersion=2.0
HTTP:HTTP Payload, URL: http://servername.domain.com/powershell

HTTP:Response, HTTP/1.1, Status: Unauthorized, URL: http://servername.domain.com/powershell, Using Kerberos Authentication

To verify the proxy setting, we used the following netsh winhttp command from a command prompt:

netsh winhttp show proxy

We then removed the proxy setting with:

netsh winhttp reset proxy

After we removed the proxy, our tools opened right up and connected fine.

So, if you’re getting Access Denied errors in the connect attempts when opening the Exchange Management Tools, look for a proxy server getting in the way…

However, what do you do if you need to have a proxy setting for external bound traffic from the server?

It turns out that there is a bypass-list setting that can be used with netsh winhttp set proxy. The full reference of the netsh winhttp commands are found here: http://technet.microsoft.com/en-us/library/cc731131(WS.10).aspx#BKMK_5

Looking at the set proxy commands, we can see there is an example of using the bypass-list setting (bypassing proxy for any contoso.com connections):
set proxy proxy-server="http=myproxy;https=sproxy:88" bypass-list="*.contoso.com"

So, if you need to use a proxy in your winhttp settings, use the bypass-list to bypass the internal domain name being used by your Exchange servers.

More Exchange 2010 Management Tools Fun

Richard Roddy [MSFT] — Mon, 26 Jul 2010 19:33:54 GMT

Yet another Exchange 2010 Management Tools error case, with the error: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer.

In this case, all of the troubleshooting used to resolve the issue previously (see my post http://blogs.technet.com/b/richardroddy/archive/2010/06/29/exchange-2010-management-tools-error.aspx) had been done and we were still getting the errors.

In this case, the issue did come back to the web.config file(s) again. For this customer, redirection was enabled for the Default Web Site to send it to the OWA path), which when done through the IIS Management Console, enables it for all sub-directories under the web site. Even after turning the redirection off for the Powershell virutal directory in the IIS interface, we still had problems.

I ended up tracking this one down to the web.config file in the Powershell virtual directory. In the web.config, we had sections for <httpErrors> and <httpRedirect> (although it was set to false). We removed these sections from the file and then we could use the Exchange Management Tools again.

So, among the list of things to check if you’re running into these errors, along with the information in http://msexchangeteam.com/archive/2010/02/04/453946.aspx, see if you have <httpRedirect> or <httpErrors> sections in the web.config file in the Powershell. If so, delete them.

Exchange 2010 and the Exchange Trusted Subsystem

Richard Roddy [MSFT] — Mon, 12 Jul 2010 20:12:55 GMT

In this case, the problem was initially manifesting itself as the following error when trying to create a new mailbox in the first database created with installation of the server:
Active Directory operation failed on adserver.domain.com. This error is not retriable. Additional information: Insufficient access rights to perform the operation.
Active directory response: 00002098: SecErr: DSID-03150E8A, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
The user has insufficient access rights.

Further it was found that the customer got a similar error when trying to create a new database:
Active Directory operation failed on adserver.domain.com. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-03152492, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
+ CategoryInfo : NotSpecified: (0:Int32) [New-MailboxDatabase], ADOperationException
+ FullyQualifiedErrorId : 36F6B416,Microsoft.Exchange.Management.SystemConfigurationTasks.NewMailboxDatabase

This ended up coming down to a case of the Exchange Trusted Subsystem not having the correct permissions on the Databases container in AD to create the new database. The questions here might be, “What is the Exchange Trusted Subsystem? And why does its permissions matter? Shouldn’t the permissions of the administrator on the container be what’s important?”

Well, to answer these questions, all commands in Exchange 2010 are now actually executed by the Exchange Trusted Subsystem, after going through user access checks in Role Based Access Control (RBAC).

More about RBAC and permissions can be looked at here:
http://technet.microsoft.com/en-us/library/dd298183.aspx
http://technet.microsoft.com/en-us/library/dd638106.aspx

The second article notes:
If RBAC allows an action to proceed, the action is performed in the context of the Exchange Trusted Subsystem and not the user's context. The Exchange Trusted Subsystem is a highly privileged universal security group (USG) that has read/write access to every Exchange-related object in the Exchange organization. It's also a member of the Administrators local security group and the Exchange Windows Permissions USG, which enables Exchange to create and manage Active Directory objects.

So, how did we figure out that the Exchange Trusted Subsystem permissions were the problem?

A DSACLS of the Databases container showed the Exchange Trusted Subsystem group missing from the Permissions on the container, so I could only wonder where else it is missing. It should have an inherited Full Control there, inherited from the CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com container.

The following document references the Exchange 2010 Deployment permissions:
http://technet.microsoft.com/en-us/library/ee681663.aspx

This gets set during setup /prepareAD.

In this case, setup /prepareAD had been run again to try and solve the issue, but did not work. So why not?

When we checked, we found that we did have the Exchange Trusted Subsystem group with full control on the Microsoft Exchange container. That’s why the prepareAD didn’t fix it.

So that means that somewhere between the Microsoft Exchange container and the Databases container, the permissions inheritance was disabled. You can see the Inheritance setting in the Advanced area on the Security for each container. You can use ADSIEdit to open the Configuration container, and then drill down to these containers:

Services
Microsoft Exchange
First Organization – (check here)
Administrative Groups – (check here)
Exchange Administrative Group (FYDIBOHF23SPDLT) – (check here)

Here is an example of what this looks like:

If the permissions inheritance is disabled on one of the ones I have indicated to check, it needs to be re-enabled, and the permissions should then be fixed.

In our case, it was missing at the Administrative Groups level. Once that was fixed, new mailboxes and new databases could be created without error.

Exchange 2010 Management Tools Error

Richard Roddy [MSFT] — Tue, 29 Jun 2010 21:25:16 GMT

This past week I was working with a customer who would receive the following error starting the Exchange 2010 Management Console or Exchange Management Shell:

Getting the error "Connecting to remote server failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid." when opening the Exchange Management Console or the Exchange Management Shell.

We went through everything under this topic in the post “Troubleshooting Exchange 2010 Management Tools startup issues” and were still running into a dead end. After a load of searching and digging through prior cases here and internal support notes, I finally found a prior case where they found a web.config and redirect.htm present in the file location of the Default Web Site (the wwwroot folder). So, my customer and I looked there and lo and behold, we had a web.config file there. We renamed the web.config and immediately the Exchange Management tools were able to connect to the server and function properly.

We investigated the web.config that was present a little further, and we found it had the following line in it:
<identity impersonate="true" />

Researching this a little bit, I found the following description:
Impersonation enabled. In this instance, ASP.NET impersonates the token passed to it by IIS, which is either an authenticated user or the anonymous Internet user account.

Looking back at the tools startup troubleshooting post, one of the things mentioned is this:
If the user that is attempting to connect is not Remote PowerShell enabled. To check if a user is enabled for Remote PowerShell, you need to open the Exchange Management Shell with an account that has been enabled, and run the following query.

(Get-User <username>).RemotePowershellEnabled

This will return a True or False. If the output shows False, the user is not enabled for Remote PowerShell.

It appears that in this case, the impersonation being enabled through the web.config was causing us to use the IIS IUSR account for our actions, which is used for anonymous access, and is not able to be enabled for Remote Powershell. Removing the web.config put us back to passing our user credentials for Remote Powershell access and fixed the problem.

MSExchange ADAccess (DSAccess) errors and the “Manage auditing and security” right

Richard Roddy [MSFT] — Wed, 16 Jun 2010 17:27:00 GMT

(Edited 8/8/2011 to add additional error that might be found in the 2114 description.

This is something I’ve ended up having to resolve multiple times with customers, so I felt it would be good to get a post out about it.

In this case, the customer had an environment of Exchange 2003, 2007, and 2010. The Exchange 2007 and 2010 servers, with the exception of one Exchange 2007 mailbox server, were throwing errors such as these:

Event Type: Error
Event Source: MSExchange ADAccess
Event Category: Topology
Event ID: 2114
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=4600). Topology discovery failed, error 0x80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, "Microsoft LDAP Error Codes." Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

The above event may have the following error instead:
Topology discovery failed, error 0x80040a02 (DSC_E_NO_SUITABLE_CDC).

Event Type: Error
Event Source: MSExchange ADAccess
Event Category: General
Event ID: 2604
Description:
Process MSEXCHANGEADTOPOLOGY (PID=4600). When updating security for a remote procedure call (RPC) access for the Exchange Active Directory Topology service, Exchange could not retrieve the security descriptor for Exchange server object SERVER - Error code=80040a01.
The Exchange Active Directory Topology service will continue with limited permissions.

Event Type: Error
Event Source: MSExchange ADAccess
Event Category: General
Event ID: 2501
Description:
Process MSEXCHANGEADTOPOLOGY (PID=4600). The site monitor API was unable to verify the site name for this Exchange computer - Call=HrSearch Error code=80040a01. Make sure that Exchange server is correctly registered on the DNS server.

At this point, the next thing done was to look for a recent 2080 event and see what the Exchange server was seeing as far as domain controllers were concerned. The 2080 looked like this:

Event Type: Information
Event Source: MSExchange ADAccess
Event Category: Topology
Event ID: 2080

Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=2252). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
dc1.domain.COM CDG 1 7 7 1 0 0 1 7 1
dc2.domain.COM CDG 1 7 7 1 0 0 1 7 1
Out-of-site:
dc3.domain.COM CDG 1 7 7 1 0 0 1 7 1
dc4.domain.COM CDG 1 7 7 1 0 0 1 7 1

For well-versed Exchange folks, the problem in this 2080 is fairly obvious, the Exchange server is missing the SACL (Manage auditing and security log) right on the DC’s.

So, then the question was why is it missing, and this is what trips up many people, and is the reason I decided to write this.

Whenever I get one of these cases, the first thing I like to do is go to one of the Domain Controllers and run Resultant Set of Policy (RSOP.MSC). We then drill down to the User Rights Assignment, as seen below.

Here we can see the Manage auditing and security log right, can see what accounts are listed in the right, and what the Source GPO is that it is coming from. By default, the Exchange Servers (Exchange 2007 and 2010) group and Exchange Enterprise Servers (Exchange 2003) group are added to this right in the Default Domain Controllers Policy. This occurs during the setup process. For Exchange 2003, this is done during the setup /domainprep process, and for Exchange 2007 and 2010, this is done during setup /preparedomain.

So, what happens if the Default Domain Controllers Policy is not the policy that is applying this right to your domain controllers, as shown in the RSOP output. If this is the case, you need to make sure that the policy that is responsible for applying the right grants the Exchange Servers (or Exchange Enterprise Servers) group the right, or edit the Group Policy Links for you Domain Controllers OU so that the Default Domain Controllers Policy is applied.

In my latest customer’s case, we saw in RSOP that the Default Domain Policy was being applied instead of the Default Domain Controllers Policy. This was because the Default Domain Controllers Policy link had been removed from the OU and the Default Domain Policy was being applied instead. And in his Default Domain Policy, the Exchange Enterprise Servers (EES) group (Exchange 2003 group) had been granted the right. We also found that his one working Exchange 2007 server had been added to the Exchange Domain Servers group (again, Exchange 2003 group) which is then part of the EES group. The server membership in the Exchange 2003 group(s) is why the one Exchange 2007 server was still working but the other Exchange 2007 and 2010 servers were not. To fix this, we linked the Default Domain Controllers Policy to the Domain Controllers container, removed the link to the Default Domain Policy from the container, and then ran ‘gpupdate /force’ on the DC to apply the policy. Once we did this, all of the Exchange Servers now worked properly.

Exchange 2007 Management Shell Errors

Richard Roddy [MSFT] — Fri, 02 Apr 2010 19:21:18 GMT

My latest adventure was in the form of errors trying to run just about any command in the Exchange Management Shell (EMS).

The problem came in conjunction with a case where Exchange 2007 was blown away and reinstalled, with some setup problems along the way, as well as some intermittent mail flow and OWA problems. This was a combo CAS, Hub, and Mailbox server, and given the problems encountered by the prior engineers, I wasn’t sure what I could trust regarding the soundness of the install.

However, back to the EMS errors. Here is the kind of errors we were seeing:

[PS] C:\Documents and Settings\Administrator>Get-ExchangeServer
WARNING: An unexpected error has occurred and debug information is being
generated: Could not load file or assembly 'Microsoft.Exchange.Extensibility.Internal, Version=8.0.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
Get-ExchangeServer : Could not load file or assembly 'Microsoft.Exchange.Extensibility.Internal, Version=8.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
At line:1 char:19
+ Get-ExchangeServer <<<<
+ CategoryInfo : NotSpecified: (:) [Get-ExchangeServer], FileNotFoundException
+ FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.Exchange.Management.SystemConfigurationTasks.GetExchangeServer

However, all functions in the Exchange Management Console (EMC) worked fine.

To be able to remove and reinstall the Exchange Management Tools however, requires removing all of the Exchange Roles as well, so we first went through a lot of troubleshooting regarding seeing is assemblies were properly referenced in the registry, etc.

After completely uninstalling the server, and reinstalling just the Management Tools, we still got the same type of error in the EMS. So next I wanted to remove Powershell and reinstall it, but we were not able to find the Windows update (926139) listed in Add/Remove programs. Since I had output from our Support Diagnostics Platform (SDP) tool, I went searching for the Powershell update in the hotfixes list we get from the tool. One particular entry I found was very interesting:

02/14/2010 05:04:44 AM Install SYSTEM AutomaticUpdates KB968930
Windows PowerShell 2.0 and WinRM 2.0 for Windows Server 2003 x64 Edition (KB968930)

Powershell 2.0 on an Exchange 2007 server. This is not a good thing, at least not until Exchange 2007 SP2 is applied.

So, we went back into Add/Remove Programs, and did not see any reference to this update number. However, another entry caught my eye: Windows Management Framework

It turns out that the Windows Management Framework is the Powershell 2.0 and WinRM 2.0 package.

We removed Windows Management Framework, fired up the EMS, and now our commands worked perfectly!

So, if you ever encounter errors in Powershell like we saw above on your Exchange 2007 server, and you haven’t applied SP2, look for Powershell 2.0 or Windows Management Framework installed on the system and remove it. Or you can go ahead and update to Exchange 2007 SP2.

Thanks to a comment from Aleksandar of http://powershellers.blogspot.com/, I can now note that if SP2 is applied to your Exchange 2007 server or your management workstation, Powershell 2 is now supported and should work fine.

Clustered Mailbox Server will not come online on one node of cluster

Richard Roddy [MSFT] — Mon, 07 Dec 2009 19:23:14 GMT

In this case, we had an Exchange Server 2007 Continuous Cluster Replication (CCR) setup, and if we either failed on the first node, or attempted to move the Clustered Mailbox Server (CMS) to the other node, the Information Store resource and all of the resources for the Storage Groups would hang in an Online Pending state on the second node.

Taking a look at the cluster log showed the following when the Exchange Information Store resource was attempting to come online on the destination node:

00000bdc.00001050::2009/11/19-17:20:26.478 WARN Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] DwWaitServiceStateChangeToTargetState: service 'MSExchangeIS' current state is: running.
00000bdc.00001050::2009/11/19-17:20:26.478 WARN Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] DwStartService: service 'MSExchangeIS' was started.

…………………………………..

00000bdc.00001580::2009/11/19-18:48:53.859 WARN Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] Reporting state 129 (OnlinePending).
00000bdc.00001580::2009/11/19-18:50:23.858 WARN Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] Reporting state 129 (OnlinePending).
…………………………………..

00000bdc.00001580::2009/11/19-18:57:53.856 WARN Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] Reporting state 129 (OnlinePending).
00000888.00000898::2009/11/19-18:58:16.371 WARN [INIT] The cluster service is shutting down.
00000888.00000898::2009/11/19-18:58:16.371 WARN [FM] Shutdown: Failover Manager requested to shutdown groups.
00000bdc.00001050::2009/11/19-18:58:16.528 WARN Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] EcStoreStartServer() returned 1
00000bdc.00001050::2009/11/19-18:58:16.528 ERR Microsoft Exchange Information Store <Exchange Information Store Instance (CMSNAME)>: [EXRES] EventLogging: Exchange Information Store Instance (TNS-MAIL-AP): The RPC call to the service to bring the cluster resource online failed. Error Code: 1.

I searched through our vast repository of previous cases for Exchange and cluster and OnlinePending and resource online failed, and found a prior case with the same symptoms. In that case, and it turned out to be the same cause in ours, the McAfee Groupshield service was stuck in a ‘Starting’ state. When Exchange-aware Antivirus in enabled on a server, if the anitvirus does not start, it will block the Information Store from being available. In the case of our Exchange cluster, this in turn kept the Information Store resource from reaching an ‘Online’ state.

To confirm this was the problem, we set the following registry value to 0 (zero) on the node we were failing on:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\VirusScan\Enabled

This tells the Exchange Information Store to disable virus scanning, effectively disabling Groupshield in this case. Once we did this, the Information Store resource and storage group resources were able to come online on this node.

The customer then worked with Groupshield to fix the problem that was causing Groupshield not to come online successfully.

Export-Mailbox using a filter does not export anything

Richard Roddy [MSFT] — Wed, 18 Nov 2009 18:59:29 GMT

Recently I was working with a customer on a case with Exchange 2007 where they were trying to use Export-Mailbox to remove specific items from a user’s mailbox, as documented here:

http://technet.microsoft.com/en-us/library/aa998579.aspx

In this case, we were attempting to remove all items matching certain subject keywords from the Inbox. So, the command we were running looked something like this:

Export-Mailbox -Identity username -IncludeFolders "\Inbox" -TargetMailbox mytargetmailbox -TargetFolder "Test" -SubjectKeywords "[Suspected spam] Alert" -DeleteContent

The command ran without any errors, and appeared to work on exporting messages, but when it was done, no messages appeared in the target mailbox, and the messages that should have been removed from the mailbox were not removed.

In further testing, we found that we could use Export-Mailbox to export the entire Inbox to the target, and we could also export from a couple of other users’ mailboxes using the subject keywords successfully.

At this point, I checked with one of the Escalation Engineers (the guys that actually look through code) here and they suggested that if the mailbox that worked was on a different store, the most likely cause was corruption of the content index for the store the problem mailbox is on.

So, given that it appeared the problem was corruption of the content index, our solution was to rebuild the index. We used the information found here (In #2 under “To Rebuild the Full-Text Index Catalog Using the ResetSearchIndex.ps1 Script”) to do so:

http://technet.microsoft.com/en-us/library/aa995966.aspx

After rebuilding the index, the Export-Mailbox command worked perfectly using the SubjectKeywords. So, if you find that you’re having problems exporting from mailboxes using keywords, you might try resetting the index catalog.