Within the past couple of weeks I have witnessed multiple instances where the RpcClientAccess service is experiencing high CPU utilization on the CAS server and in some instances also on a public folder server. I captured performance data from these systems to get a better picture of what is happening on the server. The analysis of this data does not show any unusual activity other than this process utilizing more CPU than normal. This tells me that the process itself must be doing something that is causing these extra cycles.
To determine what is happening within the RPC Client Access service we need to review the Rpc Client Access logs on the server. While parsing thru this log file I found a high volume of PublicLogon requests coming into this CAS server for a public folder database. These logon attempts are typical behavior as the CAS server will then redirect these requests to the mailbox server hosting the database. But taking a closer look at these requests and I see the database is last mounted on the expected mailbox server, however the client is being redirected to another server. (If you only have one public folder database you will simply see a high volume of requests coming into you CAS server and the CPU will most likely spike to near 100 percent). The log entry looks like the following:
2012-02-15T14:00:51.707Z,6,1,/o=EXCHORG/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Exchange User,,MFCMapi.exe,14.0.6109.5000,Classic,,,ncacn_http,,PublicLogon,1144 (rop::WrongServer),00:00:00.0621272,"Logon: Public, in database c83d21d5-7228-455e-978d-81d6649863a1 last mounted on MBX1.contoso.com at 1/28/2012 8:01:35 PM, currently Mounted; Redirected: alternate server requested, suggested new server: /o=EXCHORG/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=MBX2",RopHandler: Logon:
This issue is caused because the RpcClientAccess service on the mailbox server hosting the public folder database is not responding to client connections. If we take a look at the RPC Client Access logs on this server we should see minimal activity. This is a known issue that is documented in KB 2535105 (Outlook client applications cannot connect to public folders after you install Exchange Server 2010 SP1). The RPC Client Access service acts as a proxy for public folder availability requests and these requests are not completed. Eventually new connections for these EWS calls cannot be made because the connection group limit is reached. The KB article provides a registry key that is available starting with SP1 RU4 that disables these available service calls by the RPC Client Access service.
A .NET hotfix that resolves the underlying issue has been released since the posting of the Exchange KB, KB2497453 (FIX: Application may stop responding after several HTTP requests are aborted if you use the HttpWebRequest class to send requests in the .NET Framework). Install this hotfix on the mailbox server hosting the public folder database and reboot the mailbox server. Once this server is back online we need to restart the Rpc Client Access service on the CAS server to clear the cache. Then when we review the Rpc Client Access logs on the CAS server we should see clients being redirected to the server where the public folder database is mounted.
This one is helpful too...
many thanks for your post. we are experiencing the exact same issue on our cluster.
many users getting reprompted for details over and over again due to high cpu.
will apply that hotfix tonight.
thanks Jim (and google for finding jims post).
KB2497453 is an older patch which has been superseded by KB2478662. KB2478662 is installed. Yet we do still see the same issue occurring. Should we still install KB2497453?
This post really help me , thank you
I'm very much interested on this, mine exchnage 2010 SP3 RU2. these day I see high cpu usage on the two CAS/HUB servers the spike reaches 100% and users will be disconnected , and the ping result to these servers become worst and even 'Request timed out. so is this solution applicable for Exchange 2010 SP3 RU2.
I highly appreciate your support on this'
Yes, this applies to Exchange 2010 SP1 and later. The feature was added to SP1 which introduces the potential issue.
Useful and we have implemented the same.
Hi Jim, thanks fo the valueble information. May I know how do we verify via log/event that the registry diable the available service call?
Couple of questions:
Is the Registry Key only modified on the Public Folder servers? (I have seen some other articles with argument on this)
Is the .NET fix only applied on the public folder servers, or all roles?