Mike Lagase

Saving the Exchange world one day at a time.....

The effects of paged pool token memory usage and Outlook 2007’s shared folders feature

The effects of paged pool token memory usage and Outlook 2007’s shared folders feature

  • Comments 1
  • Likes

In Exchange 2003, we are bound by 32-bit architecture memory limitations and paged pool memory just so happens to be one of them. This is a very common issue that we in Exchange support see is paged pool memory depletion related to excessive Token usage.

For the most part, paged pool memory troubleshooting for Exchange 2003 can be found in http://support.microsoft.com/kb/912376. This article goes over what access tokens are, what symptoms are seen when paged pool memory usage is low, how token sizes are generally calculated, and a script that will allow you to export group memberships for users that are mail enabled. It does not go over some of the other reasons why paged pool memory may increase on any given Exchange 2003 server.

There is also a great document “Addressing Problems Due to Access Token Limitations” at http://download.microsoft.com/download/8/f/3/8f36dfe4-47d0-4775-ad5a-5614384921aa/AccessTokenLimitation.doc that goes over Access Tokens in much greater detail.

If you had used the scripts in 912376 to export the users group membership, you might end up with data in Excel similar to the screenshot listed below.

image

Notice that we have a slew of users that have really high group membership. This can and will affect overall Token sizes when connecting to an Exchange server.

If you ever wanted to calculate the approximate size of a users access token, the formula as listed in 912376 is the following:

[12 x number of user rights] + [token overhead] + [44 x number of group memberships] = token size in bytes

Trying to find the maximum amount of security groups a user can belong to before we double the size, we would come up with the following (4096-500 / 44 = 81.70). This means that a user needs to be a part of a maximum of approximately 80 security groups before the token size doubles to 8192. This a rough guesstimate, but this should be enough to estimate overall token sizes for users in general terms if you wanted to. Microsoft does not recommend more than 100MB of paged pool memory for token allocations as it may impact Exchange and other applications installed on the server.

One of the reasons that I believe that there are many security groups that are mail enabled in any organization is because this is a default behavior of creating groups in Active Directory Users & Computers (ADUC). When you create a group in ADUC, the default type is a security group, not distribution, so over time, you can end up with users that are a part of many groups that are of type security which is a bad thing. If you let this go long enough, the users access token can bloat to high values causing paged pool memory issues caused by excessive Toke usage.

One of the first things you need to do is ensure that the hotfix in http://support.microsoft.com/kb/912480 is applied on the Exchange server. This hotfix decreases the amount of connections that Outlook users make to an Exchange server.

Each connection that a client makes to a server will increase overall token usage for that user due to that fact that we need to expand all group memberships to ensure that you have permissions to the requested resource and you don’t have any specific deny in any of the many groups they are a part of. If you establish a lot of TCP connections to a single process such as the store.exe process, you could exhaust paged pool memory resources on the server very easily every time a user needs to verify that they have permissions to access resources in users mailboxes. The more connections you have open, the more token usage that will be consumed.

So now I would like to bring to light a new feature in Outlook 2007 to help cache shared folders such as Calendars, Contacts, Tasks, and Notes. This specific feature is Download shared folders (excludes mail folders) which is turned on by default in any new Outlook 2007 profiles. This allows easier and faster access to other users mailbox folders since the data will be cached locally in their Outlook profile. Here is what the option looks like in a users outlook profile.

image

For each shared folder that is cached in a user profile, an additional connection will be opened to the Exchange server to sync that data to the users OST file. If a user is opening a lot of other folders, then you will see many connections to the Exchange server. To make it even better, http://support.microsoft.com/kb/955572 introduced a new feature where you can cache other users mail folders such as the Inbox by adding a value to the registry on each client computer called CacheOthersMail. Once this is implemented, any user can now have the ability to cache any other users mailbox folders should they have access to them.

As you can see, this can significantly increase the amount of connections to any given Exchange server, thus increasing overall token usage. If you have users that have high group memberships, then it may be advantageous to disable these features. This can be accomplished organization wide via Group Policy by disabling the “Synchronizing data in shared folders” option listed in http://technet.microsoft.com/en-us/library/cc179175(office.12).aspx. Disabling this feature will now cause the Outlook clients to make direct RPC connections to the Exchange server since that folder data is no longer cached locally, so it may take just a bit longer to pull up the data they are looking for.

If you are migrating your users from Outlook 2003 to Outlook 2007 and are noticing increased page pool Toke memory usage on an Exchange server, then this may be one of the reasons for it.

One way to determine if the outlook client has many connections open to the Exchange server is to run a netstat –ao on the Exchange server and pipe it to a text file. If you view the list of connections from single client IP addresses to the Store process, which in this case was running under port 1674, you may see many established connections as shown below.

TCP    EXSERVER:1674       CLIENTCOMPUTER:3196     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3202     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3205     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3210     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3219     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3234     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3235     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3238     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3239     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3241     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3246     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3252     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3253     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3257     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3258     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3260     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3261     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3263     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3264     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3277     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3278     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3280     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3282     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3283     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3284     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3285     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3286     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3293     ESTABLISHED     4944
TCP    EXSERVER:1674       CLIENTCOMPUTER:3301     ESTABLISHED     4944

If this type pattern is seen for various clients, then this feature may be affecting your Exchange servers paged pool memory usage which could potentially lead to unresponsive servers.

In one of the cases that I had worked, disabling this feature via group policy dropped the runtime paged pool memory usage 40-50MB. This is a huge memory savings that may keep your server afloat until you migrate to Exchange 2007 or Exchange 2010. Of course, those versions of Exchange are not susceptible to these type nonpaged or paged pool memory issues since we have moved away from the 32-bit platform and will only support the 64-bit versions of Exchange in production.

Once you have migrated these users to later versions of Exchange, you can then remove the group policy to enable this feature again.

Of course there are many other reasons for excessive Toke usage on an Exchange server, but this is just one of those new features that may bite you in the end.

Until next time…

Comments
  • <p>Great Post Mike......!</p> <p>Thanks for sharing :)</p>

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment