Man what a week. Some customers are seeing that when opening someone else's calendar that it takes a long time at first but when they go back to the calendar it is fine for a while and then they see the slowness again. If you are seeing this, we may know why.

(Credit: A lot of this is from Stephen Griffin's internal document. His blog: http://blogs.msdn.com/Stephen_Griffin)

*** Problem Description ***

When viewing someone else's Calendar, contact folder, etc, there may be delays before the folder can be viewed. Once the folder has been viewed, switching away and back is fairly quick, but after a period of time, accessing the folder is again slow. This is especially long if the number of items in the calendar is over 5000.

*** Possible Reasons ***

When Outlook accesses someone else's folders, it applies a view which restricts the user from viewing private items.

The act of applying a view to a folder creates search folders in the store. Once a search folder is created, it is cached for later use. If we try to create a search folder and discover there is one in the cache we can use, we don't create the search folder and instead used the cached search folder. This allows subsequent viewings to be fairly quick.

By default, we don't cache all search folders forever. Caching too many search folders would cause server side delays associated with updating the search folders:

216076 XADM: Accessing Information Store Folders May Become Slow

http://support.microsoft.com/?id=216076

However, if we don't cache *enough* of our search folders, we may run into the problem from the other direction.

Suppose we're configured to keep 11 search folders (views) per folder. Suppose Frank has a calendar folder that he shares out to 15 other users. Sally accessed the folder and sees a delay while her search folder is built. After it's built, access is quick. Then Sally doesn't look at the folder for a day and 11 other users access it. A search folder will be built for each of them. Since we can only cache 11 search folders, when the 11th user hits the folder, we'll delete the search folder we built for Sally. Now, the next time Sally hits the folder, she'll have to wait while we build her search folder.

Suppose we configure Frank's Calendar folder to cache 20 views instead. Then Sally and the other 14 users can all hit the folder, and only 15 search folders will be created. Since 15 is less than 20, we never have to cycle out a view, so access is quick for everyone after the initial hit to create the search folders.

The default is 11, set store wide. This can be viewed by using ADSIEdit to view the Store Object and look at the msExchMaxCachedViews attribute. (dn: CN=Mailbox Store,CN=Storage Group,CN=InformationStore,CN=Server NAME,CN=Servers,CN=AG Name,CN=Administrative Groups,CN=Orgname,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=Company,DC=com) This is set to 11 by default. In some customer's instances we have seen that increasing this to a higher number (but no more than 50) is necessary.

How do we verify that this is what you are seeing? It would require that you take the Information Store DB offline that contains the mailbox that is being accessed. Then we would run isinteg -s <ServerName> -dump -l <logfilename>. If this output is sent to Microsoft they can confirm what you are seeing. (Hint: We are looking for a high number of SearchFIDs values.) Or, see the perfmon counter below.

Also, it turns out that in Outlook 2003 we are creating too many calendar search folders (indexes).

896418 Description of the Outlook 2003 post-Service Pack 1 hotfix package:

http://support.microsoft.com/?id=896418

This fix resolves the issue where we are creating too many views against another user's calendar.

The outlook 2003 fix used in conjunction with the increasing of the msExchMaxCachedViews attribute has been the resolution for many of our customers who have seen this.

Oh, and having additional locales on the client side can create even more views as well.

Q: What is the impact on the server?
A: Creating too many indexes over and over can place a load on the servers. Having too many indexes stored can also slow down the clients accessing the folders as well. Gradually increasing the msExchMaxCachedViews attribute is recommended while keeping an eye on the performance of the server. It is suggested that 50 be the maximum number. It is not suggested that you go to this number first. I would increase the number by 5 or 10 each time you increase the number.

Q: Is hotfix 896418 a scenario that only affects Outlook 2003 clients?
A: If they have 98%+ Outlook 2002\XP users and only a handful of Outlook 2003 clients, then it is not likely that they will run across this particular bug, because this is strictly a 2003 bug.

Q: How can we make the Event IDs show up for the Restricted Views on Folders?
A: They can set "9030 Views" under
"HKLM\System\CurrentControlSet\Services\MSExchangeIS\Diagnostics\9000 Private" to 1 and get event logs when views are created.
This will show the following:
MessageID=1167
Severity=Informational
Facility=Views
SymbolicName=msgidNewRest
Language=English
%1 created a new restricted view on folder %2 on database "%3".

Q: Is there a Performance Monitor counter that will show us the number of Restricted Views that are created?
A: Yes - But only if you are running Exchange 2003 SP1.
MSExchangeIS Mailbox
Restricted View Cache Miss Rate - is the rate at which restricted views are created in the mailbox store...

Q: How do I get the number of items in the Calendar?
A: Go to the Calendar. Click View \ Arrange by \ Current View \ By Category. (This may take a while - we are creating a new view). Then look at the Bottom Left hand side - #### Items should show there. To do this without creating another view you could use MFCMAPI.

My largest customer ran across this one. Here is the scenario:

Exchange 2003 Mixed mode with mostly 5.5 servers
New profiles created against a mailbox on an Exchange 2003 server can't send mail when in cached mode.
New profiles created against a mailbox on an Exchange 5.5 server do work in cached mode.

So, it turns out that the clients would create an "instant NDR" because a key was missing. The customer compared the two profiles stored in the registry at HKEY_CURRENT_USER\Software\Microsoft\Windows Messaging Subsystem\Profiles\<Profile name>\13dbb0c8aa05101a9bb000aa002fc45a and we found out that the primary value that was missing was "01026687". This key is used to contain the routing. It is created from the GWART in Exchange 5.5 and pulled form the AD in Exchange 2003. For some reason when you have a TON of address types, either the server doesn't send the data or the client doesn't accept it. To be honest, I really don't know which it is, but I was told that it had to do with a 32K limit. But that wasn't proven to me. What I do know is that this key only seems to get created at the profile generation time and not get modified later. So if you don't get his the first time you create the profile, you are pretty much in bad shape. We determined this with Regmon.

How many Address types were in this customer's scenario? Well using WinRoute we determined that there were about 2400 Address types. 5 different connectors had 400 X400 address types on them and this was really not necessary. Really all they needed was 1 per connector. This was left over from their old X400 cloud days. We got rid of 2000 of these addresses and it all started working. Before doing this the GWART0.mta was almost 1MB!

Now the question of the hour... Should this be fixed? Well, if you were the software company that wrote this and had a bunch of other issues that needed to be prioritized, where would this fall? In my scenario, the customer was more than happy with the workaround. It sort of is like creating over 1000 empty Routing Groups for no reason and then complaining to Microsoft that you hit the limit. Should they fix it? No. How many people are going to hit that limit? Probably not that many. How many people will be affected by the code if you make the fix and it messes up something else? Probaby everyone. So the decision is not to make the change, because the workaround is sufficient for the majority of the users. If our largest customers hit this limit all the time, then we would definitely make the change if there was no workaround.

Sound fair?

Restricted Views and Shared Calendars

Too Many Address Types Can Cause Cached Mode to Not Work