Recently we've seen some cases in Exchange Support where the error event 623 gets generated immediately at the start of the Information Store service. So far we've only witnessed this on some Exchange 2003 Servers. Please note that this is specifically about this event being generated after the IS start. There were other causes of even 623 that we had fixes for already.
When this behavior occurs, you may see the Information Store appear to take upwards of 45 minutes to fully respond at service startup. Monitoring "Version Buckets Allocated" (viewable with Show Advanced Counters - see Nageshs' excellent post here: http://msexchangeteam.com/archive/2006/04/19/425722.aspx) will show the Store is immediately running high (over 70%) and until the number falls the Information Store will be unresponsive to clients and ESM. After Version Buckets Allocated falls, the server responds fine and no other issues are observed. 623 errors go away. Restarting a 3rd party server that ties into users' mailboxes (if present) or restarting the Information Store service may cause the issue to occur again.
This problem occurs because of a large amount of hidden search folders that have been created by applications (other than Microsoft Exchange) that have access to users' mailboxes. When the Information Store starts, it becomes available to the host of 3rd party applications which might reconnect and want to sync the contents of the search folders at the same time. These search folder updates can then result in the search folders for a user's mailbox to all be updated at the same time. When a mailbox has a large item count in the Inbox folder (more than 5,000 items) you can experience higher than normal store CPU % utilization and Version Buckets Allocated spikes which can lead to version store out of memory problems. Depending on the type of search performed, the impact can be greater or smaller. Once version store cache has been depleted, the offending transaction gets canceled or it times out and is rolled back and everything moves along as if nothing happened. That's why the event 623 eventually corrects itself.
To avoid this scenario, there are a few things you can do to monitor this:
So - how do you do #2 above?
You'll have to run: isinteg -s servername -dump -l logfilename
Then open up the "logfilename" file and look for the following:
 Folder FID=0028-00000002451E Parent FID=0028-00000002451B Root FID=0028-00000002451A Folder Type=1 Msg Count=29232 Msgs Unread=112 Msgs Submitted=0 Rcv Count=4 Subfolders=0 Name=Inbox Comment= Restriction= Search FIDs=0028-000008859B57,0028-000008859B60,0028-000008859B67,0028-000008859B54,0028-000008859B5D,0028-000008859B59,0028-00000C8C48C3,0028-000008859B4E,0028-000008859B4C,0028-000008859B58,0028-00000C8C2DF3,0001-0000001995E3,0028-000008859B55,0028-000008859B5E,0028-000008859B53,0028-000008859B4D,0028-000008859B4B,0028-00000C649EB1,0028-00000C8C48E5,0028-000008859B66,0028-000008859B69,0028-000008859B56,0028-000008859B5F,0028-00000C64A1EA,0028-000008859B65,0028-000008859B50,0028-00000C8C48D6,0028-000008859B5A,0028-000008859B64,0028-00000C8C48CE,0028-000008859B52,0028-000008859B4A,0028-000008859B68,0028-000008726E8B,0001-000000197413,0001-000000197C59,0001-000000198A12,0001-0000001CF526,0001-0000001CF53B,0001-0000002284E0 Scope FIDs(search folder only)= Recursive FIDs= Search Backlinks=0001-000000031BEA,0028-000006CD9DC2,0028-000007594A53,0028-0000075DEE07,0028-00000857AB81,0028-00000A027DBC,0028-000008726E8B,0001-000000198A12,0001-000000197C59,0001-000000197413,0028-00000C8C48D6,0028-000008859B59,0028-000008859B55,0001-0000001995E3,0028-000008859B69,0028-000008859B66,0028-00000C8C48C3,0028-000008859B67,0028-000008859B65,0028-000008859B64,0028-000008859B68,0028-000008859B57,0028-000008859B53,0028-000008859B50,0028-00000C8C48CE,0028-00000C8C48E5,0028-000008859B4B,0028-000008859B52,0028-000008859B4D,0028-000008859B58,0028-000008859B5E,0028-000008859B54,0028-000008859B5A,0028-000008859B56,0028-000008859B4E,0001-0000001CF526,0001-0000001CF53B,0028-000008859B4A,0001-0000002284E0,0028-00000C8C2DF3,0028-000008859B5D,0028-00000C64A1EA,0028-000008859B60,0028-000008859B5F,0028-000008859B4C
What we're looking at here is a high number of Search folders (Search FIDs above) and Search Backlinks that - when they have to generate or update - have to scan over 29000 items each (MsgCount above). This is the crux of the 623 version store problem at startup that you might be seeing.
At this time, we do not have a simple solution for this problem... If you have this problem and have identified it using the above step #2 (as situation described in step #1 can be solved by reorganizing the folders/number of items), please contact our Exchange support line. Once we have a better way of resolving this, we'll post about it here.
For more information on Search Folders, review:
KB260322 - How To Search Folders with the SetSearchCriteria Method http://support.microsoft.com/kb/260322/en-us
Best Practices for Exchange 2003 Search Folders (there are several subsections here to look at as well) http://technet.microsoft.com/en-us/library/aa997533.aspx
Creating Search Folders: http://msdn.microsoft.com/en-us/library/ms878645.aspx
Exchange Store Search Folders: http://msdn.microsoft.com/en-us/library/aa123899.aspx
- Jeff Stokes, Dave Goldman, Michael Blanton