Thoughts from the EPS Windows Server Performance Team
At least once a week, someone on the Performance team will get a customer call concerning hangs or resource depletion on their file server. The file server in question is used for user home folder storage and users are accessing Outlook Personal Storage (.pst) files stored on the server from their client. The issue will manifest as either a server hang, or PagedPool depletion (Event ID 2020). Oftentimes the issue will occur first thing in the morning - when users are logging on and launching Outlook. In especially severe cases, the issue occurs several times daily. Sometimes the server will hang for a few minutes and then continue operating for a few minutes - and then hang again. Rinse & repeat. The users are frustrated because of slow access to their data, the server administrators are frustrated because they are tasked with fixing the problem, and upper management is frustrated because everyone else is frustrated.
If you're in this situation - there's good news ... and very bad news. The good news is that this problem is very common and is a known issue. The very bad news (from the customer's standpoint) is that PST files on a LAN/WAN is an unsupported configuration. Some customers are very surprised to hear this but Network Stored PST files have been unsupported since the days of Exchange 4.0. Microsoft KB Article 297019 goes into some detail about the effects of Network PST files:
"A .pst file is a file-access-driven method of message storage. File-access-driven means that the computer uses special file access commands that the operating system provides to read and write data to the file.
This is not efficient on WAN or LAN links because WAN/LAN links use network-access-driven methods, commands the operating system provides to send data to or receive from another networked computer. If there is a remote .pst (over a network link), Microsoft Outlook tries to use the file commands to read from the file or write to the file, but the operating system then has to send those commands over the network because the file is not on the local computer. This creates a great deal of overhead and increases the time it takes to read and write to the file. Additionally, the use of a .pst file over a network connection may result in a corrupted .pst file if the connection degrades or fails."
Let's use an example to illustrate the problem and also follow the problem through to its end result.
Let's say that a user sends an e-mail message to 500 users within the company. All of these users have their e-mail delivered directly to their PST file which is stored on the File Server. Some of these 500 users may need to extend their PST files to receive it. To extend a PST, an extra allocation on disk has to be made via NTFS. This locks out the whole volume while free space is allocated and the Master File Table (MFT) is updated. While this is happening for each user, all I/O for the other 499 users is on hold.
Allocating free space can take an extended time, especially if the disk is fragmented. Now factor in multiple users extending their PST's in the same timeframe, and significant periods of MFT lockout might be observed, which in turn is seen as inability to access any other file on the volume, resulting in queueing in the server service work queues, and sometimes SRV 2019, 2020, 2021, or 2022 events being logged. This scenario might overload the disk(s).
Setting aside the example of one email being sent to a group of users, imagine if you had a couple of hundred users who each have two or three PST files. These users have been with the company for a while, and they rarely (if ever!) delete their email from their PST files. The files continue to grow in size - let's use an average of 1 GB as the size of the PST file. Now consider that when each user launches Outlook, they make a request for two (or three) files, each of them being about 1 GB in size. Then consider what happens when 200 users all launch Outlook around the same time when they get to work. 200 x 3 x 1 = 600 GB of data being requested at the same time. That's an awful lot of Disk & Network I/O to process simultaneously. This is a very common scenario - the file server "freezing" for a few minutes at a time while it tries to service these requests.
The queuing in the server service work queues is what causes this temporary hang. The server service uses work items to handle I/O requests that come in over the network - for example: a request to extend a PST file. These work items are queued in the server service work queues, and from there they are handled by the server service worker threads. The work items are allocated from a kernel resource called Non-Paged Pool (NPP).
The server service sends these I/O requests down to the disk subsystem. If, for reasons mentioned above, the disk subsystem does not respond in time, the incoming I/O requests are queued via work items in the server work queues. Since these work items are allocated from NPP, eventually this resource runs empty. Running out of NPP causes systems to hang eventually (logging an Event ID 2019 in the process).
Digging down into this from more of a troubleshooting perspective, we can usually see issues caused by the PST files manifested in Poolmon and Perfmon captures. For example, we may see the LSwn pool tag allocation climbing in a Poolmon trace. These allocations are made by SRV.SYS. The size of the allocation is configurable via the SizReqBuf registry value. One allocation is made for each work item used by the server service. When looking at this through Perfmon, you will notice a steady decrease in the "Available Work Items" counter. If Available Work Items reaches zero, then clients may experience difficulties accessing files (any files, not just the PST files!). You may also experience 2019 errors if the problem lies with LSwn allocations (Non-Paged Pool depletion)
Another tag that highlights the issues with the PST files is the MmSt tag. This tag represents Mm section object prototype PTEs - a memory management-related structure used for mapped files. Put a different way, this is the pool tag that is used to map the OS memory used to track shared files. MmSt issues often manifest as Paged Pool depletion (Event ID 2020).
Is there any server-side tweaking that can be done to mitigate some of these effects? Yes. Is there any guarantee that this will resolve the problem completely and indefinitely? No. As an environment continues to scale up, the problem will continue to manifest itself despite all the tweaking that we can do. At some point, the tweaking itself may contribute to the problem because we've reached a point where the server simply cannot handle the workload.
(Many thanks to Rob and Kevin from our CPR team for their technical input!)
- CC Hameed
...I Hope! Well done to the performance team for their recent Windows Server articles, in particular
Windows Server Performance Team started a blog not so long ago... and one of subjects they talked about...
Seems to me that most IT folks are at least dimly aware of this...but what's the option? Installing backup software on every workstation and dealing with open files? Not likely.
This is kind of an "eat your veggies our you'll get fat" message. Sure we all know it's true, but that's not going to make us give up ice cream.
Well, I think this is a good example of what email has become. Semi-useless and a hassle. Every company has distro lists that have all the employees and departments in it that they send announcements to. I can see that being the case a while back, but that kind of stuff needs to be posted to an intranet. Share a file with a coworker? Same thing or in a network share. Want to send a quick note? How about a IM, because that is quick.
You get a better sense if they got message as well, because they will most likely respond quickly or have a status message up. Use email for things like outside communications. That will keep the mailstore smaller. The only thing that might be bad is the mobile factor of chat, that is still limited, once that goes off it is seamless
Great writeup! I've ran into this a few times in my travels and knew it was bad and moved the files back to the local machine, but didn't really delve into the nuts and bolts of what was happening.
I think this is a very valid concern from a performance point of view and I can see what it's not supported. Really useful article.
There also other operational reasons not to have PST files on network shares:
1.) Increase in storage requirements due to loss of Single Instance Storage, lower storage efficiency of PST files over the IS. All you are actually doing is moving the storage headache and not solving it.
2.) Open files on the server that you need to backup
3.) No access to PST file when no connected to the network server unless running offline mode which adds extra issue.
PST files should really only be used to store users personal e-mail and stuff they don't actually care about IMHO. There is still a lot of user education that needs to occur here.
Whilst I'll admit this is a very valid argument and .pst's should not be on a network store, please suggest a viable alternative? Put it on the local machine, cool. Now how are you going to back it up? I completely agree with JeremyinNC.
Technically obvious? Yes.
Funtionally practical? Nope. Many keep their some of their most valued older content on pst. If the HD goes on C:\ (or accidential delete) there's virtually no recovery. Try managing that.
Dale - surely you just backup your exchange storage instead? The PST file is just the local repository of this stuff anyway. The only issue you have then is of Auto-archived files.
I've always known that you're not supposed to use PST files across the network (LAN or WAN), but up until recently I did not have the specific proof as to the magnitude of problems it can cause (outsi ...
It must be nice to have a monopoly and live in a completely unregulated world. In the real world, where the rest of us live, email is, in some cases, both required to be backed up and discoverable in litigation.
It like going to your doctor and saying, "it hurts when I breathe." Your doctor says, "don't breathe then."
You Muppets!!! FIX THE PROBLEM!
Securing RPC Over HTTP Using ISA Server 2006 Survey Reveals Widespread Inadequacies in Email Outage Prevention
At least I now know the technical reasons why it shouldn't be done - i had figured out the bandwidth requirements but it's useful for the technical information.
@keith - no, the pst is not a copy of the mail in the information store - pst's are mail that has been removed from the server and stored locally. OST files are copies of the mail on the server (and this can be recreated if lost)
However, alternative solutions would be nice. For a company that is concerned with keeping down the size of the mailstore due to the time to backup and disk space AND the fact that there is an archive function, it is natural to set users up to archive mail to the c drive and by default, to the worst possible place - hidden away in documents and settings. Not only is it buried deep within a folder structure, any corruption/recreation of the profile means the email is deleted.
The fact that users want a backup of their data in case the hard disk dies, also ensures that email is stored on a network drive - MS say this isn't recommended, but the alternatives of purchasing backup software for every desktop in an enterprise (and maintaining this) is just not practical or cost effective.
@Peter - yes, archiving off personal emails and stuff they don't care is great but users just don't do that either because it's too complicated or they can't be bothered.
As a user - i see disk space as being cheap and therefore no reason to archive mail off - especially as google gives me 2gb of data - why should my IT dept be so mean to only give me 300mb? (OR worse - why don't I just forward my mail to gmail to let them store it all)
As a tech - yes disk space is cheap, but the time taken to backup or restore, or the tape capacity size is just too great to allow more than 300mb per user (for typical largish companies)
One of the possible solutions is to use the pstbackup addin from microsoft at http://tinyurl.com/4s68u which takes a backup copy of pst files to somewhere else on the network for recovery purposes. This worked in 2003/xp but not sure if it works in 2007.
It would be nice if this could be deployed as a policy (or built into outlook) with the autoarchive set to archive to a different location not in the user profile and then have the pstbackup routine dump to a network share that does not get backed up (but that doesn't help for dr purposes if the entire site burns down and no backup has been taken of archived mail)
One of resosultion is use IMAP4 instead POP3 then all mail can be stored on server, even workstation disk will be dowb.
Hoy en cosas interesantes: El video de Bill Gates que aun no habias visto, 25 Videos de SCOM 2007 para