Thoughts from the EPS Windows Server Performance Team
At least once a week, someone on the Performance team will get a customer call concerning hangs or resource depletion on their file server. The file server in question is used for user home folder storage and users are accessing Outlook Personal Storage (.pst) files stored on the server from their client. The issue will manifest as either a server hang, or PagedPool depletion (Event ID 2020). Oftentimes the issue will occur first thing in the morning - when users are logging on and launching Outlook. In especially severe cases, the issue occurs several times daily. Sometimes the server will hang for a few minutes and then continue operating for a few minutes - and then hang again. Rinse & repeat. The users are frustrated because of slow access to their data, the server administrators are frustrated because they are tasked with fixing the problem, and upper management is frustrated because everyone else is frustrated.
If you're in this situation - there's good news ... and very bad news. The good news is that this problem is very common and is a known issue. The very bad news (from the customer's standpoint) is that PST files on a LAN/WAN is an unsupported configuration. Some customers are very surprised to hear this but Network Stored PST files have been unsupported since the days of Exchange 4.0. Microsoft KB Article 297019 goes into some detail about the effects of Network PST files:
"A .pst file is a file-access-driven method of message storage. File-access-driven means that the computer uses special file access commands that the operating system provides to read and write data to the file.
This is not efficient on WAN or LAN links because WAN/LAN links use network-access-driven methods, commands the operating system provides to send data to or receive from another networked computer. If there is a remote .pst (over a network link), Microsoft Outlook tries to use the file commands to read from the file or write to the file, but the operating system then has to send those commands over the network because the file is not on the local computer. This creates a great deal of overhead and increases the time it takes to read and write to the file. Additionally, the use of a .pst file over a network connection may result in a corrupted .pst file if the connection degrades or fails."
Let's use an example to illustrate the problem and also follow the problem through to its end result.
Let's say that a user sends an e-mail message to 500 users within the company. All of these users have their e-mail delivered directly to their PST file which is stored on the File Server. Some of these 500 users may need to extend their PST files to receive it. To extend a PST, an extra allocation on disk has to be made via NTFS. This locks out the whole volume while free space is allocated and the Master File Table (MFT) is updated. While this is happening for each user, all I/O for the other 499 users is on hold.
Allocating free space can take an extended time, especially if the disk is fragmented. Now factor in multiple users extending their PST's in the same timeframe, and significant periods of MFT lockout might be observed, which in turn is seen as inability to access any other file on the volume, resulting in queueing in the server service work queues, and sometimes SRV 2019, 2020, 2021, or 2022 events being logged. This scenario might overload the disk(s).
Setting aside the example of one email being sent to a group of users, imagine if you had a couple of hundred users who each have two or three PST files. These users have been with the company for a while, and they rarely (if ever!) delete their email from their PST files. The files continue to grow in size - let's use an average of 1 GB as the size of the PST file. Now consider that when each user launches Outlook, they make a request for two (or three) files, each of them being about 1 GB in size. Then consider what happens when 200 users all launch Outlook around the same time when they get to work. 200 x 3 x 1 = 600 GB of data being requested at the same time. That's an awful lot of Disk & Network I/O to process simultaneously. This is a very common scenario - the file server "freezing" for a few minutes at a time while it tries to service these requests.
The queuing in the server service work queues is what causes this temporary hang. The server service uses work items to handle I/O requests that come in over the network - for example: a request to extend a PST file. These work items are queued in the server service work queues, and from there they are handled by the server service worker threads. The work items are allocated from a kernel resource called Non-Paged Pool (NPP).
The server service sends these I/O requests down to the disk subsystem. If, for reasons mentioned above, the disk subsystem does not respond in time, the incoming I/O requests are queued via work items in the server work queues. Since these work items are allocated from NPP, eventually this resource runs empty. Running out of NPP causes systems to hang eventually (logging an Event ID 2019 in the process).
Digging down into this from more of a troubleshooting perspective, we can usually see issues caused by the PST files manifested in Poolmon and Perfmon captures. For example, we may see the LSwn pool tag allocation climbing in a Poolmon trace. These allocations are made by SRV.SYS. The size of the allocation is configurable via the SizReqBuf registry value. One allocation is made for each work item used by the server service. When looking at this through Perfmon, you will notice a steady decrease in the "Available Work Items" counter. If Available Work Items reaches zero, then clients may experience difficulties accessing files (any files, not just the PST files!). You may also experience 2019 errors if the problem lies with LSwn allocations (Non-Paged Pool depletion)
Another tag that highlights the issues with the PST files is the MmSt tag. This tag represents Mm section object prototype PTEs - a memory management-related structure used for mapped files. Put a different way, this is the pool tag that is used to map the OS memory used to track shared files. MmSt issues often manifest as Paged Pool depletion (Event ID 2020).
Is there any server-side tweaking that can be done to mitigate some of these effects? Yes. Is there any guarantee that this will resolve the problem completely and indefinitely? No. As an environment continues to scale up, the problem will continue to manifest itself despite all the tweaking that we can do. At some point, the tweaking itself may contribute to the problem because we've reached a point where the server simply cannot handle the workload.
(Many thanks to Rob and Kevin from our CPR team for their technical input!)
Additional Resources:
- CC Hameed
Can someone please tell me how to recover the Network PST's and put them back on the local HD?
We want to take approx. 20 users network PST's and put them back on their laptops and use the PST backup utility but I want to do it right.
Thanks!!
Sandy,
For your e-mail move to disk, see:
http://office.microsoft.com/en-us/outlook/HA010875321033.aspx
The Personal Folders Backup tool is designed for use in Outlook 2002 and later and the operating systems that support each respective Outlook version. The tool provides a quick and easy way to back up the Outlook information of your choice to your hard disk or network server or share.
After you back up your information, you can copy these duplicates of your Outlook data to a removable media such as a CD or DVD. The backup files are exact copies of the original files and are saved in the same file format. You can receive periodic reminders to back up your files.
The PST backup tool is at the link above.
Mark
My question is how did the people who "have known this all along" learn it? This post only came out a year ago, and we've been putting our client's PST files on their network share for close to 5 years. We did/have done it because there is a severe quota limit on the Exchange server, and we have no sway (let alone control) over its configuration - only the file/print servers.
How could we have know this was a bad idea early enough to not only not not recommend it, but prevent users from doing it themselves? Are we the only server administrators who don't go perusing through random KB articles in our "free time", but rather tend to wait until a problem arises to search for this stuff?
If PSTs on the network is so bad, why does MS allow it? They prevented OE local folders from residing on the network with version 6, why didn't they do that with Outlook? At the very least, they could have popped up a warning about it.
I have to agree with Alexandre here. How would the server problem be any different if a large number of Access databases residing on a network share were being worked on at the same time? Why is it that we can't increase the Paged Pool memory? Our server processor load never goes above 15%, so our problem seems solely in this rather artificial limit.
We have users who need access to their "local" mail files from multiple machines in different locations (not at the same time), so truly local (C: drive) PST files aren't really feasible. As I said above, we CAN'T increase their Exchange quota.
Do any of the know-it-alls here have a realistic solution for our situation?
Even without much knowledge/experience in it, we're considering trying to set up an IMAP server that is only for online storage and is restricted in sending/receiving mail. Every user would need to then have another account, but many of our problems would go away (maybe to be replaced with others?). Does anyone have any comments on this idea??
Casper - although the post on our blog is only a year old, the original KB (297019) has been public since 2001. The article has been updated over time to include the newer versions of Outlook as they have been released. The PST files were created back in 1990 by the Exchange Server 4.0 team with the intent of letting a person store a copy of messages on their local machine. They also serve as a local message store for users who do not have access to an Exchange server - for example those who configure Outlook as their ISP Mail client.
With respect to the situation that you are running into - a vaulting solution or other sort of archival solution would seem to be a good solution as you indicate that there is a "severe quota limit" on the Exchange server. Being able to access that archive from a web browser addresses the issue of being able to get to the mail from any machine that can access that site and more importantly it allows the administrators to centrally manage and adhere to email & document retention policies.
- CC
One simple solution that is now possible from datamills is to locate for each user the PST file on his/her workstation without losing central control.
Placing the PST on users’ notebooks/desktops reduces the daily network traffic and server load dramatically, as there is no need for Outlook to open and search PSTs across the network.
Then for corporate backup/archive it is possible to use the new PST2PST incremental backup/archive (http://www.datamills.com) to incrementally store new mail onto central organization storage.
This is also a great solution for traveling users, or for disaster situations as users have all their mail data on their notebook regardless of Exchange.
Dave Green
After all these posts and one year of commenting, can anyone tell me this:
(15 computers use Outlook .pst files, each over 4GB; Network is Ethernet 100MB/s)
Is this safe to move .pst files to some Linux file server? I am not asking for 100% safety! It is not possible after all. And will that slow down network little or much?
So: Yes or No?
Thanx a lot for ur answers.
One word: Sharepoint.
Sharepoint requires a PST file be used for access via Outlook. Why would Microsoft use PST with a Sharepoint implementation that's supposed to be used over a LAN/WAN situation is beyond me. After reading all the information here, I'll have to move the PST files to my Terminal Server, because I can't get to Sharepoint without using PST in Outlook.
I found the solution. During more than 6 years I used files .PST in a volume Novell Netware and I never had problems. On this year we decided to migrate for Microsoft File Server and.... @ #$% ¨&¨ $ Therefore the solution is to put PSTs in one File server Novell.
Malden,
For 15 users with 4GB you are looking at slow performance for both Outlook and network.
It makes much more sense to leave the PSTs on the desktops and to look for "incremental PST backup" software to daily backup the PSTs onto the Linux server.
I guess that your daily changes are less than 40MB which is 0.1% of the 4GB PST, meaning with PSTs on the desktops and incremental PST backup you can expect your network to operate smoothly even if your PSTs and number of user grows x100 over.
Preston
I just started working with the PST files, and it keep saying that its not a Win32 application. Why do I do?
Surely Cached mode (.OST) would resolve the network load issue as I presume that cached mode would predominantly work with changes and not the whole pst file? I guess that this may be why cached mode was introduced?
Indeed some of the searching facilities need you to be in cached mode; cached mode also allows you to take public folder information offline for laptops users too (great!). (Need to add a favourites link to the public folders you want to have access to offline).
Unlike PST, OST is not a solution to solve email storage space. It's merely a solution to synchronize a local storage with mailbox. And if the mailbox is limited in size so is the OST.
PST is still the only viable solution that scales very well for enterprise. With Outlook 2003, each PST can be virtually unlimited in size (unlike Outlook 2000 with 2GB limit, the limit is virtual at several terra bytes). Keeping PSTs on desktop is the key to speed and scalability. No other solution can give this kind of scalability and agility.
Is there still apossibillity to open pst files if located on a local network (LAN) or WAN net work
For recover pst file-<a href="http://www.recoverytoolbox.com/pst_viewer.html">pst viewer</a>,All of your contacts, mails, messages, tasks and calendars are stored on mail server and there's no way to extract it offline,will help you to restore your data from files with *.pst and *.ost extension,tool will work under all supported versions of Microsoft Windows operating system, as well as with Microsoft Outlook,pst viewer can retrieve all contents as a number of files in *.vcf, *.txt and *.eml formats.
Once someone implement a company wide .pst solution across the network it's uncanny how the implementation can turn your IT operations department upside down and riddle it with fallout ranging from backup, rdp, and overall visibility issues that are totally unsolvable. If you can prevent .pst files from being saved and update across the network it would be your best bet in maintaining your department’s sanity. One such solution offered up to us was to get a GPO in place to prevent .pst from being saved on network shares.