Thoughts from the EPS Windows Server Performance Team
At least once a week, someone on the Performance team will get a customer call concerning hangs or resource depletion on their file server. The file server in question is used for user home folder storage and users are accessing Outlook Personal Storage (.pst) files stored on the server from their client. The issue will manifest as either a server hang, or PagedPool depletion (Event ID 2020). Oftentimes the issue will occur first thing in the morning - when users are logging on and launching Outlook. In especially severe cases, the issue occurs several times daily. Sometimes the server will hang for a few minutes and then continue operating for a few minutes - and then hang again. Rinse & repeat. The users are frustrated because of slow access to their data, the server administrators are frustrated because they are tasked with fixing the problem, and upper management is frustrated because everyone else is frustrated.
If you're in this situation - there's good news ... and very bad news. The good news is that this problem is very common and is a known issue. The very bad news (from the customer's standpoint) is that PST files on a LAN/WAN is an unsupported configuration. Some customers are very surprised to hear this but Network Stored PST files have been unsupported since the days of Exchange 4.0. Microsoft KB Article 297019 goes into some detail about the effects of Network PST files:
"A .pst file is a file-access-driven method of message storage. File-access-driven means that the computer uses special file access commands that the operating system provides to read and write data to the file.
This is not efficient on WAN or LAN links because WAN/LAN links use network-access-driven methods, commands the operating system provides to send data to or receive from another networked computer. If there is a remote .pst (over a network link), Microsoft Outlook tries to use the file commands to read from the file or write to the file, but the operating system then has to send those commands over the network because the file is not on the local computer. This creates a great deal of overhead and increases the time it takes to read and write to the file. Additionally, the use of a .pst file over a network connection may result in a corrupted .pst file if the connection degrades or fails."
Let's use an example to illustrate the problem and also follow the problem through to its end result.
Let's say that a user sends an e-mail message to 500 users within the company. All of these users have their e-mail delivered directly to their PST file which is stored on the File Server. Some of these 500 users may need to extend their PST files to receive it. To extend a PST, an extra allocation on disk has to be made via NTFS. This locks out the whole volume while free space is allocated and the Master File Table (MFT) is updated. While this is happening for each user, all I/O for the other 499 users is on hold.
Allocating free space can take an extended time, especially if the disk is fragmented. Now factor in multiple users extending their PST's in the same timeframe, and significant periods of MFT lockout might be observed, which in turn is seen as inability to access any other file on the volume, resulting in queueing in the server service work queues, and sometimes SRV 2019, 2020, 2021, or 2022 events being logged. This scenario might overload the disk(s).
Setting aside the example of one email being sent to a group of users, imagine if you had a couple of hundred users who each have two or three PST files. These users have been with the company for a while, and they rarely (if ever!) delete their email from their PST files. The files continue to grow in size - let's use an average of 1 GB as the size of the PST file. Now consider that when each user launches Outlook, they make a request for two (or three) files, each of them being about 1 GB in size. Then consider what happens when 200 users all launch Outlook around the same time when they get to work. 200 x 3 x 1 = 600 GB of data being requested at the same time. That's an awful lot of Disk & Network I/O to process simultaneously. This is a very common scenario - the file server "freezing" for a few minutes at a time while it tries to service these requests.
The queuing in the server service work queues is what causes this temporary hang. The server service uses work items to handle I/O requests that come in over the network - for example: a request to extend a PST file. These work items are queued in the server service work queues, and from there they are handled by the server service worker threads. The work items are allocated from a kernel resource called Non-Paged Pool (NPP).
The server service sends these I/O requests down to the disk subsystem. If, for reasons mentioned above, the disk subsystem does not respond in time, the incoming I/O requests are queued via work items in the server work queues. Since these work items are allocated from NPP, eventually this resource runs empty. Running out of NPP causes systems to hang eventually (logging an Event ID 2019 in the process).
Digging down into this from more of a troubleshooting perspective, we can usually see issues caused by the PST files manifested in Poolmon and Perfmon captures. For example, we may see the LSwn pool tag allocation climbing in a Poolmon trace. These allocations are made by SRV.SYS. The size of the allocation is configurable via the SizReqBuf registry value. One allocation is made for each work item used by the server service. When looking at this through Perfmon, you will notice a steady decrease in the "Available Work Items" counter. If Available Work Items reaches zero, then clients may experience difficulties accessing files (any files, not just the PST files!). You may also experience 2019 errors if the problem lies with LSwn allocations (Non-Paged Pool depletion)
Another tag that highlights the issues with the PST files is the MmSt tag. This tag represents Mm section object prototype PTEs - a memory management-related structure used for mapped files. Put a different way, this is the pool tag that is used to map the OS memory used to track shared files. MmSt issues often manifest as Paged Pool depletion (Event ID 2020).
Is there any server-side tweaking that can be done to mitigate some of these effects? Yes. Is there any guarantee that this will resolve the problem completely and indefinitely? No. As an environment continues to scale up, the problem will continue to manifest itself despite all the tweaking that we can do. At some point, the tweaking itself may contribute to the problem because we've reached a point where the server simply cannot handle the workload.
(Many thanks to Rob and Kevin from our CPR team for their technical input!)
- CC Hameed
I'd be very interested to know whether people have found a 64 bit implementation, with its much larger paged & non-paged pools, solves the problem of them running out.
We had our file system living fairly happily on a six year old Windows 2000 cluster, which included about 2,000 PSTs spread over three volumes. We have now begun a migration of the system to a new much faster (dual quad core, 8GB RAM) Windows 2003 cluster, but once the three PST volumes were moved over we found that when it's all on one node the system quickly runs out of non-paged memory (something that doesn't happen on the old system). Spreading the PSTs across the cluster (so each has about 1,000) has made it stable, but of course it's not a redundant cluster like that.
I wish it was the paged pool running out as there seems to be quite a lot you can do to tweak it, but with the non-paged there seems to be nothing other than not using /3GB.
I understand Microsoft doesn't like PSTs over the network, but I think there's a couple more fundamental things that need addressing:
Firstly, noting that other commenters have been in this situation, what is the difference between Windows 2000 and 2003 memory management that means the former will work and the latter doesn't?
It's one thing when something has never worked, but to implement a solution that is effectively 'the same, but faster' and find that it can't handle something the slower one could is particularly nasty and not something it's actually terribly easy to plan for.
Secondly, as many have noted, we administrators have to deal with the world as we find it. I personally inherited the situation in which I find myself and cannot wave a wand to make 2,000 PSTs disappear and am in a position whereby the supposed performance enhancement has actually made it worse, something managers dislike a great deal. With that in mind a response from Microsoft that helps us into a better position so that we can get rid of PSTs in a business-friendly way rather than the current blunt answer would be very helpful.
Avoid the use of pst files. If you needs to use pst files, then save it locally in the workstation. Use a network backup solution like Arcerve Backup in order to backup this files.
Another solution is to have Email Archiving. Email archiving have several benefits:
- central repository
- easy to backup
- small storage sizes
- quick indexing searchs
- legal compilance
- you avoid internal users trying to steal company email
- integration with outlook (or other email clients)
- support for VLD (very large databases)
You can use EAS; Veritas or any other solution you can afford. With email archive solution the PST issue are things from the past.
Let me tell you I keep some old old pst files in a Buffalo Terastation radi 5 with XFS and works very well with 10 users accesing pst files concurrently. Maybe XFS is better than NTFS for pst storage, remember XFS don`t need to be defragmented and have different access routines.
MCP MCSA MCSE MCTS A+ CST HPSAN HPCZ ETRUST
so what are the alternative for saving and storing emails?
After we moved PST files to workstations, as Microsoft recommends, we implemented the EdgeSafe backup solution that was mentioned in a former post.
The server related problems have been obviously solved. Plus we have gained the ability do "image" recovery of PST files and at the same time have the ability to recover deleted items.
I never saw an answer to storing the PST file on a Windows Home Server.
We are a small (two people) office using Outlook 2003. One computer uses Outlook with the PST on WHS. The other computer uses OLFolders to access Outlook PST on WHS. This allows complete sharing of the PST info. Everything seems to work fine.
Question: Sometimes, mail from a Yahoo.com account fails to arrive via POP3 but the ISP says it was delivered to our mail box. Is this a familiar problem to anyone?
Jim - Accessing .PST files stored on Windows Home Server is unsupported (http://support.microsoft.com/kb/955690).
Very interesting thread - again like most IT staff - this one snuck up and caught us. We have about 100 users with up to 1.2gb .pst files
these .pst files all lived happily on a Winnt server in the users work areas.
We migrated up to Windows 2003 and now this bug bites us hard although randomly and haphazardly every day.
I know some have asked - but no answers have been forthcoming .
Why is it that Windows NT or Windows 2000 eats this sort of access for breakfast and yet - Windows 2003 spits and coughs and cant swallow.
For mine - A huge backward step from Microsoft - now where is my Windows Server 2000 disk
I suspect that over the course of time, various tuning settings were applied on your old server(s) to the server service, and memory management areas - so it may appear that Windows 2003 is less reliable, when in fact it just hasn't been tuned. Take a look at the KB articles referenced at the end of the post - they have some tuning settings.
However - the underlying message remains the same - although you can tweak and tweak the server to allow for this sort of file access, at some point you run out of tweaks ...
This might be a good time to start talking about cleanup and getting rid of those .PST files.
thanks for your response - so I tend to disagree
To be honest the NT server was a bog standard NT Server 4.0 install with Service Pack 6 applied - t did nothing but serve data storage - I built it years ago with no tweaks no special adjustments - nothing out of normal
The Windows 2003 machine has 20 times the disk storage - 4 times the memory - 4 times the number of CPU's - much faster CPU's at that
Win2003 server must be using more resources OR allocating less resources - At this stage I am seeking none of the listed telltale events in the event log
There are a lot of suggestions for buying some archive solution or add storage and keep everything in the exchange store. Which would be fine if I had the resources. But since I don't or won't for some time I have to live within my means which means pst files. And keeping them on the local drive is a terrible solution in my opinion. You have to back them up somehow and doing it from the desktop is adding more complexity which violates the KISS rule of which I believe in wholeheartedly. I am going to tweak the page pool settings and see what happens. I will let this post know how it goes.
so how can put email info on a server
my email server is a comercial one.
I dont have exchange so how can i put the email info on a network share
This is a very interesting article. I also have situation which is giving me headache. For about 4 months now, I’m working for a company with 4 locations. All the locations have an Exchange 2007 running on a windows 2003 (64) servers. I was surprised with the fact that about 75% of the employees have a mailbox size between 5 GB and 17 GB on the server. They are working with office 2007 and because they are mobile workers they have also an ost file on their laptop. Further they have a pst file for archive purpose which must be available every time of the day. These pst file are located on the network and they are growing. The company exists of 180 employees. As you see I need a solution to reduce the size of the mailboxes without decreasing the functionality they have now. Has anyone any ideas for me.
Large exchange stores = large OST files (cached exchange mode) = frequent OST file corruption over slow links or dial-up VPN connections. I cant win. However, I am moving away from PST files in part because of the lockups described in this article.
I believe that some of the reasons why the old win2k/WinNT servers dont exhibit the same issues is due to the number of extra services that 2k3 is running.
For example, look at Volume Shadow Copy services. If you take snapshots then you will most likely see your 2020 and 2019 event IDs happen near the snapshot times. If you use backup software that makes use of volume shadow copy then you can see similar issues. Just watch the poolmon during snapshots or when backup software is running for some scary data.
Then take that and throw in things like user quotas, bloated antivirus software, and tracking of larger volumes. I only started to see this issue after updating my antivirus software... Of course there is no such thing as a simple virus scanner anymore. Its all 'security solutions' that eat up resources and make my new server slower than my old servers.
I think I really need to migrate my storage to 64 bit host servers. Unfortunately thats not in the cards for several years.
On a side note... Despite my problemsI love love love volume shadow copy. Even with the extra resources it uses I wouldn't give it up after using it for the last two years. I've shown so many users the 'previous versions' capability and my calls for tape backup restores have gone to almost zero. In my mind VSS makes up for having to create a dedicated PST server.
We have Outlook mail client with PST files for the last 11+ years, serving our email infrastructure very well. It is true that once in a while we had a corrupt PST, but with 7,000 users and a total 12 TBytes of data we did not want to change the way we work, nor betting on another database solution consolidating so much data.
So we decided to take another small step and secure our PST file, so if a failure occurs we will be able to recover the PST file. After searching for alternatives we decided to simply backup the PST file, and used PST2PST backup which is centrally managed and runs the backup seamlessly. With 7,000 members that is the key point for us.
The notebook users with their PST files are now secured, and PST files are gradually being moved from the network to workstations.
Interesting article but you could have suggested some alternatives. Obviously users cannot store these files locally, as they contain their archived correspondence for the last few years. Clearly these also can't be imported back into Exchange. Using your scenario as an example, 200 users with 3 1Gb PST files each = 600Gb - and what about mailbox limits - these will also go out the window? Archiving solutions are expensive .... can we use another exchange back end server with archive mailboxes somehow?