Troubleshooting Event ID 333 Errors

In previous posts we've discussed the basics of memory management including an overview of kernel and user memory, pool resources as well as the /3GB switch.   Continuing our discussion of memory management, we are going to examine an issue that we have been seeing more of on the Performance team - the Event ID 333 message.  As system memory increases, and applications and roles of the server become more important, access to the disk becomes more competitive between applications and the operating system.  As a result, you can start to see the Event ID 333 logged, and you are likely seeing other performance issues including server hangs, pauses, sluggish response times and more.  So today we're going to discuss what the Event ID 333 is, possible causes and common solutions for both 32-bit and 64-bit systems.

The first thing to understand is what exactly an Event ID 333 is.  The event ID 333 is a System event error log that occurs when the registry is unable to complete a flush operation to the disk.  There are several reasons that this can fail and we'll discuss them below.  So, what does the Event ID 333 look like?  In the system event log, you see at least one, and more likely multiple instances of the following:

Event ID 333’s are new to Windows Server 2003 Service Pack 1 and are written to the system event log if the Operating System is not able to flush out or write to the registry hive.  The symptoms that accompany an Event ID 333 can vary between server hangs, “Insufficient resources exist to complete the requested service” errors, SQL Databases stopping and starting, database queries are slow and sluggish operating system performance.  The apparent slow server performance is why the Performance team is engaged on the majority of the issues.  As mentioned previously, the Event ID 333 can occur on both x86 and x64 systems.

Now that we have a little background on what the Event ID 333 is and why it exists, we’re going to take a look at the causes. There are a number of different reasons for why a server is logging the Event ID 333's - the majority of the issues are caused by one of the following:

• The disk hosting the system partition is not keeping up with the load (high disk queue lengths).  This can be determined using a Performance Monitor capture.  This can occur on both x64 and x86 systems.
• There is memory pressure.  In many cases there will also be issues with low Paged (Event 2020) or NonPaged (Event 2019) pool memory or low System PTE's.  Using Performance Monitor can help to determine if you are low on System PTE's.  This is more likely to occur on x86 systems dues to the memory limitations of the 32-bit address space
• A filter driver is keeping the registry from being flushed.  This can occur on both x64 and x86 systems
• A User account granted the “Lock system pages in memory” user right.  This can occur on both x64 and x86 systems

So - how do we go about resolving the Event ID 333 problem?  Depending on the particular scenario that is causing your issue, then you should perform one of the following:

Disk issues:

1. Enable Enhanced Performance Options for the physical disks that host the Operating System itself
2. Update your disk controller firmware and drivers.  Engage your hardware vendor to ensure that you are running the latest supported version and also to verify if there are any known issues.
3. Determine what process is taxing the disk by using Performance Monitor logging.  Examine the Process Object and review each processes I/O Data bytes/sec.  If a process exhibits an unusual amount of I/O compared to your server's baseline performance, then investigate that process.  For non-Microsoft processes, contact the software vendor.

Maximize Kernel Memory and System PTE's:

1. If you are using the /3GB switch on a 32-bit system, do you really need it?  As we discussed in our post on /3GB this switch may not be necessary. 
   • Some software vendors recommend implementing the switch for their applications - verify with them that they have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in their image header.  If they do not, then the application is only able to use the standard 2GB of user-mode memory - but the kernel is now limited to 1GB for no reason. 
   • You can also check the virtual memory of the executable that is believed to take advantage of /3GB.  If the process is not exceeding 1.8GB of Virtual Memory then it is unlikely to be able to take advantage of the extra virtual memory.
   • If the /3GB switch is required (for example Exchange Servers that are configured as Mailbox Servers) then add the /USERVA=2970 switch to give a portion of the User-mode memory to the Kernel to help guard against PTE Depletion.  Microsoft KB Article 316739 discusses the use of the /USERVA switch.
2. Adjust the allocation and use of PagedPool memory by making some registry changes that are outlined in Microsoft KB Article 312362.
3. Disable the Hot Add Memory feature to re-allocate reserved Paged Pool memory back to the Operating System.  When the Hot Add Memory feature is enabled, the operating system pre-allocates kernel resources to handle any future memory that may be added to the computer.  Kernel resources are allocated based on the capabilities of the computer instead of on the RAM that is actually installed.  The kernel may allocate significant resources to RAM that may never be installed.  Therefore, the Hot Add Memory feature may cause the maximum size of the paged pool to be much smaller than expected.

A filter driver is preventing the registry from being flushed:

A quick note on filter drivers - removing or disabling filter drivers without understanding the impact to the system can result in unexpected system behaviors including system hangs or bugchecks.  Examples of common programs that use filter and kernel drivers include Anti-virus software, Backup Software (including Microsoft Volume Shadow Copy) and Version-2 printer drivers.  For information on how to temporarily disable filter drivers, refer to Microsoft KB Article 816071.  Before disabling the filter drivers however, check to see if some of the 3rd party drivers are simply outdated.  The easiest way to check on these 3rd party filter drivers is to use our MPS Reporting utility to capture this information.

1. Download the MPSRPT_SETUPPerf.exe file from the Microsoft Product Support Reporting Tools page to the local machine.
2. Run the MPS Reports executable
3. Once the report has finished, you can extract the PStat.txt file from the .CAB file that is created and review the list of 3rd party filter / kernel drivers to see which ones may be outdated.  Contact the vendor for the filter driver to get the latest supported version and also to discuss any potential issues with the driver.

"Lock Pages in Memory" user right:

On x64 systems, the likelihood of a server running out of kernel memory or System PTE's is far less than on an x86 system.  However, we have seen the Event 333 error occur on x64 systems as well.  Using the "!vm" debugger command when reviewing a memory dump of the server may indicate that the server is low on physical memory, even though performance monitor data indicates that there is plenty of available memory.  The sample output below illustrates this:

15: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 2095394 ( 8381576 Kb)
Page File: \??\C:\pagefile.sys
Current: 4249600 Kb Free Space: 4154172 Kb
Minimum: 4249600 Kb Maximum: 4249600 Kb
Available Pages: 868200 ( 3472800 Kb)
ResAvail Pages: 250 ( 1000 Kb)

********** Running out of physical memory **********

To work around this behavior on x64 platforms or on servers with 4GB or less of physical RAM use the following steps:

1. Click on Start --> Run
2. Type Secpol.msc and click OK
3. Expand "Local Policies" then click on "User Rights Assignment"
4. Double click on "Lock pages in memory"
5. Highlight any user accounts listed and click on "Remove"
6. Click OK after all accounts have been removed
7. Reboot the system

NOTE: By default, the Windows Operating system does not grant this user right to any accounts.  The “Lock pages in memory” right is granted to the account used for SQL Services by the SQL 2005 RTM/SP1 Enterprise Edition install on 32bit systems.  If you are using SQL Enterprise on 32-bit servers with more than 4GB of RAM, the Lock Pages in Memory right is needed.  To help reduce the occurrence of the Event ID 333's on these systems, ensure the user account you are using for the SQL services is only used for SQL. Check the access right for “Lock pages in memory” and only list the account used for SQL.  If System, or any other accounts are listed, remove them.  For x64 systems, remove all accounts listed.

Well that does it for this post.  As you can see the Event ID 333 has several causes and is usually one of many symptoms seen on servers experiencing performance problems.  By troubleshooting and eliminating Event ID 333’s, the overall performance of the server should improve.  Keep an eye on the Microsoft Knowledgebase - in the coming weeks, the majority of this information should be available in a KB Article.

- Aurthur Anderson