Thoughts from the EPS Windows Server Performance Team
Useful Microsoft Blogs
MEMORY MANAGEMENT – EVENT ID 333
Description: The first thing to understand is what exactly an Event ID 333 is. The event ID 333 is a System event error log that occurs when the registry is unable to complete a flush operation to the disk. There are several reasons that this can fail and we'll discuss them below. So, what does the Event ID 333 look like? In the system event log, you see at least one, and more likely multiple instances of the following:
Event ID 333’s are new to Windows Server 2003 Service Pack 1 and are written to the system event log if the Operating System is not able to flush out or write to the registry hive. The symptoms that accompany an Event ID 333 can vary between server hangs, “Insufficient resources exist to complete the requested service” errors, SQL Databases stopping and starting, database queries are slow and sluggish operating system performance. The apparent slow server performance is the reason why the Performance team is engaged on the majority of the issues. As mentioned previously, the Event ID 333 can occur on both x86 and x64 systems.
Now that we have a little background on what the Event ID 333 is and why it exists, we’re going to take a look at the causes.
Scoping the Issue:
There are a number of different reasons for why a server is logging the Event ID 333's - the majority of the issues are caused by one of the following:
Data Gathering: In all instances, collecting either MPS Reports with the General, Internet and Networking, Business Networks and Server Components diagnostics, or a Performance-oriented MSDT manifest must be done. Additional data required may include the following:
Troubleshooting / Resolution: After you have gathered this data, review the following:
Maximize Kernel Memory and System PTE's:
A filter driver is preventing the registry from being flushed:
A quick note on filter drivers - removing or disabling filter drivers without understanding the impact to the system can result in unexpected system behaviors including system hangs or bugchecks. Examples of common programs that use filter and kernel drivers include Anti-virus software, Backup Software (including Microsoft Volume Shadow Copy) and Version-2 printer drivers. For information on how to temporarily disable filter drivers, refer to Microsoft KB Article 816071. Before disabling the filter drivers however, check to see if some of the 3rd party drivers are simply outdated. The easiest way to check on these 3rd party filter drivers is to use our MPS Reporting utility to capture this information.
"Lock Pages in Memory" user right:
On x64 systems, the likelihood of a server running out of kernel memory or System PTE's is far less than on an x86 system. However, we have seen the Event 333 error occur on x64 systems as well. Using the "!vm" debugger command when reviewing a memory dump of the server may indicate that the server is low on physical memory, even though performance monitor data indicates that there is plenty of available memory. The sample output below illustrates this:
15: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 2095394 ( 8381576 Kb)
Page File: \??\C:\pagefile.sys
Current: 4249600 Kb Free Space: 4154172 Kb
Minimum: 4249600 Kb Maximum: 4249600 Kb
Available Pages: 868200 ( 3472800 Kb)
ResAvail Pages: 250 ( 1000 Kb)
********** Running out of physical memory **********
To work around this behavior on x64 platforms or on servers with 4GB or less of physical RAM use the following steps:
NOTE: By default, the Windows Operating system does not grant this user right to any accounts. The “Lock pages in memory” right is granted to the account used for SQL Services by the SQL 2005 RTM/SP1 Enterprise Edition install on 32bit systems. If you are using SQL Enterprise on 32-bit servers with more than 4GB of RAM, the Lock Pages in Memory right is needed. To help reduce the occurrence of the Event ID 333's on these systems, ensure the user account you are using for the SQL services is only used for SQL. Check the access right for “Lock pages in memory” and only list the account used for SQL. If System, or any other accounts are listed, remove them. For x64 systems, remove all accounts listed.