APPLIES TO

• 32-bit (x86) Windows Server 2003 

Important This blog contains information about how to modify the registry. Make sure to back up the registry before you modify it. Make sure that you know how to restore the registry if a problem occurs. For more information about how to back up, restore, and modify the registry, click the following article number to view the article in the Microsoft Knowledge Base:
256986 Description of the Microsoft Windows registry
http://support.microsoft.com/?id=256986

Problem description:
-----------------------

When you use a Microsoft 32-bit Windows Server 2003-based computer, you might experience one or more of the following symptoms:

- Server hangs/unresponsive. 

**** Note:  There are many different root causes for this behavior.

-"Delayed Write Failed"

** Note:  There are at least 2 different root causes for this behavior.

-Blank events in the Event Viewer.

** Note:  There are at least 2 different root causes for this behavior.

-Task Manager shows up blank.  The tabs of Applications, Processes, and Performance didn’t have any information.

-You might not be able to RDP

**** Note:  There are many different root causes for this behavior.

- “Insufficient system resources exist to complete the requested service.”

**** Note:  There are many different root causes for this behavior.

- You might not be able to logon to the system after adding the /3gb switch in the boot.ini

-Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: MM/DD/YYYY
Time: HH:MM:SS AM
User: N/A
Computer:
Description:
Application popup: ApplicationName.exe - Application Error: The application failed to
initialize properly (0xc0000142). Click on OK to terminate the application.

Note: Where ApplicationName.exe could be anything such as cmd.exe.

*** Note:  There are at least 3 different root causes for this behavior.

-Event Type: Warning
Event Source: PerfOS
Event Category: None
Event ID: 2012
Date: MM/DD/YYYY
Time: HH:MM:SS AM
User: N/A
Computer: ServerName
Description:
Unable to get system process information from system. Status code returned is data
DWORD 0.

**** Note:  There are many different root causes for this behavior.

-Event Type: Information
Event Source: dmio
Event Category: None
Event ID: 30
Date: MM/DD/YYYY
Time: HH:MM:SS AM
User: N/A
Computer: ServerName
Description:
dmio: Harddisk1 write error at block *********: status 0xc000009a

    Note:  0xc000009a = STATUS_INSUFFICIENT_RESOURCES

** Note:  There are at least 2 different root causes for this behavior.

Cause(s):
--------------

1.  System PTEs are set too low to start out with. 

2.  32-bit Windows Server 2003 running with /3GB

3.  PagedPoolSize being set too high

4.  High end video cards on the servers.

5.  Hot add memory (DynamicMemory)

6.  Microsoft drivers

7.  Third party (3rd party) drivers that consume a lot of System PTEs

Tools:

------

1. Perfmon (built-in to the O.S.)
For those that are new to perfmon, the Performance Analysis of Logs (PAL) Tool might make your life a little easier.
http://www.codeplex.com/PAL

Note:  If you are on Windows Server 2003 RTM:

894067 The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003
http://support.microsoft.com/?id=894067

What to look for:

Memory – Free System Page Table Entries 

- Max value right after a reboot.

A server should have at least 15,000 system PTEs available right after a reboot.

- Min value after the system has been up for a few days, depending how often the problem occurs.

A server should have at least 5,000 system PTEs available at all times.

2. Kernel memory dump.

972110 How to generate a kernel dump file or a complete memory dump file in Windows Server 2003
http://support.microsoft.com/?id=972110

969028 How to generate a kernel or a complete memory dump file in Windows Server 2008
http://support.microsoft.com/?id=969028

3.  Debugging Tools for Windows

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

4. Windows Server 2003 RTM hotfixes:
-831899 You receive a "stop 0x000000DA" error message on a blue screen after you enable PTE tracking on a Windows Server 2003-based node
http://support.microsoft.com/?id=831899

-833905 Your Windows Server 2003-based server stops responding if a mirrored dynamic disk volume uses two hard disks that are connected to two different hard disk controllers
http://support.microsoft.com/?id=833905

Windows Server 2003 Service Pack 1 hotfixes:
None

Windows Server 2003 Service Pack 2 hotfixes:
None

If you are running Symantec (Veritas) Backups (BackupExec or NetBackup)

955033 Stop 0x0000003F or STOP 0x000000D8 occurs on Windows Server 2003 computer with Veritas Backup installed
http://support.microsoft.com/?id=955033

Troubleshooting:
-------------------

Proactive action plan (to try preventing the problem):
------------------------------------------------------

1.  Check to make sure that the registry key is set to the default.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

SystemPages (dword) 0 (Hex) (default)

Note:  You might have servers with newer video cards, set this with 0x000c3000 (hex).  This is an old value that is not applicable for Windows Server 2003.

2.  If you are using the /3gb switch you should be using the /USERVA=3030 or /USERVA=2920 in the boot.ini.

3. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

PagedPoolSize (dword) ______ (default is 0 (hex)) will take away from Free System PTE’s (SystemPages).  You have to have a fine balancing act between the two on 32-bit Windows Server. 

4. On servers except for Terminal Servers running in Application Mode, you really don’t need high end graphics.

In the boot.ini, add /BaseVideo

In Device Manager, go to Display adapters, change the video card to “Standard VGA Graphics Adapter”.

5.  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

DynamicMemory (dword) 1 (hex)

6. Microsoft drivers

i.e.  If you have a HBA and the hba driver uses Storport.sys, i.e.

932755 An updated Storport storage driver (version 5.2.3790.4021) is available for
Windows Server 2003
http://support.microsoft.com/?id=932755

7. Check with your OEM hardware vendor for updated drivers/firmware's.

8.  Check the version of termdd.sys in c:\windows\system32\drivers

Make sure it’s the version for Windows Server 2003 Service Pack 2, there have been some customers where the upgrade to SP2 from SP1 didn’t update this driver.

9.  If those items above do not help, you will need to grab a kernel only memory dump.

WARNING:  I cannot emphasize enough that the items above #1-#4 should be taken care of before proceeding with this step.  We have had numerous Administrators trying to troubleshoot a driver issue when in reality the system was just not booting up with enough Free System PTE’s.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory
Management
TrackPtes (DWORD) 1 (Hex)

Caution:  You could run into the following problem by enabling this on Virtual PC 2004:

837339 "Stop 0x000000DA" error occurs on a host computer after you start a virtual machine in Virtual PC 2004
http://support.microsoft.com/?id=837339

!vm 1

// You are going to be looking for warnings like the one below:

********** Running out of system PTEs **************
******* 202 system PTE allocations have failed ******

Note:  If your server has less than 505 Free System PTE’s, more than likely your system will hang.

!pte

!sysptes

!sysptes 4

More information:

The third-party products that this blog discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

References:

247904 How to Configure the Paged Address Pool and System Page Table Entry Memory Areas
http://support.microsoft.com/?id=247904

256004 How to troubleshoot "STOP 0x0000003F" and "STOP 0x000000D8" error messages in Windows Server 2003, Windows XP, and Windows 2000
http://support.microsoft.com/?id=256004

311901 The effects of 4GT tuning on system Page Table Entries
http://support.microsoft.com/?id=311901

316739 How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
http://support.microsoft.com/?id=316739

Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=ed0e8084-abf7-4c00-ba6a-7d658cdb052a

What Are These System PTEs?
http://msdn.microsoft.com/en-us/magazine/dd420461.aspx

Bug Check 0x3F: NO_MORE_SYSTEM_PTES
http://msdn.microsoft.com/en-us/library/ms793941.aspx

Bug Check 0xD8: DRIVER_USED_EXCESSIVE_PTES
http://msdn.microsoft.com/en-us/library/ms795903.aspx

SystemPages set too high
http://technet.microsoft.com/en-us/library/aa996786.aspx

!sysptes
http://msdn.microsoft.com/en-us/library/cc267106.aspx

Additionally the two books that Mark Russinovich and David Solomon co-wrote have good information regarding the System PTEs:
Windows Internals 4th Edition http://www.microsoft.com/learning/en/us/Book.aspx?ID=6710&locale=en-us
Windows Internals 5th Edition http://www.microsoft.com/learning/en/us/book.aspx?ID=12069&locale=en-us