To DEP or not to DEP …

To DEP or not to DEP …

  • Comments 14
  • Likes

In my previous posting on Access Violations, I briefly mentioned Data Execution Prevention (DEP).  I have recently had the opportunity to work on a couple of customer issues that caused me to dig a bit deeper into the workings of DEP, so I figured that I would pass this knowledge on.  To begin with, some quick background on DEP.  Data Execution Prevention, or DEP, is Microsoft's software implementation that takes advantage of hardware NX  or XD support.  NX stands for No Execute and XD stands for Execute Disabled and are the ability for the processor to mark physical memory locations with a flag indicating whether or not the data in that location should be executable or not.  NX is AMD's implementation and XD is Intel's, but they are basically the same thing.  This software support requires the Windows PAE kernel be installed, but this should happen automatically, so you don't have to set the /PAE switch in your Boot.ini.  What all of this means is that with DEP, the operating system has the ability to block certain code from executing on the system.  DEP was first introduced with Windows XP Service Pack 2 and has been included in every Microsoft OS and service pack since then.

With hardware enforced DEP, all memory spaces are automatically marked as non-executable unless they are explicitly told they are being allocated for executable code.  This flag is set on a per-page basis and is set via a bit in the page table entry (PTE) for that page.  If something tries to execute code from a memory region that is marked as non-executable, the hardware feature passes and exception to DEP within Windows and lets it know that this is happening. DEP then causes an assert within the code stack that is executing, which causes it to fail with an access violation, which should look pretty much like the following:

In the past, this was not enforced and code could execute from basically anywhere.  This allowed virus and malware writers to exploit a buffer overflow, and spew a string of executable code out into an unprotected data region.  It could then execute it from that location uncontested. Those of you who remember the outbreaks of Blaster and Sasser – those are prime examples of using this sort of exploit.  By combining processor NX or XD support with Windows OS support, this type of vulnerability should be largely mitigated.

Sometimes an innocent application will trigger DEP simply due to faulty coding.  We often see this on older applications or things like shareware.  It is usually not intentional and never caused a problem in the old days, but now that security is paramount, inefficient (and sometimes sloppy!) memory management can cause some serious issues.  The right answer of course is for the application vendor to rewrite the portion of the app that is triggering DEP, but that is of course not likely in the case of older applications or shareware applications.  In this case, you can exempt the application for DEP monitoring so that DEP ignores it.  As long as you trust the application in question and know it is not really doing anything malicious, exempting it from DEP should not be a problem. Here is what the GUI looks like:

You can add a program to the exemption list by simply clicking Add and browsing to the .EXE file in question.  However, there are a couple of other ways to disable DEP for a specific application beyond using the GUI.  The first is by changing the Application Compatibility settings for the application in the registry.  To do this, browse to the following key in the registry:  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers.  For each application for which you want to disable DEP, you create a string value where the name of the value is the full path to the executable.  You would then set the value data to “DisableNXShowUI” as shown below.

If you have several applications for which you want to disable DEP across your environment, it may be worthwhile to use the Application Compatibility Toolkit to deploy a custom Compatibility Database (see the TechNet article on Resolving Application Compatibility Issues with Compatibility Administrator for more details).

Turning our attention back to the boot.ini for a second before we wrap up, you may have noticed an entry in your Boot.ini saying Optout or Optin, like this:

[boot loader] 
[operating systems] 
multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Windows Server 2003, Standard" /fastdetect /noexecute=optout

The 'noexecute' value dictates what state DEP operates under. There are four different entries possible:

  • Optin:  DEP is enabled for Windows system binaries and any application that 'opts in'.  By default, only Windows binaries will be covered.  This is the value set if you choose the option 'Turn on DEP for essential Windows programs and services only' listed in the screenshot above.
  • Optout:  DEP is enabled for all processes, not just Windows binaries.  An application or process can be 'opted out' on an individual basis.  The Application Compatibility Toolkit can be used to create shims to opt-out apps, then deployed on your network.  This option is set if you choose 'Turn on DEP for all programs and services except those I select', like in the screenshot above.
  • AlwaysOn:  DEP is on for all processes, period. You cannot exempt processes from DEP monitoring, and any Application Compatibility Toolkit shims do not apply.
  • AlwaysOff:  Totally disables DEP regardless of hardware support. In addition, the PAE kernel will not be installed unless /PAE is put in the boot.ini.

Please note that these last two values must be set manually.

With that, we’ve come to the end of this post.  Hopefully you find this information useful!

- Tim Newton

Share this post :
Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Thanks!  I was looking for that reg key!

  • I thought it would work.

    I need to add an exception on a few hundred machines so doing it via the registry seems the easiest way.

    When I run the installed program I don't get a "DEP has closed ..." window but an error message about registering xwpdlx20.ocx.  If I add the program to the DEP list via the GUI I have no problems but I can't do that on hundreds of machines.

  • What performance impact does DEP have on a Windows 2003, SQL Server 2000 environment?

  • Hi Rsw8n,

     DEP should have no noticeable performance penalty. With hardware NX or XD support, all the OS really has to do is keep an eye on which apps it should monitor or not. I have never seen any sort of formal testing done on DEP performance, but it has been enabled by default ever since SP2 for XP and I have never seen nor heard of any performance penalty at all.


  • I am in the exact same boat as Ramona, all be it 2 years later:

    I just set our PCs (XP Pro SP3) to OptOut, and there's one app that doesn't work with DEP (ISIS Pro scanning software).  So i added it as an exclusion via a regedit:

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers]

    "C:\\DocuWare\\ISIS PRO.exe"="DisableNXShowUI"

    and pushed the registry change via GPO.  This ADDED the exe name to the exceptions list (Sys Properties, Performance Options, DEP tab) with a little check in the box and everything.  Only problem is, the software does NOT behave as though it is excluded from DEP:  I have even used Process Explorer (MS Sysinternals) to verify that despite being added to the exceptions list, ISIS Pro is running with DEP enabled.

    You know how we fix it?  Log into each machine as an admin, UNCHECK the box (apply), and RECHECK the box (apply again).  But like Ramona said, this isn't really an option for hundreds of PCs...

    any clue as to why the registry setting that worked for so many others is giving me the bird?  I'm baffled that the system seems to think this exception has been made, yet behavior does not reflect registry/system settings.

    Thanks for the responses.

    ~Baffled Sys Admin

  • @UpAndComing

    Just a thought... Do you reboot the system after changing the registry settings (and before manually tweaking the UI)?

  • Fantastic! Excellent! I was struggling with API documentation to find out how to do this and now you solved all my problems! Thank you so very much!

  • thank you! that regkey fixed the last annoyance left by dep, i'd run mplayerc.exe and when i exited, i'd get the error msg about not being able to read some memory, i knew it was left over from when i had added it as an exception to dep before i had turned off dep. it was so easy, all i had to do was go to that key and to layers and bam there it was mplayerc.exe, removed that entry and now it works like it's supposed to again. thank you again! no one else on the net had any info that actually helped..

  • Thanks for an extremely helpful explanation.

  • Old thread but figured I'd ask.  Is there anyway to disable DEP for an Outlook Add-In.  I have a legacy app that utilizes a third party DLL that seemingly performs a task DEP disagrees with.

    If I turn off DEP globally in Outlook 2010 via the Outlook options panel it works fine but I'd like to ensure it's specific to my Add-in for security reasons.

    In the System Properties Opt-Out list I've tried selecting the DLL of the add-in, the offending DLL I utilize that causes the DEP exception, the MCVBVM60.DLL virtual machine that runs it, and even Outlook.EXE, but none of them seem to do the trick.

    I was just wondering if you'd have any ideas.

  • Hi Tim,

    Am I missing the link here ? DEP exclusion seems to work only when using the GUI to configure an exception.

    When using only the reg add, DEP doesn't seem to know of the configured exception.

    Used the reg add on Server 2008 R2 (deployed via SCCM 2007/MDT2010).



  • thanks dude, excellent post! i really appreciate it when peeps take the time to explain the tech

  • Hi,

    Thank you very much for this post very useful, short and efficient ;)

  • Nice Post :-)