Robert Hensing's Blog

Software Security . . . and stuff.

DEP on Vista exposed!

DEP on Vista exposed!

  • Comments 6
  • Likes

This is going to be a looooong blog - but its one that I've been meaning to post for a loooong time now.  My hope is that you will learn something you didn't peviously know about DEP and Vista or both and that you will, after reading this blog, re-configure your machine to perhaps make it a bit safer to use when surfing the Internet. :) 

I'm going to be talking about DEP on Vista (both 32bit and 64bit SKUs), primarily in its hardware form, and explain what the default configuration is for 32bit and 64bit Vista and why the defaults are the way they are.  I will then make some recommendations on how you can change some things to enhance your online safety with several high-risk applications.

What is DEP?
DEP (Data Execution Prevention)
 is a slightly overloaded term that refers to a set of hardware and software technologies that we have implemented (with support from Intel / AMD) to make it harder to exploit security vulnerabilities on Windows.  On Vista - if your processor supports hardware enforced 'no execute' or 'execute disable' bits (i.e. 'NX' on AMD and 'XD' on Intel CPU's) you will have hardware DEP enabled by default for most Windows applications with a few notable exceptions.  If your processor does not support the 'no execute' bit - you'll only get software DEP protections which is still better than nothing but not as good as hardware DEP and easier to bypass / defeat.  (NOTE:  This is essentially the same behavior that was introduced on XP Service Pack 2).  Hardware DEP works by setting a special bit in a PTE (page table entry) which is an object used by the VMM (virtual memory manager) to map virtual memory addresses to physical memory addresses.  If an attempt is made to execute code from a virtual memory page that has been marked as non-executable (via the bit being set in the PTE for that page), the processor will raise an exception and the OS can then end the application or bring down the entire OS (if the exception is raised in kernel mode code).  Hardware DEP + ASLR is a good thing when used together as it signficantly raises the bar for those seeking to exploit security vulnerabilities like the recent ANI file 0-day.

How does Vista make use of DEP by default?
Before I answer that - I want to talk about the two different architectures that Vista supports presently.  On 32bit versions of Vista - you can only run 32bit applications (obviously).  But on 64bit versions of Vista you can run both 32bit applications and 64bit applications side by side.  It is worth noting here that for 64bit processes running on 64bit versions of Vista - DEP is enabled all the time and any changes to the system wide DEP policy (more on this in a second) are ignored for 64bit processes since they are always opted in to DEP by defualt.  It is also worth noting here that just because you are running a 64bit version of Vista on your shiney new semi-expensive 64bit hardware, it doesn't necessarily mean you are using the 64bit versions of various Windows applications (like Internet Explorer, Windows Media Player etc.) and are thus 'protected' by this feature!!!

To explore your default system wide DEP policy for 32bit applications (remember - the system wide DEP policy only applies to 32bit applications) on Vista simply click start, then type 'system' and then scroll down and click the search result entitled 'system'.  On the new window that pops up click 'Advanced System Settings', elevate it to admin when prompted and then click the 'Settings' button under 'Performance'.  Finally - click the 'Data Execution Prevention' tab (whew - glad we made that easy to find . . .grr).  Yes I know there are probably 642 other ways to get to this same dialog box - that's what we all "love" about Windows right? :)

Now - if you are running Vista on a CPU that supports hardware DEP you will see a dialog box that looks like this:

This setting corresponds to the DEP policy we call 'OptIn' which means that DEP is only enforced for 32bit applications that specifically opt-in to the system wide DEP policy at compile time by using the /NXCOMPAT linker flag.  It is worth noting here that, if you are a developer, you can update your existing binaries without recompiling them using 'link /edit /nxcompat filename' to get them to opt-in to DEP if there is no reason they shouldn't.

A far easier way to view your system-wide DEP policy would be to run the new 'bcdedit.exe' from an elevated CMD shell - the output of which is shown below: 


Windows Boot Manager


identifier              {bootmgr}

device                  partition=C:

description             Windows Boot Manager

locale                  en-US

inherit                 {globalsettings}

default                 {current}

displayorder            {current}

toolsdisplayorder       {memdiag}

timeout                 30

Windows Boot Loader


identifier              {current}

device                  partition=C:

path                    \Windows\system32\winload.exe

description             Microsoft Windows Vista

locale                  en-US

inherit                 {bootloadersettings}

osdevice                partition=C:

systemroot              \Windows

resumeobject            {470198ad-e159-11db-9baf-00d0b71a8ddd}

nx                      OptIn

You can see above that the 'nx' policy is set to 'OptIn' on my 64bit Vista box by default.

Now let's examine the effects of this system-wide DEP policy on a 64bit version of Windows Vista Business edition running on DEP enabled hardware. 

To do this I have reconfigured Task Damager Manager to have the processes tab list the DEP status of each process by clicking the 'View' menu and choosing 'Select Columns' and then scrolling down to the very bottom and checking 'Data Execution Prevention'.  After configuring Task Damager Manager this way I started a variety of 32bit and 64bit Internet facing applications side by side so that you can see the effects of this policy on them in the far right column below:


For those who have never run a 64bit version of Windows, the first thing you will notice in the above output is that Task Damager Manager distinguishes 32bit and 64bit processes for you by appending a "*32" to the end of the process name for 32bit processes.  You can see in the above output that I have started several 32bit applications - some intentionally - some not so intentionally.

Let's start with Explorer - on 64bit versions of Vista - by default your shell, Explorer, is a 64bit version of the binary (explorer.exe) and thus it has DEP enabled by default.  But as it turns out - we have provided a 32bit version of this binary and many other system binaries in a special folder for you to shoot yourself in the foot with use.  Why?  For the same reason we do anything seemingly stupid - for backwards compatibility and application compatibility reasons beyond our control. :).  This article describes one very important reason for providing a 32bit version of Explorer.  If you have a 64bit version of Vista and you install an application like oh say Winzip - you will notice that when you right click in the shell, the cool shell integration ('Add to Zip' menu option) doesn't show up in the 64bit version fo Explorer - you have to run the 32bit version of the shell to get that shell extension loaded (because Winzip is a 32bit app).  The reason is simple - you can't load 32bit DLL's inside of 64bit host processes - so the workaround is to run the 32bit version of Explorer whenever you want to see your shell extensions. 
UPDATE: I was informed on 4/6/2007 that there is a version of Winzip 11 which does contain 64bit shell extensions: but the point remains that many popular applications do not yet have 64bit shell extensions.

Moving on - you will see both a 64bit version of Internet Explorer and a 32bit version running side by side.  You will also note that the 64bit version of Internet Explorer has DEP enabled while the 32bit version does not.  Doh!  Having DEP enabled in Internet Explorer by default for everyone on the planet would have been pretty damn useful during the few days of active ANI file exploitation before we released the security update.  It's a good thing 64bit versions of Vista default to running the 64bit version of Internet Explorer by default right?  Oh wait - they don't.  What what WHAT!?  Why on Earth not?!  Well, for the same reason we do anything seemingly stupid - for backwards compatibility and application compatibility reasons beyond our control. :)  I mentioned earlier that 32bit DLL's can't load inside of 64bit processes.  Think about Active X controls - what are they?  They are libraries that load in a process.  More precisely they are more than likely *32bit* libraries that load in a process.  What is everyone's *favorite* ActiveX control these days?  It's "Youtube" (as my wife would call it) or more accurately "Flash" to us 'geeks' <G> - so let's see what happens when we try to install Flash on a 64bit version of Internet Explorer with DEP enabled:


Ooooohhh that's right!!!  No one makes 64bit ActiveX controls because 99.9999% of the world is running a 32bit version of Windows.  Hmph.  Well now you know why the 32bit version of Internet Explorer is the one that you get 'by default' on Vista when you click the 'Internet' icon on the start menu (because it's the only one that would work with the majority of the worlds AX controls) - but why is DEP disabled by *default* for 32bit IE?  Why doesn't it opt-in to the system wide DEP policy?  Don't you take security seriously?  Well, again, it's for the same reason we do anything seemingly stupid - for backwards compatibility and application compatibility reasons beyond our control. :)  It turns out that by the time we were getting ready to RTM Vista - Adobe had not yet released broadly - versions of the popular ActiveX controls that worked with DEP . . . ActiveX controls used by the majority of today's popular web sites.  If you enabled DEP for the 32bit version fo IE - it would crash the instant the old versions of these controls loaded.  This would make for an un-pleasant out of box user experience - so the tough call was made to ship IE in Vista with DEP disabled.  In the time since Vista has shipped - Adobe has corrected this situation and has released versions of their ActiveX controls that work with DEP so you should be able to finally opt the 32bit version of Internet Explorer into DEP using the instructions provided by Mikehow here.

Now - you will note that the 32bit version of Windows Media Player and Outlook 2007 are also not opted-in to DEP by default so the instructions provided in Mike's blog won't help you there (and do I really need to repeat the reason why?) with the default system-wide DEP policy.

(NOTE:  I was actually pleasantly surprised to see the 32bit version of Live Messenger 8.1 opted-in to DEP by default this morning when I re-ran my tests as I believe during the beta it was not opted-in by default.)

So here you are - reading this blog from your 64bit Vista installation realizing that:

  1. Your highest risk Internet facing 32bit applications (Internet Explorer, mail clients etc.) are probably not configured to use DEP by default - even on Vista x64!
  2. You are actually running high risk 32bit versions of common applications on your spiffy new 64bit version of Vista by default and not the DEP-enabled-by-default 64bit versions like you may have assumed you would be on this SKU.

What can be done to get DEP creamy goodness all the time for all processes both 32bit and 64bit alike on Vista? 

The simplest thing you can do is to change the system-wide DEP policy and reboot (for it to take effect).  The easiest way to do this is to run a copy of CMD elevated and to type the following (or you can go through the overly complex and not-at-all-easy-to-find steps I outlined above):

bcdedit.exe /set nx OptOut

This tells Vista to enable DEP for all 32bit applications unless they specifically opt-out of it with the /NXCOMPAT linker flag.  That's pretty decent and it will force all of the applications in the screenshot above to use DEP (i.e. you will see 'Enabled' across the board after the reboot for your 32bit applications).  This is how I roll . . . it's the first setting I change after I install Vista on any machine I touch. :)  Maybe someday we'll be in the position to be able to flip this on with a service pack or something (dare to dream).

An interesting thing to try is to switch the policy over to opt-out as described above and then to try and opt a 64bit application out of DEP. :) 

I tried - Vista was un-impressed:

If, for some reason, you don't want to change the default system-wide DEP policy - you can actually tweak the registry a bit to force certain applications to opt-in to DEP even though they weren't compiled to do so.  This is accomplished using the venerable 'Image File Execution Options' registry key . . . below is a sample .REG file that you can run to DEP enable the 32bit version of some common apps immediately (well - the next time you launch the app).

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\explorer.exe]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\iexplore.exe]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\outlook.exe]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\wmplayer.exe]

NOTE:  I don't recommend using the registry hack above as I have been told that there are certain things having to do with "ATL thunks" that don't happen when you opt a process into DEP using the above registry key that DO happen when it's opted in by the system wide DEP policy with the net result being that its better to use the system-wide policy.  BUT - the system wide DEP policy requires a reboot - whereas these registry keys cause the DEP policy to be enforced the next time the application is re-started.

There, I said it.


  • PingBack from

  • Hey Robert, please see this article: and tell me if whatever's written in the "Preventing exploits" and "x86-64 specific features" sections REGARDING DEP is correct or incorrect. And please correct it if its wrong. : )

  • 'someone' - your write-up in Wikipedia looks really good!!!!

    well done.

  • Hi, I’m Eric Lawrence from the Internet Explorer Security Team. With the RSA security conference kicking

  • Hello, this is Rob Hensing. I work with the SWI team at Microsoft. One focus of my job is looking for

  • Oui, vous avez bien lu le titre. Pourquoi Adobe Reader 9.0 ? Jetez un &#339;il sur l'article de Robert

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment