Pushing the Limits of Windows: Physical Memory

re: Pushing the Limits of Windows: Physical Memory

Phileosophos — Mon, 21 Jul 2008 23:18:58 GMT

It seems to me that you omitted something. To wit, the contribution of SP2 when dealing with Windows XP. I built a system with 4 GB of RAM roughly three years ago because I wanted the extra RAM for pro audio/video applications. By your article, my system should have had about 3.75 GB of free RAM under the 32-bit version of Windows XP (in light of my video card's 256 MB).

But that's only true with SP1. From what I've been able to gather, when Microsoft added the DEP feature to SP2, they made what strikes me as a stupid decision: to lop off the top gigabyte of memory altogether, rather than deal with driver issues that could be caused by interactions with DEP. When I boot into WinXP SP1 on that machine, I do have 3.75 GB of RAM available; when I boot WinXP SP2 on the very same hardware, using the very same drivers, I have 2.75 GB available.

I have received conflicting answers to my support inquiries over the years from Microsoft personnel. Can you perhaps comment on the top gigabyte of RAM when using WinXP SP2 or later?

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Mon, 21 Jul 2008 23:37:30 GMT

The behavior I describe in the post is the same for all releases of Windows client systems, including Windows XP, so if there's a difference in behavior between SP1 and SP2, it has to do with a video driver or some other factor. You can confirm that by using the Meminfo tool to look at physical memory ranges and the device manager to see what's using the the missing ranges.

re: Pushing the Limits of Windows: Physical Memory

Tom — Mon, 21 Jul 2008 23:53:51 GMT

Great article, thanks! It's always fun to see Task Manager screenshots from monster systems. :) One thing I wondered while reading this article: are computer companies afraid to put 64-bit Windows on machines, or do they just not know any better? At my company we bought several Dell PCs with 4GB last year. They came with 32-bit Vista, which reports only 3.1GB as being available. (With Vista SP1 it now reports 4GB in some places, as you mentioned...)

By the way, I noticed one typo:

"two 1MB video cards" (should be 1GB)

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Tue, 22 Jul 2008 00:02:48 GMT

Thanks for the feedback and for pointing out the typo.

re: Pushing the Limits of Windows: Physical Memory

Beric Holt — Tue, 22 Jul 2008 03:10:39 GMT

Excellent article Mark. I understood the x86 vs x64 memory issues before, but this explains much more in detail, and shows us how to actually discover exactly how much memory we are loosing on the x86 platform.

Thanks!

re: Pushing the Limits of Windows: Physical Memory

Chris D — Tue, 22 Jul 2008 07:05:21 GMT

As always great article. Looking forward to more.

re: Pushing the Limits of Windows: Physical Memory

Phileosophos — Tue, 22 Jul 2008 07:25:08 GMT

First, thanks for the response. But second, what I find just doesn't match what you say.

For the record, I can't use the meminfo utility you suggest because it requires Vista or later. I quote from the bottom of the linked page: "Please note that MemInfo requires Windows Vista or higher as well as full (elevated) administrative privileges."

Sadly, device manager is no more illuminating, insofar as the memory shown in use is identical across both OS installs and yet SP1 install 3.5 GB of memory while SP2 reports 2.5 GB of memory--I've since upgraded to video that use 512 MB instead of the 256 MB I mentioned originally.

What I find most interesting is that in order to make the most of my 4 GB of physical RAM, I must (1) run WinXP SP1, and (2) use the /PAE switch in my BOOT.INI file. Only by using the /PAE switch does WinXP SP1 report the extra RAM and make it available. That just doesn't track with your statement that the behavior "...is the same for all releases of Windows client systems, including Windows XP, so if there's a difference in behavior between SP1 and SP2, it has to do with a video driver or some other factor."

I would *LOVE* to pursue this further somehow. You're the brightest and most thorough developer I've had the privilege of chatting with, and this is an issue on which I would relish closure.

re: Pushing the Limits of Windows: Physical Memory

Sarath — Tue, 22 Jul 2008 08:13:20 GMT

Total Physical Memory 4,096.00 MB

Available Physical Memory 2.09 GB

4 GB RAM only 2.09 GB addressed :(

re: Pushing the Limits of Windows: Physical Memory

Corwin — Tue, 22 Jul 2008 10:23:24 GMT

Hi.

I believe that Computer shops are holding back 64 bit OS'es because customer support. Home users have many more or less different gadgets that dont have 64 bit drives. Also - there is still lots of old 16-bit software what they need to run for some reason (mostly sentimental reasons).

re: Pushing the Limits of Windows: Physical Memory

Chris Nahr — Tue, 22 Jul 2008 10:36:54 GMT

Good article, but I'd like to add one thing: you may have to manually enable memory remapping in the BIOS to access all of 4+ GB RAM in a 64-bit system. Otherwise, part of it may be obscured by memory-mapped devices at the same location.

re: Pushing the Limits of Windows: Physical Memory

Thessan — Tue, 22 Jul 2008 11:20:45 GMT

Thanks! It would be interesting to hear your thought on the windows swapfile too (most likely you've already talked about this somewhere and I've missed it). I've heard a lot of different theories on how to handle the seap on system with large amounts of RAM. Some claim it's better to have no swapfile, while others claim it's better to let Windows handle it.

re: Pushing the Limits of Windows: Physical Memory

Craig — Tue, 22 Jul 2008 13:41:53 GMT

Excellent article as always! Very informative.

re: Pushing the Limits of Windows: Physical Memory

TechGuy — Tue, 22 Jul 2008 14:42:23 GMT

@Phileosophos: "From what I've been able to gather, when Microsoft added the DEP feature to SP2.."

FYI: From what I can remember, when DEP is enabled, /PAE (physical address extension) is also (implicitely) enabled, which may affect available RAM. (depends on system)

re: Pushing the Limits of Windows: Physical Memory

Jose Olcese — Tue, 22 Jul 2008 16:56:17 GMT

Mark,

Excellent article!

One question, in the system that you show where 7E700000 to FFFFFFFF is reserved for PCI, who's responsible for this reservation? OS? Bios? who?

Thanks!

re: Pushing the Limits of Windows: Physical Memory

Fleet Command — Tue, 22 Jul 2008 17:21:05 GMT

Hi Mark

Correct me if I am wrong:

Devices call Windows and say "Oi, Windows! If anyone passed you a pointer to the 4096th MB of the physical memory, that someone is actually addressing our device memory! We just didn't think you have that much memory, now do you?" And Windows goes back and says: "OK. No problem. And no! I don't think any computer with more than 512MB RAM ever exist!"

Does that sound about right?

re: Pushing the Limits of Windows: Physical Memory

Warp — Tue, 22 Jul 2008 18:11:57 GMT

In another forum someone commented that the first image in this page represents the minimum hardware requirements for the next version of Windows. That made me LOL.

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Tue, 22 Jul 2008 19:03:18 GMT

Phileosophos: TechGuy is correct - XP SP2 enables PAE by default because it's required for DEP. Prior to XP SP2, if you specified the /PAE switch manually, Windows didn't truncate addresses above 4GB. During XP SP2 testing the driver problems were discovered and so the change was made then. I've updated the post to mention this.

re: Pushing the Limits of Windows: Physical Memory

David — Tue, 22 Jul 2008 19:27:05 GMT

That 2tb 64 processor system looks a bit underutilized.

re: Pushing the Limits of Windows: Physical Memory

raino — Tue, 22 Jul 2008 19:58:48 GMT

Great post as always, I'm really enjoying this kind of technical writings.

re: Pushing the Limits of Windows: Physical Memory

jrronimo — Tue, 22 Jul 2008 20:57:18 GMT

Great post Mark. It really gives a background to things I half-knew before, heh. I've been running 64-bit Vista since the day it was made available to beta testers and it was certainly worth it.

re: Pushing the Limits of Windows: Physical Memory

John — Tue, 22 Jul 2008 22:16:31 GMT

*g* Vista x64 can address 4 times more RAM than "Server" Standard x64

re: Pushing the Limits of Windows: Physical Memory

mvadu — Tue, 22 Jul 2008 23:16:52 GMT

Great post.. And why you are showing Task Manager screenshots instead of Process Explorer!! :)

re: Pushing the Limits of Windows: Physical Memory

GrofLuigi — Tue, 22 Jul 2008 23:47:27 GMT

Great article! One question though:

Is the 'taken' memory actually used (do the actual I/O calls go through actual memory cells) or the address space is the only thing needed? In other words, would the system behave the same if there was no actual memory (or maybe just a dummy memory chip with no cells)?

Now that I think about it, this wouldn't be possible with graphic cards' allocations, but what about other devices?

re: Pushing the Limits of Windows: Physical Memory

TaylorMichaelL — Tue, 22 Jul 2008 23:57:22 GMT

I can't get over the initial screenshot. On the 2TB system 95GB of memory is already in use. They must be running some hefty apps.

Video card menory

Greg Fri — Wed, 23 Jul 2008 01:11:44 GMT

Very nice article, so if I have a video card with more on-board memory, say 512MB, will 512MB also be set a side of my main systems memory? Will your article elaborate on the trade offs regarding video card memory, how its used, and why more may be better or worse in some cases - hard core gaming vs business use only. If I just use my computer for business use,is it better to have less video graphics card memory? - thank you

re: Pushing the Limits of Windows: Physical Memory

Alex — Wed, 23 Jul 2008 03:10:33 GMT

Meminfo appears to require Vista ...

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Wed, 23 Jul 2008 03:25:52 GMT

You're right, meminfo requires Vista.

re: Pushing the Limits of Windows: Physical Memory

Stephen Reese — Wed, 23 Jul 2008 08:50:17 GMT

I don't think 8 gigs is by any means too much. Heck that's how much I have in order to ensure that notepad is opening fast enough on my Vista box ;-).

re: Pushing the Limits of Windows: Physical Memory

Mark Edwards — Wed, 23 Jul 2008 14:11:15 GMT

Hmm, a 2Tb system, with what appeared to be 64 threads. Could that have been a HP Integrity Superdome running IA64 architecture?

As always, Mark, a good read!

re: Pushing the Limits of Windows: Physical Memory

Steve Boyko — Wed, 23 Jul 2008 14:42:25 GMT

Like thessan, I would love to see an article on the Windows swap file. I have read conflicting opinions on whether it is really required in memory-rich systems, and I would like your expert insight into this.

re: Pushing the Limits of Windows: Physical Memory

WEB Developer — Wed, 23 Jul 2008 16:45:02 GMT

Is there any reasons to put into 32 bit computer 4GB memory? I think that developers who do it spend users' money.

re: Pushing the Limits of Windows: Physical Memory

Vadmyst — Wed, 23 Jul 2008 17:15:03 GMT

What about Vista ability to use USB flash storage for page files? Does this affect memory consumption or memory addressability?

Great article

re: Pushing the Limits of Windows: Physical Memory

edwardk — Wed, 23 Jul 2008 19:00:00 GMT

I'm actually dealing with an interesting, related topic currently. My customer is looking to deploy Vista to 18,000 desktops currently running Windows 2000 with 1 GB of memory. They're looking for confirmation that their application portfolio will perform similarly under Vista as it did under w2k. RAM would be the obvious potential bottleneck. Would you say that monitoring available memory over time is an adequate measure to analyze?

re: Pushing the Limits of Windows: Physical Memory

Security Man — Wed, 23 Jul 2008 21:27:55 GMT

Thanks for the post. I'm looking forward to read a next bunch of information.

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Thu, 24 Jul 2008 11:54:05 GMT

Mark Edwards: good call, it's a superdome.

re: Pushing the Limits of Windows: Physical Memory

stuart robertson — Thu, 24 Jul 2008 12:10:36 GMT

Phileosophos did you read kb888137 as this seems to answer your question re sp1 and sp2 memory

re: Pushing the Limits of Windows: Physical Memory

Jojo99 — Thu, 24 Jul 2008 12:56:50 GMT

This is what I see on the performance tab for my Win32 SP3 system. My available physical memory always stays around 1GB (out of 2GB).

I've always been curious about what "System Cache" is. What does this memory represent and how is it managed? Why is the number so high here?

Totals

Handles 23245

Threads 879

Processes 83

Commit Charge (K)

Total 1535908

Limit 5009600

Peak 3269464

Physical Memory (K)

Total 2095528

Available 971408

System Cache 890664

Kernel Memory (K)

Total 128680

Paged 98196

Nonpaged 30484

re: Pushing the Limits of Windows: Physical Memory

Vadim — Thu, 24 Jul 2008 14:34:21 GMT

Mark,

Thanks for your analysis. I have a practical question though. As someone has already pointed out a 64-bit Vista Ultimate can address several times more memory than Windows Server 2008 standard 32-bit or 63-bit. My question is about Windows Server 2008 32-bit in two flavors: Standard and Enperprise. The Enterprise version supports up to 64Gb despite being 32-bit. Is it possible to enable support for more than 4Gb of RAM in the Standard version? For example, I don't a feature set of the Enterprise version but I need more RAM supported on a 32-bit system.

Thanks.

re: Pushing the Limits of Windows: Physical Memory

Wayne Robinson — Thu, 24 Jul 2008 15:31:33 GMT

Mark you are a GOD! Thank You for an excellent read :)

re: Pushing the Limits of Windows: Physical Memory

DaveC — Thu, 24 Jul 2008 22:05:41 GMT

Mark, I just discovered your BLOG and this wonderful and informative post. Please keep up the great writings - you are now in my blog subscriptions!

PS - Is there anyway that you would post a full sized screen shot of that 64proc / 2TB RAM taskmanager? i would love to pass that around.

re: Pushing the Limits of Windows: Physical Memory

William Maynard — Fri, 25 Jul 2008 21:15:35 GMT

This is the first time I have been here, and would like to know if you can provide me with an address to where I might be able to download a disk compactor for my hard drive. I am running HP Windows Vista Home Basic with 14 1GB, of which I have 117GB left. Sometimes I'll have 123GB, then it will drop to 107GB, and then go up to varies degrees of memory. My email address is: billyblue3@verizon.net or mybillyblue@yahoo.com.

I WILL appreciate anything you can do to accommodate me. Thank you and have a nice day.

Respectfully,

William Maynard

re: Pushing the Limits of Windows: Physical Memory

Ales — Sun, 27 Jul 2008 00:23:12 GMT

Hi Mark,

great blog post! I wonder if you're willing to address Large Pages Support in Windows 2003/2008 in one of your following articles. I experimented with Oracle10g and Large Pages (on IBM machine with 32GB of RAM, Windows 2003 SP2, x64) and I don't know how can one confirm that for example Oracle process memory was allocated with large pages or not!? Can we use some perf. monitor counter or sysinternals tool for that? Even after I configured Oracle and setup Oracle server process to run under user with "Lock Pages In memory" privilege I observed considerable page faults for Oracle process - hence I suspect that process allocated memory with "ordinary" memory pages.

Regards,

Ales

re: Pushing the Limits of Windows: Physical Memory

Quux — Sun, 27 Jul 2008 21:14:12 GMT

Excellent article Mark; I wish I'd know of it yesterday!

As a rule of thumb, I use Perfmon and the Memory, Page Reads/sec counter. If that graph doesn't tend to spend most of its time near zero, with the occasional and inevitable spikes being short and typically under 100, I assume the system has enough physical RAM. The higher that counter's average goes (over long periods of time and at normal usage for the system), the more strongly I consider buying more ram or running less programs concurrently.

Your thoughts on this as a rule of thumb?

re: Pushing the Limits of Windows: Physical Memory

Anna N — Mon, 28 Jul 2008 12:19:32 GMT

My rule of the thumb, 1GB for XP on subnotebook (more RAM means higher power and shorter battery life), 2GB for every other XP, 4GB for Vista. More than 4GB only on 64bit Vista AND only if you have profiled demanding application you need to use and it needs more RAM.

Yes, there's a reason to have 4GB under 32bit OS, and that is because most graphic cards have 512MB or less, so even after drivers' memory holes you have usually much more than 2GB available, and that's useful for caching (so for casual usage, too), especially under Vista. 4GB is (on current platforms) better than 3GB because of dual channel memory so 2x2GB means better memory performance than 2GB+1GB. This may change on triple-channel Nehalem, maybe 3x1GB will be better.

re: Pushing the Limits of Windows: Physical Memory

Kenact — Mon, 28 Jul 2008 15:37:13 GMT

"What about Vista ability to use USB flash storage for page files?"

Actually, that capability has been available since Windows recognized USB. I used to run my paging to an 8gb USB memory stick on a Windows 2k Terminal Server. It worked just fine. One caveat, use a memory stick that was designed to be used for paging.

re: Pushing the Limits of Windows: Physical Memory

Steve Munson — Mon, 28 Jul 2008 17:55:22 GMT

Fantastic article, which now leads to some questions... I have an ASUS P4S8X motherboard running XP Pro SP2, with an ATI Sapphire AGP graphics card with 512MB on it, (64MB aperture on the motherboard) and the system has 3x1GB RAM (all 3 motherboard slots full and at max capacity). I had been having difficulty with IE prior to going from 2GB to 3GB system RAM, where it would do really strange things to the display window, like flash the borders and scroll bars, and stop repainting at all - you could move the window, but it wouldn't repaint the part of the desktop that you moved it from. I suspected video drivers, but couldn't be too sure. I added the extra gigabyte of RAM figuring it sure wouldn't hurt with as many windows as I have open at one time. Much of the trouble went away, but has now returned. I thus tried to change the motherboard graphics aperature to 512MB, and that locked up the system such that I couldn't even boot to the POST screen. I had to reset the CMOS and reconfigure to get it working again at the 64MB setting. I'm going to have to take a closer look at the video drivers, but I'm real curious about why the graphics aperture change was such a significant piece of grief that I couldn't even get past POST. I use a 24" LCD with a 1920 x 1280 resolution for my monitor, and I also had a 19" CRT, so my system still thinks I have two active monitors (and thus two instances of the exe for the vid driver). I have a copy of Vista Ultimate that I'm planning to install on my next system, although that might be awhile before I can afford the new machine (I'm not going to settle for any cpu less than a Q9450 quad-core with 8 GB RAM). The question is, do I waste any time on researching my XP difficulties, or do I save the time and trouble and just tough it out until I get to Vista Ultimate 64? I might learn something valuable though, so NOT toughing it out is certainly attractive from that perspective.

Steve

re: Pushing the Limits of Windows: Physical Memory

Guest — Mon, 28 Jul 2008 18:56:10 GMT

How does the flash memory with Vista work into this RAM usage? i.e. readyboost

re: Pushing the Limits of Windows: Physical Memory

netskink — Mon, 28 Jul 2008 20:34:32 GMT

Where can we get the meminfo -r program for XP? The one on the linked webpage is for Vista only.

re: Pushing the Limits of Windows: Physical Memory

ASilver76 — Mon, 28 Jul 2008 22:08:44 GMT

Question:

Let's say I'm running a 32-bit version of Winodws - for example, Windows XP SP2. I have installed 4GB memory, and only between 2.5-3.34 GB are accessible to the OS. The question I have is whether or not the remaining memory is still acceable for other system operations - for example, if I created a 1GB RAM disk, leaving me with 3GB, would Windows still recognize the RAM disk and the 3GB, or would it indicate that the RAM disk and only 2GB of free memory exists? I was under the impression that the additional "wasted memory" could, in fact, be repurposed in just such a manner, and would not affect the Windows perception of "available memory" at all. Is this correct?

re: Pushing the Limits of Windows: Physical Memory

Peter Geany — Tue, 29 Jul 2008 14:04:10 GMT

Hi Mark another excellent article thanks for posting it. Like some other posters above, I would like to hear what you recommend for pagefile / swapfile sizes. There is very little really solid advice out there and what is documented is causing confusion as its all about how with little why and much is left unsaid. I look after server systems with up to 64GB of RAM mostly 32 bit but we are slowly moving to 64bit Server 2003R2 as we refresh and because we have reached the 2 GB limit for some of our processes.

We occasionally have alerts from our older 2003 sp1 machines with 8GB of RAM and a 4GB page file alerting that the page file is full, yet the server shows that there is still 500Mb or more physical memory available. It happened on a 32GB 32bit system also when as far as I could tell only 12 GB of memory was being used at the time.

My colleges feel that we should extend the pagefile beyond 4GB to 8GB, however I recall from the Sysinternals course I attended in London with yourself and David Solomon 18 months ago that maybe this is not the right thing to do on a 32bit system. As my memory is a little fuzzy now and systems have been updated and advice may have changed, some definitive written advice would be welcome.

re: Pushing the Limits of Windows: Physical Memory

Ian Boyd — Wed, 30 Jul 2008 01:29:23 GMT

ASilver76: "if you have a system with 3GB or more of memory and you are running a 32-bit Windows client, you may not be getting the benefit of all of the RAM"

No, the system isn't even using it. It's not a practical limitation, it's a limitation imposed to maintain stability. The RAM is available to the computer, but it is moved above 4GB. Since some drivers (somehow) can't handle memory above 4GB, doing so would cause problems for users. So MS made the decision on 32-bit client operating systems to just not allow it.

If you want the benefit of all the RAM, use a 64-bit operating system.

re: Pushing the Limits of Windows: Physical Memory

Ian Boyd — Wed, 30 Jul 2008 02:44:38 GMT

How is it that 32-bit drivers could even be given a memory address above 4GB? If it's a 32-bit pointer, the maximum value is 32-bit.

Unless pointer has been typedef'd to a 64-bit value for a long time now?

re: Pushing the Limits of Windows: Physical Memory

David Willmore — Thu, 31 Jul 2008 00:36:36 GMT

Who cares what physical memory mapping looks like? Isn't that what PAE is for? To extend page table mappings to point to >4GB addresses and, thus, allow them to be mapped into addresses <4GB? It seems that your post could go a little further into disambiguating the user memory map from the physical memory map. Maybe I've been running Unix for too long.

User processes don't need PCI address space mapped into them. Device drivers would, but not user processes.

On 32 bit XP SP2, basically, they nerfed PAE support?

re: Pushing the Limits of Windows: Physical Memory

Rowan Hawkins — Thu, 31 Jul 2008 05:39:30 GMT

I have a question about the memory hole mapped for video drivers. Is this memory hole mapped on systems that have dedicated video memory or only for ones with video memory accessed as part of the systems total available ram?

I could understand making a mapping for addressing the physical ram making on a system that used shared memory, but the other choice sounds like Memory Management by the Department of Redundancy Department.

re: Pushing the Limits of Windows: Physical Memory

Robbie Mosaic Fan — Sun, 03 Aug 2008 00:39:48 GMT

Great article, taught me quite some things. Thanks Mark!

re: Pushing the Limits of Windows: Physical Memory

Arlona — Mon, 04 Aug 2008 21:30:28 GMT

I still need to know in plain and simple terms, how do I add more memory to my HP Vista. At this time my memory is at 0.99

I need help soon.

Thank you

Arlona

How to enable mem above 3G on Win XP

Linda Walsh — Tue, 05 Aug 2008 01:48:18 GMT

How do I enable more than 3G on WinXP? It seems that with Windows SP1 or SP2, a patch was put in to limit memory to 3GB -- not 3.'x'. But

exactly 3Gig. I've seen this confirmed by other users. Something to do with driver security was the reason given.

If I reboot into 32-bit linux, I see 3.5-3.7GB memory, and if I enable the PAE addressing, I see the full 4GB.

So there are two separate issues -- one is using addresses greater than 4GB to remap physical memory that overlaps device or BIOS reserved memory -- that's something that MS should have left enabled when using the "/PAE" boot switch. If a user had a problematic driver, they could just 'not use' the PAE switch -- but better yet -- would be a RAMDISK that used any memory that would need >32bit addressing - that way 'drivers' could be shielded from ever seeing a >32bit addr, but users could still use the excess mem with only the OS seeing it.

But the need for PAE mode doesn't explain the 3GB limit. I'm talking exactly 3GB -- not 3.1 or 3.2GB, but 3*1024MB (3072). The only info I could find was that memory was limited in SP2 to make certain security checks on memory addressing easier or some such nonsense that didn't seem to make alot of sense.

So how can I undo this limitation -- I do have a PAE enabled processor, and I'd even be so bold as to want to try PAE addressing mode, but even if it wasn't stable, I could at least get an extra 15-20% of memory from memory 'around' the holes reserved by the BIOS.

Memory upgrade

Richard Jones — Tue, 05 Aug 2008 18:46:54 GMT

Hi Mark, thank you for an informative article. I'm a relative beginner to computer upgrading and found the information very useful.

I have a question, if I may?

I'm running Vista Home Premium 32bit and have upped the RAM to 4Gb -(2x2Gb matched pair). Bios shows the full 4Gb as present but when both sticks are inserted, the computer will not boot past the POST screen. Inserting one stick only (either stick, either slot)allows a normal boot so I take that to mean both sticks and slots are functioning correctly?

A search of the web has explained a great deal re: memory utilization but I can find nothing on this specific issue. Can you shed any light for me please?

Thank you in anticipation

Richard Jones

re: Pushing the Limits of Windows: Physical Memory

Thebitstream — Thu, 07 Aug 2008 20:46:57 GMT

My laptop have 2GB of RAM and cannot be expanded often then I use ReadyBoost to add some speed on the system.

Due to a superfetch in Vista 32bit can this lead do more memory utilization?

re: Pushing the Limits of Windows: Physical Memory

Magumi — Fri, 08 Aug 2008 04:32:28 GMT

Great article, as always. I just wonder how long it will take before we have 2TB of RAM in our laptops. :-)

re: Pushing the Limits of Windows: Physical Memory

Ken Schaefer — Fri, 08 Aug 2008 06:32:49 GMT

To those asking about Readyboost - Readyboost is, effectively, a cache. Instead of reading data from your hard disk (which takes a relatively long time), data can be read from a flash-based cache with near-zero seek time. A file system filter handles this - it intercepts read calls for files and if the file is in the Readyboost cache, returns it.

This has nothing to do with memory address space (virtual address space, system address space etc). Having a Readyboost cache doesn't change the amount of virtual addresses that the OS will use.

The same with the pagefile. The pagefile is for storing data that doesn't fit into physical RAM (i.e. it is "paged" to the pagefile to be stored, and retrieve when required). Having a pagefile doesn't change the virtual address space that the OS is programmed to use.

@David Willmore - PAE is more than just expanding the amount of address space. NX requires PAE, because the fact that a memory page is not executable has to be stored somewhere. See: http://en.wikipedia.org/wiki/Physical_Address_Extension and http://blogs.msdn.com/oldnewthing/archive/2004/08/20/217683.aspx

HTH

re: Pushing the Limits of Windows: Physical Memory

Alex — Fri, 08 Aug 2008 14:20:02 GMT

Mark,

Great article and love your Internal book series. I have a general question for you- I had the oppurtunity to read your Internal books in tandem with William Stallings Operating System book- both terrifc reads and great when paired together. Do you know any other great books like these to read? Thanks

re: Pushing the Limits of Windows: Physical Memory

Robert Gifford — Mon, 11 Aug 2008 14:57:33 GMT

I Have two PNY 8 Gig. 2.0 Flash Drives . When I first got them they worked fine with ReadyBoost I had to re-format From FAT to NFTS Now ReadyBoost does not reconize them As being ReadyBoost capable . In other word when you open prop.Box to activate ReadyBoost the choice is not there??

re: Pushing the Limits of Windows: Physical Memory

Paolo — Tue, 12 Aug 2008 18:59:28 GMT

I have a strange behaviour:

The computer property page reports 5,00GB RAM with a Window Server 2008 datacenter.

BUT, if I look at the activity manager, it reports only 3966MB in use.

Any suggestion?

TIA

Paolo

re: Pushing the Limits of Windows: Physical Memory

Santhosh — Tue, 12 Aug 2008 21:46:56 GMT

Hi Mark,

PFN database represents each page of memory with a 28-byte data structure, the PFN database on a 128GB system requires about 930MB.

I have a question on the above excerpt of the blog.

If 128 GB system requires about 930 MB, then why is the 4GB system occupying 960 MB. Effectively, shouldn't it just occupy 29 MB.

Please clarify my confusion.

- Santhosh

re: Pushing the Limits of Windows: Physical Memory

dave — Tue, 12 Aug 2008 23:25:07 GMT

re: Ian Boyd

>How is it that 32-bit drivers could even

> be given a memory address above 4GB? If it's

>a 32-bit pointer, the maximum value is 32-bit.

>Unless pointer has been typedef'd to a 64-bit >value for a long time now?

"32 bit system" means "a virtual address is 32 bits". The physical address size is somewhat wider (36 bits in Pentium Pro and onwards, since mid-1990s). Likewise, 32-bit code is code that's running in an address space where the VA size is 32 bits.

For software addressing memory, the memory management unit maps 32-bit virtual into (larger) physical addresses.

For devices addressing memory via physical addresses, the driver has to not attempt to store physical addresses in 32-bit variables. That's been the case 'for a long time', but I suppose not all driver writers got that memo, which is why the artificial restriction exists in the client versions.

re: Pushing the Limits of Windows: Physical Memory

dave — Tue, 12 Aug 2008 23:30:24 GMT

How is it that 'datacenter' versions of 32-bit Windows Server 2003 are able to support 128 GB?

I was under the impression that the Pentium family had a 36-bit physical address space, which works out as 64 GB. Where did the extra bit come from?

Did more recent versions of the processor grow another bit? Maybe there's some feature of a 64-bit-capable processor in 32-bit mode that I missed?

Thanks/dave

re: Pushing the Limits of Windows: Physical Memory

jugge — Wed, 13 Aug 2008 12:31:56 GMT

PAE is implementation specific. The PTE is 64 bit in PAE mode, so there are a lot more bits in it that can be used for addressing. How many, depends on the specific CPU. Modern 64bit CPUs have a wider addressbus, and can therefore also make use of more of those bits.

re: Pushing the Limits of Windows: Physical Memory

Rick — Wed, 13 Aug 2008 16:14:58 GMT

Great clarifying paper which epxlains some often misunderstood points, i think.

however, surely due to my lacking of some basis about the internals of windows memory menagement, i failed to totally understand the consequences in x64 windows systems of the reservation, done by chipset/bios if i have well understood, of some physical addresses ranges under FFFFFFFF, for devices:

if those ranges of physical addresses are statically reserved, how can a x64 win. system really use all of the physical ram ? ok, in those system also memory above 4GB can be used, but it sounded to me that that memory reserved for devices is anyway lost.

for example, in that mentioned 4GB of ram laptop with vista x64: ok it's shown that those 500MB reserved to devices below FFFFFFFF are then addressed, unlike 32bit windows, above that boundary; but if physical ram still is 4GB, how can the totality of it be avaible to the user and system purposes, if part of it it's statically reserved to devices ?

or maybe i misunderstood the definition of "physical address", related to Windows mem. management, which is implicit in this paper and it happens something like: "physical" addresses cover not only the physical ram but also the amount of disk space allocated to the file(s) pagefile.sys, so that regions reserved to devices can be swappped to disk and really 100% of physical ram can be used, if needed.

can someone help me to really understand the point ?

thanks to Mr Russinovich for the paper and to him and/or to everyone else reading my post for his/her attention

Lots of memory and 32 bit

Brandon — Wed, 13 Aug 2008 16:15:57 GMT

A while ago we had a problem with a 32bit Enterprise version of 2003 plus SQL 2005 Enterprise. The performance wasn't as good as we expected it to be. Microsoft SQL consultants had previously said that the pagefile is not needed for a SQL server with enough (16 GB) RAM. So we set it up with 3 or 4 GB.

Then we found this: http://msdn.microsoft.com/en-us/library/ms178067.aspx

there it says "Under Windows Server 2003, dynamic AWE memory is supported on servers with less physical memory than the configured virtual memory limit. Windows 2000 required that the available physical memory exceed the virtual memory in order to enable AWE memory."

Based upon this description we decided to do the normal 2.5x RAM sized pagefile (24GB), so that the physical memory would be smaller than the virtual memory limit... and performance improved, even though it wasn't running low on memory.

Why? Do we always need to have a larger pagefile than memory in a 32 bit environment? It just seems like a waste of space since SQL should stay in memory.

Thanks,

Brandon

re: Pushing the Limits of Windows: Physical Memory

Rick — Wed, 13 Aug 2008 16:21:24 GMT

i forgot in my previous comment: can anyone suggest a tool to see the layout of physical memory, like that "meminfo" mentioned in thi paper, but working on nt5.x class systems ?

thanks in advance

re: Pushing the Limits of Windows: Physical Memory

David Solomon — Mon, 25 Aug 2008 04:42:45 GMT

re: Brandon's commment "Based upon this description we decided to do the normal 2.5x RAM sized pagefile (24GB)"

Fyi, although many MS formulas/defaults use RAM*n to size the pagefile, it's actually nonsense - the more RAM you have, the smaller pagefile you need to run a given set of apps. Windows won't use more pagefile space if it's there (it just means the system has a larger commit limit).

--David Solomon

coauthor, Windows Internals (MS Press)

www.solsem.com

re: Pushing the Limits of Windows: Physical Memory

David Solomon — Mon, 25 Aug 2008 04:45:46 GMT

re: Rick's question "if those ranges of physical addresses are statically reserved, how can a x64 win. system really use all of the physical ram?"

the physical addresses are RESERVED by devices - the actual RAM is not taken up by that; however, since some amount of physical *addresses* are used up by (non-RAM) devices, the physical address for the upper part of RAM is above FFFFFFFF (e.g. the highest value you can have with 32-bits). And since 64-bit Windows can go beyond that number, that's why it can "use" (or maybe more clearly stated "address") all the RAM.

Hope that clears it up...

--Dave Solomon

coauthor, Windows Internals

www.solsem.com

re: Pushing the Limits of Windows: Physical Memory

Luiz Santos — Wed, 27 Aug 2008 19:56:51 GMT

Hi Mark,

I have server with Windows Standard 64-bit with 16 GB of memory. I just set page file for "none", but after reestart I notice that in Task Manager it keeps saying that I am using 6 GB of memory and what is funny is that if i go to Performance Monitor I can see that the Page FIle counter is always 0. Do you know why Task Manager shows that?

Regards,

Gus

re: Pushing the Limits of Windows: Physical Memory

Jay — Thu, 28 Aug 2008 05:08:00 GMT

So does this mean vista 32bit or xp 32bit can make full use of 3GB's? If it all shows in task manager and it gets cached so is it all really being used? It must be if it gets cached correct?

re: Pushing the Limits of Windows: Physical Memory

rsbroad — Thu, 28 Aug 2008 19:47:03 GMT

The Intel Pentium has a 36 bit address bus. The 36 bit address bus is connected to the external Northbridge Chip, which is also called the Memory Controller. The Northbridge Chip has a second 40 bit address bus. This second 40 bit address bus is connected to the actual dram (dynamic ram) chips.

The AMD Athelon has a 40 bit address bus. The Athelon has the Memory Controller circuitry integral to the CPU chip.

In both cases of Intel and AMD:

The maximum address space in 32 bit mode is 2^32, 4 gigabytes.

The maximum address space in 64 bit mode is 2^36, 64 gigabytes.

In both cases of 32 and 64 bit:

Any memory access outside of the maximum address space requires the much dreaded bank switching.

Bank switching works. It might seem like repairing furniture with duct tape, but it works. Therefor a 32 bit mode processor can conceivably access 2^40 bytes, 1 tetrabytes, or 1,099 gigabytes.

Actually, bank switching works except when it doesn't work. Therefor Windows 32 bit is limited to 2^32 bytes. In other words, Windows 32 bit would access 2^40 bytes if it could, but it can't, so it doesn't. Blame drivers.

The practical challange of installing huge amounts of ram is in the electronics, not the software. Supplying power to 2^40 bytes of ram is science fiction. Supplying address signals and data signals to so much ram is fantasy.

At any moment in time, Intel and AMD may increase or decrease the width of the Memory Controller address bus. Older Northbridge Cips had less than 40 bit addressing. Newer (as yet unannounced) chips may have more bits of addressing. Smaller devices, such as notebook computers, may have smaller Memory Controller chips in order to save power.

There is no way to recover the lost dram in 32 bit Windows. But 3.2 gigabytes ought to be enough for everybody.

re: Pushing the Limits of Windows: Physical Memory

Nick Kotarski — Thu, 28 Aug 2008 22:27:00 GMT

Hi Mark,

Another very interesting article.

Shame that Meminfo won't work on XP.

Nick

re: Pushing the Limits of Windows: Physical Memory

mrAndy — Fri, 29 Aug 2008 00:02:30 GMT

Have an acer laptop. Ugraded from 1gb to 4gb (2x 2gb) RAM. however, the system seems to be missing 2gb. some scanning software identifies the missing 2gb so the RAM must be working.

And yes the Upgrade is compatable with my system.

Any ideas????

re: Pushing the Limits of Windows: Physical Memory

prni — Mon, 01 Sep 2008 19:19:26 GMT

So why not leave a way to allow access to above-4gb for those who are willing to risk it ?

Pretty soon the h/w vendors would update their drivers...

re: Pushing the Limits of Windows: Physical Memory

prni — Mon, 01 Sep 2008 19:27:27 GMT

Hi Mark,

Can you elaborate on the relationship between virtual and physical adresses in relation to this discussion, i.e. since each process has it's own 32 bit virual adress space, could different processes use more than 4 gb of memory if the old PAE behaviour was brought back ?

Thanks

Per

re: Pushing the Limits of Windows: Physical Memory

Bryan Fairchild — Fri, 05 Sep 2008 07:15:26 GMT

Great article Mark, Thanks for defining the 4GB barrier and how 32bit and 64bit OS's deal with it. I noticed you've got two GTX 280s, very nice! I just built my power house system but had to settle for one 280. I too realized quickly after getting my new system built that I needed to ditch 32bit. 64bit Vista runs great.

re: Pushing the Limits of Windows: Physical Memory

Robo — Sun, 07 Sep 2008 12:38:43 GMT

Hello Mark, very useful article.

Could you comment a bit on the LargePageMinimum fix (KB270715). The latest nvidia drivers documentation implies that the fix is still needed for PCI Express video cards and K8/Phenom cpus. What are the implications for the memory manager/os performance?

Adobe Flash Player

Mike — Thu, 11 Sep 2008 01:14:20 GMT

Adobe does not have a flash player for our 64 bit computers. We use the internet to get new infromation on building products. Most of the sites require Flash Player to view the products. Since Adobe does not appear interested in providing a player for 64 bit machines do you know of a alternative Player that will work.

re: Pushing the Limits of Windows: Physical Memory

Rick — Thu, 18 Sep 2008 11:56:03 GMT

I want to thank Mr. Solomon for his kind and clear answer to my question

(question originated, as i said, from my holes in the knowledge of mem. management in win., not from something missing in the paper)

I'm also honored that the coauthor of "Windows Internals" took a bit of time to answer my question, thanks again.

re: Pushing the Limits of Windows: Physical Memory

const — Sun, 21 Sep 2008 21:58:04 GMT

Why Windows itself does not use memory above 4GB for internal needs? With no dependences of h/w vendors drivers...

May be internal buffers, PFN database or something else?

re: Pushing the Limits of Windows: Physical Memory

Alex — Wed, 24 Sep 2008 15:42:10 GMT

Mark, I know sysinternals for a long time and always wonder, where do you find this kind of information? Are they public trhough C or C++ system API/dlls?

re: Pushing the Limits of Windows: Physical Memory

Sudeep Kumar K S — Fri, 26 Sep 2008 18:50:24 GMT

Very informative blog.Good work. I was working on capacity planning for VMWare and was analysing usage statistics for the current server. I was wondering how I can find out the Average memory utilization and the peak memory utilization. Looks like I should find out available MBytes and then subtract those values from the actual MBytes of memory in the server and then find out the avareage of that and peak from that

re: Pushing the Limits of Windows: Physical Memory

Yuhong Bao — Sat, 04 Oct 2008 10:46:35 GMT

BTW, I talked to Larry Osterman about adding the ability for 32-bit client versions of Windows to address more than 4 GB of RAM via PAE, needless to say that unfortunately failed.

re: Pushing the Limits of Windows: Physical Memory

Evan Teran — Tue, 07 Oct 2008 19:46:13 GMT

Thank you for providing accurate information on this subject. There is so much mis-information out there being spread by "experts" that it's not even funny.

One question though. Why couldn't the RAM that gets displaced to above the 4GB mark be used for something guaranteed to be safe? Basically, you could create a separate pool for this RAM and only use it for things that are known to never use 32-bit DMA and such. Like for example, user space code pages?

re: Pushing the Limits of Windows: Physical Memory

PLY — Sat, 18 Oct 2008 03:55:38 GMT

OK - then am I wasting my money if I purchase a Windows XP Pro system with 4GB of RAM and a 1.5 GB video card?

re: Pushing the Limits of Windows: Physical Memory

George Birbilis — Wed, 22 Oct 2008 12:36:03 GMT

why not have the OS make a ramdrive at the extra memory and use it for speeding-up the system, similar to how Vista can use flash memory devices for system speedup?

re: Pushing the Limits of Windows: Physical Memory

Jazmin — Wed, 29 Oct 2008 00:54:17 GMT

Recently I have installed Windows 2008 32bits on my board and it reports 1.34GB of Installed System Memory (RAM) and physically I have 1GB of Memory.

I have checked Physically Memory field from msInfo.exe tool and it indicates that system has ~ 1GB.

Do you know how I can figurate why windows increase the RAM installed?

Jazmin

Sandman — Mon, 10 Nov 2008 20:19:03 GMT

Jaz, you likely have only 1GB (too little, btw) RAM installed, but I suspect you also have a video card with onboard RAM.

This extra RAM on the video card receives an address for mapping, and may be what you are seeing as reported/used memory mapped.

If you were to install further RAM to the full limit of 4GB, you would find that your memory in the machine can be discovered by utility tools and found to total perhaps ~4.3 GB, including both RAM and video, but only 3.5 GB will be available to the system due to this same video card taking up some of the RAM addresses.

Why Windows XP 32bit cannot see 4GB RAM?

nagyandor — Tue, 11 Nov 2008 17:32:49 GMT

Here is my experience and I am looking for the answer.

Configuration:

Base: Asus T3-P5G965

Proc: Intel E6600

Video: Asus X1350

Mem: Corsair Twin2X 800 DDR2 4x1GB

BIOS: memory remap feature is enabled

(If enabled, only 2GB memory is available without PAE kernel. With this option, PAE is available.

If disabled, 2.8GB memory is available without PAE kernel. With this option, PAE is not available.)

Windows XP 32bit is not able to manage the whole physical memory, altough PAE is working as you can see in the following screenshots:

https://www.netacademia.net/sqlimage.axd?name=tudastarpic&RecordId=24348&FName=winxp32.PNG

https://www.netacademia.net/sqlimage.axd?name=tudastarpicorig&RecordId=24348&FName=taskman.PNG

https://www.netacademia.net/sqlimage.axd?name=tudastarpicorig&RecordId=24348&FName=mem.PNG

Linux:

I tried Fedora 9 and 10 x86 distribution, and the basic install OS can show 2GB memory. However, I change to the PAE versioned kernel, the whole 4GB memory is available.

Windows 2003 Server R2 Enterprise:

This Windows edition can manage the whole 4GB memory like the PAE enabled linux.

https://www.netacademia.net/sqlimage.axd?name=tudastarpic&RecordId=24378&FName=w2k3r2Ent32_props.PNG

https://www.netacademia.net/sqlimage.axd?name=tudastarpic&RecordId=24378&FName=w2k3r2Ent32_taskman.PNG

Here is the boot.ini with the default switches:

[operating systems]

multi(0)disk(0)rdisk(0)partition(4)\WINDOWS="Windows Server 2003, Enterprise" /noexecute=optout /fastdetect

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect /NoExecute=OptIn

So it is sure my hardware is capable to manage 4GB RAM on the 32 bit OSs (Fedora linux, Win2K3R2Ent32)

Why the Windows XP 32bit cannot give me the whole memory while the PAE versioned kernel is used automatically as shown in the first screenshot?

Have somebody ever seen Windows XP 32bit with 4GB RAM? Or Have somebody ever seen anybody to experience this?

Thank you for your response!

re: Breaking the 3GB limit in 32-bit WindowsXP

Athena — Fri, 28 Nov 2008 05:43:11 GMT

Yes -- I have similar problems.

The part mark seems to be unaware of is that consumer versions of XP-32-bit disabled memory above 3GB in SP2 due to driver problems. Used to be documented in some KB article -- supposedly, in SP1, it wasn't broken -- but something else was -- which you could have gotten a patch for at that time if you wanted the full 4GB. But they never ported the patch to SP3 and left it with a 3G limit. Doesn't seem to matter whether I use the map-3G-to-user-space switch or not. I also have a PAE processor that has no problem with seeing 4GB of memory under linux, but MS seemed determine to cripple XP -- apparently they were afraid of some drivers not handling I/O to devices over 4G -- but in Linux they solved that with 'bounce' buffers. A bit slower -- as it's like "paging" memory pages in and out of 'high-memory', but if your option is to limit the memory to 3G and have 4G of memory page on and off of a 1GB-disk page file, vs. paging the 'usable' memory between low<->high memory, guess which is ALOT faster and better use of the hardware? Sigh. If only WinXP was open source and could be fixed. (*hint*? -- hey, the impossible sometimes happens; Bush is staying out Obama's way and letting him deal with the financial crisis even though Obama isn't president yet. If we can get that type of cooperation out of Bush, Microsoft open-sourcing XP should be a comparative slam-dunk :-))

re: Pushing the Limits of Windows: Physical Memory

Steve W — Sun, 30 Nov 2008 02:45:33 GMT

I've got a Toshiba M400 Tablet PC with Vista Ultimate (32-bit), 4 Gigs of 667 MHz RAM, and 45% free on a 7200 RPM, 180 GB hard drive. When it was brand new it responded much more quickly and videos played smoothly. However, it has become rather pokey of late. It's sloooow to respond to mouse clicks (especially the Start Menu), and videos of all formats stutter in WMP (but not as much in WMC).

Someone recommended that I downgrade to Windows XP Tablet 2005. Their rationale was that as an offshot XP Pro, XP Tablet 2005 can make better use of my Gigs of RAM. I don't need all the features Vista has such as Aero and better handwriting recognition (the tablet function is usualy used to illustrate during PowerPoint presentations).

Does this recommendation make sense?

re: Pushing the Limits of Windows: Physical Memory

David Solomon — Sun, 30 Nov 2008 18:06:15 GMT

re: Athena's comment: "The part mark seems to be unaware of is that consumer versions of XP-32-bit disabled memory above 3GB in SP2 due to driver problems"

WRONG. There is no such limitation in XP 32-bit.

Mark's article is correct - re-read the sentence "While 4GB is the licensed limit for 32-bit client SKUs, the effective limit is actually lower and dependent on the system's chipset and connected devices."

So XP 32-bit will not access PHYSICAL MEMORY that happens to fall above the 4GB PHYSICAL ADDRESS barrier. So depending on placement of device memory in the physical address range (such as video cards, PCI addresses), you might get 3.1, 3.2, 3.3, close to 4GB - who knows. But there is no 3GB limit!

re: Pushing the Limits of Windows: Physical Memory

DAKS — Tue, 02 Dec 2008 22:36:43 GMT

I have the same issue that nagyandor has with WinXP SP 3. I have 4GB RAM but windows only reports 2GB. Why only 2GB compared to others showing over 3GB?

re: Pushing the Limits of Windows: Physical Memory

Thief^ — Fri, 05 Dec 2008 16:40:37 GMT

Re: DAKS

"I have the same issue that nagyandor has with WinXP SP 3. I have 4GB RAM but windows only reports 2GB. Why only 2GB compared to others showing over 3GB?"

Because your motherboard chipset has decided to reserve 2GB, and theirs hasn't. If you update your chipset drivers, you could see more.

re: Why Windows XP 32bit cannot see 4GB RAM?

nagyandor — Wed, 10 Dec 2008 00:35:45 GMT

I still cannot find the answer to my question in the title.

Why can I see the whole 4GB memory in linux and Win2k3r2 with PAE and why not in WinXP with PAE?

(see my previous comment)

Thanks,

Andor

re: Memory Issue

Tayyab — Wed, 17 Dec 2008 19:44:24 GMT

I am using xp, sp 2, media centre edition, i upgraded my memory to 4G, now it shows me 2.99G in system properties, and 4096Mb in system info, i-e physical memory,

Need solution

re: Mem issue

tayyab — Wed, 17 Dec 2008 19:46:07 GMT

Same problem, i just need solution, i installed 4 gb of ram, and it shows 2.99 Gb in system properties, while in system info it displays 4 Gb, help plz

re: Pushing the Limits of Windows: Physical Memory

r00t — Thu, 08 Jan 2009 06:38:51 GMT

do you have 32 or 64 bit xp version?

re: Pushing the Limits of Windows: Physical Memory

Magnus369 — Fri, 09 Jan 2009 22:00:16 GMT

Is there any way of actually finding out why a system is claiming much more ram in use than shown by currently running processes? In this case, I have a sql server running on windows 2003, with 8GB of ram installed. I had to add the /PAE and /3GB switches to the boot ini to get the system to see the full 8GB. In the task manager, I see the full 8GB of ram available, with about 10GB of swap. Now, as far as ram in use, it was roughly 7GB, with another 6GB of swap commited. Digging through the task manager's process list, I could only add up about 3GB of ram in use. Is there anyway of finding out where that additional ram is? or is this a case of just having to reboot the system to reclaim it?

re: Pushing the Limits of Windows: Physical Memory

Octavius — Sun, 11 Jan 2009 00:17:56 GMT

I recently purchased a "economic"Packard Bell PC overseas.It has a Pentium dual core E2180 2.0 GhZ with a Sis 671 chipset with Mirage3(32bit/64Mb)video card and 1Gb DDR2-667MhZ RAM and 32-bit Windows Vista "starter".This OS was the most OBSURD thing ive ever seen.It limited me to 3 programs open at once and didnt have nor support over half the features of Vista home basic.I found out that this Os was intended for entry level,low-cost PC´s in developing countries.Furthermore,the Os was NOT upgradable.So,i went and got myself Vista ultimateSP1 32-bit and Formatted my drive including the 8gig partition for recovery and the Os works fine,better than expected.I also noticed the 980Mb instead of the full 1gb RAM issue as everyone else,so i decided to get myself 4GB of RAM(x2-2GB)and a 128 bit,512 MB 9500 GT.I went to the SiS Page and the motherboard and proccesor support 64-bitOS.So my question is,if I buy the video card and the 4GB of RAM am I going to see a performance increase or should I upgrade to 64Bit first?

re: Pushing the Limits of Windows: Physical Memory

Maarten — Wed, 14 Jan 2009 16:29:58 GMT

Thanks for publishing this. I knew about the artificial memory addressing limit of 4GB in 32 bit Windows versions, but this is the first time I read about why Microsoft decided to implement this limit.

I can certainly understand going over 4GB would be a risk when some drivers will choke on it, but I suspect the marketing reasons you hinted at are at least as important. Otherwise they could at least have implemented a registry or BCD flag to enable true PAE.

For me switching to 64 bit Vista is not an option yet, because some apps I use don't run (as well) on it.

Anyway, thanks again for sharing this!

re: Pushing the Limits of Windows: Physical Memory

Catto — Fri, 23 Jan 2009 17:56:31 GMT

Hey Now Mark,

Great post. Lots of good info.

Thx 4 the info,

Catto

re: Pushing the Limits of Windows: Physical Memory

tcsenter — Wed, 28 Jan 2009 21:37:26 GMT

Andor,

The answer to your question is explained fully under the section titled "Windows Client Memory Limits". Beginning with Service Pack 2 for Windows XP, the PAE kernel in 32-bit client versions of Windows (e.g. XP and Vista) was modified to effectively limit total physical address space to 4GB, the same as without the PAE kernel. However, this modification to the PAE kernel was not applied to 32-bit enterprise and database server SKUs (though entry-level server editions have the same limit).

re: Pushing the Limits of Windows: Physical Memory

Nihar — Tue, 03 Feb 2009 09:37:58 GMT

Mark,

I have one question. I got this info from the shop owner of laptops. I went to a shop to purchase a laptop with 4GB RAM. The shop owner told me that if you are purchasing a laptop with 4GB of RAM then go for a laptop that has 64-bit windows OS because it uses the full 4 GB and if you are going for 32-bit Laptop then go for 3GB and don't go for 4 GB because, it doesn't fully utilize 4 GB.

Is that true? I followed his advice and purchased Sony Vaio 3GB RAM Laptop with Windows Home Vista.

Please let me know.

re: Pushing the Limits of Windows: Physical Memory

operagost — Tue, 03 Feb 2009 19:55:23 GMT

"So why not leave a way to allow access to above-4gb for those who are willing to risk it ?"

Because that would not help sell copies of 64-bit Vista.

re: Pushing the Limits of Windows: Physical Memory

tcsenter — Sat, 14 Feb 2009 02:41:21 GMT

Nihar,

64-bit OS is not a guarantee. The hardware (CPU and chipset) must also support addressing beyond 32-bit. Nearly all CPUs released in the past few years are 64-bit ready, but there are still several chipsets in use that are hard limited to 32-bit addressing. In such cases, a 64-bit OS can be installed because a 64-bit ready CPU will run it, but system address space is still limited to the 'old' 32-bit standard by the chipset.

Unfortunately, this was all too common until several months ago. Major OEMs like HP, Dell, and others were configuring notebooks (and desktops to a lesser extent) with Vista 64-bit offering up to 4GB RAM, but the chipsets were often the same 'old' 32-bit architecture (Intel 945 Series was one of the more commonly used). This went on for a good year before OEMs began to shift a larger percentage of their product lines toward newer chipsets with larger addressing support. Many OEMs are still shipping notebooks with 32-bit chipsets, but its becoming the exception.

So you can't make too many assumptions about the hardware from the OS. If the system hardware can support MORE than 4GB RAM (e.g. 8GB), that is a sure-fire indication the chipset can exceed 32-bit addressing limits. But if the system hardware is advertised to support 4GB (MAX), you'll need to dig deeper and ask more questions. Some companies set an arbitrary support limit at 4GB in marketing specs even though the hardware is capable of supporting more. In other cases, 4GB could be a real limit of the hardware rather than an arbitrary marketing decision.

re: Pushing the Limits of Windows: Physical Memory

EL — Wed, 18 Feb 2009 04:37:48 GMT

Hi, Nice article!! Very informative.

However, I don't understand the principle about the limit of 4GB vs devices. Here's what you said:

[quote]

If a system has 4GB RAM and devices, like video, audio and network adapters, that implement windows into their device memory that sum to 500MB, 500MB of the 4GB of RAM will reside above the 4GB address boundary, as seen below:[/quote]

Like you said, you have a system with 4GB of RAM.

If video,audio and network adapters took 500MB of 4GB of RAM, why you can go above 4GB of RAM (beyond the physical limit) to be able to use the "stolen" 500MB ? After all, they are reserved/used by the devices and I just don't understand how it would be possible to use the "phantom" space above the physical limit of 4GB.

Thank you in advance for clarifications!

re: Pushing the Limits of Windows: Physical Memory

tcsenter — Thu, 19 Feb 2009 05:14:16 GMT

EL,

The quoted statement may lack some clarity. The 500MB address space required for devices would displace an equal amount of RAM, requiring the RAM to be placed above 4GB so that device memory can occupy the space below 4GB. The 500MB RAM then becomes inaccessible to an OS which is limited to 32-bit physical address space. i.e. it cannot be utilized

With 36-bit PAE or 64-bit OS, the 500MB RAM which is displaced above 4GB can be utilized. I hope that helps.

re: Pushing the Limits of Windows: Physical Memory

Klaus — Mon, 23 Feb 2009 05:50:25 GMT

About PAE. So when Microsoft supports 128 GB in their 32bit OSes that is 36+1 = 37-bit PAE?

re: Pushing the Limits of Windows: Physical Memory

dopey — Tue, 07 Apr 2009 21:36:17 GMT

Great article! I never knew why windows at up those gigabytes of ram before, and now I do!

Thank you!

/subscribing

re: Pushing the Limits of Windows: Physical Memory

erez — Wed, 13 May 2009 04:41:06 GMT

Let me ask a "reverse" quetion. Given all of the above discussion, is there *any* non-datacenter (I think you refer to it as "client" above) 32 bit Windows based OS that can use more then 3.x GB? I have a friend that does edits of huge images, and Photoshop is a huge memory consumer. From my reading about Windows 7, it has the same limitations -- as far as memory sizes...

re: Pushing the Limits of Windows: Physical Memory

Stavi — Fri, 29 May 2009 13:39:09 GMT

Erez has mentioned W7. Why W7 x32 has the same 4GB limit when all the drivers should have WHQL certification from Vista upwards?

When all the drivers are certified the risk of enabling more than 4GB is almost none.

Seems to me like marketing...

re: Pushing the Limits of Windows: Physical Memory

SamCPP — Mon, 01 Jun 2009 02:49:45 GMT

"What they found was that many of the systems would crash, hang, or become unbootable because some device drivers, commonly those for video and audio devices that are found typically on clients but not servers, were not programmed to expect physical addresses larger than 4GB. As a result, the drivers truncated such addresses, resulting in memory corruptions and corruption side effects. Server systems commonly have more generic devices and with simpler and more stable drivers, and therefore hadn't generally surfaced these problems. The problematic client driver ecosystem led to the decision for client SKUs to ignore physical memory that resides above 4GB, even though they can theoretically address it."

What a pathetic way around the problem. Instead on instituting REAL driver certifications, they fudge the driver rules to accommodate broken drivers. 64-bit windows is still not mainstream yet 4GB memory IS extremely common nowadays yet we can't use 1GB+ of it.

re: Pushing the Limits of Windows: Physical Memory

Auto — Mon, 06 Jul 2009 19:06:10 GMT

I think that Windows Vista 32-bit can surely manage more than 4 GB with PAE, but there is only a license problem :-<

Have a look at this very GOOD article:

http://www.geoffchappell.com/viewer.htm?doc=notes/windows/license/memory.htm

"Pushing the Limits of Windows" ?

I think they want to push the 64-bit version of the operative system :-)

Stay well.

Regearding Superfetch

Mikael Karlsson — Tue, 14 Jul 2009 00:30:57 GMT

Please make a little article about Superfetch in Windows 7 and how/why it has been nutered, and touch on if there's anything the user can do to make it work more similarly to what it does under Vista. We're a few people here and there that has been in the dark over this for a long while now. Please.

re: Pushing the Limits of Windows: Physical Memory

James Sutherland — Tue, 14 Jul 2009 00:37:23 GMT

I've known about the "memory hole" issue and having machines with 4 Gb of RAM only showing around 3 Gb under 32 bit Windows; what surprised me recently was noticing the System Properties pane showing 18 Gb on my home PC (Vista Ultimate x64), despite only having 3 Gb of actual RAM installed! Presumably just an odd cosmetic bug - if the memory management subsytem really believed it had an imaginary extra 15 Gb, things would collapse almost instantly - but bizarre.

I have a few imaging tasks (analyzing 3D images - CT scan data, as it happens) which are running into 2 and 3 Gb address space limitations; switching from the 2/2 split to 3/1 helped, but wasn't quite enough for everything. I'm starting to move our systems to the x64 version of Windows where possible, but with existing machines and data that can be a pain. As others have said here, I do wish more hardware vendors would ship x64 Windows!

re: Pushing the Limits of Windows: Physical Memory

Mehmet Iyigun — Tue, 14 Jul 2009 05:10:57 GMT

This is in response to Mikael Karlsson's comments regarding Superfetch.

During Windows7 development, we made a number of improvements to how Superfetch manages memory. Many of these changes were directly in response to customer comments. At a high level, some of these improvements are as follows:

(1) Be quieter: Even though Superfetch always utilizes low-priority I/O for its memory population in order to avoid interfering with foreground activity, we found that many users get annoyed at hearing the disk activity and seeing the disk light blink. In Windows7, Superfetch is a lot more respectful of user presence.

(2) Be more selective: In Windows7, Superfetch still populates the OS cache with frequently-accessed data from the disk and prioritizes RAM contents, but the underlying algorithms have been improved over Vista. As a result, Superfetch now typically prefetches a smaller, but more relevant volume of data from the disk and prioritizes memory more effectively.

Overall, our results (from a number of users over weeks) indicate that disk activity due to Superfetch is significantly lower in Windows7 compared to Vista while system responsiveness is much improved due to fewer hard page faults from the pagefile and other files.

There is no way to make Windows7 Superfetch bahave like its Vista predecessor.

Is there a specific scenario you're worried about?

Regearding Superfetch

Mikael Karlsson — Wed, 15 Jul 2009 03:26:45 GMT

I appreciate your response very much mr Mehmet Iyigun, thank you.

I fully expected some changes due to user feedback even if much of what I have seen users comment on Superfetch has largely been based on misunderstandings. I had hoped though that the changes were not going to be quite this drastic. To explain my concern I think that I do best to describe two situations.

A: When Windows has started up and everything has become responsive, Superfetch continues its unobtrusive work by loading roughly 6-7GB of data into the otherwise unused RAM of my humble system. After or even during this is being done I can start up larger applications like Photoshop or my latest played games and they typically load up just as fast as if I had just run them minutes ago. This makes me feel very happy about having fitted my primary system with 8GB over a year ago with already then quite inexpensive memory modules. Inexpensive means that most anyone running a 64bit Windows on a desktop system can also afford having 8GB of RAM, and greatly benefit from it thanks to Superfetch.

B: After Windows has started up and all the little apps has done so as well, Superfetch does very little for about six minutes, only then does it noticeably start to preload anything, but even then not much more than some 500MB of data into my 6-7GB of unused RAM. If I start for instance Photoshop in advance of those six minutes it starts as slowly as ever, and even after those measly 500MB have loaded none of my recently and often run games does not start up any bit noticeably faster than had Superfetch been turned off. Unless I carry out some file copy operations I rarely see more than half of the RAM being used at all for a whole day. In this situation the system might as well have had 4GB rather than 8GB. I would not notice the difference on a typical day, in stark contrast to situation A.

I hope that Superfetch might become in some manner somewhat configurable in the not too distant future. The 64bit evolution is beginning to roll in earnest and it's not too unusual for home user desktops of today to be able to sport even more than 8GB of RAM. Solid state drives won't become fast enough or affordable enough to make preemptive file caching less interesting for several years still.

re: Pushing the Limits of Windows: Physical Memory

TK — Tue, 28 Jul 2009 14:35:05 GMT

It must be possible to write a ramdrive.sys that accesses the memory controller driver directly to allow access to PAE remapped ram above the 4G barrier that 32 bit windows is scared of using and use the upper unused RAM as the primary paging file drive letter for virtual memory use, even if it was just 1G it would be a significant performance increase over vista's use of harddrive based paging file.

Re: Superfetch

Rune — Tue, 28 Jul 2009 17:13:17 GMT

Mehmet Iyigun, what I have always tried to find out (but never had time to explore myself) is: Does Superfetch benefit from ReadyBoost?

My guess is that it doesn't. It would be a neat feature to have SF load stuff from a flash thingy rather than completely strangle my hard drives.

From my POV, SF adds very little benefit. If I want to launch PS, I will launch PS. If I want to launch VS, I will launch VS. I do not need someone making educated guesses as to what I will do or not. My machine at work (XP incidentally) stays on most of the time, so everything is usually in the cache anyway. On my previous work machine I had Vista, and that machine I booted every day and SF was very annoying. At one point I had presented some pictures to friends, after which it would load a bunch of 18MB big raw files every time I started Vista...

re: Pushing the Limits of Windows: Physical Memory

Harold — Thu, 06 Aug 2009 06:38:56 GMT

Something confused me for a moment, so I believe you have a misprint you'll want to correct. The memory limit for Windows 7 will be 192GB rather than 192MB. BTW excellent article.

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Thu, 06 Aug 2009 14:31:32 GMT

@Harold

Thanks for the feedback and for pointing out the typo, which I've fixed.

re: Pushing the Limits of Windows: Physical Memory

Mark Cleary — Sat, 15 Aug 2009 15:28:58 GMT

Carmen Crincoli's blog http://blogs.msdn.com/carmencr/archive/2004/08/06/210093.aspx says you can use that extra memory by specifying both /PAE and /NOEXECUTE in boot.ini.

<quote> When you use the /NoExecute switch on these OS’s, ntldr now loads the PAE kernel, but in a special mode. You don’t get access to over 4GB of RAM, and more importantly, your drivers don’t get physical addresses over the 4GB mark either. <snip/> If you add the /PAE switch, you get the normal PAE behavior <snip/>

While both /PAE and /NoExecute use the same kernel file (ntkrnlpa.exe or ntkrpamp.exe) and address translation mechanism, you need both switches in place to enable both features. </quote>

re: Pushing the Limits of Windows: Physical Memory

Gregory — Tue, 18 Aug 2009 10:14:42 GMT

Mark Cleary: Not in XP SP2 or SP3. Not in Vista, and very unlikely in Win7 either.

You know what the real problem is? The real problem is that people have forgotten what an utter pain EMS (the expanded memory spec) was to work with, and PAE is nothing more than a jazzed up EMS. But worse? Worse is MSFT not allowing us to make up our minds. Far better would be MSFT enabling the use of PAE and memory addresses >4GB, and then disclaiming all support.

Like that's going to work in real life, though.

re: Pushing the Limits of Windows: Physical Memory

Jamie Hanrahan — Mon, 24 Aug 2009 00:17:20 GMT

Gregory: > PAE is nothing more than a jazzed up EMS.

Sorry but that is completely wrong. PAE and EMS have nothing to do with each other.

PAE is nothing more than virtual memory address translation just as x86 has always done, with the tables expanded to three stages of lookup instead of two, and with support for wider physical addresses. Without PAE there are two lookup tables (page table and page directory); PAE adds a third table, the PDPT, and makes all three tables 64 bits wide instead of 32.

And far from being an "utter pain", PAE has no impact at all on the app programmer. Anyone who tells you an app has to be modified to "take advantage of PAE" does not know whereof they speak.

Maybe they're thinking of AWE, but AWE and PAE are completely independent things.

And guess what? Address translation on x64 starts with the PAE scheme and adds a *fourth* table, the "page map level 4" table. So if you think PAE is "nothing more than a jazzed up EMS" then you must think x64 is even worse.

Take a look at http://www.intel.com/Assets/PDF/manual/253668.pdf . Look at figures 4-2 (x86 without PAE), 4-5 (x86 with PAE), and 4-8 (x64). Now please explain why you think PAE is "nothing more than a jazzed-up EMS"?

re: Pushing the Limits of Windows: Physical Memory

Geoff Chappell — Wed, 26 Aug 2009 06:38:02 GMT

"If you want [to be able to use more than 4GB in Vista] without contrivance, then pester Microsoft for an upgrade of the license data or at least for a credible, detailed reasoning of its policy for licensing your use of your computer's memory. ... [C]onsider Windows Server 2008. For the loader and kernel in Windows Vista SP1 (and, by the way, for the overwhelming majority of all executables), the corresponding executable in Windows Server 2008 is exactly the same, byte for byte. Yet Microsoft sells 32-bit Windows Server 2008 for use with as much as 64GB of memory. Does Microsoft really mean to say that when it re-badges these same executables as Windows Vista SP1, they suddenly acquire an architectural limit of 4GB? Or is it that a driver for Windows Server 2008 is safe for using with memory above 4GB as long as you don't let it interact with the identical executables from Windows Vista SP1?"

http://www.geoffchappell.com/viewer.htm?doc=notes/windows/license/memory.htm

re: Pushing the Limits of Windows: Physical Memory

Geoff Chappell — Tue, 01 Sep 2009 09:05:22 GMT

Just to be clear ... where the post dated August 25, 2009 says "by Geoff Chappell", it must be some other Geoff Chappell than the author of the page at the cited URL. I, the author, did not post that.

It looks like someone has copied from a third website and posted here under my name. I would never extract with that "... [C]" construction.

To whoever did post it, even if your name is Geoff Chappell, please don't link your name to my website. You're not me, and you should be glad of that.

Geoff Chappell.

re: Pushing the Limits of Windows: Physical Memory

MTeixeira — Mon, 07 Sep 2009 20:02:33 GMT

Thanks for another excelent article,

a bit offtopic, why do we still have 4KBytes pages (since 80386 maybe earlier?), if we had say 1MByte pages we would need a much smaller PFN, and more importantly less address remaping by the OS/CPU (I am not sure how many real memory fetchs must be done to convert a VM address to a physical one - 3?)

re: Pushing the Limits of Windows: Physical Memory

John Campbell — Fri, 09 Oct 2009 23:48:32 GMT

Hi Mark,

on my 32-bit vista, 3GB installed gives 2.86GB.

However installing 64-bit vista and 64-bit windows 7 RC still gives me 2.86GB usable. Why is that? Would really really appreciate your help on how to solve it. Sony is refusing to help!

re: Pushing the Limits of Windows: Physical Memory

cheap computers — Thu, 15 Oct 2009 11:15:32 GMT

I too realized quickly after getting my new system built that I needed to ditch 32bit.

re: Pushing the Limits of Windows: Physical Memory

Itsme — Sat, 17 Oct 2009 22:21:34 GMT

I'm glad to hear that the SuperFetch feature is less broken or maybe even fixed in Windows 7.

In concept it's a great idea. The vista iplementation was so so so bad, with it being so eager to swap out something that I was using five minutes ago and would probably be using again in five minutes for something that I use every week or two. It's like the motto was do something, do anything, just stay busy.

To the guy who was complaining about it not being aggressive enough I say so what. For every person who thinks it's not aggressive enough, twenty people think it's too aggressive.

re: Pushing the Limits of Windows: Physical Memory

gareth — Thu, 29 Oct 2009 19:41:00 GMT

I'm really going to repeat a question that Santosh asked about the PFN and didn't seem to get answered.

In the example it shows a 960MB PFN -- it explains that the 960MB is required to address 32GB of RAM.

What it doesn't explain is whether the PFN is static in size or if it is scaled based on the amount of physical RAM.

The obvious question then is when you implement 4GT and/or userVA is the PFN re-sized to leave more space for the kernel.

re: Pushing the Limits of Windows: Physical Memory

markrussinovich — Thu, 29 Oct 2009 21:30:19 GMT

@gareth

The PFN database is sized according to physical RAM, with one entry per page. 4GT has no effect on physical memory, just the split of the virtual address space between the kernel and user mode.

re: Pushing the Limits of Windows: Physical Memory

KonradK — Wed, 04 Nov 2009 08:33:21 GMT

I cannot find clear answer to one question. If I have Windows XP SP2 or SP3 and exactly 4GB of RAM installed and use /PAE switch, will I have 4GB available or is /PAE switch completely ineffectual since SP2?

In other words: is a 4GB limit for XP an address space limit or it is an amount of usable memory limit?

re: Pushing the Limits of Windows: Physical Memory

Thomas Vogler — Sun, 08 Nov 2009 10:13:49 GMT

Thank you for that insightful article. But this all is and never was an issue for me with my 8 GB Notebook. Gaahh - wait - the label reads "MacBook Pro" - oops, wrong forum. I converted. :-)

re: Pushing the Limits of Windows: Physical Memory

Thomas SAmuel — Wed, 11 Nov 2009 11:06:21 GMT

Hi Mark,

Greetings!

My HP Proliant DL 380 G4 server, has 6 GB of memory with Windows 2003 enterprise edition server(it was showing). I reconfigured RAID 5(added 3 more Hdd`s) and reinstalled windows 2003 enterprise edition server itself( same as earlier), now its showing only 3.5 GB of memory. No other changes made in server. Please help

Thanks in advance

Thomas Samuel Anish

re: Pushing the Limits of Windows: Physical Memory

EW — Wed, 11 Nov 2009 21:41:03 GMT

Thomas Samuel,

Edit boot.ini and add the /PAE switch at the end of line in the section "operating systems".

eg.

[boot loader]

timeout=3

default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS

[operating systems]

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise - 6GB" /noexecute=optout /fastdetect /PAE

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise - 4GB" /noexecute=optout /fastdetect

Greetz.

re: Pushing the Limits of Windows: Physical Memory

Alef Sin — Sat, 21 Nov 2009 10:28:19 GMT

As usual, great article Mark!

I'm trying to understand what's going on on my laptop: I'm running Windows 7 64bits and I have 4GB of RAM. The "Total memroy" reported by Windows is only 3447MB and the resource monitor is telling me that there is a 649MB "Hardware Reserved" area in the address space. Here is the meminfo report:

Physical Memory Range: 0000000000001000 to 000000000009F000 (158 pages, 632 KB)

Physical Memory Range: 0000000000100000 to 00000000D77D0000 (882384 pages, 3529536 KB)

MmHighestPhysicalPage: 882640

Given that I'm running a 64bits OS, I don't understand why only 3.3GB of RAM is available to the OS. Could it be a chipset limitaion? (this is an old intel 945 chipset).

re: Pushing the Limits of Windows: Physical Memory

molotov — Sat, 21 Nov 2009 14:58:39 GMT

Hi Alef,

This thread seems to discuss what you are dealing with. It does appear to be related to the chipset you're using.

http://social.technet.microsoft.com/Forums/en/w7itproperf/thread/7d63ee09-b33e-4ba6-a9fe-85b928f99ba6