Pushing the Limits of Windows: Processes and Threads

re: Pushing the Limits of Windows: Processes and Threads

Sanket Patel — Thu, 09 Jul 2009 03:08:34 GMT

Hi Mark, I am looking at a VMMap snap of Notepad.exe on Win7 x64, and I am seeing a "Thread Stack (Wow64)" with a commit of 224K. Why does a 64-bit process have a Wow64 stack?

re: Pushing the Limits of Windows: Processes and Threads

markrussinovich — Thu, 09 Jul 2009 03:15:33 GMT

I don't see that. Email me the .mmp file and I'll take a look.

re: Pushing the Limits of Windows: Processes and Threads

James Bray — Thu, 09 Jul 2009 14:59:44 GMT

Great article.

I love the VMMap utility too. I'm a software developer myself, and the system I help maintain runs under Interix (SUA/SFU). We were having a problem recently whereby one of our applications was vaporizing under heavy load. We finally discovered that it was running out of stack space because, under Interix, the maximum stack size *is* hardcoded (possibly a GCC thing). I just reran our application using VMMap and it shows the stack growing from ~100K to 16MB. I only wish we'd had this utility then :-)

James

re: Pushing the Limits of Windows: Processes and Threads

zizebra — Thu, 09 Jul 2009 15:19:28 GMT

You never cease to amaze me with interesting articles you write. I thought I was at the helm of my profession till I discovered your articles and I discovered I was old enough but only a child. I cant seem to understand one thing though, why dont we have built alerts/counters for critical resources when they go so low before something can be done. Windows always gets the blame for poor programming out there.

re: Pushing the Limits of Windows: Process and Threads

Ross Presser — Thu, 09 Jul 2009 17:44:57 GMT

Yesterday I posted a comment showing that my Windows XP 32-bit system, running on a 64-bit capable processor but installed as 32-bit Windows XP, not booted with the /3GB or any other special switch, but with 3GB of RAM, can produce over 50,000 threads with "testlimit -t" (no -n). That comment has not shown up here. What's the explanation?

re: Pushing the Limits of Windows: Process and Threads

vadmyst — Thu, 09 Jul 2009 23:25:47 GMT

zizebra,

I sometimes get the an error about low virtual memory, do you mean that kind of warnings/alerts?

However it would be nice to have a built-in tool that could somehow report status of particular resources on Windows.

For instance, CPU is over 80% or available memory is low for considerable amount of time - issue warning

re: Pushing the Limits of Windows: Process and Threads

Andrei Volkov — Thu, 09 Jul 2009 23:33:01 GMT

I wonder what's the limit on the number of fibers? Like say if I want to use a fiber per each concurrent user session of my web app...

re: Pushing the Limits of Windows: Process and Threads

Spiff — Thu, 09 Jul 2009 23:41:08 GMT

Nice article. Please forgive my amateur question, but is it possible for me to "Empty Working Set" directly in my program? Some users complain about the seemingly high Working Set usage of my program when viewed in Task Manager, and I notice that "Empty Working Set" reduces it quite nicely. :) So I thought maybe I could programmatically trim it down during program startup.

re: Pushing the Limits of Windows: Process and Threads

markrussinovich — Fri, 10 Jul 2009 06:46:12 GMT

@Ross Presser:

Take a look in VMMap at the stack reserves.

re: Pushing the Limits of Windows: Process and Threads

Tony — Fri, 10 Jul 2009 11:52:18 GMT

I always thought that ASLR stands for "Address Space Layout Randomization" not "Address Space Load Randomization".

typo?

mingbo wan — Fri, 10 Jul 2009 12:27:15 GMT

The 64-bit stack has a reserve of 256MB (except that on systems prior to Vista, the initial thread’s 64-bit stack is 1MB).

32-bit Testlimit was able to create 3,204 threads on 64-bit Windows, which given that each thread uses 1MB+256MB of address space for stack

should be 256KB?

re: Pushing the Limits of Windows: Process and Threads

markrussinovich — Fri, 10 Jul 2009 15:22:10 GMT

@mingbo wan:

No, because a 32-bit thread has both 32-bit and 64-bit stack reserves.

re: Pushing the Limits of Windows: Process and Threads

markrussinovich — Fri, 10 Jul 2009 15:25:39 GMT

@Tony:

Good catch, I've fixed the text. ASLR in Vista originally stood for Address Space Load Randomization, so I accidentally use that sometimes.

re: Pushing the Limits of Windows: Process and Threads

mlynch — Fri, 10 Jul 2009 20:40:36 GMT

I just want to make sure you understood what mingbo wan is saying, because I'm confused as well:

"The 64-bit stack has a reserve of *256MB* (except that on systems prior to Vista, the initial thread’s 64-bit stack is 1MB)."

...

"32-bit Testlimit was able to create 3,204 threads on 64-bit Windows, which given that each thread uses *1MB+256MB* of address space for stack (again, except the first on versions of Windows prior to Vista, which uses 1MB+1MB), is exactly what you’d expect."

mingbo wan was saying that these should read 256 *kilobytes* not megabytes. If you wrote MB deliberately, then I'm a bit confused because your accompanying VMMap output shows the 64-bit thread stack as 256 KB and mathematically 256 MB doesn't add up.

3,204 threads * (1 MB + 256 MB stack) = 813 GB address space

whereas

3,204 threads * (1 MB + 256 KB stack) = 4,005 MB = ~ 4 GB address space

Which is roughly the expected address space size of a 32-bit process on 64-bit Windows. Is it really not a typo?

re: Pushing the Limits of Windows: Process and Threads

markrussinovich — Fri, 10 Jul 2009 20:52:38 GMT

@mlynch

Ah, I see, sorry! It was a typo. Fixed.

re: Pushing the Limits of Windows: Process and Threads

Marcel — Mon, 13 Jul 2009 15:28:52 GMT

Interessting article, but I've got one question regarding the guard page(s) you mentioned. How many are there usually and what's going on if I write

char x[32768];

x[32767] = 0xff;

in my code while most of the stack area occupied by the x array was still uncommited? Potentially I think the x[32767] could point to memory AFTER the guard pages, but what happens then?

re: Pushing the Limits of Windows: Process and Threads

Michael Grier [MSFT] — Mon, 13 Jul 2009 20:40:42 GMT

Marcel: the compiler emits instructions to touch all of the stack pages in the frame (if it's larger than one page) to address this issue. One more reason not to have large stack allocated buffers if you can help it.

re: Pushing the Limits of Windows: Process and Threads

Marcel — Tue, 14 Jul 2009 14:05:47 GMT

@Micheal Grier: You're right, thanks a lot! I've disassembled and reverse engineered a good deal of code in my time, but never noticed this behaviour before. Weird, but very interesting.

re: Pushing the Limits of Windows: Process and Threads

dave — Tue, 14 Jul 2009 16:12:06 GMT

>Please forgive my amateur question, but is it possible for me to "Empty Working Set" directly in my program?

Sure is. There is a Win32 function confusingly called "EmptyWorkingSet" which does what you want :-)

This is a relatively newly-documented function, I think. You have always been able to achieve the same effect by SetProcessWorkingSetSize(hProc, -1, -1)

(Side note: I just looked up the defn of SetProcessWorkingSetSize in the SDK doc. Who let the words "physical RAM memory" pass review? Bah...)

Generally speaking, it may be useful for a service to trim its working set after once-only initialization, and possibly when it becomes idle, if it's a mostly-idle sort of service. But you have to take care to avoid pessimising performance: releasing real memory is not an automatic performance win.

re: Pushing the Limits of Windows: Process and Threads

arnshea — Tue, 14 Jul 2009 22:40:24 GMT

I've been eagerly following this series. It's amazing.

One minor issue, the article mentions that the PE specifies an initial commit size of 4k for the thread stack but I believe in hex 0x1000 = 8k? BTW, VMMap shows 8k...

re: Pushing the Limits of Windows: Process and Threads

arnshea — Tue, 14 Jul 2009 22:48:20 GMT

About my previous statement, it looks like dumpbin is showing the total initial memory committed for thread stack space - 4k for stack, 4k for guard page. Sorry about that!

re: Pushing the Limits of Windows: Process and Threads

Scott Doty — Fri, 17 Jul 2009 18:09:40 GMT

"Unlike some UNIX variants" --

Hahahaha, that UNIX(tm) stuff (and even Unix (no tm) stuff still gets under your skin, hmmmmm? ;)

Thank the gawds of computing that we had Unixes to run, back when VMS/WNT/Windows/whatever were still trying to play catch up...

Hope life has been good to ya these past 10 years -- and my condolences re: Microsoft Vista and Microsoft Windows Vista 7, but $MSFT richly-deserves to crumble after all the uncompetitive crap we've put up from them all these years...

Sincerely,

A Sincere Fedora Fanatic.

p.s. btw, UNIX(r) is a registered trademark of The Open Group. That sort of thing will become important, as $MSFT starts looking for UNIX(r) certification, hehe...

Keep smiling! :)

Scotty doesn'

Yonah — Wed, 22 Jul 2009 12:29:51 GMT

Scotty, I like the fact that in response to Mark's technical criticism of Linux, all you could really do is offer a "works better for me" citation and make fun of Mark's credentials and education. Like most zealots you are heavy on dogma, light on substance. From that Usenet posting, looks like the only thing that hasn't matured in those 10 years is you.

re: Pushing the Limits of Windows: Processes and Threads

Chuck Desylva — Sat, 25 Jul 2009 01:56:47 GMT

Fantasitic post Mark. Thanks! Here is a crazy request: How about the same info for the XBox360? A close up architectural review like this should prove very insightful.

I am particularly interested in how one can reduce the huge network footprint the 360 has. It literally cripples other PCs to a crawl in a home network when it is using the network like downloading content.

re: Pushing the Limits of Windows: Processes and Threads

Will — Tue, 28 Jul 2009 04:04:08 GMT

Mark,

I have one simple question related to paging. I have paging off because I have 8 GB of RAM and I figure that's enough for me to run without paging; however, I still get tons of page faults. Why? There's no page file, there shouldn't be anything to retrieve from disk back to memory.

Thanks.

re: Pushing the Limits of Windows: Processes and Threads

Ross Presser — Thu, 30 Jul 2009 18:22:02 GMT

Now that VMMap can actually RUN on my 32-bit Windows XP, thanks to today's update, I can do as you asked and look at the stack resources.

I ran testlimit -t and hit control-S after a few seconds; the thread count was already up to 7100. I started VMMap and attached to the testlimit process.

The first thread stack was Size=256K, committed=12K; the remaining 7199 or so thread stacks were all size=64K, committed=8k. Total stack shown in the upper summary area had size=461,056K, committed=57,612K; free showed Size=1,595,268K.

If I let it run and it continues at this pace, I compute it should generate at least 30,000 threads. Letting it run ...

Created 30645 threads. Lasterror: 8

My recollection is that when I tried this right after your blog post, I got over 50,000 threads. Later today I am going to shut down all unnecessary programs and services and try this again.

Given this result, do you think you should revise your blog entry? Like I said, I haven't done anything to tweak the system to get these results. It's a stock Dell system; 3GB of RAM; the processor is 64-bit capable but running 32-bit Windows XP; no /3GB or any other boot switches.

(Since the last few times I've posted to this blog, my comment has been ignored, I am going to post this same comment as a blog entry on my LJ, rpresser.livejournal.com.)

re: Pushing the Limits of Windows: Processes and Threads

Larry — Wed, 05 Aug 2009 16:40:09 GMT

Thanks for the great post Dr. Russinovich

re: Pushing the Limits of Windows: Processes and Threads

Brian — Thu, 06 Aug 2009 17:26:47 GMT

Will,

Techically you have the pagefile turned off, but paging is still very active.

EXEs are not brought into memory in full, they are brought into memory as needed in pages. So if a program attempts to access code/data that is not resident in memory yet, then a page fault occurs and the system loads the page.

Even with the pagefile disabled, if the system starts to see memory pressure, then it can take code/data and discard it (because it can just load it again from the exe/dll if needed).

re: Pushing the Limits of Windows: Processes and Threads

Colomoto — Thu, 06 Aug 2009 17:49:18 GMT

Great article to look behind the scenes of the different Windows flavors.

re: Pushing the Limits of Windows: Processes and Threads

Hex — Mon, 10 Aug 2009 18:38:55 GMT

Hello Mark,

I did my own test running multiple copies of IE 7 on Vista Enterprise SP2 (32-bit). And my result differs from your for 32-bit systems. I was running about 2150 threads when GDI died because of GDI objects limit.

I can even show the video I've made from Vmware(it's russian locale):

http://s785.photobucket.com/albums/yy136/hexello/?action=view&current= VistaVMMovie.flv

The better picture can be achieved if press "full size" and then resize the screen about 1.5 times.

See the threads count shown by Task Manager.

re: Pushing the Limits of Windows: Processes and Threads

Yimin Chen — Sun, 30 Aug 2009 18:15:57 GMT

Using VMMap to attach to a managed process, i saw there was one thread with 2 guard pages. What is that means?

Here is the content copied from the saved .mmp file (lines 17A000 and 175000 are marked as Read/Write/Guard):

1 80000 0 0 0 4096 0 0 2956 0 8192 1 Thread Stack;

1 81000 0 0 0 995328 995328 0 2956 4 131072 1 Thread Stack;

1 174000 0 0 0 4096 0 0 2956 0 8192 1 Thread Stack;

1 175000 0 0 0 4096 4096 0 2956 260 131072 1 Thread Stack;

1 176000 0 0 0 16384 16384 16384 2956 4 131072 1 Thread Stack;

1 17a000 0 0 0 8192 8192 0 2956 260 131072 1 Thread Stack;

1 17c000 0 0 0 16384 16384 16384 2956 4 131072 1 Thread Stack;

re: Pushing the Limits of Windows: Processes and Threads

Cem PAYZIN — Sat, 05 Sep 2009 01:23:02 GMT

What about Timer limits. I heard that Win32 has a limit of 10.000 per process and 32.000 for desktop. Do you know anyting about Timer limits on x64 Windows like Windows 2008 x64 ?

re: Pushing the Limits of Windows: Processes and Threads

Hannan — Sat, 05 Sep 2009 09:08:47 GMT

I love the VMMap utility too. I'm a software developer myself, this article is useful for me to learn better about VM .I want to share it with more and more people by add this links in my foler/URL.

re: Pushing the Limits of Windows: Processes and Threads

Fred Swalef — Tue, 15 Sep 2009 23:41:31 GMT

Appreciated the article and of course the tools you have provided. It really helps when your trying to work out issues/ troubleshooting the whys and wherefores. Remember when windows first came out and you could just monitor the resources you were using? Thanks again Mark you are an asset to IT community.

re: Pushing the Limits of Windows: Processes and Threads

Raymond Chen — Thu, 01 Oct 2009 17:51:55 GMT

Note that strictly speaking, you can have more processes than threads if you use zombie processes (which have no threads). Mind you, zombie processes aren't runnable...

re: Pushing the Limits of Windows: Processes and Threads

markrussinovich — Thu, 01 Oct 2009 18:59:28 GMT

@Raymond: good point, you'll see that if the last thread in the process exits, but an application has a handle open to the process or a driver has a reference to the process object.

re: Pushing the Limits of Windows: Processes and Threads

Osvaldo D Santos — Sun, 04 Oct 2009 01:31:08 GMT

Hi Mark,

Could you please give me an example of a process and a thread in a simple program?

Thank you very much.

Regards.

re: Pushing the Limits of Windows: Processes and Threads

Tom Slavens — Thu, 05 Nov 2009 18:45:14 GMT

Mark,

Someone already asked about fiber limits, but I didn't see any response to it and didn't find a later article on fibers. Could you shed any light on fiber limits? Thanks.

re: Pushing the Limits of Windows: Processes and Threads

jam — Fri, 13 Nov 2009 12:55:52 GMT

Mark,

I have an unusual problem with my OpenGL program regarding "Empty Working Set". Maybe you can help.

On a 8GB XP64 workstation and 4 core, if I use a large amount of memory with my application (lets say about 4GB) suddenly, the rendering performance is slowing down dramatically (is getting a sort of stalled). First, after minimizing the window (enforcing EmptyWorkingSet) the renderer performs quite well again as he initially did.

What can be the reason for this phenomenon. In case the application only uses about 2 GB the performance level is unchanged. I switched of the swap memory file as well.

Thax!