Microsoft.com Operations

We are the operations team that runs the Microsoft.com sites.

Blogs

Running Microsoft.com on 64 Bit…The Dependencies, The Goodness, the Gotcha’s

  • Comments 5
  • Likes

As part of our charter to be a very early adopter of emerging technologies, Microsoft.com is always pushing the envelope…we seem to live on the cutting edge and we venture out into the bleeding edge on a semi-regular basis. While the bleeding edge can be a bit hairy, (and messy…get out the band aides and sutures!!) at times, I can think of few better ways to provide better feedback into the core product teams.

 

Such is the case with the 64 bit architecture. We began testing the Windows Server x64 Edition of the O/S in early April 2004 with a single “white box” AMD64 reference machine and finally rolled it out across the board for www.microsoft.com at the end of April 2005.

 

It became evident early on that one of the compelling features of the x64 platform was the ability of running the x86 O/S on it with no performance hit. That fact alone simplified the hardware investment. We could continue to run pretty much “as is” in production while the x64 testing was proceeding.

 

The key takeaways from a hardware migration perspective were:

 

  1. Validate and test new platform in lab on a single server.
  2. Re-validate same scenarios with a single server taking live traffic.
  3. Phase in the new platform at our convenience since the x64 H/W can run the x86 O/S with no perf hit.

 

There were initial concerns around how we would port all of the legacy software applications to this new platform. The WoW64 layer virtually eliminated the need to touch or recompile any of the legacy code or 32bit ISAPIs. This was HUGE as I’m sure you can appreciate. The x64 bit processors have x86 registers so the perf hit was again negligible.

 

Nirvana’s new song may well be titled “Heaven Is Running Native x64”, but our software migrations did have some dependencies that required us to sing the song “Running 32bit IIS Worker Processes Under WoW64”. (Yikes was that a metaphorical stretch or what…)

Those dependencies were:

 

  1. ASP.Net 1.1 – there is no 64bit flavor for version 1.1 of the Framework – this new 64bit flavor is there in 2.0
  2. ISAPI extensions & filters – again leagacy code compiled years ago on x86
  3. Custom COM components – registered on the site that older ASP app use.

 

For details on how to set a metabase key that tells IIS to use 32bit worker processes instead of native 64bit go to this KB article “KB895976. How to run 32b it worker processes on 64-bit” http://support.microsoft.com/default.aspx?scid=kb;en-us;895976

 

The software migration gave us these key takeaways:

 

  1. Zero code changes required to move to x64 platform.
  2. Legacy ISAPI/COM components migrated without issue.
  3. ASP.Ney 1.1 apps migrated without issue.
  4. 32-bit processes running under WoW64 get 4Gb of addressable virtual memory.
  5. kernel memory isn’t sacrificed to squeeze out more VM for usermode processes. (Even with 32bit IIS worker processes, the kernel runs native 64bit.)

 

OK so aside from the fact that we really “like” to test drive new stuff…in production…(strap yourself in, hold on and take deep breath!). What are the compelling Performance reasons for going to x64? It turns out that they are substantial, primarily because of the larger virtual memory space available to IIS worker processes.

 

  1. App pools that recycling every 5 minutes on x86 leveled off and stopped recycling all together.
  2. Memory leaks or ASP.Net cache abuse from poorly behaving application were able to be identified. (2GB address space make its difficult to differentiate between an app leaking memory and one that was caching as part of its normal behavior.)
  3. 64bit addressing results in possibility of 2^64 unique address combinations (16 exabytes).  X64 O/S uses 43 bits for addressing giving the kernel 8TB of addressable memory which leaves the other 8TB for user mode processes. Since the 32 bit processed no longer need to share ½ of the addressing space with the kernal, there is the full 4GB addressing space to use running under WoW64.
  4. Processor utilization improved dramatically, with average proc utilization of the x64 was 34% vs 64% for x86, primarily because apppools were no longer recycling or had settled down to recycling over weeks not minutes.
  5. x86 servers had noticeable CPU spikes to 100% for a sustained time period, due to one or more apppools recycling because they had run out of virtual memory. X64 server did not see these spikes because apppools are not running out of VM.

 

To give you a quick comparison:

X86 ASP  req/sec 7.85, Response time 244ms

X86 ISAPI req/sec 110.85, Response time 248ms

X86 Static req/sec 41.9, Response time 135ms

X86 Static (cached) req/sec 47.11 Response time 1ms

 

X64 ASP  req/sec 7.41, Response time 53ms

X64 ISAPI req/sec 125.43, Response time 18ms

X64 Static req/sec 31.01, Response time 3ms

X64 Static (cached) req/sec 54.51 Response time 1ms

 

(These figures generated using (SPA) Server Performance Advisor)

 

So to sum the x64 performance gains:

  1. Response time significantly improved for apps due to larger VM space
  2. Significant drop in CPU due to less frequent worker process recycling
  3. Server consolidation enhanced due to bette4r performance per server.

 

So I can hear some of you now saying, “All this is great stuff. But are you telling me there were NO problems?”

 

Actually, there were none. OK there were a few. We did have a couple of hiccups along the way and here are our Gotcha’s

 

1.    Make sure to verify 3rd party dependencies, especially x64 compatible drivers for things like Antivirus Software; Backup Software, Imaging/Deployment Software, and common admin tools that use filter drivers like filemon and regmon.

2.    Wow64 file system and registry redirection behaviors work a little differently in x64.

a.    File system redirection – a 32bit process that attempts to access c:\windows\system32 get redirected by the Wow64 layer into c:\windows\syswow64. This holds all the 32 bit Windows binaries and prevents a 32bit process from trying to load a 64 bit binary…same goes for any scripts or tools running in a 32 bit process.

3.    Registry Reirection

a.     Works similar to the file system redirection. A 32 bit process trying to read or write to HKEY_LOCAL_MACHINE\Software gets redirected to HKEY_LOCAL_MACHINE\Software\WoW6432Node. This is more behind the scenes goodness that allow separate configurations to be maintained for 32/64 bit processes. Remember: any custom setting or keys set in this node may need to exist in both keys.

4.    Script Host Bitness (cscript.exe/wscrpit.exe)

a.    Scripts with dependencies on x86 COM objects need to be run with 32bit cscript or wscript. Specify either he 32bit or 64bit version of the scripting host.

 

X64 has had a huge beneficial impact on running www.microsoft.com. We welcome your questions about what we did, what we used, the methodology we used, our choice of musical metaphors or <your question goes here>.  

Comments
  • Gotcha number 1 can take unusual forms. Some of the third parties who provide things like defraggers seem to think that editions of their products that support XP Pro x64 should cost more than editions that only support XP Pro x86. The community needs to educate these providers on what going 64 means and does not necessarily mean. If much of that happens with Vista x64 there will be some speed bumps.

  • Do you have any stats on 32bit vs 64bit SQL Server (2000 or 2005)?

  • With the SPA response time comparison, it does look like X64 had a significant improvement.  But what was the CPU speed/configuration and amount RAM between the server hardware?  It seems like the improvement came from large increase in addressable memory.

  • I was reading up on 64 bit and comparsions to 32 bit. This is a few links with some highlights that stuck

  • I was reading up on 64 bit and comparsions to 32 bit. This is a few links with some highlights that stuck

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment