Torgo 3000

A change of direction... sort of

Well folks, I'm still employed at MS (or rather re-employed at MS after having been canned with the rest of the group).  And I'm working on an as-yet-unannounced project.  So as long as the muzzle is still on, I'll try to cover some FSX stuff and some general technical art stuff.  I don't know how often I'll be posting (it's not like it can be any less frequent than it has been in the past ; P ) but I will try to get some useful information out there about making art for FSX or making art for games in general.  I've been doing a lot of shader writing recently, which I am really excited about (expecially now that the years of legacy code and requirements for supporting backwards-compatibility restrictions have been lifted).  It's been very freeing to be able to write a shader that does something incredibly ridiculous like use 5 separate world space cube-map lookups (diffuse w/ alpha, normal, specular, reflection and refraction).  Whether a shader like that would actually ever be used in production is totally beside the point.  It's just nice to see something really cool in the viewport, because there's always a way to optimize and make something almost as cool for half the performance cost.

I will say one thing.  Performance is about priority number one from day one on the new project.  I definitely learned my lessons from FSX.  Creating a flexible content-driven shader system when there is already mountains of legacy content not necessarily authored specifically for that system, and no time to go back and touch every single asset is a BAD idea.  If you're in games and you are carrying forward legacy content and you decide to change your shader or draw system, make absolutely sure you can either touch manually or at least do some kind of scripting pass on every piece of content.  Better yet, design a schema-based extensible shader system right from the start that can handle creating new values with defaults.

Maybe my next post will be a performance post-mortem on the Flight Simulator engine and content system.  If you've got topics you'd like to get info on, feel free to post in the comments, I'll try to cover them.  And when I can talk about what I'm working on now, I'll let you know.

A really great look at performance in FSX...

Lotus has a really great post about aircraft performance in FSX.  I highly reccomend reading the whole post.  It's a bit technical, but does put into practical terms a lot of the ideas about how to create performant aircraft (especially for multiplayer).  Draw calls are not always the single performance killer and reducing draw calls is not a silver bullet for creating objects that perform well in the engine.  I talk a little about this in my "When is a draw call not a draw call" post.  Lotus does a really great job of showing practical numbers from existing addon aircraft.

The reason for some of his findings has to do with what is being updated and sent to the GPU from the CPU.  If each of the draw calls in an aircraft is close to 65,000 vertices, then each draw call is sending almost an entirely full vertex buffer update to the card every frame.  While it is still only a single draw call, it is a huge amount of data being sent across the pipe.  So 35 draw calls with 1000 vertices each does not equal 35 draw calls with 60,000 vertices each.  And if you are using poor UV and smoothing techniques, then it is very easy to get a 25,000 polygon model close to that limit for each draw call.

Another comparison I'd love to see is the animation hierarchy in the aircraft.  An aircraft can be skinned and have 15 draw calls, but have 10 parallel animation hierarchies that are each 20 levels deep.  The amount of data needing to be updated during each frame becomes gigantic (something on the order of 2100 matrix transforms per frame).  If that same aircraft were animated with a flat hierarchy (not always easy or even possible to do) it would have about 200 transform matrix updates each frame.

All in all, it's a great post and a very informative read.  Excellent work, Lotus!

Jetway IK tutorial

I captured a tutorial about creating a Jetway using our IK system and Gmax over a year ago.  Due to software weirdness and a general lack of time, I was unable to render the video out.  However, thanks to Nick Whittome (a genius at getting me to do stuff and a prodigy at helping me out when I can't get it done on my own), the video has been rendered and is now posted at the FSDeveloper website.  You can grab the video directly from this link.  Extra special thanks to Nick and Arno for getting me to do this and providing a place for it to live.  It's a little improvised, so if you can sit through some "Ums" and "Uhs", there's decent content in there.

I also see that Nick has posted requests for future videos.  So hopefully I can get through some others over the coming months / years.

New Acceleration Pack missions.

We just released three new multiplayer racing missions for Acceleration on FSInsider.  I haven't tried them out yet, but I hear they're a lot of fun.  Plus, you can't beat the price!

SDK MaxScript tools.

There has been some confusion about what tools will work with which version of 3ds Max and I hope to set the record straight today (for those who might be using our SDK and 3ds Max).  In general, the SDK tools that were released for 3ds Max for FSX and FSX Acceleration (FSX SP2 SDK) were created and tested on 3ds Max 9 32-bit on 32-bit XP.  Since then, most of us have switched over to 64-bit Vista and are running 32-bit 3ds Max 9 on that platform.  All of the above configurations are supported.  Unfortunately, at this time, no 64-bit versions of 3ds Max are supported.  Additionally, 3ds Max 2009 is not supported at all (neither 32-bit nor 64-bit).  3ds Max 2008 32-bit (on XP and Vista) should be supported, but extensive testing has not been performed yet.

Here are the supported setups:

32-bit XP with 32-bit 3ds Max 9
32-bit Vista with 32-bit 3ds Max 9
64-bit Vista with 32-bit 3ds Max 9
32-bit XP with 32-bit 3ds Max 2008 (Supported, but not fully tested)
32-bit Vista with 32-bit 3ds Max 2008 (Supported, but not fully tested)
64-bit Vista with 32-bit 3ds Max 2008 (Supported, but not fully tested)

The next SDK we ship will support whatever the most recent version of 3ds Max is at that time (we will likely try to support a version or two back as well, but at this point no promises are being made).  Addtionally, we are still working on the alternative solution to Gmax.

Having said all of that, I know that the MaxScript tools shipped with FSX SP2 were a little buggy.  My bad (though I will say that the fact that I was the sole designer, developer and tester on the MaxScript tool suite pretty much guaranteed a certain amount of failure ;) )  I am making a real effort this time around to make all of the tools work as well out of house as they do in house. We also have more resources dedicated to guaranteeing a better user experience for developers.  As far as MaxScript tools are concerned, my philosophy is if we can use it in house, then you should be able to use it equally well wherever you are.

Cool new things...

I must say, I am really excited about where we are heading as a team in terms of tools and content creation pipeline.  I think we've finally gotten on track.  And by "on track", I mean we're actually on the rails.  We may still be sitting in the station, but at least we aren't wandering around in the weeds trying to figure out where the track actually is.

I can't promise or disclose too much, but I will say that there are some really cool things happening in the studio right now that we hope to make available to 3rd party developers as well as SDK users.  In terms of Digital Content Creation tools, I am making a serious effort to try to expose all of our in house tools to 3rd party content creators.  Ideally, you should be able to use a tool out of house exactly how we use it in house.  We haven't made that effort much previously, but we are definitely focused on it now.  ESP really helps to be a driving factor in terms of tools and pipeline.  For 3rd party developers on our entertainment products, I think you're going to see some serious improvements in our SDK thanks in large part to the development of ESP.

As for the "Gmax replacement issue", I can't say much about that right now.  It's definitely on our mind and something we are striving very hard to make happen.  I'll try to keep you updated as much as I can here.  Also, be sure to go to the FSinsider developer's corner as we are updating that more regularly.  Additionally, the FSDeveloper site is a great place to gather and share information.

Let's blow up the exporter real good!

I first came across our vertex limit issue when trying to debug the export of the trike model.  For some reason when we tried to export the Aircreation, it would continously throw up errors about exceeding the max vertex limit.  Huh?  We'd never run into this problem before.  Why were we suddenly seeing this?

Well, in our quest to up the ante in terms of visual quality, we ended up creating a model that exceeded the 65,000 vertex limit for a single draw call.  "But it's only got 27,000 polygons and 15,000 vertices!" we thought to ourselves.  What we didn't realize at the time was that a) Non-perfect UV mapping and smoothing can convert single vertices in 3ds Max or Gmax into many vertices on the graphics card and b) in the quest to reduce draw calls, we had actually run into one big draw call.  Too big, in fact.

So lets look at issue a).  At first blush, the polygon and vertex count should not be an issue at all.  One of our vertex buffers can hold roughly 65,000 vertices.  The trike should easily fit into one of those.  However, as we started to look at the UV mapping and smoothing, we realized that we were indeed exceeding the limit after processing.  Everywhere there is a seam in the UV map, those vertices get doubled on the graphics card.  If it is the corner of a seam, they get tripled or more.  The way this particular model was UVed, it had very unique seams around most of the geometry.  This resulted in a whole bunch of the vertices being doubled or tripled.  So now our vertex count was around 30,000.  Additionally, every vertex that is on the edge of two smoothing groups also has to get doubled, tripled if it is one the edge of 3 smoothing groups.  And not all of the UV seams lined up with smoothing seams.  Suddenly our vertex counts were upwards of 40,000 to 50,000.  Oh, and we also had a bug where objects with normal maps were passing double the vertex counts.  Oops, now we had exceeded the 65,000 vertex buffer. 

Which brings us to issue b).  In our quest to reduce draw calls we had created a single texture sheet for the Aircreation.  So now the entire model was being fitted into a single vertex buffer, which it couldn't fit into.

So how did we solve the problem?  Well, once we animated some parts, that broke up the draw calls.  Also, once we applied some different materials with different parameters, that also broke up the draw calls.  So if you're trying to export a model with 100,000 polygons and you are running into the vertex limit, first of all, shame on you.  ; )  But I know, it's fun to push the limits, so after getting over the initial shame, if you want to load huge models into Flightsim, you'll have to make sure that they are broken up into consumable chunks.  So even if you want the same texture and material, you may have to create a second material and change specular power to be 56 instead of 55 (or something minimal like that) and apply it to half of your gigantic model.  Then it will export as two draw calls and fill two vertex buffers.

So now you ask "Why didn't you automatically break up huge objects along vertex buffer seams so we don't have to do this hack?"  Well, unfortunately it was one of those things we just didn't get to.  And I'm sure you'd rather have a new sim engine or graphics feature than an exporter feature that has a simple work-around, right?

Hmmm... er, what?

There seems to be a consensus amongst people I know (based on highly scientific poll results of course) that 2007 was one of the most strange, stressful and just plain bizarre years on record.  Maybe it's just that all my friends are hitting their mid-thirties and freaking out, I don't know.  At any rate, after a much needed hibernation on top of a not-so-much needed year of chaos, disorder and growth, I think I've finally got the cobwebs out of my brain and am ready to start blogging and being active again.  So consider this the first shot across the bow (and no, it probably won't be a machine gun shot, more like a muzzle-loader, but at least it's a shot).  I'll follow this up with a post about our exporter and how you can blow it up real good.

Performance Art 4: Always use a texture and bones ain't free.

For this issue of the ever-continuing "Performance Art" series, I'll talk a little about flat-shaded or colored polygons and the use (or overuse) of bones in a model.

Always use a texture:

In the past, when the memory on graphics cards was very small, we tried to save texture space by flat-shading or assigning a single color to groups of polygons.  This provided color information to parts of a model that didn't need the detail of a texture and saved memory in the process.  However, with the advent of pixel shaders and graphics cards with a gig of memory on them, this is actually a performance killer now.  The reason for this is simple, every unique color is a new constant switch.  And every constant switch is a new draw call.  So those 100 flat shaded objects in our models that we thought would save texture space are now actually causing 100 new draw calls.  Sure, they're cheap draw calls, but they are draw calls none the less.  So how do we fix this problem?  Simple.  Save a small space in the texture sheet to have 4x4 blocks of solid color.  Anything that needs to reference that color, simply take all the UVs for those parts and shrink them way down to fit in the center of that 4x4 block (with room on the edges to compensate for mips).  Voila, it's still a single draw call and you get the benefit of saving texture space since you're only using the equivalent of a 64x64 texture sheet for a full 256 colors.  That can easily be fit in the gaps of a standard 1024x1024 texture sheet.

Bones Ain't Free (especially not ones in a big 'ol hierarchy)

Bones are a wonderful thing.  They allow us to do animation on the vertex level without having to store transforms for each vertex in a mesh (anyone want to try storing matrix transforms for 1000 vertices?  Okay, I know, you can do it in a texture, but that's out of scope for this discussion and still not free!)  They also allow us to combine disparate animated parts in a model into a single draw call.  However, there is a cost to having bones in a model.  The hierarchy must be traversed recursively for each update for each frame.  If you have a single node or possibly two in the hierarchy (like is the case for most of our autoskinning), then this is relatively cheap. Update parent node, update child node with parent's transforms, update child node with child's transforms, draw.  3 transforms and a draw.  Well adding a single node to the chain drastically increases the complexity of the updates: Update parent node, update child1 node with parent's transforms, update child2 node with parent's transforms, update child1 with child1's transforms, update child2 with child1's transforms, update child2 with child2's transforms, draw.  It's a geometric progression that goes from 3 updates for 2 bones to 6 for 3 to 12 for 4 to 24 for 5.  And that's every frame.  I've seen some third party add-ons that have literally hundreds of bones, many with hierarchies 3 to 6 bones deep (and sometimes more).  That's great for saving draw calls, but not so great for performance overall.  That would be the equivalent of making an aircraft that has every single Half-Life 2 character you might see onscreen at once as the wings.  And that's only one instance of that aircraft.  Imagine multi-player...

Hopefully this helps.  Keep your hierarchies simple and make sure to always have a texture.  We let out a lot of rope with this version of Flight Simulator.  As you watch us swinging from it, please listen to what we've got to say, that way only one of us has to hang.

First preview of Acceleration

Gamespot put up our E3 preview video for the Acceleration expansion pack for FSX.  You can watch it or download it.

 Enjoy!

Performance Art 3: Polygons don't matter.

As next gen technology came online, people started repeating this mantra:  "Polygons don't matter anymore."  For the longest time (okay, about a week) I was stuck thinking, "Awesome, we can throw any amount of geometry at the new cards and they'll just handle it.  Sweet!"  The part that the infamous "they" left off of that Mantra was: "...vertices do!"

So polygons don't matter anymore.  That is absolutely true.  However vertices do.  They matter absolutely.  Here's a handy little excercise to see what I mean by this.  Open up 3ds Max or Gmax (or follow along in your mind).  Create a box.  Convert it to an editable mesh and apply a UVW Unwrap modifier.  Collapse it back to a mesh and type in "getNumTverts $" in the Max Script listener window.  In the listener output window, you should see the number 8 appear.  That means it has 8 texture vertices.  That makes sense, 8 physical vertices and 8 texture vertices, right?  Now apply a UVW Map modifier to the box and choose "face" as the mapping type.  Collapse it back to a mesh and type in "getNumTverts $" in the Max Script listener. You should now see the number 36 appear in the listener output.  Huh?  36 texture vertices on a simple box?  This is because any texture vertex that is not welded gets duplicated.  That happens in the game as well.  It also happens for shading groups.  We do do some optimization and welding when we convert the geometry to a model, however any hard edge in the UVW Mapping will always cause a split in vertices.

So what this means is that even though your polygon count may be low, your vertex count may be sky high.  Like I said, we do optimize pretty heavily on export, but we can't catch every case and if the model is authored poorly from the start (completely unique texture vertices for all the faces for example) you can wind up with four times as many vertices as you intended.

So why does the vertex count matter?  Well, because all of the geometry is put onto the graphics card via vertex streams and vertex buffers.  A vertex buffer is basically 64k, which translates to ~64,000 vertices stored per buffer.  Every buffer is a single draw call.  Sometimes we fill the buffer up all the way, sometimes we don't (bad batching).  However, let's assume the best case scenario and imagine that we are batching everything perfectly.  We create a building that we wan't to appear in a scene 1,000 times.  That building has 1000 polygons.  Okay, that's a little high, but not bad for a high-detailed model.  But due to poor modeling, UVing and smoothing, the building actually has 2400 vertices in it.  64,000 / 2400 = 26 buildings per draw call.  1000 / 26 = 38.4 or 39 draw calls for those buildings.  Even though it's a perfect batch and a perfect scenario, we still require 39 draw calls for that single building 1000 times.  Let's imagine that the building was well authored and optimized and actually only had 1200 vertices in it (a more reasonable scenario).  64,000 / 1200 = 53 buildings per draw call.  1000 / 53 = 18.8 or 19 draw calls.  That's a pretty significant reduction. Especially if you have 200 variations of buildings you want to draw (200 * 39 = 7800 draw calls, 200 * 19 = 3800 draw calls).  These are all still excessive numbers, but you get the point (and also can see how creating high-polygon models with bad vertex optimization can kill the framerate quick).

So what can we do about this?  The first thing to do is author good models.  Yes we have our fair share of bad models.  As I said, we've got legacy that we just can't update every release, so we chip away at it release by release.  Make sure your smoothing groups are used wisely.  If it's a rounded object, 1 smoothing group may do.  If it's not, try to create as few smoothing groups as makes sense.  Also, weld your texture vertices and align and overlap things as much as possible.  If you have two faces overlapping in the UV space, make sure the common verts are welded.  Also, try to UV things as contiguously as possible.  If you are creating a building, create one seam on it and have all four faces lined consecutively on the texture sheet (like splitting a cylinder and flattening it out).  If the front and back and sides are identical, then make sure to weld all of the overlapping vertices.  Like I said, we do perform some automatic welding when threshholds are close, but if they aren't close in the first place, then we can't weld them.

The second thing to do is batch up parts of models when possible.  If you have several buildings with the same window type, batch up those windows with the same texture and the same material.  Since they're small in vertex count, they will all likely fit into a single draw call.  So rather than have 1200 vertex buildings that cause 19 draw calls, create smaller groups that can batch up smartly.  1 draw call for windows, 1 draw call for doors, 1 draw call for awnings and then 1 draw call for the simple building geometry that is left.  As the material batching comes online, this will be much easier to do.  But it's good practice to start thinking about this now.

Performance Art 2: When is a draw call not a draw call?

So we know that draw calls are a major killer for us.  So you would basically assume that the more draw calls there are, the worse the performance.  Well, this is not always true.  There's a funny scenario happening in the Adrenaline Expansion Pack right now.  Without the material batching, there's one area that is causing over 12,000 draw calls.  This is about 12x the USRDA recommended daily dosage of draw calls for any individual.  So it's a slide show right?  Actually, it's running at about 15-20 frames per second on most people's machines.  Which brings me to part 2 of the Performance Art thread:  When is a draw call not a draw call?

 Okay, a draw call is always a draw call.  Every new draw call involves the CPU sending information to the GPU to be rendered.  And believe me, we have our own overhead for each draw call (things like pull on ground and other procedures to make sure everything is rendering in the correct place).  However, if you have several thousand objects in a scene that all have the exact same material and textures, and are using a simple material with only diffuse and night textures, they can all be ripped out in a row without a single material switch.  Which means there is virtually no processing between each draw call except the new vertex information (in this case, incredibly simple apartment complexes of 100 vertices or less).  And on top of that, the pixel shader requires about 3 instructions.  So for 8500 of the 12000 draw calls, there is not a single material switch, very simple vertex data being sent to the card and almost no GPU calculations in the pixel shader.

So what's the lesson here?  Well the first is that we still need to do material batching, and when we do, those 8500 draw calls are going to become a handful.  The second lesson is that a draw call is not always super expensive.  If you author your content smartly (reusing textures and material settings exactly, using fewer vertices in the model and creating simple materials) you can rip through draw calls incredibly quickly.  And the benefit to authoring content that way is that when material batching comes in, all of those objects suddenly batch up into a few draw calls.  The last lesson is that we are really kicking it up a notch for the expansion pack.  Seattle, with all of the autogen cranked and traffic turned all the way up is still only about 5000 to 6000 draw calls.  I know, I say "only" like that's a low number.  It's still extremely excessive and hopefully the material batching work will really knock those numbers down and knock the framerates up.

Performance art.

Okay, so I'm not going to get dressed in a sheet and speak in a made-up language while someone projects images from my childhood on me. I'll leave that to the qualified people.  Instead I'm going to talk about performance art in the sense of art that is performant.  Having finished SP1 and looked very intensively at art and performance, saying that I've learned a lot is an understatement.  Having witnessed and worked on so many new systems for FSX (the new shader system, incredibly dense and varied autogen, new water, living world traffic, etc.) I should have looked at performance much differently.  We took the traditional route of "Let's get it all functioning and then we'll fix performance in the end."  Unfortunately, when you're dealing with a whole new shader system and an engine that is extremely draw call intensive, the art must be created up front with performance in mind.

Hopefully, through this blog and other venues, I can help people who make content for Flight Simulator X start off with performance in mind.  I will admit openly that we still have a bit of performance work left to do.  Hopefully, a lot of it will come in with the DX10 update (including more performance work for DX9) and the Xpack.  A lot of it will be how we author art content and a lot of it will be actual engine changes to draw larger batches with fewer draw calls.  However, despite this performance work, add-ons are a huge part of our product and they are notoriously bad on performance.  And we often get blamed for poor authoring choices that result in badly performing 3rd party content.  (We also share the blame for not calling out performance in the SDK and also for issuing too many draw calls to start with.)

The number one killer for us (and just about every single DX9 game out there) is draw calls.  There is a really cool little app called NVPerfHUD that will let you step through the scene's draw calls as well as profile a whole bunch of different performance metrics (sorry, it works in debug only).  When we load up FSX in NVPerfHUD on a decent graphics card (6800-ish) the graphics card is 100% idle most of the time.  Unless you get really close to an object with an expensive shader, the graphics card is never taxed.  This is because it is always waiting for the CPU to process and send the draw calls, which it handily processes with ease.

In DX9, every time you have a material switch or texture change, you are issuing a new draw call.  So if you change a single constant on the FSX material --> new draw call.  If you use a different texture --> new draw call.  So for that third party aircraft with 11 1024x1024 textures.  That's a guaranteed 11 draw calls (not to mention something like 50 or 60 megs of texture space used up with the night, spec and normal maps).  Then, if each of those textures get split into 3 different materials (glass, metal and rubber), that's 33 draw calls.  Then if that aircraft becomes an AI plane and shows up 10 times in the scene, that's 330 draw calls, a full third of the recommended draw call budget.  If you are adding an autogen 2 object to the scene, if you have more than one draw call per object, performance is going to suffer significantly.

So how do you reduce draw calls?  Well, we are taking steps right now to try to batch up all objects with like materials. We already do this with some of the autogen and the road traffic.  We are looking to do this with everything we can (ripping out the fixed function pipeline is going to be a huge help for this).  This will significantly reduce draw calls on a lot of objects.  However, it still won't fix poorly authored content (which believe me, we have plenty of on our own already and are working hard to fix).  If you're authoring autogen content or buildings, try to pack as many objects onto 1024 or smaller sheets and make sure that those objects all have the exact same FSX material constants.  Also make sure that each object is referencing a single texture sheet.  This will guarantee that each object is only a single draw call.  And when material batching comes in, dissimilar objects that reference the same texture sheet and the same material constants will get batched and drawn together.

Addtionally, as often as possible, use the textures to change the shader values as opposed to using the shader constants themselves.  If you control reflection in the diffuse alpha and specular in the specular color and specular alpha, then you should be able to set the reflection and specular power term constants at max and use the texture values to adjust accordingly.  You should be able to get rubber, metal, plastic and other surfaces in a single draw call just by adjusting the various portions of the textures.  The only thing you should need new material constants for is something like glass, chrome or an object that is transparent or glows.

Also, make sure that all of the material textures are grouped.  Put all the chrome on one sheet, because then you can have a single chrome material and a single chrome draw call.  Do the same with glass, metal, etc.  If you have 11 1024x1024 textures, and each of them has a chrome section, then you are forcing 11 new draw calls when you could have one.

Over the next few weeks, I'll talk more about draw calls and how to reduce them as well as other performance bottlenecks and pitfalls to avoid.

FSX SP1 is live!

You can download the English, French, Italian, German, Spanish or Polish versions from our website. The Japanese version is a day behind and should be up soon. I think you will be pleasantly surprised with the performance improvements in SP1.

A lot of work went into making this update really worthwhile. We found some pretty major bugs. For example, the old FS9 aircraft that we hadn't updated were using an insane amount of draw calls. The Dash-8 was 225 draw calls and is now 14. The Cherokee was 245 and is now 13. The MD83 was 375 and is now 12. So between those three aircraft, we were at about the recommended number of draw calls for a scene. Needless to say, turning on AI aircraft should be a lot more performant. We also batched the autogen a lot better and fixed a whole bunch of other performance related bugs (as well as started to offload some tasks like terrain rendering to the other cores of a multi-proc machine).

Unfortunately, the fresnel ramp is still a bit messed up. It sort of works, but doesn't really sample as it's supposed to. I wouldn't recommend using it for anything except special effects (which is what it's really designed to be used for anyway). We overused it in FSX and will be scaling back significantly on its use moving forward. It's really ideal for things like two-toned paint schemes and controlling reflections. Because it is a multiply, it doesn't make much sense on specular (unless it is really colored) because all it does is dim the specular down. At any rate, we will be fixing that bug with the DX10 update.

Have fun with SP1! I think you'll like it.

New FSInsider site and multi-mon patch.

The new FSInsider site has gone live.  There's a total redesign including a neat new look, more content and more regularly updated content.  If you're interested in finding out what we do here, it's a great place to start.

Also, there's a Flightsim specific Vista patch that addresses multi-mon support (previously a massive bug in Vista and a key feature in FS).  Here's the 64-bit version.  One of our developers noted this:

The link at the bottom of that website (http://support.microsoft.com/kb/933590/en-us) is interesting to look at if you want to see how many system files they had to change to fix this problem.

It's good to hear they're willing to put out the effort to support our users.