Torgo 3000

Cool new things...

I must say, I am really excited about where we are heading as a team in terms of tools and content creation pipeline.  I think we've finally gotten on track.  And by "on track", I mean we're actually on the rails.  We may still be sitting in the station, but at least we aren't wandering around in the weeds trying to figure out where the track actually is.

I can't promise or disclose too much, but I will say that there are some really cool things happening in the studio right now that we hope to make available to 3rd party developers as well as SDK users.  In terms of Digital Content Creation tools, I am making a serious effort to try to expose all of our in house tools to 3rd party content creators.  Ideally, you should be able to use a tool out of house exactly how we use it in house.  We haven't made that effort much previously, but we are definitely focused on it now.  ESP really helps to be a driving factor in terms of tools and pipeline.  For 3rd party developers on our entertainment products, I think you're going to see some serious improvements in our SDK thanks in large part to the development of ESP.

As for the "Gmax replacement issue", I can't say much about that right now.  It's definitely on our mind and something we are striving very hard to make happen.  I'll try to keep you updated as much as I can here.  Also, be sure to go to the FSinsider developer's corner as we are updating that more regularly.  Additionally, the FSDeveloper site is a great place to gather and share information.

Let's blow up the exporter real good!

I first came across our vertex limit issue when trying to debug the export of the trike model.  For some reason when we tried to export the Aircreation, it would continously throw up errors about exceeding the max vertex limit.  Huh?  We'd never run into this problem before.  Why were we suddenly seeing this?

Well, in our quest to up the ante in terms of visual quality, we ended up creating a model that exceeded the 65,000 vertex limit for a single draw call.  "But it's only got 27,000 polygons and 15,000 vertices!" we thought to ourselves.  What we didn't realize at the time was that a) Non-perfect UV mapping and smoothing can convert single vertices in 3ds Max or Gmax into many vertices on the graphics card and b) in the quest to reduce draw calls, we had actually run into one big draw call.  Too big, in fact.

So lets look at issue a).  At first blush, the polygon and vertex count should not be an issue at all.  One of our vertex buffers can hold roughly 65,000 vertices.  The trike should easily fit into one of those.  However, as we started to look at the UV mapping and smoothing, we realized that we were indeed exceeding the limit after processing.  Everywhere there is a seam in the UV map, those vertices get doubled on the graphics card.  If it is the corner of a seam, they get tripled or more.  The way this particular model was UVed, it had very unique seams around most of the geometry.  This resulted in a whole bunch of the vertices being doubled or tripled.  So now our vertex count was around 30,000.  Additionally, every vertex that is on the edge of two smoothing groups also has to get doubled, tripled if it is one the edge of 3 smoothing groups.  And not all of the UV seams lined up with smoothing seams.  Suddenly our vertex counts were upwards of 40,000 to 50,000.  Oh, and we also had a bug where objects with normal maps were passing double the vertex counts.  Oops, now we had exceeded the 65,000 vertex buffer. 

Which brings us to issue b).  In our quest to reduce draw calls we had created a single texture sheet for the Aircreation.  So now the entire model was being fitted into a single vertex buffer, which it couldn't fit into.

So how did we solve the problem?  Well, once we animated some parts, that broke up the draw calls.  Also, once we applied some different materials with different parameters, that also broke up the draw calls.  So if you're trying to export a model with 100,000 polygons and you are running into the vertex limit, first of all, shame on you.  ; )  But I know, it's fun to push the limits, so after getting over the initial shame, if you want to load huge models into Flightsim, you'll have to make sure that they are broken up into consumable chunks.  So even if you want the same texture and material, you may have to create a second material and change specular power to be 56 instead of 55 (or something minimal like that) and apply it to half of your gigantic model.  Then it will export as two draw calls and fill two vertex buffers.

So now you ask "Why didn't you automatically break up huge objects along vertex buffer seams so we don't have to do this hack?"  Well, unfortunately it was one of those things we just didn't get to.  And I'm sure you'd rather have a new sim engine or graphics feature than an exporter feature that has a simple work-around, right?

Hmmm... er, what?

There seems to be a consensus amongst people I know (based on highly scientific poll results of course) that 2007 was one of the most strange, stressful and just plain bizarre years on record.  Maybe it's just that all my friends are hitting their mid-thirties and freaking out, I don't know.  At any rate, after a much needed hibernation on top of a not-so-much needed year of chaos, disorder and growth, I think I've finally got the cobwebs out of my brain and am ready to start blogging and being active again.  So consider this the first shot across the bow (and no, it probably won't be a machine gun shot, more like a muzzle-loader, but at least it's a shot).  I'll follow this up with a post about our exporter and how you can blow it up real good.

Performance Art 4: Always use a texture and bones ain't free.

For this issue of the ever-continuing "Performance Art" series, I'll talk a little about flat-shaded or colored polygons and the use (or overuse) of bones in a model.

Always use a texture:

In the past, when the memory on graphics cards was very small, we tried to save texture space by flat-shading or assigning a single color to groups of polygons.  This provided color information to parts of a model that didn't need the detail of a texture and saved memory in the process.  However, with the advent of pixel shaders and graphics cards with a gig of memory on them, this is actually a performance killer now.  The reason for this is simple, every unique color is a new constant switch.  And every constant switch is a new draw call.  So those 100 flat shaded objects in our models that we thought would save texture space are now actually causing 100 new draw calls.  Sure, they're cheap draw calls, but they are draw calls none the less.  So how do we fix this problem?  Simple.  Save a small space in the texture sheet to have 4x4 blocks of solid color.  Anything that needs to reference that color, simply take all the UVs for those parts and shrink them way down to fit in the center of that 4x4 block (with room on the edges to compensate for mips).  Voila, it's still a single draw call and you get the benefit of saving texture space since you're only using the equivalent of a 64x64 texture sheet for a full 256 colors.  That can easily be fit in the gaps of a standard 1024x1024 texture sheet.

Bones Ain't Free (especially not ones in a big 'ol hierarchy)

Bones are a wonderful thing.  They allow us to do animation on the vertex level without having to store transforms for each vertex in a mesh (anyone want to try storing matrix transforms for 1000 vertices?  Okay, I know, you can do it in a texture, but that's out of scope for this discussion and still not free!)  They also allow us to combine disparate animated parts in a model into a single draw call.  However, there is a cost to having bones in a model.  The hierarchy must be traversed recursively for each update for each frame.  If you have a single node or possibly two in the hierarchy (like is the case for most of our autoskinning), then this is relatively cheap. Update parent node, update child node with parent's transforms, update child node with child's transforms, draw.  3 transforms and a draw.  Well adding a single node to the chain drastically increases the complexity of the updates: Update parent node, update child1 node with parent's transforms, update child2 node with parent's transforms, update child1 with child1's transforms, update child2 with child1's transforms, update child2 with child2's transforms, draw.  It's a geometric progression that goes from 3 updates for 2 bones to 6 for 3 to 12 for 4 to 24 for 5.  And that's every frame.  I've seen some third party add-ons that have literally hundreds of bones, many with hierarchies 3 to 6 bones deep (and sometimes more).  That's great for saving draw calls, but not so great for performance overall.  That would be the equivalent of making an aircraft that has every single Half-Life 2 character you might see onscreen at once as the wings.  And that's only one instance of that aircraft.  Imagine multi-player...

Hopefully this helps.  Keep your hierarchies simple and make sure to always have a texture.  We let out a lot of rope with this version of Flight Simulator.  As you watch us swinging from it, please listen to what we've got to say, that way only one of us has to hang.

First preview of Acceleration

Gamespot put up our E3 preview video for the Acceleration expansion pack for FSX.  You can watch it or download it.

 Enjoy!

Performance Art 3: Polygons don't matter.

As next gen technology came online, people started repeating this mantra:  "Polygons don't matter anymore."  For the longest time (okay, about a week) I was stuck thinking, "Awesome, we can throw any amount of geometry at the new cards and they'll just handle it.  Sweet!"  The part that the infamous "they" left off of that Mantra was: "...vertices do!"

So polygons don't matter anymore.  That is absolutely true.  However vertices do.  They matter absolutely.  Here's a handy little excercise to see what I mean by this.  Open up 3ds Max or Gmax (or follow along in your mind).  Create a box.  Convert it to an editable mesh and apply a UVW Unwrap modifier.  Collapse it back to a mesh and type in "getNumTverts $" in the Max Script listener window.  In the listener output window, you should see the number 8 appear.  That means it has 8 texture vertices.  That makes sense, 8 physical vertices and 8 texture vertices, right?  Now apply a UVW Map modifier to the box and choose "face" as the mapping type.  Collapse it back to a mesh and type in "getNumTverts $" in the Max Script listener. You should now see the number 36 appear in the listener output.  Huh?  36 texture vertices on a simple box?  This is because any texture vertex that is not welded gets duplicated.  That happens in the game as well.  It also happens for shading groups.  We do do some optimization and welding when we convert the geometry to a model, however any hard edge in the UVW Mapping will always cause a split in vertices.

So what this means is that even though your polygon count may be low, your vertex count may be sky high.  Like I said, we do optimize pretty heavily on export, but we can't catch every case and if the model is authored poorly from the start (completely unique texture vertices for all the faces for example) you can wind up with four times as many vertices as you intended.

So why does the vertex count matter?  Well, because all of the geometry is put onto the graphics card via vertex streams and vertex buffers.  A vertex buffer is basically 64k, which translates to ~64,000 vertices stored per buffer.  Every buffer is a single draw call.  Sometimes we fill the buffer up all the way, sometimes we don't (bad batching).  However, let's assume the best case scenario and imagine that we are batching everything perfectly.  We create a building that we wan't to appear in a scene 1,000 times.  That building has 1000 polygons.  Okay, that's a little high, but not bad for a high-detailed model.  But due to poor modeling, UVing and smoothing, the building actually has 2400 vertices in it.  64,000 / 2400 = 26 buildings per draw call.  1000 / 26 = 38.4 or 39 draw calls for those buildings.  Even though it's a perfect batch and a perfect scenario, we still require 39 draw calls for that single building 1000 times.  Let's imagine that the building was well authored and optimized and actually only had 1200 vertices in it (a more reasonable scenario).  64,000 / 1200 = 53 buildings per draw call.  1000 / 53 = 18.8 or 19 draw calls.  That's a pretty significant reduction. Especially if you have 200 variations of buildings you want to draw (200 * 39 = 7800 draw calls, 200 * 19 = 3800 draw calls).  These are all still excessive numbers, but you get the point (and also can see how creating high-polygon models with bad vertex optimization can kill the framerate quick).

So what can we do about this?  The first thing to do is author good models.  Yes we have our fair share of bad models.  As I said, we've got legacy that we just can't update every release, so we chip away at it release by release.  Make sure your smoothing groups are used wisely.  If it's a rounded object, 1 smoothing group may do.  If it's not, try to create as few smoothing groups as makes sense.  Also, weld your texture vertices and align and overlap things as much as possible.  If you have two faces overlapping in the UV space, make sure the common verts are welded.  Also, try to UV things as contiguously as possible.  If you are creating a building, create one seam on it and have all four faces lined consecutively on the texture sheet (like splitting a cylinder and flattening it out).  If the front and back and sides are identical, then make sure to weld all of the overlapping vertices.  Like I said, we do perform some automatic welding when threshholds are close, but if they aren't close in the first place, then we can't weld them.

The second thing to do is batch up parts of models when possible.  If you have several buildings with the same window type, batch up those windows with the same texture and the same material.  Since they're small in vertex count, they will all likely fit into a single draw call.  So rather than have 1200 vertex buildings that cause 19 draw calls, create smaller groups that can batch up smartly.  1 draw call for windows, 1 draw call for doors, 1 draw call for awnings and then 1 draw call for the simple building geometry that is left.  As the material batching comes online, this will be much easier to do.  But it's good practice to start thinking about this now.

Performance Art 2: When is a draw call not a draw call?

So we know that draw calls are a major killer for us.  So you would basically assume that the more draw calls there are, the worse the performance.  Well, this is not always true.  There's a funny scenario happening in the Adrenaline Expansion Pack right now.  Without the material batching, there's one area that is causing over 12,000 draw calls.  This is about 12x the USRDA recommended daily dosage of draw calls for any individual.  So it's a slide show right?  Actually, it's running at about 15-20 frames per second on most people's machines.  Which brings me to part 2 of the Performance Art thread:  When is a draw call not a draw call?

 Okay, a draw call is always a draw call.  Every new draw call involves the CPU sending information to the GPU to be rendered.  And believe me, we have our own overhead for each draw call (things like pull on ground and other procedures to make sure everything is rendering in the correct place).  However, if you have several thousand objects in a scene that all have the exact same material and textures, and are using a simple material with only diffuse and night textures, they can all be ripped out in a row without a single material switch.  Which means there is virtually no processing between each draw call except the new vertex information (in this case, incredibly simple apartment complexes of 100 vertices or less).  And on top of that, the pixel shader requires about 3 instructions.  So for 8500 of the 12000 draw calls, there is not a single material switch, very simple vertex data being sent to the card and almost no GPU calculations in the pixel shader.

So what's the lesson here?  Well the first is that we still need to do material batching, and when we do, those 8500 draw calls are going to become a handful.  The second lesson is that a draw call is not always super expensive.  If you author your content smartly (reusing textures and material settings exactly, using fewer vertices in the model and creating simple materials) you can rip through draw calls incredibly quickly.  And the benefit to authoring content that way is that when material batching comes in, all of those objects suddenly batch up into a few draw calls.  The last lesson is that we are really kicking it up a notch for the expansion pack.  Seattle, with all of the autogen cranked and traffic turned all the way up is still only about 5000 to 6000 draw calls.  I know, I say "only" like that's a low number.  It's still extremely excessive and hopefully the material batching work will really knock those numbers down and knock the framerates up.

Performance art.

Okay, so I'm not going to get dressed in a sheet and speak in a made-up language while someone projects images from my childhood on me. I'll leave that to the qualified people.  Instead I'm going to talk about performance art in the sense of art that is performant.  Having finished SP1 and looked very intensively at art and performance, saying that I've learned a lot is an understatement.  Having witnessed and worked on so many new systems for FSX (the new shader system, incredibly dense and varied autogen, new water, living world traffic, etc.) I should have looked at performance much differently.  We took the traditional route of "Let's get it all functioning and then we'll fix performance in the end."  Unfortunately, when you're dealing with a whole new shader system and an engine that is extremely draw call intensive, the art must be created up front with performance in mind.

Hopefully, through this blog and other venues, I can help people who make content for Flight Simulator X start off with performance in mind.  I will admit openly that we still have a bit of performance work left to do.  Hopefully, a lot of it will come in with the DX10 update (including more performance work for DX9) and the Xpack.  A lot of it will be how we author art content and a lot of it will be actual engine changes to draw larger batches with fewer draw calls.  However, despite this performance work, add-ons are a huge part of our product and they are notoriously bad on performance.  And we often get blamed for poor authoring choices that result in badly performing 3rd party content.  (We also share the blame for not calling out performance in the SDK and also for issuing too many draw calls to start with.)

The number one killer for us (and just about every single DX9 game out there) is draw calls.  There is a really cool little app called NVPerfHUD that will let you step through the scene's draw calls as well as profile a whole bunch of different performance metrics (sorry, it works in debug only).  When we load up FSX in NVPerfHUD on a decent graphics card (6800-ish) the graphics card is 100% idle most of the time.  Unless you get really close to an object with an expensive shader, the graphics card is never taxed.  This is because it is always waiting for the CPU to process and send the draw calls, which it handily processes with ease.

In DX9, every time you have a material switch or texture change, you are issuing a new draw call.  So if you change a single constant on the FSX material --> new draw call.  If you use a different texture --> new draw call.  So for that third party aircraft with 11 1024x1024 textures.  That's a guaranteed 11 draw calls (not to mention something like 50 or 60 megs of texture space used up with the night, spec and normal maps).  Then, if each of those textures get split into 3 different materials (glass, metal and rubber), that's 33 draw calls.  Then if that aircraft becomes an AI plane and shows up 10 times in the scene, that's 330 draw calls, a full third of the recommended draw call budget.  If you are adding an autogen 2 object to the scene, if you have more than one draw call per object, performance is going to suffer significantly.

So how do you reduce draw calls?  Well, we are taking steps right now to try to batch up all objects with like materials. We already do this with some of the autogen and the road traffic.  We are looking to do this with everything we can (ripping out the fixed function pipeline is going to be a huge help for this).  This will significantly reduce draw calls on a lot of objects.  However, it still won't fix poorly authored content (which believe me, we have plenty of on our own already and are working hard to fix).  If you're authoring autogen content or buildings, try to pack as many objects onto 1024 or smaller sheets and make sure that those objects all have the exact same FSX material constants.  Also make sure that each object is referencing a single texture sheet.  This will guarantee that each object is only a single draw call.  And when material batching comes in, dissimilar objects that reference the same texture sheet and the same material constants will get batched and drawn together.

Addtionally, as often as possible, use the textures to change the shader values as opposed to using the shader constants themselves.  If you control reflection in the diffuse alpha and specular in the specular color and specular alpha, then you should be able to set the reflection and specular power term constants at max and use the texture values to adjust accordingly.  You should be able to get rubber, metal, plastic and other surfaces in a single draw call just by adjusting the various portions of the textures.  The only thing you should need new material constants for is something like glass, chrome or an object that is transparent or glows.

Also, make sure that all of the material textures are grouped.  Put all the chrome on one sheet, because then you can have a single chrome material and a single chrome draw call.  Do the same with glass, metal, etc.  If you have 11 1024x1024 textures, and each of them has a chrome section, then you are forcing 11 new draw calls when you could have one.

Over the next few weeks, I'll talk more about draw calls and how to reduce them as well as other performance bottlenecks and pitfalls to avoid.

FSX SP1 is live!

You can download the English, French, Italian, German, Spanish or Polish versions from our website. The Japanese version is a day behind and should be up soon. I think you will be pleasantly surprised with the performance improvements in SP1.

A lot of work went into making this update really worthwhile. We found some pretty major bugs. For example, the old FS9 aircraft that we hadn't updated were using an insane amount of draw calls. The Dash-8 was 225 draw calls and is now 14. The Cherokee was 245 and is now 13. The MD83 was 375 and is now 12. So between those three aircraft, we were at about the recommended number of draw calls for a scene. Needless to say, turning on AI aircraft should be a lot more performant. We also batched the autogen a lot better and fixed a whole bunch of other performance related bugs (as well as started to offload some tasks like terrain rendering to the other cores of a multi-proc machine).

Unfortunately, the fresnel ramp is still a bit messed up. It sort of works, but doesn't really sample as it's supposed to. I wouldn't recommend using it for anything except special effects (which is what it's really designed to be used for anyway). We overused it in FSX and will be scaling back significantly on its use moving forward. It's really ideal for things like two-toned paint schemes and controlling reflections. Because it is a multiply, it doesn't make much sense on specular (unless it is really colored) because all it does is dim the specular down. At any rate, we will be fixing that bug with the DX10 update.

Have fun with SP1! I think you'll like it.

New FSInsider site and multi-mon patch.

The new FSInsider site has gone live.  There's a total redesign including a neat new look, more content and more regularly updated content.  If you're interested in finding out what we do here, it's a great place to start.

Also, there's a Flightsim specific Vista patch that addresses multi-mon support (previously a massive bug in Vista and a key feature in FS).  Here's the 64-bit version.  One of our developers noted this:

The link at the bottom of that website (http://support.microsoft.com/kb/933590/en-us) is interesting to look at if you want to see how many system files they had to change to fix this problem.

It's good to hear they're willing to put out the effort to support our users.

I never thought the blurries would be a problem with Trainsim....

...but I guess we're going to have to worry about them there as well.  Check out this record breaking high speed train in France.  Tres grande vitesse indeed!

Oh what a difference the CPU makes.

So I got this new machine at work.  It's one of them new-fangled Core 2 Duo machines that Intel's making.  And man does it show just how CPU bound we really are with all of our draw calls.  The machine specs are: Intel Core 2 Duo CPU (a huge leap ahead of the dual Athlons in my other two machines), 2 gigs of ram and a GeForce 7900 (a bit faster than the 6800 and ATI x1600 in my other machines).  I was shocked to turn all of the settings up completely (including autogen) and find myself getting 15 FPS solid.  And this is on FSX retail.  This wasn't even on SP1 with all of the performance enhancements.  Granted, the whole machine is a step up, but the huge difference in CPUs seems to be the really big win and it makes sense.  Wer're pushing way too many draw calls onto the CPU for FSX retail.  Fortunately, SP1 addresses a lot of those issues, as well as takes advantage of both (or all four if you're really rich) cores on the CPU.

So hooray for CPUs!  And for those of you who don't want to buy a new CPU, hooray for SP1.  Look for it soon.  I'll post as soon as we get done testing it and get it ready for release.

IK Craziness

I've heard from our good buddy Nick that the IK on the jetway is a bit confusing, even with the documentation provided.  I could see how that would be the case since IK in general is a pretty steep learning curve and the jetway is a pretty complex bit of IK that 3ds Max itself can't even visualize.  Sliding joints only work in Max with the HD solver, and there's no way we could support that in the engine.  You can set the sliding joint limitations with HI or Limb solvers, but they are pretty hokey with bones.  And in order to reduce the draw calls on the jetways from 13 to 2, we had to skin the jetways with bones.

Fortunately, part of the SP1 update is the autoskinning of animations (to reduce draw calls, the dreaded CPU killer) and the inclusion of skinning on shader 1.1.  What does this mean?  It means that you can get rid of the bones on the jetway and still have it only take 2 draw calls (1 for the jetway and 1 for the ground markings).  Unfortunately, if you don't understand IK, or haven't worked with it extensively, this may be a very difficult task.  Because of this, Nick has conviced me to create a video showing how I set up the jetway using Gmax.  Some time in the next couple of weeks I'm going to sit down and capture a video that goes over the jetway as well as some other basic IK techniques for rigging characters and Nick will post it somewhere that everyone can access it.  I'll keep you updated as I progress.

Why so quiet?

We've all been very hard at work on a variety of simultaneous projects in this studio.  As you know, we announced the next version of Train Simulator a little while back and we've been hinting at / talking about the SP1 update for Flight Sim as well.  Well, on Friday we offically announced a confirmation of the DX10 update as well as something brand new, the first ever Flightsim Expansion Pack.  I'm really excited about the expansion pack, as it's not just a content update.  There's actually some really cool new features involved, including multi-player air racing, something that I saw a demo of a couple of weeks ago and had my mind blown.  It looked like a blast!

So the SP1 update will come out first, and includes a bunch of updates focused mainly on fixing performance and content.  This is really key for us, as I know performance is one of the big grumblers right now (and rightly so).  DX10 will likely include even more performance fixes and even some neat enhancements.  And the expansion pack will be able to take advantage of both of those updates right out of the gate.  I guess what this all points to is a much more connected development team that is committed to listening to what our users have to say and responding sooner than the next full release.  So you can expect two updates and an expansion pack from us just this year. 

Also note that Hal did some interviews with a bunch of the sim community sites about SP1 and the future releases and the results can be found on Avsim, Flightsim.com, SimFlight and SimHQ.

Problems with your skin.

You may be thinking about eczema, but I'm thinking about that funny thing that happens when you skin up a model and then export it into Flightsim only to discover something has gone horribly awry and your model suddenly looks like something out of a Japanese horror film.  Usually this is caused by having animated bones with scale applied to them.  Or by forgetting to make sure all of your vertices have skin weights applied to them.  Or by some other unknown gremlin in the system.  However, if you are having that problem on your models, and you have more than 128 skinned bones in that model, you may have just discovered a bug in our engine that we are working on fixing.

Apparently, only the first 128 scene graph nodes can be exported as skinned.  What does this mean exactly?  Anything with animation on it gets exported as part of the scene graph.  You can export as many animated parts as you want.  However, in the current incarnation of Flightsim, only the first 128 nodes can be part of a skin.  "Why didn't you find this bug before you shipped?", you ask.  Well, in the interest of trying to maintain some performance,  we kept our bone counts down below 128 per model.  So we as artists never actually ran into this issue.  Leave it to the third parties to push our system to its breaking point.  ; )

The fix for this should be part of the SP1 that we are working on right now.  I believe we are trying to bump the limit up to 1000 skinned bones and are also working on implementing skinning on 1.0 and 1.1 cards as well.  Which means more people will see more bones and will see them on more cards.