Testing, testing, 1.. 2... 3...

  • Comments 12
  • Likes

There's a sentiment I've seen echoed on various forums regarding Flight Simulator, with regards to testing-- especially beta testing. This sentiment is captured over at Avsim by a poster who claims "...nobody outside the MS building is a beta tester. You are a marketing tool and an extra set of eyes."  (part of a longer thread). The poster was previously involved in testing a console title.

Now, I'm *not* a tester. I'm an artist. We make bugs, we don't find them.[:)]  But I do happen to sit next to somone who is a tester, so I feel fully qualified to shoot my mouth off about the testing process here in the ACES game studio.

I imagine that console testing might very well happen as Mr._Al describes. I don't know. My experience deals with PC titles, and even there I can tell you with confidence that testing a title like Flight Simulator is a different kettle of fish than testing Half Life 2.

PC titles have to worry about a broad spectrum of configurations (how video cards/ sound cards/ memory/ CPU/, etc interact) whereas consoles don't. That means configuration testing is a part of the external testing process-- an important part. Even Microsoft doesn't have an infinite number of machines to test various configurations on-- we use a representative sample (which is a lot of machines) instead. Having a wide variety of configurations used in the "wild" (external test) as it were, drastically increases the odds that you'll catch a major crashing or hanging bug. In a sense that might be considered an "extra set of eyes," but the feedback and input (via crash logs and written feedback) are not the only way external beta testers affect final quality, and are critical to a successful release.

Simulations-- like Flight Simulator, are different than other PC titles. Flight Sim, for example, is a pretty big sandbox-- fly anywhere (except the poles...), any time of day, in a variety of aircraft technologies. You look at a game/sim like Forza, for example, and yes it's pretty realistic. Tons of cars, tons of detail. Yes, it edges into the simulations space. But...can you drive from Lillestrøm to Sørumsand? Umm... nope. You're locked into the (very detailed) areas provided. The limitation of Forza and the breadth of Flight Simulator require different testing models, differing areas of expertise. We have a great group of full time testers. They come from a variety of backgrounds, but they are certainly subject matter experts in their various areas of expertise, and yes, a bunch of 'em are pilots. That being said, we bring various outside experts onto our Beta as necessary and useful; meteorologists, air traffic controllers, pilots, and the like. Sometimes those experts are also part of our existing user community. They are listened to.

There's also the 3rd party. The mod/add-on community for Flightsim is quite large and contains both professional and amateur components, some of which have large (passionate and vocal) followings. While the team's focus has to be on the core product, we invest a significant amount of time and resources into supporting backwards compatability. Part of that back compat testing is done in house to be sure, but we invite many representative samples of add-on developers from multiple areas and "hardcore" users to our Betas to provide feedback and input precisely on those areas.

All those people have ideas and feedback. The vast majority of bugs logged are found by our internal teams (they'd better, since that's their fulltime job-- to find bugs), but the Flight Sim Beta has the highest "valid" (usually meaning "non duplicated") bugs found by external process. That ain't 'cause the code's crappy. It's 'cause the Beta testers are both informed and passionate about what they do. They care about what they're doing and give great feedback. Which makes better product.

That being said, the rap sometimes given to the testers is that they "didn't do their job, just look at this bug (fill in your own example here)!" to which I'd say;

"Hold on there Sparky! Not so fast!"

I've mentioned before this post that there are "bugs" and there are incomplete features, and that one is often confused with the other. I'm not walking some fine semantics line when I say this, it's just the nature of the beast. In part, it's why we have version numbers: we build upon what's gone before, and take advantage of new opportunities. This is not meant to excuse poor or incomplete implementation: sometimes we're guilty of that. Usually it's because time and resources don't allow for more, and we decided that something is better than nothing. That choice isn't always the right choice, but on the whole I think we've done okay.

Let me bring this post back to where I started. Here at ACES, our Beta testers (and our fulltime testers too!) have a more than average impact on the development of our products. Their suggestions and input have changed feature implementation and increased quality. Not every suggestion leads to an immediate implementation, but don't assume that because it's not in the product that it wasn't asked for, or remarked upon. (For that matter, don't assume that the team doesn't argue feature set back and forth as well. We do.) Our Beta testers are more than a "marketing tool." (we ask them to sign an NDA, and frankly would prefer them not to really mention they've tested the product at all), and certainly more than "an extra set of eyes" (although when you've got a whole planet to deliver, you need every set of eyes you can get!).


There's a lively comment or two (including the person originally quoted) in the comments section of this post, and a follow up from me with a little more exposition in it, please check 'em out...



  • Thanks for setting the record straight, J - It's great knowing that public testers *who should remain anonymous* are benefitial to the success of the franchise. True, FS is nothing like Half Life, Battlefield, etc. and really needs to be sampled on many various PC config's to see what is to be expected - what better way to do this than a public sampling of testers? :-)



  • Thanks for the clarification, Jason. One of these days it's going to sink into some folks minds that they can't get away with trying to buffalo with BS anymore... ;)

  • O and Bill-- thanks for the comments. The poster in question (Big_Al on Avsim) had an experience doing some testing for a console title, and what he describes may very be accurate for some situations-- but not for titles like FS or TS or CFS.

    Also, that sort of talk usually leads to unfair characterizations of the people who test the products we make-- amateur *or* pro.



  • I worked on PC and console games, thanks.

    And you never actually addressed my comments. You just said exactly what I said regarding the hardware compatability issues and extra sets of eyes. But you failed to mention whether the bugs found by the external testers would actually get fixed and whether they were not builds behind the internal testers. Also at what part of the development cycle were the external testers given a build?

    There is no BS. The bugs found by the inhouse staff sometimes is tough to demand a fix, especially around a deadline and especially if you are going to release a product in 2 months. As for bugs found by external testers? 99% of them won't even be addressed, that is a fact.

  • I have more experience working with PC games, but thanks for making judgements about me. The simple fact is, you repeated everything I said regarding an extra set of eyes and compatability tests. BUT you did not address anything regarding the importance of the bugs found by an external testing team and whether or not if they were found, it would be too late in the development cycle to address them. Having meetings with producers and programmers when the marketing and production department all wants to get the product out to stores ASAP, and demanding fixes to certain bugs found by internal testers is tough enough. Bugs from external testers thus takes a back seat, and being an extra set of eyes and compatability testing is NOT the core testing process and only last minute. Thus you are not as important as the internal team which was my entire point.

  • oops, I started a comment again because the site was running slow... I guess you get my point.

  • Howdy big_al, thanks for dropping by!

    "I have more experience working with PC games, but thanks for making judgements... "

    Please don't misunderstand my intent-- the post I read from you said that you had worked on a console title-- I was offering up your "bonafides", so to speak so that people would know that I *wasn't* saying that you were spouting bs. :)

    Forgive me if you feel that I didn't address your points. Let me try to do so here:

    "But you failed to mention whether the bugs found by the external testers would actually get fixed and whether they were not builds behind the internal testers. "

    They do. The FS beta test, for example, has a high "valid" bug find rate as compared with other betas we're familiar with. Finding bugs is useless unless they get fixed. Many externally found beta bugs get fixed, and they range the spectrum from feature suggestion to crashing bugs, config related or no.

    A good deal of 3rd party back compat bugs are also found and fixed directly from the external feedback.

    Not all bugs are fixed or found.

    Some "bugs" are in fact feature suggestions. The features in question are "fixed" or changed depending on severity, time, and resources available, but external testing provides signifigant impact to the product.

    "As for bugs found by external testers? 99% of them won't even be addressed, that is a fact."

    Not true here at ACES, in my experience.
    I certainly understand that that may have been the case in your experience, but products like Flight Sim have broad well educated and passionate customer base. We respond to that input.

    "BUT you did not address anything regarding the importance of the bugs found by an external testing team and whether or not if they were found, it would be too late in the development cycle to address them."

    Alphas, Betas, or whatever, are released at "milestone" points reflecting signifigant enough development change to be worthwhile, so yes, beta testers tend to be "behind" the daily builds. An Alpha or Beta is cut and a group formed when there is signifigant amounts of features to test, in a relatively stable fashion. This means they are later rather than earlier in the development cycle.

    They are not so late, however, as to render external feedback useless. We try and cut an Alpha as early as possible in the cycle.

    One final point; the Beta process is both time consuming and expensive.If it was merely window dressing and/or only for configuration testing, we could probably get by with outsourcing, or without it or it's associated costs at all.

    I hope that helps answer some of your questions.



  • Thanks for the reply. I never claimed to have the inside scoop at microsoft, from what I've heard, they don't treat their testers very well, like EA. But at least they listen to feedback from external testers. At my old company we had a hard enough time fighting to get our bugs fixed. But external testers got the later beta builds at a time when features were way too late to be asking for.

    Anyway, thanks for the inside scoop, it is good to hear at least one division of microsoft listens to their customers...


  • Alex,

    You're welcome. One clarification:

    "But external testers got the later beta builds at a time when features were way too late to be asking for. "

    Usually by the time any program (Flight Simulator included) hits Beta feature areas are "locked."

    Bugs are first priority to be fixed. New feature suggestions come second, although there's often a line that gets crossed between fixing a bug and extending a feature. :)

    The feature set for products like Flight Simulator are derived from the very beginning-- based off where we left last time, and off the copious amounts of external feedback we receive.



  • Very interesting thread, for outsiders of the game industry as me it´s really exciting to learn how a testing process develops.

    By the way what´s exactly an alpha version? I guess it is an early version of the product which is only tested in house, or maybe the first build?

    Anyway, I´m having great fun with all the discussion around The Topic (you know what I´m talking about), even more fun than with the sim itself :) (naaaa, not true)

  • Hi Jorge,

    Alpha comes before Beta. :)

    There's a pretty good description of the process here:

    (you may have to copy and paste the link)

  • Hi!

    Thanks for insight!

    I can imagine that a product like FS would require external group of experts to help in some system testing (I've heard lot's of people in core FS team were pilots though?)

    "I'm *not* a tester. I'm an artist. We make bugs, we don't find them."

    Can I quote you from time to time?:)

    I happen to be QA lead for some project lately and it gave me huge headaches (being an artist originally). Do you guys have that ongoing problem with numerous versions of RC builds (like RC10beta1).. ;) you don't have to answer that hehe..

    Happy New Year, can't wait for FS10, I know it will be great!

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment