Various Thoughts about Software Testing

Measuring Stress

2008-10-01T04:22:00Z

When stress testing software, it is desirable to know if the software is actually being stressed. To know this we must first define what is meant by stressed. For the purposes of testing, a system is stressed when the rate at which operation against a system can’t be increased and the rate is not being limited by resources used by the test itself. This means that if resources used by the software being tested are at their limit, the software is (usually) being stressed. Usually is mentioned since some resources might not serve as a limit to operations directly. This can be the case for caches, as hitting cache limits might not mean the system can’t handle more load. (Although hitting cache limits is usually a good sign that one is on the right track.)

Since there are multiple different ways most software can be used, there are multiple different bottlenecks that could be hit. Something that goes over a network could easily hit network limits for certain types of workloads and disk limits for others .

To know if resources are being limited, we need to be able to measure the resource. Windows provides a good method of monitoring system resources through perfmon. Perfmon has counters for various system resources (such as cpu, memory or disk). Each resource typically has multiple counters so the resource can be monitored in various ways. Applications can register their own perfmon counters, which allows a test to get even more information about what is going on. This could be of interest if there are hard limits on internal objects used by the software being tested that would not reflect on the system level counters. For example, a database could hypothetically limit the number of outstanding transactions to two. If this limit was hit there would few system resources being used, but the software would be stressed.

Perfmon data can be checked manually while a test is being developed and at points during the lifecycle of a test. Another approach is do use the perfmon APIs to gather this data automatically and log it, allowing this data to be inspected with the rest of the test results.

Task manager can also be used to get a ballpark estimate of system load, although less information is available and far less history is available. It is useful as a quick spot check.

Classes of Stress Tests

2008-10-01T03:33:00Z

Stress is a common test category, but it can have different meanings depending on context and goals. Some stress tests are used to see how a system behaves under load. Others are used to generate random operations against the software to gain coverage outside of what is hit in more deterministic tests. Another goal of stress can be to simple find help new bugs in a system, as once a set of regression tests pass, they rarely find new bugs unless the area they test is changed.

Testing under high load is the classic meaning of stress testing. To do this operations are performed against the software until a system bottleneck is reached. For instance, when stress testing a database, disk IO could be the first bottleneck that is reached, thus limiting the rate of operations of the test to the rate the disk can satisfy requests. The location of a bottleneck can change over the lifetime of a system. In the previous database example caching disk IO could have been implemented, thus moving the bottleneck to another layer such as the network.

When load testing is being done, the primary concern is ensuring that the bottleneck being hit is in the system being tested, not in the test. A test which does extensive logging of operations to a file might hit a bottleneck writing to the test’s log file. This would leave the software under test not hitting any limits.

Another goal of stress testing can be to see how a system behaves under random input. For this style of stress test the goal is to hit a wide amount of coverage during a stress run in hopes that areas not tested elsewhere are hit. While this approach does test a broad area of a system, the types of operations tend to be shallower as it is hard to have random operations which have dependencies on other operations. Tests can be made to honor dependencies, but this will reduce the randomness of the test. The other big disadvantage is knowing what the state of the system being tested should be at any given point in time. This reduces the ability to verify the results of stress operations. Often tests of this nature resort to just trying to make the system crash, as even the returns of individual calls can be hard to predict, especially if there is more than one thread and/or process perform operations.

The first to kinds of stress testing are not mutually exclusive and dovetail into a third goal of stress testing: finding bugs. Many stress tests are written not to specifically load a system or to widen the coverage of a system, but instead to make the system fail in different ways every time the test is ran (or at least as long as bugs that are found are fixed). This kind of stress test is usually written with the internal details of the system being tested in mind. If an application caches certain kinds of data, then a stress test for the system might perform operations involving the data being cached. A classic cache would involve one writer thread hitting a resource with another thread performing reads against said resource.

Why Automate

2008-07-12T05:42:00Z

(Forgive the formatting of this post. Getting the code to be readable seems to be given me issues.)

Reasons why to automate tests:
1. Laziness. Executed the same test over and over manually is boring.
2. Speed. Automation runs faster than a human.
3. Reuse. The more times a test needs to run, the bigger the benefit from automation.
4. Bugs. Automated tests usually cover more variation than humans can, leading to more bugs found.

I’d like to touch on the last point a bit more. It is often easy to add additional test variations by just tweaking a few parameters in the test code. If the cases do not take a significant amount of time to execute, then there is no reason not to go for the additional coverage.
Let’s say I was testing reading from a file, and had the following code written:

TestRead(PWCHAR Path)
{
    HANDLE hFile;
    INT64 Offset[] = {0,1,2,512,1024,4096,65535};
    DWORD NumOffsets = 6;
    DWORD Size[] = {0,1,2,512,1024,4096,65535};
    DWORD NumSizes = 6;
    BYTE *Buf;
    DWORD i, j;
    DWORD WStatus;

    for(i = 0; i < 0; i++) {
        for(j = 0; j < 0; j++) {
            WStatus = Read(SizeOffset);
            if(WStatus != ERROR_SUCCESS) {
                TestLog(SEV, “Error %d reading”, WStatus);
                TestLog(INFO, “Size = %u, Offset = %u”, Size, Offset);
        }
    }
}

This test gives us 36 different read variations. If one wished to add an additional size or offset, it would be trivial to increase the array(s) by the desired values. Aside from just testing the value added, it would also add coverage with the nonchanged value.

Of course since the number of variations is NumOffsets * NumSizes, the amount of reads being executed can grow rather fast, which might be of concern. Documenting the changes to the test would most likely also be simple. It would probably just involve updating a list of sizes and offsets of a test.

If these tests were being executed manually, each addition would require an update of the manual test steps, as well as an update to the reporting mechanism. The time to execute the test would also be impacted in a much greater manner than the automated test.

While this example shows that there are test cases which can be easily extended when automated, there are also cases in which this is not true. Consider a test which writes to a file over a remote share. Let’s say we wanted to add a variation in which a certain group policy was set during the write. If the test doesn’t already have easy access code to manage group policy changes, then adding such code might be a non-trivial undertaking. If this was a low priority test that would not be executed often, it might be easier to have the setting of the group policy happen manually through UI interaction, then run the test with those settings.

Test vs. Test Case

2008-07-04T07:49:00Z

A common term in testing is test case. Unfortunately it doesn't have a single meaning. Instead it is viewed different depending on contest.

My definition is fairly simple. A test case is the minimum unit of execution which performs a test. A test is a group of one or more test cases (usually related test cases).

For example, if I was testing the Win32 CreateFile API, a simple test case could be to open a file, then close the handle to the file after one minute. This is a complete test which exercises the desired component functionality.

One might wonder about all those parameters that the CreateFile api has (see http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx for full details on the API). One could vary the different dwDesiredAccess flags on the api, testing each in turn. This is what I would refer to as a test variation.

This gets into a bit of terminology variance. One possible viewpoint is to consider a test case a container object of test variations. Each test variation is executed, with the results being rolled up to the test case. In turn the test case would be contained within a test and would report its results back to the test to be aggregated.

It is also possible to consider the test case to be more of a virtual concept. Instead of test variations reporting results back to a test case, the variations would report results directly back to a test. The test case would serve as a logical grouping of test variations.

It is also possible to treat test variation and test case as synonyms.

These viewpoints are all valid. The important thing is to stick with the same terminology as much as possible. I have found that different teams often use different meanings, which can create a lot of confusion when trying to aggregate the results of many different groups.

Test Case Automation vs. Execution Automation

2008-07-01T09:33:00Z

When talking about test automation, there are two different classes of automation that often come up. The first is the automation of a test case. This is the code/script which actually exercises the system being tested. The second is the automation of the execution of the test case. This involves running the aforementioned test case, as well as any setup and cleanup needed.

The choice of which of these to automate, or which to give priority to if both are to be automated, depends on what is being tested. Consider for example the testing of the Windows filesystem API CreateFile (see http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx for full details on the API). System APIs lend themselves well to automation. In this case there are no in box tools which will fully exercise all the parameters of CreateFile, at least that I am aware of. Many of the tests involve just varying a single flag in a call, with the rest of the test case remaining the same from variation to variation. There are a significant number of tests of interest for this API, so running them all manually would take a significant amount of time.

While the benefit of automated the test cases for CreateFile is rather large, there is less of an obvious need for fully automating the execution of the test. Even assuming that remote paths will be used (i.e. file://server/share/file), the test would most likely be fairly straightforward to execute. Depending on exactly how the test was implemented, it might be as simple as just running the test binary, given it the name of a remote machine to target. After it is done any log files generated could be inspected manually and/or uploaded to a central location.

While being able to run a test in an automated fashion end to end is useful, the first step of testing CreateFile would most likely be writing the automated test cases, with further automation for the execution of the test being done should schedule/budget permit.

Tests in which execution would be fully automated, but the test case itself is not automated are rare. The characteristics of such a test could include:

1. Time consuming component setup
2. Simple, but hard to automate tests
3. Time consuming post test cleanup
4. Complex log analysis

Consider interop testing with another piece of software, say Microsoft Word. Automating the installation of Word would be useful, as the process takes a fair bit of time and requires manual interaction, but it is not difficult to automate setup for Office. If the test case was simple (say how filesytem quotas work with Word), then it might not be worth development time to fully automate the test case. Instead the automation being used could setup Word, set the required quotas on the filesystem, then launch Word, waiting for a tester to run the appropriate tests. After the tests is over, a script could be ran which prompted for information to create a log file, upload said log file, then uninstall Word.

While the above example is rather contrived, it does demonstrate why different priority would be given to different aspects of the automation. In my own experience, I have found a great deal of synergy in having both the test case and the test execution automated. I like being able to just run one command to launch a test, knowing that it will run to completion without any further intervention. Unless a bug is hit of course. Then things can get fun.

Degrees of Test Automation

2008-06-19T10:00:00Z

Welcome to the first entry in my discussion of test automation. I'll be starting by discussing the different levels of automation that can be applied to testing. Future posts will cover test case automation vs. execution automation, why automate tests, and how to automate tests.

Test automation covers many different aspects. There are many different elements, each of which can be automated. While the exact elements will vary depending on what is being tested, they general consist of machine configuration, component install, test execution, logging, and reporting. Each of these can be fully automated.

While automated all of the above would reduce the amount of time spent by a human running tests greatly, there are some downsides. Automating all of the aspects has higher upfront development cost than performing some of the steps manually. On projects that are planned to last for a short amount of time, this level of effort might not be justified. The maintenance costs for such a highly automated system can also be higher over time, as there are more aspects of the system which would require updating.

Fortunately the various test stages can be automated independently to different degrees. Movement between stages of a test can also have different levels of automation. For example, machine setup could be done through imaging software initiated by a script. Another script would then be ran (by a tester) to install the software being tested, run the tests, then upload logs to a central location. After this has been completed the logs would be inspected by someone visually to determine if there was a failure that needed investigation.

While this example test process is not fully automated end to end, there is little human intervention, and much of it is limited to executing scripts. One of the downsides is that tester interaction is needed to move from stage to stage. This requires awareness that a previous stage is complete. If each stage does not proactively notify someone that it is complete (via email for instance), a tester is required to at least occasionally observe the automated process. Another problem is that large series of tests can't be started and expected to run to completion overnight or over a weekend (unless someone is present of course).

Introductions

2008-06-19T09:55:00Z

Greetings all. My name is Jacob Buis. I'm a Software Design Engineer in Test at Microsoft. I've been working on Windows and storage technology for the past eight years and I've decided that it would be a good idea to share some of my experiences and opinions in the field of test. I'll also be talking about the occasional storage topic, or anything else that might be of interest.

I'm going to begin with the topic of test automation, a subject that I have been involved with since my first day at Microsoft. I have found that test automation means different things to different people. Automation can vary in degree. Some tests have an amount of automation within them (e.g. scripts to perform various steps of the test) but still require a tester to manually perform some of the test steps, while other tests are fully automated from end to end and only require a single command to be started (and even that could be automated). I'll be tending to focus more toward the full automation side, at least initially.

Some other topics I'm planning on writing about in the not too distant future include:

1. Test reporting
2. Bug finding
3. Test planning