Measuring Stress

jacobb — Wed, 01 Oct 2008 04:22:00 GMT

When stress testing software, it is desirable to know if the software is actually being stressed. To know this we must first define what is meant by stressed. For the purposes of testing, a system is stressed when the rate at which operation against a system can’t be increased and the rate is not being limited by resources used by the test itself. This means that if resources used by the software being tested are at their limit, the software is (usually) being stressed. Usually is mentioned since some resources might not serve as a limit to operations directly. This can be the case for caches, as hitting cache limits might not mean the system can’t handle more load. (Although hitting cache limits is usually a good sign that one is on the right track.)

Since there are multiple different ways most software can be used, there are multiple different bottlenecks that could be hit. Something that goes over a network could easily hit network limits for certain types of workloads and disk limits for others .

To know if resources are being limited, we need to be able to measure the resource. Windows provides a good method of monitoring system resources through perfmon. Perfmon has counters for various system resources (such as cpu, memory or disk). Each resource typically has multiple counters so the resource can be monitored in various ways. Applications can register their own perfmon counters, which allows a test to get even more information about what is going on. This could be of interest if there are hard limits on internal objects used by the software being tested that would not reflect on the system level counters. For example, a database could hypothetically limit the number of outstanding transactions to two. If this limit was hit there would few system resources being used, but the software would be stressed.

Perfmon data can be checked manually while a test is being developed and at points during the lifecycle of a test. Another approach is do use the perfmon APIs to gather this data automatically and log it, allowing this data to be inspected with the rest of the test results.

Task manager can also be used to get a ballpark estimate of system load, although less information is available and far less history is available. It is useful as a quick spot check.

Classes of Stress Tests

jacobb — Wed, 01 Oct 2008 03:33:00 GMT

Stress is a common test category, but it can have different meanings depending on context and goals. Some stress tests are used to see how a system behaves under load. Others are used to generate random operations against the software to gain coverage outside of what is hit in more deterministic tests. Another goal of stress can be to simple find help new bugs in a system, as once a set of regression tests pass, they rarely find new bugs unless the area they test is changed.

Testing under high load is the classic meaning of stress testing. To do this operations are performed against the software until a system bottleneck is reached. For instance, when stress testing a database, disk IO could be the first bottleneck that is reached, thus limiting the rate of operations of the test to the rate the disk can satisfy requests. The location of a bottleneck can change over the lifetime of a system. In the previous database example caching disk IO could have been implemented, thus moving the bottleneck to another layer such as the network.

When load testing is being done, the primary concern is ensuring that the bottleneck being hit is in the system being tested, not in the test. A test which does extensive logging of operations to a file might hit a bottleneck writing to the test’s log file. This would leave the software under test not hitting any limits.

Another goal of stress testing can be to see how a system behaves under random input. For this style of stress test the goal is to hit a wide amount of coverage during a stress run in hopes that areas not tested elsewhere are hit. While this approach does test a broad area of a system, the types of operations tend to be shallower as it is hard to have random operations which have dependencies on other operations. Tests can be made to honor dependencies, but this will reduce the randomness of the test. The other big disadvantage is knowing what the state of the system being tested should be at any given point in time. This reduces the ability to verify the results of stress operations. Often tests of this nature resort to just trying to make the system crash, as even the returns of individual calls can be hard to predict, especially if there is more than one thread and/or process perform operations.

The first to kinds of stress testing are not mutually exclusive and dovetail into a third goal of stress testing: finding bugs. Many stress tests are written not to specifically load a system or to widen the coverage of a system, but instead to make the system fail in different ways every time the test is ran (or at least as long as bugs that are found are fixed). This kind of stress test is usually written with the internal details of the system being tested in mind. If an application caches certain kinds of data, then a stress test for the system might perform operations involving the data being cached. A classic cache would involve one writer thread hitting a resource with another thread performing reads against said resource.

Various Thoughts about Software Testing : Stress

Measuring Stress

Classes of Stress Tests