Test Case Automation vs. Execution Automation

jacobb — Tue, 01 Jul 2008 09:33:00 GMT

When talking about test automation, there are two different classes of automation that often come up. The first is the automation of a test case. This is the code/script which actually exercises the system being tested. The second is the automation of the execution of the test case. This involves running the aforementioned test case, as well as any setup and cleanup needed.

The choice of which of these to automate, or which to give priority to if both are to be automated, depends on what is being tested. Consider for example the testing of the Windows filesystem API CreateFile (see http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx for full details on the API). System APIs lend themselves well to automation. In this case there are no in box tools which will fully exercise all the parameters of CreateFile, at least that I am aware of. Many of the tests involve just varying a single flag in a call, with the rest of the test case remaining the same from variation to variation. There are a significant number of tests of interest for this API, so running them all manually would take a significant amount of time.

While the benefit of automated the test cases for CreateFile is rather large, there is less of an obvious need for fully automating the execution of the test. Even assuming that remote paths will be used (i.e. file://server/share/file), the test would most likely be fairly straightforward to execute. Depending on exactly how the test was implemented, it might be as simple as just running the test binary, given it the name of a remote machine to target. After it is done any log files generated could be inspected manually and/or uploaded to a central location.

While being able to run a test in an automated fashion end to end is useful, the first step of testing CreateFile would most likely be writing the automated test cases, with further automation for the execution of the test being done should schedule/budget permit.

Tests in which execution would be fully automated, but the test case itself is not automated are rare. The characteristics of such a test could include:

1. Time consuming component setup
2. Simple, but hard to automate tests
3. Time consuming post test cleanup
4. Complex log analysis

Consider interop testing with another piece of software, say Microsoft Word. Automating the installation of Word would be useful, as the process takes a fair bit of time and requires manual interaction, but it is not difficult to automate setup for Office. If the test case was simple (say how filesytem quotas work with Word), then it might not be worth development time to fully automate the test case. Instead the automation being used could setup Word, set the required quotas on the filesystem, then launch Word, waiting for a tester to run the appropriate tests. After the tests is over, a script could be ran which prompted for information to create a log file, upload said log file, then uninstall Word.

While the above example is rather contrived, it does demonstrate why different priority would be given to different aspects of the automation. In my own experience, I have found a great deal of synergy in having both the test case and the test execution automated. I like being able to just run one command to launch a test, knowing that it will run to completion without any further intervention. Unless a bug is hit of course. Then things can get fun.

Degrees of Test Automation

jacobb — Thu, 19 Jun 2008 10:00:00 GMT

Welcome to the first entry in my discussion of test automation. I'll be starting by discussing the different levels of automation that can be applied to testing. Future posts will cover test case automation vs. execution automation, why automate tests, and how to automate tests.

Test automation covers many different aspects. There are many different elements, each of which can be automated. While the exact elements will vary depending on what is being tested, they general consist of machine configuration, component install, test execution, logging, and reporting. Each of these can be fully automated.

While automated all of the above would reduce the amount of time spent by a human running tests greatly, there are some downsides. Automating all of the aspects has higher upfront development cost than performing some of the steps manually. On projects that are planned to last for a short amount of time, this level of effort might not be justified. The maintenance costs for such a highly automated system can also be higher over time, as there are more aspects of the system which would require updating.

Fortunately the various test stages can be automated independently to different degrees. Movement between stages of a test can also have different levels of automation. For example, machine setup could be done through imaging software initiated by a script. Another script would then be ran (by a tester) to install the software being tested, run the tests, then upload logs to a central location. After this has been completed the logs would be inspected by someone visually to determine if there was a failure that needed investigation.

While this example test process is not fully automated end to end, there is little human intervention, and much of it is limited to executing scripts. One of the downsides is that tester interaction is needed to move from stage to stage. This requires awareness that a previous stage is complete. If each stage does not proactively notify someone that it is complete (via email for instance), a tester is required to at least occasionally observe the automated process. Another problem is that large series of tests can't be started and expected to run to completion overnight or over a weekend (unless someone is present of course).

Various Thoughts about Software Testing : Automation

Test Case Automation vs. Execution Automation

Degrees of Test Automation