Test Case Automation vs. Execution Automation
When talking about test automation, there are two different classes of automation that often come up. The first is the automation of a test case. This is the code/script which actually exercises the system being tested. The second is the automation of the execution of the test case. This involves running the aforementioned test case, as well as any setup and cleanup needed.
The choice of which of these to automate, or which to give priority to if both are to be automated, depends on what is being tested. Consider for example the testing of the Windows filesystem API CreateFile (see http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx for full details on the API). System APIs lend themselves well to automation. In this case there are no in box tools which will fully exercise all the parameters of CreateFile, at least that I am aware of. Many of the tests involve just varying a single flag in a call, with the rest of the test case remaining the same from variation to variation. There are a significant number of tests of interest for this API, so running them all manually would take a significant amount of time.
While the benefit of automated the test cases for CreateFile is rather large, there is less of an obvious need for fully automating the execution of the test. Even assuming that remote paths will be used (i.e. file://server/share/file), the test would most likely be fairly straightforward to execute. Depending on exactly how the test was implemented, it might be as simple as just running the test binary, given it the name of a remote machine to target. After it is done any log files generated could be inspected manually and/or uploaded to a central location.
While being able to run a test in an automated fashion end to end is useful, the first step of testing CreateFile would most likely be writing the automated test cases, with further automation for the execution of the test being done should schedule/budget permit.
Tests in which execution would be fully automated, but the test case itself is not automated are rare. The characteristics of such a test could include:
- 1. Time consuming component setup
- 2. Simple, but hard to automate tests
- 3. Time consuming post test cleanup
- 4. Complex log analysis
Consider interop testing with another piece of software, say Microsoft Word. Automating the installation of Word would be useful, as the process takes a fair bit of time and requires manual interaction, but it is not difficult to automate setup for Office. If the test case was simple (say how filesytem quotas work with Word), then it might not be worth development time to fully automate the test case. Instead the automation being used could setup Word, set the required quotas on the filesystem, then launch Word, waiting for a tester to run the appropriate tests. After the tests is over, a script could be ran which prompted for information to create a log file, upload said log file, then uninstall Word.
While the above example is rather contrived, it does demonstrate why different priority would be given to different aspects of the automation. In my own experience, I have found a great deal of synergy in having both the test case and the test execution automated. I like being able to just run one command to launch a test, knowing that it will run to completion without any further intervention. Unless a bug is hit of course. Then things can get fun.
