Degrees of Test Automation
Welcome to the first entry in my discussion of test automation. I'll be starting by discussing the different levels of automation that can be applied to testing. Future posts will cover test case automation vs. execution automation, why automate tests, and how to automate tests.
Test automation covers many different aspects. There are many different elements, each of which can be automated. While the exact elements will vary depending on what is being tested, they general consist of machine configuration, component install, test execution, logging, and reporting. Each of these can be fully automated.
While automated all of the above would reduce the amount of time spent by a human running tests greatly, there are some downsides. Automating all of the aspects has higher upfront development cost than performing some of the steps manually. On projects that are planned to last for a short amount of time, this level of effort might not be justified. The maintenance costs for such a highly automated system can also be higher over time, as there are more aspects of the system which would require updating.
Fortunately the various test stages can be automated independently to different degrees. Movement between stages of a test can also have different levels of automation. For example, machine setup could be done through imaging software initiated by a script. Another script would then be ran (by a tester) to install the software being tested, run the tests, then upload logs to a central location. After this has been completed the logs would be inspected by someone visually to determine if there was a failure that needed investigation.
While this example test process is not fully automated end to end, there is little human intervention, and much of it is limited to executing scripts. One of the downsides is that tester interaction is needed to move from stage to stage. This requires awareness that a previous stage is complete. If each stage does not proactively notify someone that it is complete (via email for instance), a tester is required to at least occasionally observe the automated process. Another problem is that large series of tests can't be started and expected to run to completion overnight or over a weekend (unless someone is present of course).
