(Forgive the formatting of this post. Getting the code to be readable seems to be given me issues.)
Reasons why to automate tests:
1. Laziness. Executed the same test over and over manually is boring.
2. Speed. Automation runs faster than a human.
3. Reuse. The more times a test needs to run, the bigger the benefit from automation.
4. Bugs. Automated tests usually cover more variation than humans can, leading to more bugs found.
I’d like to touch on the last point a bit more. It is often easy to add additional test variations by just tweaking a few parameters in the test code. If the cases do not take a significant amount of time to execute, then there is no reason not to go for the additional coverage.
Let’s say I was testing reading from a file, and had the following code written:
TestRead(PWCHAR Path)
{
HANDLE hFile;
INT64 Offset[] = {0,1,2,512,1024,4096,65535};
DWORD NumOffsets = 6;
DWORD Size[] = {0,1,2,512,1024,4096,65535};
DWORD NumSizes = 6;
BYTE *Buf;
DWORD i, j;
DWORD WStatus;
for(i = 0; i < 0; i++) {
for(j = 0; j < 0; j++) {
WStatus = Read(SizeOffset);
if(WStatus != ERROR_SUCCESS) {
TestLog(SEV, “Error %d reading”, WStatus);
TestLog(INFO, “Size = %u, Offset = %u”, Size, Offset);
}
}
}
This test gives us 36 different read variations. If one wished to add an additional size or offset, it would be trivial to increase the array(s) by the desired values. Aside from just testing the value added, it would also add coverage with the nonchanged value.
Of course since the number of variations is NumOffsets * NumSizes, the amount of reads being executed can grow rather fast, which might be of concern. Documenting the changes to the test would most likely also be simple. It would probably just involve updating a list of sizes and offsets of a test.
If these tests were being executed manually, each addition would require an update of the manual test steps, as well as an update to the reporting mechanism. The time to execute the test would also be impacted in a much greater manner than the automated test.
While this example shows that there are test cases which can be easily extended when automated, there are also cases in which this is not true. Consider a test which writes to a file over a remote share. Let’s say we wanted to add a variation in which a certain group policy was set during the write. If the test doesn’t already have easy access code to manage group policy changes, then adding such code might be a non-trivial undertaking. If this was a low priority test that would not be executed often, it might be easier to have the setting of the group policy happen manually through UI interaction, then run the test with those settings.
