testiloquent
Testing-Essentials ▪ Think Like a Tester ▪ Test Strategy ▪ Test Tooling, Automation ▪ Test Analysis and -Design ▪ Performing Tests and Reporting ▪ Appendix
Test Automation
The work done by hand … is the bedrock of engineering skill. Machines and robots … evolve as we transfer our skills and craftsmanship to them. In other words, craftsmanship is achieved by learning the basic principles of manufacturing through manual work, then applying them on the factory floor to steadily make improvements. This cycle of improvement in both human skills and technologies is the essence of Toyota’s jidoka. Toyota - Company Information, Vision & Philosophy
The connotations of the term test automation promises more than automation in testing can deliver. Just as programming cannot be automated, so can testing not be automated. In both professions, automating tasks for which computers are better equipped than humans can provide benefits.
We must keep in mind that automation is software. Software must be designed, built and maintained by someone. While great advances are being made in artificial intelligence, to date that someone will be a human being.
Programming and testing are knowledge work. Neither exists without the problem-solving, analytical and creative skills of professionals. Human professionals.
Considerations for Automation in Testing
Test automation is normally associated with programming scripts that automate activities involved in interacting with the product and reporting results.
‘No-code’ test automation tools - software with which users can automate without the need to write code - are available, and are now becoming more capable by leveraging the power of large language models. The promise of ‘no-code’ automation is that by relieving the burden of scripting, it can make automation more accessible, thus potentially giving people more freedom to focus on creative analysis and problem-solving.
Convenience can come with a high price-tag, however, and even the most expensive tool won’t be the ideal choice for every purpose. Moreover, the convenience can lure decision-makers into over-reliance on a particular tool. This can back-fire when the company selling the ‘no-code’ automation tool goes out of business, makes unfavourable pricing model changes, or introduces an update that removes a popular feature. As the scripts developed with ‘no-code’ tools may be stored in a proprietary file format, ultimately, ‘no-code’ merely means ‘not our code’.
There is currently no single tool that is suitable for automating all types of tests in any and all project environments.
What Purpose Will Automation Serve?
When defining your test strategy, consider how automation might help you achieve your mission.
Automate to answer new questions
Automation can make it possible to answer questions about the system that would otherwise go unanswered. Ideally, the answers to these questions will deliver more value to project stakeholders than the cost of creating and maintaining automated tests.
Lest we forget: testing is not an end in itself. To make informed decisions about software systems, decision-makers need information about the system, and businesses pay for testing to provide this information. Your automated tests are not fit for purpose if the test results are ignored, because perhaps they answer questions no one is asking, deliver answers that are not credible, or that decision-makers find unwieldy.
Take the project situation into account to choose a suitable automation approach. Rolling out a full-scale test automation effort if the conditions for effective test automation are not met would not be expedient.
Automate for increased efficiency
Test automation - done well - is worth pursuing. It can support development by saving time, reducing waste and increasing confidence.
Examples:
- Regression tests can be good automation candidates. They are used to check for undesired side-effects of changes to the system or anything it depends on. Automating necessary regression tests can reduce the overall time it takes to deliver value to customers.
- Implementing automated checks for processes in the pipeline can give development teams early feedback which might not be feasible or even possible without automation.
Who Will Automate Tests?
Are the Right Skills Available?
As Weinberg said, poor testing may be worse than not testing, and automation amplifies this effect.
Successful test automation demands skillful test analysis and design, as well as skillful software design and development. People with these skills will need to be hired or trained.
Will the Skills Be Available Long-Term?
The time and effort that test automation demands should not be underestimated. Test automation is a long-term endeavour - the work does not end once the scripts have been written. For as long as the test scripts are in use, the people responsible for automation will need to have sufficient time available to keep them up-to-date and well maintained.
Consider the opportunity cost of automation. When developers are responsible for automating tests, they have less time available for building new software or other important tasks. When testers automate tests, time spent on writing and maintaining scripts is time not spent on tests that humans are better equipped to perform than computers.
Should This Test Be Automated?
Further reading:
When Should a Test be Automated? by Brian Marick. TLDR:
- Automating this test and running it once will cost more than simply running it manually once. How much more?
- An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
- During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
Oracles
Not every potentially valuable test can / should be automated. Tests for which unambiguous oracles are available is where automation can shine - the script depends on explicit oracles as a reference to identify and flag potential bugs.
Testability and Automation
Architecting and designing testability into a product from the start go a long way toward reducing the cost of testing and making maintenance of the software easier (that is, more cost effective).
Software that is highly testable also tends to be well designed. – McDonald, Musson, Smith, Bean, Catlett, Kilty, Williams
The TMA (Too Much Automation) Trap
It took me a long time, but I finally realized that I was over-automating, that only some of the tests I created should be automated. Some of the tests I was automating not only did not find bugs when they were rerun, they had no significant prospect of doing so. Automating them was not a rational decision. – Marick
Tooling for Test Automation
It’s likely that when you start your internship, there will be one or two test automation tools available to you. Your team members can give you specific information on the technology stack and tools in use, which - hopefully - will be the right tools for the job.
Expensive test tools can cause a lot of trouble if they are badly designed, unreliable, or force testers to test in ways they wouldn’t have chosen except that it’s convenient for the tool. If such a tool were cheap, we wouldn’t hesitate to throw it away. But if it’s expensive, the person who bought it doesn’t want to look like a fool - and thus becomes a bigger fool. – Weinberg.
Alternative Applications for Automation in Testing
While - as Marick explains - not every test ought to be automated, some testing activities include tasks that computers do better than humans.
Another skill that you will develop with time is to realize which tasks are economical to automate, as illustrated here:
Reaping the benefits of automation in testing need not mean investing in costly software. Instead, look to tools pre-installed on your computer or freely available online (if your employer’s policies permit). More on that in the lesson on test tooling.
Previous: Test Tooling and Automation ▪ Next: About Test Tooling.