Testing-Essentials ▪ Think Like a Tester ▪ Test Strategy ▪ Test Tooling, Automation ▪ Test Analysis and -Design ▪ Performing Tests and Reporting ▪ Appendix
Give me six hours to chop down a tree and I will spend the first four sharpening the axe. – Abraham Lincoln
You can technically do testing without software other than the system under test, however, tools can help us increase our effectiveness in many ways.
Analysis, design and implementation
- produce test data (spreadsheets; state-model generators; Monte Carlo simulations; random number generators) .
- obfuscate or cleanse production data for privacy reasons (data shufflers; name replacers)
- generate interesting combinations of parameters (all-pairs or combinatorial data generators)
- generate flows through the product that cover specific conditions (state-model or flowmodel path generators)
- set up and configure the product or test environments (like continuous deployment tools; virtualization tools; or system cloning tools)
Interacting with the product
- submitting and timing transactions; perhaps for a long time; at high volume; under stress (profiling and benchmarking tools)
- encode procedures like operating the product and comparing its outputs to calculated results.
- simulate software or hardware that has not been developed yet; or that we do not have immediately available to us (mocking or stubbing tools)
- probe the internal state of the system and analyze traffic within it as testing is being performed (instrumentation; log analysis; file or process monitors; debugging tools)
Evaluation of results
- sort, filter, and parse output logs (text editors; spreadsheets; regular expressions)
- visualize output for comparative analysis (diffing, charting and graphing tools, conditional output formatting)
- develop, adapt and apply oracles that help us recognize potential problems (source file or output comparison tools; parallel or comparable algorithms; internal consistency checks within the application; statistical analysis tools)
Documenting and reporting
- record our activities and document our procedures (note-taking tools; video-recording tools; built-in logging; word processors; user interaction recording tools)
- prepare reports for our clients (mind maps; word processors; spreadsheets; presentation software)
Test management
- map out our strategies (mind maps, outline processors, word processors)
- identify what has and has not been covered by testing (coverage tools; profilers; log file analysis)
- preserve information about our products, and aid other people in future support and development (wikis; knowledge bases; shared drives)
Which tool to use and how to use it will depend on the context. Here are a few of Bach and Bolton’s rules of thumb to consider when choosing a tool:
- Tools that support many purposes are preferable to those optimized for one purpose.
Note how often “spreadsheet” software appears on the list above…
- Tools that are inexpensive (or free) are preferable to expensive tools, even in many cases where the expensive tools are more powerful.
- Tools that are supported by a large and active community are preferable to those that are not.
More users means more free support, as well as more libraries, plug-ins and other extensions for the tool.
- Tools over which we have control are generally preferable to those controlled by others. Exceptions:
When sharing work with others is important, then the tool in common use is preferable.
When configuring the tool requires know-how that’s not available.
Previous: About Test Automation. ▪ Next: Test Tool List.