testiloquent
Testing-Essentials ▪ Think Like a Tester ▪ Test Strategy ▪ Test Tooling, Automation ▪ Test Analysis and -Design ▪ Performing Tests and Reporting ▪ Appendix
What Is Testing?
Software testing is a set of activities carried out to gather (usually quality-related) information and reduce uncertainty about the product. The subset of activities carried out, who does them, and what form they take can differ greatly from case to case. The differences depend on contextual factors such as the kind of software being tested and the level at which it is being tested, the project budget, the available tools, the test team members’ availability and skills, the software development lifecycle model, the information objectives and more. We will discuss these contextual differences in more detail in upcoming chapters.
Testing Well
Poor testing may be worse than no testing at all. – Weinberg
What is a good test?
Consider the following analogy, from Myers (1979). Suppose that something’s wrong with you. You go to a doctor. He’s supposed to run tests, find out what’s wrong, and recommend corrective action. He runs test after test after test. At the end of it all, he can’t find anything wrong.
Is he a great tester or an incompetent diagnostician? If you really are sick, he’s incompetent, and all those expensive tests were a waste of time, money, and effort. In software, you’re the diagnostician. The program is the (assuredly) sick patient. – Nguyen, Kaner, Falk
The ideal tests
Perfect tests would detect every bug in the system under test (no misses) and never flag correct behaviour as a bug (no false alarms). Moreover, a perfect test would provide a clear description of the test conditions and the steps taken to produce the results. Performing the test on time, with the available resources would be possible, and its cost would be sustainable.
We know that the the first requirement can’t be fulfilled.
- Complete testing is not possible. No amount of testing will be enough to prove that all the bugs have been discovered.
- Furthermore, accurate results cannot be guaranteed. Our oracles are heuristics - useful guidelines, not infallible formulas or universal truths.
There are practical limitations to the granularity with which we can document our tests. The right amount of documentation will depend on the project situation and other context factors. However, providing information about making our method offers benefits in various common situations for example:
- In some industries, evidence of the testing done needs to be produced in case of an audit.
- If our tests miss a bug or raise a false alarm, we will want to analyze the test, find its weak spot and improve it.
- If the test needs to be reproduced, for example by a developer working on a bug-fix, or for retesting after a bug has been corrected.
The scientific model and testing
">Karl Popper (1965) argues that the correct approach to testing a scientific theory is not to try to verify it, but to seek to refute the theory - that is, to prove that it has errors. The harsher the testing, the more confidence we can have in a theory that passes it. Much of Popper’s reasoning applies directly to software testing. – Nguyen, Kaner, Falk
Testing as Popper suggests means:
- Choosing the tests you can perform that can reveal the information needed (your information objectives).
- Changing the tests as the system under test changes and matures.
The testing process
There are groups of activities that can usually be identified in test processes.
- Defining and refining the strategy is an ongoing necessity as the project progresses. The strategy keeps the testing objectives in sight to ensure alignment of the test approach. Individual testers concerned with tests for an upcoming two-week iteration may define their strategy in a conversation with the team in front of the whiteboard, and a picture of the notes will suffice. Large projects with compliance regulations to adhere to and many people to keep informed may have a formal strategy document that lays out the strategy in some detail
- Work on the strategy frequently goes hand-in-hand with Planning, where it’s determined when which test activities will be done, by whom and what dependencies are to be taken into account.
The strategy and plan inform the other test activities.
- Analysing the system under test and the context provides necessary information for designing effective tests
- Implementing and performing tests according to the design
- Monitoring the test activities, reporting results to customers and stakeholders