Every working day I do my job as a software tester I see that there are realy unspoken facts about testing topics. Some might be too clear to spoke out or people simple don’t want to talk about them! Other ones might aren’t simple worth to be mentioned – by what reasons ever! But before they are all got of people’s mind, I want to list them here :-). This is just a short list I will provide. Maybee someone has any other issue to add… Then just post me and I will add it…:-)
Unspoken fact 1: Numbers without any context are blank!
Counting and presenting any pure numbers in testing neighter shows anything about the content, the value or the quality of the testing work nor anything about the system under test. Leaving those numbers without any additional comments, they say us nothing – not even they show any progress! In my eyes the ‚testing story‘ of the persons doing the tests and interaction with the stakeholders are allways the thing which realy counts in testing!
Unspoken fact 2: Software testing alone can not improve the quality of a SUT!
With software testing alone we can not improve quality in general, but we can show, that a software doesn’t work as expected (if we compare with our test oracles)! This might be only a part of a quality connotation but of course also that’s a tester’s job.
Why? Well, what is quality? Here’s one of many definitions I like: ‚Quality is value to some person‘ (quote by Jerry Weinberg (see http://en.wikiquote.org/wiki/Gerald_Weinberg). So what quality should we improve and for which stakeholders? That’s a general question which must be answered in every organization – but not e. g. from a testing perspective alone!
Unspoken fact 3: It’s not possible to automate ‚failed‘ test cases! All we tend to automate is in a sense a regression thing!
It’s only possible to automate a manual test after its test script has at least one time been executed and passed in a way with manual testing! Otherwise there’s no excepted result for the automated test script! In other words: It only make sense to automate regression test cases (what I mean here with regression is: at least one time executed manuel tests which passed), because at this time we have at the latest our needed expected result! In my kind of view: Test automation is a part of manual testing – or in general a part of our testing work – not more and not less! But when doing it right, it’s a good tool to help doing good testing work at all!
Unspoken fact 4: When thinking on ‚test automation‘ we usually mean ‚check automation‘!
Apropos ‚test automation‘. In my eyes naming it ‚check automation‘ instead of ‚test automation‘ fits more. Why do I think so? See here for instance: http://www.satisfice.com/blog/archives/856
Unspoken fact 5: Bugs are often in areas in the SUT where nobody has thought about before!
A huge number of bugs will be found in areas in the system under test for which no testing efford has been planned before. By my expirience often exploratory testing can help to find such bugs (more early). With the knowledge we got during a exploratory testing tour we then have the option to create new test scripts in our test management tool for later (regression) use.
Unspoken fact 6: Executing ’step by step‘ test scripts will often NOT uncover importand bugs (fast)!
Writing detailed ’step by step‘ test scripts is mostly not the solution for finding important bugs in a fast way or to provide fast feedback loops (between testers, developers and all other stakeholders)! Therefore we should invest in other approaches in which we can start directly with testing with hard keeping our focus on the SUT (I think e. g. on ‚exploratory testing‘ and using ‚check lists‘). The looking ‚left and right‘ on the SUT behind the test steps of a manual test scipt might help. If we find an important thing in that way, we at least know that our manual test script is not detailed enough and we should edit it with the missing details. We can use here the possiblity of doing exploratory testing work during our script based testing. It’s shown e. g. on of James Bach’s and Michal Bolton’s ‚Rapid Software Testing‘ class (see here for example).
Unspoken fact 7: There can be better solutions for the duty of documentation our testing work!
Unfortunately we need detailed step by step test scripts to start with
test automation check automation on the one side and e. g. to protect us against a law suit on the other side. There’s often a need for accountability. On regulated envirionments and organizsations we often have to proof our testing work in detail! Executing manual test scripts with the help of a test management software can help doing this documentation for us! But the problem here is, that exactly this can defocus us from doing good testing or simple we don’t have the time to executed manual test scripts for logging our done testing work! By own experience I know: It’s hard to do both (good testing and good documentation!).
But in my point of view there are solutions for this: 1) Using check lists instead of detailed test scripts might be enough to document our testing work and showing it to managers and auditors and also to use them for the needs of test automation. 2) If using check lists is not an option, at least trying to use ‚modules‘ and ‚test parameters‘ in test cases in out test management software with predefined ’steps‘ which can be reused several times for several test scripts might be an option to simple work faster (modularisation in the test management tool).
Unspoken fact 8: Wrong use of a test management tool can prevent good testing work!
Using a test management software can unfortunately prevent testers from doing the testing work realy well when there are too hard restrctions, e. g.:
– When testers are not allowed to interact with their stakeholders (product owners, developers, etc.) and simple have to executed the ‚detailled manual test scripts‘ in silence! By experience I know that this will lead to bad testing work. As mentioned below the need of ‚asking questions‘ is the most important thing of doing good testing work!
– When testers do not have the freedeom for doing realy good testing work. What I mean is: Testing is applied epistemology (http://www.markbernstein.org/Jun0301/TestingisAppliedEpistemolo.html). Testers light the way! To do great testing work, a tester needs the freedom to explore the system under test und also for some degree she has the option for doing testing apart from the ‚happy paths‘ what the test scripts say. In additional to also execute the test scripts step by step.
– When testers are not allowed to use the test management tool as they want that it will really supports them! Examples: no test parameters in the test scripts allowed, no modularization allowed, to hard restrictions on user rights, etc. …
– When not the testing work itself is in the main focus but the documentaion of any testing work! (E. g. execution of 50 bad/week test cases counts more then execution of only 10 but realy hard/expensive test cases.) (Compare also to unspoken fakt 1).
– When the test management tool will not be used as one of the test oracles to derrive other test cases to implement from the history of others, or to use the down written test scripts to start implemention of e. g. the regression test scripts with
test automation check automation (compare to unspopken fakt 3)
– When written down test scripts are ‚written in stone‘. That means when it’s not allowed to perform any kind of changes to them. But as everyone should know, there’s a need to change them as our SUT need’s to be changed to over time.
Unspoken fact 9: Testing can have different targets!
Testing can have completely different targets! Finding (important) bugs fast (‚bug hunting‘) might be one target for a software tester or any organization which is providing ‚testing work‘. Showing that given test scenarios on the SUT will pass under curtain circumstances can be an other target in testing! These are only two examples of possible targets in testing. I guess there are thousends of targets imaginable!
Unspoken fact 10: Asking quetions is essential in testing!
We can not do good testing work without asking questions! Imagine the following: In general we can have different oracles in testing like written down manual test scripts, models for requirements, mind maps, known customer workflows, user scenarios, use cases, comparable software to our SUT, behaviour in history, the ’story‘ behind all that, our diffenent stake holders and their knowledge (product owners, customes, developers, managment people, tester collegues, etc.), and so on and so on.
The first intension to start with every testing work will allways be: Asking questions! Typical questions to start can look like ‚What is a typical workflow which our customer want to do?‘ or ‚Can you show me the main areas of this software product?‘ More detailled questions can be ‚When user1 with password y loggs in to the system in edit mode, why isn’t she allowed to edit z like comparable user2 can do?‘ The thing simple is: Without asking questions we can not build any models, we can not profit from any test oracle! So we can not understand the SUT well enough to test it in a good way! Asking questions is essntial in a testers job! It also is important ‚how‘ we asc questiongs and ‚what kind of questions‘ and so on. Karen Johnson showed a nice presentation about ‚The Art of Asking Questions‘ at the EUROStar Software Testing conference. See here to whatch the video.
Unspoken fact 11: The context needs to be concidered every time!
It’s essential to make shure that we and the people we work together every time talk about the same things! Knot knowing the exact contexts will lead to misinterpretations, missunderstandings and bad relationships between people!
Not only the point that testers have to swich between different ‚testing projects‘ (different SUT or different products for which the testing need to be done) is important. It’s also important what kind of testing work we do currently, to which people we report and ask questions, which oracles we use, which ‚test cases‘ we excute, etc. By own experience it’s often not easy to find the right way so that we can say ‚Yes, we talk about the same thing know!‘
Examples: The simple statement ‚All test cases have been executed!‘ will lead to missunderstanding or simple to wrong assumptions! Why? Management people might think: ‚Ah ok, ALL testing of ALL our software products in CURRENT release is good! No problems at all!
Why might they think so? Becuase there’s no ‚qualification‘ in the statement – no context! What SUT we are talking about? On which test environment have the test cases been executed? On which software release version are we talking about? Which kind of test cases have been executed? Is this good or bad? Who has executed them? A manual tester? A product owner? Several people together? Our ‚
Test automation Check automation tool‘? For the view of management people also interessting: What is the result of ‚all test cases have been executed!‘ ?
A better version of such a statement would be: ‚For release version 10.5.13 of our software product ZOTA testers Mike, John and Susan executed all their manual sanity checks on our test environment ‚testdb1‘! As a result ot this, 15 importand bugs have been found and raised to our bug management system!