My article Agile Methodology Is Not All About Exploratory Testing on the Scrum Alliance website is actually triggering the sort of interest I envisaged, but I would also like to further buttress my points by way of a rejoinder to the post by Huib Schoots as seen below;
Agile Testing or ‘Testing in Agile’
It is ok if the two are different to you but for me I want to as much as possible stay clear of playing with Semantics. If you believe they are different you might as well believe that Test plan is not Plan of Test.
“It is unequivocally the case that: you cannot estimate your time for exploratory testing, i.e. assign points realistically”
While disregarding with the above Huib Schoots wrote that; “The decision to ship the product (which includes a decision to stop testing) is a business decision, not a technical one.”
Stephan, we seem to agree on this one but please note that Business decision to stop testing MUST hinge on the defined acceptance criteria or validation criteria. The decision in most cases will be dependent on the test coverage of the acceptance criteria that demonstrates the level of confidence in the product. You therefore cannot know when to stop testing until some level of confidence is reposed on the product and a business ideally would make this decision when guided by some of the followings;
- completed/uncompleted high priority Test Cases
2. Test cases completed with a defined/agreed pass percentage
- When all high priority defects are fixed or when defect rate falls to an agreeable level
- Allocated Budget (if inadequate may lead to technical debt and rushed implementation)
- Release deadlines or testing window
- When the risk to stop testing is minimal and not detrimental to Return on Investment
6. Functionality, code coverage, or even requirements met
7. Beta or alpha testing completed
In view of the above, my argument against focusing solely on exploratory testing therefore is that you cannot test indefinitely, and if testing must stop at some point in Agile you must optimise the time in the spirit of Scrum by working to deliver a measured expected output that fulfils the acceptance/validation criteria.
To achieve this in a time boxed approach (Scrum) you must estimate how long it will take you to complete or achieve a level of confidence in the product. It is therefore a smart thing to estimate how long it will take to cover a story, and the time it will take to test all the stories in that sprint is the time allocated to testing a deliverable/shippable product in the sprint.
Lastly, the estimation of testing activities will help to determine the rate at which product backlog items are completed, also known as the velocity.
“You cannot plan for exploratory testing, as you do not have defined expected results.”
Huib Schoots also does not agree with the above and the idea of expected result, he therefore wrote and quoted that “Quality is value to some person”
Who is that person? In Agile/Scrum, that person can be one, two, or any of Business, Product Owner, Stakeholder, etc. They are supposedly the ‘Customers’ that the Scrum Team, Testers, or Developers want to make happy. If the customers must be happy for a team of Developers/Testers to continue to work for them there must be an expectation and that expectation cannot be met if Testing activities are not in the centre stage of Agile/Scrum. The Testers will verify by adding value to a product while Business/Product owner validate to complete the process of adding quality to a product. To verify there must be a measure of acceptance called expected resultsand to validate there must be acceptance criteria defined by the business.
“There is no defined scope for exploratory testing.”
Disagreeing with my quote above Huib Schoots wrote that “Using a coverage outline in a mind map or a simple spreadsheet, I keep track of what I have tested. “
I want to disagree with you. Keeping track of what you have tested does not necessarily mean you are working with a defined scope. For an example, you may be wondering in a bush without a destination in mind yet keep track by marking spots as you wonder around in the bush. Therefore, tracking your undefined activities is not the same as scoping.
Huib Schoots also wrote that “Exploratory testing is NOT ad hoc or unstructured when done right”
You might want to take this up with James Bach who wrote Exploratory Testing Explained (v.1.3, 4/16/03) and stated that; “Exploratory testing is also known as ad hoc testing. Unfortunately, ad hoc is too often synonymous with sloppy and careless work. So, in the early 1990s a group of test methodologists (now calling themselves the Context-Driven School) began using the term “exploratory” instead. With this new terminology, first published by Cem Kaner in his book Testing Computer Software, they sought to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline.
“There is no measure of progress, as testers cannot determine when testing is enough.”
On my assertion above Huib Schoots argues that “So testing is about making choices what to test and what not to test”. I strongly disagree. If testing is about building quality into a product by ensuring that all acceptance criteria are met Tester(s) therefore cannot make a choice solely on what to test or what not test without knowing when to stop testing. Testing cannot be stopped without knowing what scope was covered, and progress cannot be determined without a matrix of expected results.
“Still, not many Agile projects will require just two phases, like integration and regression. But it’s definitely not only exploratory testing that’s needed, as is erroneously believed in some quarters.”
My referring to ‘Phase’ above is simply referring to testing window. Yes, testing done by developer after developers have completed developing software is unit testing, and that is the unit testing phase or window depending on your choice of words. Please, don’t let us plunge into semantics but genuine arguments.
I have also noticed that some of your arguments are either supporting my assertion or presenting facts that are out of context. Not sure, but there may be a language barrier here. There was absolutely no need for the agile testing quadrants you posted to refute my claim here. Even then, looking at it holistically the Agile quadrant you posted supports my article. Remember that the title and the objective of my article on Scrum Alliance website is to present an argument that ‘Agile Methodology Is Not All About Exploratory Testing’? That it is, testing in Agile or Agile testing is not wholly about exploratory testing. I did not present argument in my article that that exploratory testing is not part of testing in Agile but that not wholly exploratory testing. In the top right an corner of the quadrant you posted, it identified Exploratory as one of the testing activities and not solely exploratory.
Finally, the article was posted to begin a productive and healthy argument about the topic that the Scrum Alliance editor pointed out to me weeks before it was posted. I do NOT believe that the article itself presents a premise for you to conclude about the level of my knowledge of software testing as you have erroneously inferred. .
Haven noticed that Huib Schoots argument is predicated largely on automation testing and the attempt to discredit rather than accommodate diverse views on the topic, I would like to suggest that he reads the following books;
- Agile Testing: A Practical Guide for Testers and Agile Teams Author(s): Lisa Crispin and Janet Gregory
- Succeeding with Agile: Software Development Using Scrum Author(s): Mike Cohn
- Agile Estimating and Planning Author(s): Mike Cohn
Dele Oluwole CSM