Did you test it long enough?

At the beginning of this year I was in a restaurant. It was possible to buy a drink. So the software, which took care for the supply, was good. Also the training of the people was good, because I bought more than I intended to. There was one problem: how can I empty the bottle with this straw? My first thought was: did you test it long enough? As a tester I have heard many variations: What were you testing? How did you spend your time? Or did you pay attention during testing?

The sound of nuisance

At eleven o’clock in the morning George joined the meeting of the software testers.

“Good morning. My name is George. Your boss asked me to help you. I know a lot about software testing, but not much about cars. “
“Hi George, my name is Jack. At the moment I have an important problem with the tires of sold cars. The tires wear off faster than it was the case with the previous models. So car dealers began calling to the head office. And my boss got the question: “Did you test it long enough?”“

George looked confused: “What is the relationship between tires and software? “
“Nowadays everything is computerized. The automated suspension takes care for a comfortable ride, so not every stone in the road is felt by the passengers. There is something wrong in the software of the suspension. We were able to pinpoint the problem: it is the combination of a wet surface and steep road. “

“What did you do with this information?”
“I went to my boss and I explained the problem. Up till now the suspension software was one of the most reliable parts of the system, so it was not tested intensively. We tested the cars on a wet flat road and a dry steep road. We did not test the on a wet and steep road, because time is money. “

“What was your answer on the question, whether you tested it long enough? “
“I told that at the end of the test all criteria were discussed in depth. Every stakeholder was involved. At that moment it was the best decision we could take. Furthermore I suggested to change the criteria for the suspension software. He agreed with me on the spot.”

´Why did you change the criteria?”
“Because the tires wore off.”
“What you are actually saying, is:
“I wait, until something unpleasant happens. Then I change the criteria.”“
“That’s right.”
“Is there a way to change the criteria before testing? “
“Sure, I checked the release notes beforehand. I noticed, that some components had been changed. But the functionality was basically the same. “
“Did you determine the impact of the changes of the components?”
“No, because the quality of the software was good and the functionality did not change.”
“The supplier probably did some refactoring. So the basic questions you should pose as a tester are: What did you change? and how can I test it? Then you can determine, whether the criteria must be adjusted and extra tests must be added.”