Category Archives: Branding software testing

Test Twilight Zone

When I grew up, there was a TV program called The Twilight Zone. It was about people getting in really strange situations. Logic and laws of nature did not seem to apply.

The reason it appealed to me was that it could happen to me, the writer. Ordinary situations became unordinary situations.

This TV program had a tune. There are 3 notes which I still remember, to create a gloomy atmosphere. Those 3 notes became the tradenark of this zone.

Then the voice over would start like:
“This is a story of a woman. Let’s call her A. A is a very good UX designer or User Experience designer. She takes care, that people can use a program on first sight.

If she would design a kitchen tool, she could easily skip the manual. The product is so intuitive the moment you see and touch it.

She thinks that this is normal for her and not for other people. But she is about to enter the Twilight Zone.”

UX
A looked to me and asked me:
“How many bugs did you find?”
I mentioned a number above 40.
She swallowed.

I had no specs and not enough domain knowledge. Only a briefing from my PO or Product Owner. Still I had found some strange things.

She said that she had found about 20 bugs.
“What did you find?” she asked.
I described some bugs I had found. There was a bug with an input validation. I had just enough domain knowledge to point this one out.
I told about the details of the elements, which had confused me as a user. And …
“You are doing UX.”, she exclaimed.
“I am only testing.”

Then the voice over would come back: telling about the UX designer’s meeting. Mentioning the morale and The Twilight Zone.

“Next.”
“Huh.”
“Next story.”
“There you read” => PM

Then the voice over would start with:
“Mister X has more than 10 year of experience in “.

Let’s skip the scary part.
Okay with you?
Works on my web site.

Mister X and I were sitting at a table. We were talking about business on Sunday. That is typical Dutch by the way. Somehow the word migration was dropped and I could not stop myself to tell my deployment plan story.

My telling triggered the attention of Mister X. I told how I merged 3 plans and how I set up a meeting to discuss the result. X asked familiar questions which I had already covered in 2 piece Q and A.

“What was your role?”
“I was a tester. I wanted this project succeed, so I became the chairman.”
[Actually I made a mistake: I was a test coordinator. But still …]
Ten minutes in my story I heard:
“You were doing project management!”

At that moment I made contact. The next time he would remember my story about the migration. I would not be another faceless tester, which is to be avoided according to James.

PS
During a meeting last week I spoke up:
“I do not have a clue, but I have a weird idea.”

I almost forget the tune.

Finding courage

After more than 10 minutes of discussion the work item was clear to the developers and the product owner. Then the standard question was posed: “Are there any more questions?”

As a tester I had digested the information. I was not sure about the solution, so I raised my hand. Everyone looked at me.

What’s Up, Doc?

Currently I am the only tester in my team. If something has impact on testing or a quality related attribute, then I talk about it. It is something some people take for granted.

In the past people started rolling their eyes, if I questioned something. Until the main stakeholder supported me. Look who’s talking?

A few years ago I heard about a demo for a certain project. I tried to invite myself. My project leader objected with: “There is nothing to test.”

I persisted and attended the demo. Every now and then I posed a question. After the demo I heard no more objections about my presence.

Invitations for the remaining demos were even sent to me. The stakeholders obviously valued my input. Look who’s listening?

No harm intended
Last month the Skype rehearsal was not that successful. I had one month left to improve the exercises. They were crucial for my workshop at TestBash NL.

During the session I zoomed in on some exercises. In hindsight they were too big to handle in 1 go. Agile people might call them epic.

By breaking them down I got digestible mini exercises. I liked the idea.

Fast feedback for me and you.
[On the melody of Tea for two]

Some exercises had the complexity of my daily work. Using simple tests I might overlook edge cases. So let me complicate things please.

At the beginning of this section I wrote that the exercises were not that successful. I expected that the exercises went more smoothly than experienced.

Luckily there was useful feedback to improve the exercises. I had something to act upon. Things could only improve now. Also by writing down my thoughts and actions in this post.

During the preparation of every talk or workshop of mine there is a moment I think: “I cannot tell this.” And then the presentation is getting better. These experiences form my word of comfort.

The Skype rehearsal reminded me of #30daysoftesting: Lisa Crispin had doodled about experimenting. She was struggling, how to fit it in.

I tweeted her:
“There is no failure. There can’t be, if your only mission was to “see what happens”. ”
@sivers

An Oracle, Two Systems, And Four Situations

On September 26, 2015 Justin Rohrman ‏tweeted:
“Anyone know of the origin of the word ‘Oracle’ in software testing? Who started using it, what other words were used prior, …”

I answered with:

  • in the past in the Greek city of Delphi there was a special person you could ask question: Oracle.
  • The answer was sometimes cryptic. When the Persians invaded Greece, the answer was to hide behind
  • wooden doors. There was a smart man, who interpreted the answer right. He advised to build
  • a fleet of wooden ships, which beat the Persian fleet.

 

An oracle is a source of information, which I can use during testing. It can be used to determine, whether the observed situation is right. Right?

A dialogue from a movie with John Cleese about choosing a direction (while driving a car):
“Right?”
“Right?!”
“Left?”
“Left!”

Are you sure?
“Hey, I think I found a bug.”
“Okay. Tell me about it.” The other tester requested me.
I told as objective as possible my observations and my conclusion.
“Did you look in the previous version?”

Some systems are so complex. A fast way to sift possible bugs was to look at the system used in production. Of course afterwards it was always possible to read the specs or ask the PO, Product Owner.

One tester was able to pinpoint the version, where a bug was introduced, within minutes. He had the latest versions of the system available.

So I learned,

  • how to install more  than 5 different releases on my PC.
  • how to rollback a release including the database.
  • how to prepare the proper database and configuration for testing.

Where did you see it?
During the test I noticed, that the test version showed status information, which was not shown in the release in production. I asked other testers, who noticed the same. So my test environment was OK. There was still a doubt, whether the customers would miss the information.

The most simple way was to ask help from the help desk.

In the afternoon a blond woman was standing at my door post:
“You are right. The situation occurs in production.”
It was time to make a bug report.

Just don’t let it happen again
“I have not enough information to test the issue.” I complained.
“Did you test in the previous version?”, another tester remarked.
“No.”
“If you reproduce the steps from the issue, then you notice, what will go wrong. The issue has been solved, if the observed situation does not occur.”

During one project I had to test a system with service desk agents and PR colleagues. There were issues to be tested. I did not have time to make test cases for them.  If I let the testers test on their own, there was a chance  that they would check, whether only the bug was removed. I needed information about similar situations and connected cases.

I opened an issue in the defect registration system and read the description. “I would test this …” Then I wrote down the test ideas in the comment. Before the test I gave some instructions to the testers:
“In the comment situations are described, which must be tested. If you are ready, write in a comment, what you did. Then you assign the issue to me.”

After the start of the test the issues came back to me. I looked at the feedback. Sometimes I realised, that more situations had to be tested. Then I added them to the comment of the issue and assigned it back. The nicest thing, that happened, was that extra test ideas were added by the other testers.

Did you have a good look at it?
During the regression test I had to test the export and import function. I had two systems installed with different databases. The only things, what I did, were export data to a file and import the file in the other system.

I had two screens, so it was easy to compare the data: I placed the systems on different screens. If there was a difference between the source and target system, then I made a print screen. The picture contained images of both screens. Hey look mom! With one finger!

Next stop was my word processor, where I pasted the image. Then I used the keyword BUG followed with a small description. Sometimes I analysed the data file to pinpoint the source of the strange situation. If the file was wrong, then the source system might have some glitches. Otherwise it was the target system.

During the stand up I mentioned, that I had found a number of particular situations. This resulted in a session, during which I spoke with a programmer. The following pattern was repeated:

  • Search for “BUG”.
  • Resize the image to show the differences between the systems.
  • If necessary, look in the file.
  • Determine, whether it was a bug.

During the bug removal phase I got two kinds of feedback from the programmer: it was fixed or it cannot be fixed. The latter surprised me, but a study of the specifications changed my view. The format of the file did not support all types of information. There were semantic limitations. Something like: country is not supported, so this cannot be exchanged.

 

Does it fit?

Fancy restaurants have little delicious dishes and standard spoons, which might not fit.

Take that
“We cannot combine the automated test and the performance test.”, Scott stated. “They don’t fit. It’s a waste of time.”
Alice looked hopeful.
George brushed it away:
“Instead of testing you are talking. You’re the one, who wastes time.”
Scott was a man, who did not mind a good discussion:
“We’ll lose a lot of time, if we do not split the tests. ”
“My decision is definite.” George answered.
“Don’t you understand?” Scott shouted.
Alice started to look white.

“You can leave the room NOW.”
“You cannot run away from decisions.”
George stood up:
“You bet. I will have a little talk with HR right now.”
He went out the room and slammed the door.
Alice looked miserable.

The door opened slowly. George looked inside and said:
“That did not go quite right.”

Take Me To St Louis
“Scott, do you know a person you never shout at and who knows as much about automated tests and performance tests as your manager?” George asked.
Scott was silent for a few moments.
“My mother. ….!? Wait a sec. You are not involving my mom in this discussion!”

“Hi mom”, Scott began.
“What is the matter, son?”, George answered.
“I want to thank for the apple pie you made for me.”
George raised an eye brow, Alice smiled.
“Did you come to talk only about the apple pie? ”
“No, for my work I have to combine automated tests and performance test.”
“I am sorry, son, but I cannot help you with this. Just do, what they told you.”

Scott was shifting gears in his head.
“A performance test for a car is to look, whether it can handle the load.”
Alice put a thumb up.
“I understand: if I can put all ingredients for an apple pie in the car, then it can handle the load.”
“No mom, that is the purpose of an automated test. That one is focused on the functionality of the car: can I open the door, put the ingredients in the car, close the door, and drive away?”
George looked puzzled:
“What kind of load do you mean?”

George nodded.
Scott made another attempt:
“A load for car software are heavy conditions. Does the car ride well with four adults, luggage for long holiday on slippery roads in a dark rainy night?”
“I only fetch my apple pie ingredients in my car during daylight, when the weather is good. ” George explained.
Scott threw his arms up in the air.
A frown appeared on Alice’s forehead.

Take 5
“You are close.”, George assured Scott.

“You know, mom, that a car has a lot of software.”, Scott stated.
“You say so, son.”, George answered. Scott continued with:
“The dashboard must show the right information at the right moment, if I am speeding on the highway. In this case the computers in the car must work very hard. That is the purpose of the performance test.

An automated test can be used to check, whether all buttons and displays of the dashboard work properly. If I drive fast, I am not using every button on the dashboard. So the automated test and performance test cannot be combined. It does not fit.”
Alice put her right hand on her chin.

“Thank you for the explanation, son. You definitely earned an apple pie.”
Alice started laughing merrily. Scott joined in.

After Alice had stopped with laughing, Scott looked at Alice:
“Do you need more arguments, Boss?”

Alice sobered up. She got a focused look:
“You know, that there is a lot of pressure. So I tried to take a shortcut. I notice, that you have a lot of anger. I assume, that you have a need to be understood. Am I right?”
Scott just nodded.
“This basically means, that a choice must be made between quality and functionality. “, Alice continued.
“At the moment Quality has the best chances.”

Alice turned away from Scott.
“George, I really want to thank you for ..”
George’s chair was empty. He had left the room.

 

Can we scale down the tests? Part 2

Sometimes you have a harmonica with the wrong scale. It is hard to scale it down.  A simple solution is to buy harmonicas in all scales. It will cost time and money, but it might save the day.

If a manager walks into your cubicle with a suggestion to scale down tests, then it is time to get the big picture before the decision to skip tests. Let’s go back to George.

Quality is an option
At eleven o’ clock in the morning George joined the meeting of the software testers. Cynthia was not looking very happy.
“What’s wrong with you, Cynthia?”, George asked.
“Management decided to skip the user acceptance test for the navigation system to save time.”, Cynthia answered.
“Did you point out the consequences?”
“Of course the interface can really be improved.”
“Can you prove it?”
“Sure.”
“OK, let’s call the PO for his opinion.”

John joined the meeting of the software testers.
“Hi Cynthia, this is supposed to be a tester only meeting.”
“You are welcome.” Cynthia reassured him. “We won’t bite you.”
“Would you give me a permission to let the other testers have a look to the navigation system?”
“No problem.”

George was studying his laptop.
“According to the user story it is possible to enter the right destination using voice commands within 10 seconds.
Your aunt asked you to come to Paris, which is in the neighborhood of Tupelo.”

First test:
“The destination is Paris.”, Pete said.
“What is destination?” was the answer of the navigation system.
“I want to go to Paris.”
“What means “I want to”?”
“Go to Paris.”
“Go to Paris go to Paris is an invalid voice command.”

Second test:
“Go to Paris.”
“Do you want to go to Paris in Arkansas, Paris in California, Paris in Idaho, Paris in Illinois, Paris in Kentucky, Paris in Maine, Paris in Michigan, Paris in Mississippi, Paris in Missouri, Paris in New York, Paris in Ohio, Paris in Pennsylvania, Paris in Tennessee, Paris in Texas, Paris in Virginia, Paris in Grant County in Wisconsin or Paris in Kenosha County in Wisconsin?”

Third test:
“Go to Paris in the neighborhood of Tupelo.”
“What is neighborhood?”
“Begin again.”
“Go to Paris, which is near Tupelo.”
“How far is near?”
“I don’t know.”
“Give an estimation.”
“I don’t know.”

Fourth test:
“Go to Paris via Tupelo.”
“Do you want to go to Paris in Arkansas, Paris in California, Paris in Idaho, Paris in Illinois, Paris in Kentucky, Paris in Maine, Paris in Michigan, Paris in Mississippi, Paris in Missouri, Paris in New York, Paris in Ohio, Paris in Pennsylvania, Paris in Tennessee, Paris in Texas, Paris in Virginia, Paris in Grant County in Wisconsin or Paris in Kenosha County in Wisconsin?”

Fifth test:
“Go to Tupelo.”
“The destination is Tupelo.”
“Go to Paris, which is closest to Tupelo.”
“Give an estimation.”
“Uuuuuh.”
”What is Uuuuuh?”

George looked at the PO:
“Do you think, that a user acceptance test is a considerable option?”

Are you appreciated as a tester?

Some people might wonder at the size of the cup. My answer is, that I got it from a famous tester. Then the following list of action points is likely to be proposed:

  • Sell it on eBay.
  • Gather proof i.e. pictures.
  • Request Huib Schoots to sign a certificate of authenticity, that he gave this cup to Han Toan Lim for the most intensive and concentrated test session at ..

At that  moment I would interrupt with:
“Time Out dude. You cannot buy appreciation; you have to earn it.”

Test right there
The consultant with a strong HR background in IT looked at me.
“I see those small wheels spinning in your head. If you can read a design document, you can probably write one.”
“I already did.”, I admitted.
He continued: “Then you can describe the flow. If you can describe the flow, then you can program.”
He looked to me with the silent question:
“Why do you not move up the ladder?”

At that moment the time slowed down to a stop. Internally I sighed for 3 seconds. Then time accelerated to the normal speed. Reality snapped back and I was confused. Only a tenth of a second had passed in reality. I saw a man looking at me and waiting for an answer. Like a stubborn school boy I stated: “I just want to test.”

Talking about talking kids
During the holiday my wife talked about the show for children: “The theme is job. So a member of the animation team asked the kids about the job of their father.” I heard, that one of my kids answered with “Not a real job.”. I groaned. Another one said: “Software tester.” My wife imitated the small, hesitant voice of animator: “A software tester?!”

Then she prepared me, that something worse would come. I steeled myself. The next kid said: “Police agent.” The voice of the animator became enthusiastic: “Police agent. Did you hear that: police agent. That is great!” My wife was not pleased. Neither was I.

Yours gratefully
On my last day in the office and my last workday I noticed, that one of the functional application  managers had not dropped by to say goodbye. So I went to his desk. The talk, that followed, was about gone times, the present time and  times to come.

During the talk we had walked to the door to the corridor. It was the door to a new future for me. It was time to say goodbye. While shaking hands the functional application manager extended his left arm and patted on my shoulder. He let his smile disappear and instead he pressed his lips together to suppress his sadness. “You fare well.”
At that very moment I really felt appreciated as a tester.

 

Can we scale down the tests?

The harmonica is a music instrument to carry with you easily. It has a disadvantage. If you have one with E scale, then it is difficult to play songs in another scale. You miss tones and it is harder to change tones. There are different solutions to this problem: you could bend tones or just buy a chromatic harmonica. But it will still be awkward to play tunes in another scale. So it is difficult to scale down.

If a manager walks into your cubicle with a suggestion to scale down tests, then it is time to get the big picture before the decision to skip tests. Let’s go back to George, my favorite fictive tester, and his fictive testers, whom he coach, and their fictive technology.

No real reason to scale down

At eleven o’clock in the morning George joined the meeting of the software testers.
“Hi every one, you had a nice weekend?”
“Yes, I just got back from Texas”, Pete said.
“The last time you advised me to sneak in testing terms instead of clarifying the test definitions and their benefits. Especially after a kind manager request to scale down the tests.”
George nodded.
“So I went to my boss and I said:
“The customer wanted a leather steering wheel. It is advisable to do a regression test: can you still steer the car in the right direction?”
My boss got the point and arranged, that I could join the car test team in Texas with the latest version of the leather steering wheel.”

Pete started fiddling with his smart phone.
“You really have to see this.
Jack, could you please open a port on the projector, so I can stream this movie?”
The screen showed a picture of the latest car model under test in a desert. The reactions of the other testers came directly:
“Nice headlights!”
“Great design.”
“Hey, I tested those side mirrors.”
George noticed: “This footage looks professional.”
Pete answered: “This is a standard procedure. There are 4 cameras recording. The one, whose images are shown on the screen, is from the observer at the start.”

“That’s me.”
On screen Pete came walking with a filled bottle of Vodka.
“Now you have to watch this and listen.”
Pete said on screen: “Hey guys, as promised: I will lose this bottle, if I do not find a bug.”
Chuckling was heard from the speakers and in the room.
Pete put the bottle on the ground and started the car.
He drove a small circle around the bottle.
“Yes, your bottle is still there.”
Another circle followed by another one.

A commanding voice was heard from the speaker:
“Enough playing around, Pete. Head to the track.”
The circling continued and the car started to gain speed and slip.
“Pete, stop the car.”
The sounds of the car became harder.
“I repeat: Pete, stop the car. ”
After 10 seconds
“Okay, hit the emergency button.”
The car stood still and Pete got out of the car dizzy.

On the screen a man with a red cap wearing a head set and a microphone came in view. He went straight to the bottle. “The seal is broken. You drank.”
Pete protested: “That’s not true.”
He pulled out a breathalyzer test from his pocket and blew hard on it.
The man with red cap snatched the test from Pete´s hands and observed the test.
He continued meekly:
“So you are sober. But what went wrong with the car?”

At the background the voice of a mechanic was heard.
After a shout of pain he said: “The steering wheel is real hot under the leather. I almost burnt my hand.”
Pete said to the red cap: “I found my bug. Give me high five.”
Before the red cap knew, what was happening, he gave a high five.
“So what is your diagnosis, tester?”
“Did you burn your hand, when you gave me a high five?”
The red cap looked astonished to his hand: “Of course not.”

“In this car model a special form of servo assisted steering is used using sensors, so car drivers do not have to grip the steering wheel. A car driver can move his hand above the wheel and the car will automatically turn.“
The red cap nodded.
“While you were preparing everything for the test drive, the leather steering wheel started heating up in a closed car in the desert. Because my hand is cooler than the leather, the sensors ignored the movement of my hand. But the sensors determined, that my hands were in another position based on the temperature measurements. With the consequence, that the car made small circles.”

The red cap was impressed.
George looked also impressed: “The next time they will pay more attention to the product risks than to the solved issues.”

 

How to convince your manager with a door handle?

“How much time do you need to test this? “ my manager demanded.
“I cannot give an estimation, because I do not know, how it has been corrected.”
Annoyance came in her voice:
“Han Toan, what are you talking about?”
A pressing silence followed.

I looked around for a simple object to make my message clear. A powerful trick I learned from a senior business consultant. My eye fell on the door.
“Suppose, that this door handle had been broken. The supplier fixed it.”
She nodded slightly.
So I continued: “What would you test, if they only replaced the door handle?
The door? The room? The floor? The wing? The building?”
The atmosphere changed in the room: she understood it.

At that moment I did not need to use plan B:
“If the building has been replaced, would you test the structure of the building before testing the door handle?
If the complete wing has been replaced, would you check the water pipes and electrical wiring before testing the door handle?
If the door has been replaced, would you also check the lock?”

Did you test it long enough?

At the beginning of this year I was in a restaurant. It was possible to buy a drink. So the software, which took care for the supply, was good. Also the training of the people was good, because I bought more than I intended to. There was one problem: how can I empty the bottle with this straw? My first thought was: did you test it long enough? As a tester I have heard many variations: What were you testing? How did you spend your time? Or did you pay attention during testing?

The sound of nuisance

At eleven o’clock in the morning George joined the meeting of the software testers.

“Good morning. My name is George. Your boss asked me to help you. I know a lot about software testing, but not much about cars. “
“Hi George, my name is Jack. At the moment I have an important problem with the tires of sold cars. The tires wear off faster than it was the case with the previous models. So car dealers began calling to the head office. And my boss got the question: “Did you test it long enough?”“

George looked confused: “What is the relationship between tires and software? “
“Nowadays everything is computerized. The automated suspension takes care for a comfortable ride, so not every stone in the road is felt by the passengers. There is something wrong in the software of the suspension. We were able to pinpoint the problem: it is the combination of a wet surface and steep road. “

“What did you do with this information?”
“I went to my boss and I explained the problem. Up till now the suspension software was one of the most reliable parts of the system, so it was not tested intensively. We tested the cars on a wet flat road and a dry steep road. We did not test the on a wet and steep road, because time is money. “

“What was your answer on the question, whether you tested it long enough? “
“I told that at the end of the test all criteria were discussed in depth. Every stakeholder was involved. At that moment it was the best decision we could take. Furthermore I suggested to change the criteria for the suspension software. He agreed with me on the spot.”

´Why did you change the criteria?”
“Because the tires wore off.”
“What you are actually saying, is:
“I wait, until something unpleasant happens. Then I change the criteria.”“
“That’s right.”
“Is there a way to change the criteria before testing? “
“Sure, I checked the release notes beforehand. I noticed, that some components had been changed. But the functionality was basically the same. “
“Did you determine the impact of the changes of the components?”
“No, because the quality of the software was good and the functionality did not change.”
“The supplier probably did some refactoring. So the basic questions you should pose as a tester are: What did you change? and how can I test it? Then you can determine, whether the criteria must be adjusted and extra tests must be added.”