Visual TDD in Test Automation – part 2

A month ago I was in the sport school. The manager had a conversation with me:

“You are using the wrong liquid to clean the machine.”
My eyes went to the automatic liquid dispenser.

She followed my view:
“You must use the hand trigger sprayer. You used the spray for the hand instead of the machine.”

Somehow I missed the hand trigger sprayer. It was clearly in view. But I came from the other side, the sprayer was hidden behind a machine.

So what was the problem?

People tend to use the most simple solution for a problem. You know, I am one of them.

For programming I use Test Driven Development. After some impressive demonstrations and real life experiences.
For me the Red Green Refactor cycle of Test Driven Development is simple.

When I used Visual Testing in the preivous blog post, I used a breakpoint.

If I set a breakpoint and debug the program, then it will stop at this breakpoint. Now I have enough time to get useful information from the program, get some coffee, or take a small nap. The latter is an advantage of blogging in free time or being jobless.

Disclaimer: I am not paid by Applitools for writing this article.

Let me visualise the steps from the last blog post:

Start: I have automation code, that opens the Login dialog.
A big rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle!

Red: make a failing test.
I added a CheckWindow method and placed a breakpoint on this line. During the debugging I manually entered the password “Cap” and resumed the program.

What I actually did, was making a screenshot of the desired situation or baseline.

Then I ran the program again. The test failed, because my code did not enter “Cap” in the username.
Why? Cap can be an abbreviation of Captain Marvel or Captain America.

Green: make enough code to pass the failing test.
Then I added code to enter “Cap” in the username textfield. Then I ran the program and it passed.
A rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle. In turn this big rectangle has an arrow with "new automated steps" to a big rectangle with a rectangle containing "Cap". The last big rectangle has an arrow with "snapshot" to a big rectangle with a rectangle containing "Cap"!

Refactor: there was no need to clean up the code, because it was a proof of concept.

Breakpoint overboard

For some people the breakpoint is really strange. So let me try to find a solution with using Test Driven Development while avoiding this point.
A breakpoint to be more precisely.

You are about to read about exploring a new situation in test automation.
Go.

Undesired baseline

And I off went.

For a fast demo I based my code on the Java code on
https://applitools.com/docs/topics/overview/walkthough-example.html#Building

I used the source code TestLogin2.java and pom.xml from this repo.  Notice the name of the class and the test.

The source code of TestLogin2.java is shown with the TestLogin2 class and test name "Login test 12"!

The really interesting coding took place is in the test01 method.

// Go to Login dialog.
driver.get(weburl);

// TODO: fill in the username

// TODO: add test.
// TODO: for making a baseline and
// TODO: comparing the screen with the baseline.
// [edited for blog post]

Now I added the CheckWindow method.

// Go to Login dialog.
driver.get(weburl);

// TODO: fill in the username

// add test.
// for making a baseline and 
// comparing the screen with the baseline.
// [edited for blog post]
eyes.checkWindow("Username filled in");

I left out the breakpoint: no red filled circle is shown.
On line 66 no breakpoint is set, so there is no red filled circle!

Then I ran the program.
On Testlogin2.java the right menu is opened and the Run TestLogin2.main() option is selected!

A message was shown, that a baseline was created.The console shows the following message: "New Baseline Created: 1 steps"!

Time to look in my free Applitools account.
The batch run "Login test 2" has been selected and the result shows a printscreen with a plus sign, thumb up sign, and a green bar!

Now I had done something really strange:
I made an undesired baseline.

Note the danger, that the passed test is accepted as passed, but that was not my intention.

That was not my goal for that day:
adding code to enter “Cap” in username.

There was a simple fix to solve this problem.

Desired baseline

Let me fill in “Cap”.

// Go to Login dialog.
driver.get(weburl);

// fill in the username
driver.findElement(By.id("username")).sendKeys("Cap");

// Test.
// add test. // for making a baseline and 
// comparing the screen with the baseline.
// [edited for blog post]
eyes.checkWindow("Username filled in");

I ran the program again.
On Testlogin2.java the right menu is opened and the Run TestLogin2.main() option is selected!

The test failed.
The console shows the following message: "Test failed : 1 steps matches=0 missing=0 mismatches=1"!

In applitools.com things were not updated yet, so I pressed the Refresh button.

The Refresh button is located under the last 30 batch runs and above Login test 2!

I opened the test result and noticed a difference. But the new screen was right.Button with two images of a form and a double pointed arrow between them is selected. In the left image or baseline; "Cap" in the username field is hightlighted in pink, in the right image or checkpoint several spaces in the username field is hightlighted in pink!

So I pressed the Thumb up button.

A message is shown± `You have successfully resolved all steps. Would you like to continu to the next step or close this editor=` with the Continue button and a Close button!
And the test ruslt became green.
The result shows a printscreen with an unequal sign, thumb up sign, and a green bar. In the upper right corner a Save button or Diskette button is shown!

There was Save button which changing size about every 4 seconds. So I pressed it/

I double checked the test result by removing the hightlighing of the differences.In the menubar a button with a stack of sheets is selected showing a menu with Diffs option which is unchecked. In the left image or baseline; "Cap" in the username field is not hightlighted, in the right image or checkpoint "Cap" in the username field is not hightlighted!

So I ran the program again.On Testlogin2.java the right menu is opened and the Run TestLogin2.main() option is selected!

The test passed this time.The console shows the following message: "All steps passed: 1 steps"!

I went to the Applitools website and pressed the Refresh button.

The Refresh button is located under the last 30 batch runs and above the batch run of Login test 2!

And I selected the last batch run.

The last batch run "Login test 2" has been not selected. The previous batch run "Login test 2" has been selected!

The green test result was shown.

The last batch run "Login test 2" has been selected and the result shows a printscreen with an equal sign, thumb up sign, and a green bar!

There were no differences.

Button with two images of a form and a double pointed arrow between them is selected. In the left image or baseline; "Cap" in the username field is not hightlighted, in the right image or checkpoint "Cap" in the username field is not hightlighted!

Questions

Now I had a test to check whether the username was equal to Cap.
But did I see whether the test fail?
No. Can I trust a test, which I did not see failing?
Probably not.

Another question:
is it good to trust a test, which status went from Green to Red?
I don’t think so.
If someone takes over the coding  of my test, then this would be really confusing: “A failed test is a good sign?!”

According to me this is the problem:
I have a useful pattern or way of working. It is Red Green Refactor.
Then I changed it into an anti pattern or bad way of working. This is Green Red Refactor.
This happened, because I wanted to avoid a breakpoint.

There is another way to explain this.
I have the habit to make pictures of the state of the game or baseline. Please have a look at a game in progress.

Same hotel board game in progress from another angle!

A hotel board game in progress!

Notice that I took pictures from different angles to be sure that the positions of all game pieces were clear.
Also to be sure that there was no hand trigger sprayer hidden somewhere somehow.

Suppose one of my kids would protest: “We were playing another game.” This is actually a bad baseline.

Then I could changed to this picture:A game of Settlers of Catan in progress!

Right game, but right state?
Looking at the boats I don’t think so.A board game of Settlers of Catan is shown with land and and sea tiles. At the edges of the sea files there are ships transporting people and spices!

Visual steps without a breakpoint

Let me visualise the steps taken in this very blog post:

Start: I have automation code, that opens the Login dialog.
A big rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle!

Green: make enough code to pass the failing test.
I added a CheckWindow method and ran the program.

What I actually did, was making a screenshot of the undesired situation or baseline.

When I ran the program again, the test passed.
A rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle. In turn this big rectangle has an arrow with "snapshot" to a big rectangle containing an empty rectangle!

Red: make a failing test.
Then I added code to enter “Cap” in the username textfield. Then I ran the program and the test failed.
A rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle. In turn this big rectangle has an arrow with "new automated steps" to a big rectangle with a rectangle containing "Cap". The last big rectangle has an arrow with "snapshot" to a big rectangle with a rectangle containing "Cap"!

Refactor: there was no need to clean up the code, because it was a proof of concept.

The used anti pattern is Green Red Refactor.
Highly unrecommended.

Wrap up

In this blog post I described a way to use Visual Testing without a breakpoint. I also did not really use Test Driven Development.

There were two drawbacks to this approach:

  1. The test CheckWindow was not tested. So an extra step was needed.
  2. I assumed that the change in the test result would be a sign of progress.