a Test Fuga On 2 A Flat Screens – Part 2

A single note might be forgotten; a melody might be engraved in one’s memory.

Words can be compared with music notes. In most cases a single word does not make much sense for a tester. Performance is too vague, good too ambiguous, funny too personal. In my previous blog post I described, how I had gathered useful information and created test ideas using mind maps. Now I had some groups of words, pieces of a melody. Time for music!

Let’s have a look

Now I had a lot of test ideas. The best way to combine them is to use a test charter. The first test charters were not exciting like “Explore the interface by finding the right buttons for the functions”. This sounds silly, but I could not explore, what I could not find.

So what about the two screens as mentioned in the title? During my test I had two computer screens in front of me. On one screen the Application Under Test was shown, on the other screen my Word Processor. With a turn of my head I could switch between the application and my notes.

Let’s make notes
At the start of my test session I noted relevant information like day, time, version, and test database in my document. Sometimes I described actions I had performed. I did not write all my actions, which would slow down the tests or my flow of thoughts. In case of possible bugs I would go back and describe other relevant steps for a proper repro path or reproduction path of the bug. Print screens were also useful to accelerate the note taking.

The programmer had warned me, that it was not possible to schedule the data exchange. So I only tried to look at the buttons. I found a button and clicked on it by accident. (A typical case of an automatic target seeking index finger.) This was a waste of time. But the application was still stable, so I assessed the situation. I had started the ad hoc operation.

I switched to the Windows Explorer. Maybe some traces of my action were still visible. In a subdirectory for temporary files I found interface files, which had been created shortly before. This was a bug: temporary files must be deleted after use. And a great opportunity to dissect the files.

Let’s cover

The structure of the interface files was relatively simple. So I chose a mind map to record the coverage. For every file a branch was added. For every type of record a sub branch was added to the branch of the file. If the record had passed the tests, I placed an OK icon on the branch. Otherwise a NOK icon was placed.

In case of files with many combinations I once used a spreadsheet to describe the coverage.

Let’s debrief

After the first test charter I noticed, that I spent more than the planned 1 hour. On the other hand I had started testing the most important part of the interface, the files. The one hour limit was too short for a good test session, so I extended it to two hours in later test sessions.

Then I updated my mind maps. I used a partial filled circle icon to indicate, how much I had completed the test charter. Furthermore I used similar icons to mark the progression of my test ideas. So one screen contained my mind map and the other screen my document with my notes.

Let’s ask

Whenever I had any questions, I added those to the mind maps. I marked the branches with question mark icons. A question could be like “How many attempts are made to transfer the files?” After I got the answer, I would put it in the mind map, removed the question mark icon, and updated the test charter mind map, if necessary.

During the test I asked the programmer, whether I had to test both the ad hoc service and the scheduled service in depth. He assured me, that the same code was used for both services. This saved me a lot of time. The heuristic Avoid duplication of code had been used.

Let’s update
Every day I would look to the test ideas to be tested. Then I looked to my test charters. I focused on the test charters with the highest product risks. On a daily basis the mind maps were updated with every piece of information and progression was tracked.

In the meantime the option to schedule services for the interface was gradually implemented. I started to add branches in the information mind map to describe the proper steps to start up the services. Because this process changed regularly, I modified my mind map accordingly. I noticed that my fellow testers also were struggling with services. So I put this information next to the mind maps in the knowledge management system.

Let’s use markers

During the tests I used the two markers TODO and BUG in my notes. After BUG a short description of the unwanted situation was given most of the times accompanied by a print screen. TODO was used to mark down situations, which needed further investigation. If I ran out of ideas during a test session, then I searched for TODO. At the end of the session I would file bug reports for situations marked with BUG. Afterwards BUG was replaced by the defect number and short description.

Over time my use of keywords changed. My notes were a chronological list of actions and print screens. New notes were added at the end of the document. Sometimes it was hard to reproduce bugs. So I used hash tags like in Twitter like #DoubleTime. I replaced the marker BUG with my own test tag. At the end of the document I placed #DoubleTime. Then I started to make notes to reproduce the bug. Of course not all strange situations were reproducible. I noted that and marked it with #Remarkable. In the future strange situations could be found by looking for #Remarkable in the content of the files using Windows Explorer.

This system was still awkward. Now I had two places in the file referring to the same strange situation. Then I started to use indent like this:
01-okt   Invalid value shown on screen

15-okt    used 2, 3, and 5. Not reproducible

Let’s look forward

At the end I stored the latest versions of my mind maps in the knowledge management system of my company. Because the files had been created by a non-standard program, I added images of the mind maps as well. This saved some mouse clicks for the interested reader. I also updated the steps to install the services in a proper way.

It was a nice job. I had experienced exploratory testing and learned a lot. Now it is time for me to move on.

a Test Fuga On 2 A Flat Screens

One of the most complex music pieces is a fuga. Just imagine several instances of the same melody at different progression points, which can be heard at the same moment. And together still sounding well. Don’t try this at a meeting. It is like 3 people talking at the same time and it still makes sense.

I am aware, that fugas are not everyone’s favourite music. Some readers prefer music from TV channels with names containing a minimum of Capitals. Other readers can remember the theme of their favourite movies. Coming soon in a theatre in their neighbourhood.  A theme or melody is a recurring reminder, that one listens to or watch a certain story.

A theme in software testing is that I constantly am aware, why I am testing a certain object. I have to be conscious, that I need to focus on the right things. And that is hard. Especially with bugs popping up and undocumented features being uncovered.

WYSIWYE: What You See Is What You Explore

First question I always ask is: what is the purpose of the new or changed functionality? Followed by a lot of other questions. In order to structure the obtained information I use a mind map program. The mind map contains gathered information and references to relevant files. Good news: some digital mind maps allow links to other digital files. Just click once to look at the relevant information.

In a world with an increasing number of interconnecting systems or things a test for an interface was not a surprise for me. I got specs and a minimal description of the interface. Files had to be exchanged. So I started digging:

  • What is the purpose of the interface?
  • What kind of data is contained in the files?
  • Is there a way to intercept the files?
  • What kind of program do I need to dissect the files?
  • How do I get the right data in the files?
  • What are my requirements for my test environment?
  • Etcetera.

I & SFDIPOT

There are several ways to determine, which areas must be tested. A heuristic or a fast and constructive way to determine, what can be tested, is SFDIPOT. A simple way to remember this is San Francisco DEPOT. For my fellow Dutch testers SOFT DIP, which is approved by one of the founders. Not me.

Some readers might suspect that “I” in the title of the paragraph is a reference to the author of this blog post. Other readers familiar with  SFDIPOT think, that “I” stands for Interface. It was strange to use this heuristic for the test of an interface, but it worked for me.

Next stop was to make a new mind map with the following branches using SFDIPOT:

  • Structure
    What is the interface? The interface was a set of files, which were exchanged. I had to test the system, which made the files, and test the created files. The receiving system was out of scope.
  • Function
    Does the system make the right files? I used some specifications to understand, what was contained in the files. On my PC the right tool was installed to read the records in the files.
  • Data
    What kind of information was stored in the files? Browsing through the specs I got some ideas about the content. The big question was how to cover the possible values. Which steps should I take to set the right values in the records of the files? Some record fields were fixed; others had a limited set of values. Of course there were record fields, which could contain text. Diacritic characters should be tested.
  • Interface
    How are the systems connected? I asked a programmer, whether I should test a network failure. His answer was: “Yes. Of course.” I made a note or branch in the mind map, that I should pull the network cable out of my PC during one interface test.
  • Platform
    Which operating system is used by the system, which makes the files? So my PC had Windows 7. Windows 8 and Windows 10 should also be tested.
  • Operations
    How will the interface being used? In order to support the file exchange some custom made services had to be installed. A restart of the PC could occur. This might impact the availability of the services. So a restart was planned for testing.
  • Time
    When will the interface be used? I noticed, that there were two options: ad hoc and scheduled. I asked myself, what would happen, when an ad hoc service and a scheduled service are being executed at the same time.

Whenever I ran out of ideas, then I switched back to the first mind map with the gathered information:

  • Did I use all relevant information for my test ideas?
  • There should be test ideas for time. Let me have another look.

A disclaimer: the complete set of test ideas is not described. This blog post is my experience report, how I handled the interface test.

Now I had some test ideas to turn into a theme for my Test Fuga. It was to time to structure the tests. My next blog post will cover the following steps and the Two A Flat screens.