Wednesday 27 August 2014

What's your test data strategy??

Through my years as a test manager I've seen many test strategies and plans. Most of them included a section on "test data". Often a short one mentioning the need for test data and the types of data listed in a table - that's it.

For some projects it's enough but more often I've come to realise that data is the test  - or data is driving the test. So there is a need to dig deeper and to be a little more focused on what counts as "test data".

ISTQB defines test data as: Data that exists (for example, in a database) before a test is executed, and that
affects or is affected by the component or system under test.

So far so good. There is also a definition of test data management:

The process of analyzing test data requirements, designing test data structures, creating and maintaining test data.

That is a lot more interesting because it touches upon a whole set of prerequisites for handling data - and ensuring that data is available in a controlled way. Even the most elaborate test cases, test analysis documents, test environments and skilled test team is worthless without data. And ever so often you see panicking people frantically trying to create data on the fly to pass that last important test.

So starting thinking about a structured approach to test data asking questions like:

  • Test data landscape - what's the total set of test data
  • Ownership of test data - system or organisation that "owns" data
  • Re-useability of test data - can data be handled in a smart wat
  • "Consumeable" test data - test data that can only be used once
  • Manipulation of test data - how to create data that can be used in a certain test context.
  • Primary types of test data - identify test data dependencies where one type of data is directly related to others.
These are probably some of the questions that will bring forward a lot of other relevant issues and challenges in a test environment where there are a number of integrated systems included in test. Once there are integrations outside your own physical organisation you must have a strategy - or trial and error will prove to be very expensive.

Thursday 21 August 2014

User reference group studies...

A colleague of mine showed me this video, and aside being funny (and a little disgusting) it has some points that I think is worth noticing – watch the clip before reading the rest, to prevent my comments ruining your laugh.

Remember to turn the sound on, the comments are priceless.

https://www.youtube.com/watch?v=m-wQhSuf6mM

What we see is what could be perceived as a user reference group. They are not likely to be the typical consumer of Surströmming, but they serve as excellent reference for users without knowledge of the product would experience.

Here is what I noticed:
  • There is a high enthusiasm, and expectation to this product. The group is clearly under influence of expectations build prior to the video. Be aware that you do not accidentally influence the group when setting up / explaining the concept to them.
  • The group is prepared, they have brought the tools for the task. This is very important, as the users’ opinion and reaction will be affected by lack of preparation, so make sure that the usability study is well prepared.
  • The opening of the product is catastrophic, this is definitely something that would raise a bug in the usability department. A label reading “handle and open with care” seems to be missing, or has escaped the attention of the users.
  • If product is so bad that it makes the people leave the area, you might want to rethink your design/implementation.
  • The user reference group invites in a new member – The dog. This is just a bonus for the test, make sure to capture information you get for free.
  • The user reference group shares associations about the product. Statements like “This reminds me of…” are very powerful, as it tells something about the mood and feelings that the user is experiencing. Capture them and ask questions to the story after the session.
Happy testing!
/Nicolai