Vinktank

Requirements Testing

You can test code to make sure it works as specified by the requirements, which in agile is commonly known as acceptance testing. This type of testing places a lot of faith on those that created the requirements in getting them correct. Although there is talent in creating the acceptance criteria there is little skill involved in checking that an application does what it has been specified to do, and I believe there is a misconception that this is mostly what testers do (and for some testers all they need to do). That is one reason why agile projects will try and automate these tests.

Testing Requirements

Everybody makes mistakes, even the customer or business analysts will make mistakes, so who tests the requirements? Often this is left to user testing, which in some cases only occurs for real users once the application is released. Requirements testing is where I believe testers on an agile project can make a big contribution. With enough domain knowledge it is possible to perform exploratory testing to help understand what the system does and does not do and find gaps in the requirements.  An example of a gap in requirements I have come across frequently is around importing and exporting. Both are captured as separate stories the first is the ability to import a certain format. The second is to export so another system can then import the information. After implementing both stories it is not possible to import what is exported, which is often a requirement but never stated.

To help enhance and target exploratory testing to find gaps in requirements I like to do the following:

• Goal directed testing. With other team members (ba’s, customers, real users, developers), come up with goals that a user would want to achieve with the system and then see if you can do it and understand how easy and intuitive it is to accomplish. A goal would not be “I want to log in”, but instead it should focus on what a user is wanting to do in a session. For example “check my account balance and if I have enough money pay a bill”.

• Persona based testing. In most cases there are different types of people that will be using the system. Persona’s are often used by analysts to help understand who the users of the system are, what traits they have and what they would expect the system to do. If you don’t have access to real users, then using persona’s in conjunction with goal directed testing above is an excellent way of testing requirements by looking at the application as if you were that person.

After a couple of projects where QTP has been integrated with Continuous Integration (Cruise Control, Bamboo, Hudson) I have found that tdruntestset.exe (see Integrating Quick Test Pro with Cruise Control) is not the best way to do the integration. This executable takes up a large amount of CPU time, is very sensitive to parameters passed and Quality Center structure. So instead it has been easier to create a small script that uses the Quality Center API, logs in to Quality Center, locates the Test Set (or Test Sets) required to execute, runs them and waits for the tests to finish. Upon test set completion it checks the results to see if there were any failures, if any tests failed the script fails causing the build to fail. This script gives me more control over which tests sets to run, changing the run names to match build numbers etc. The script is then called via an ant exec tag (using cscript to execute the vbscript) and integrated with your continuous integration server of choice. (I have done it with Cruise, Bamboo and Hudson).

Run Test Suite From Quality Center is a copy of a “generic” vbscript, you will need to adjust it to your environment.