Getting more out of automated acceptance tests
There are things you know, things you think you know and things you don’t know. The first two are easy to test, you know how something should behave and can check that it does, or you think you know how something does behave and test to see if you are right. Testing for something you don’t know is harder. One technique that does help is to vary the data and environments being used. Running the same tests with different data can often uncover issues that had not previously been thought of, “Ah yeah those types of customers have that extra field.” If working with an existing system, ask around the support team and business representatives and you will normally find some problematic data. Using this data in the tests is a great way to find issues.
Acceptance tests are used to verify that a story (requirement) is complete, automating these tests is common and it helps regression testing and iterative development. Even though the tests are extremely powerful and have improved quality of projects delivered, they are not being used to their full potential. Some problems I have seen with acceptance tests and data include:
- Acceptance tests are tied tightly to data which means they can only be run in a development environment or if data is seeded into another environment.
- It is difficult to improve the tests by increasing/changing the data being used
- There is a reluctance by the team to “refresh” the data or worse the data is fabricated by people with little domain knowledge.
To get around these issues and be able to reuse the automated acceptance tests, separate the data from the test intention. By doing this you will get the following benefits:
- Can have multiple data sets for different environments allowing the tests to be run in each environment
- Data can easily be reviewed by other non technical team members
- Populating the data sets can be automated separately from the tests and does not confuse the test intention
- You can find things you don’t know
The same techniques that are used to create a data layer for an application should be applied.
That is an excellent point. We just faced this issue for current project. Need to test product in different environments, but every time we have to comment off and un-comment off some test data manually before running automation scripts. It is really painful. We are planning to separate testing data from existing codes.
However I think if we are doing that way, there may be some potential problems. First of all, extra code is needed to retrieve testing data and those testing data may be from different sources and in different formats. It will make retrieving method more complicated. In the meantime, some data are not gained easily from external source by automation.
Secondly sharing is an advantage of separating testing data, which means other team can also reuse those testing data. It will definitely reduce duplication. But it will cause another problem. It becomes very hard to maintain the stability of testing data. Any modification from one team may cause other team’s automation scripts failed.