Comments on: How do you measure the success of a test strategy http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/ Pragmatic software testing & development by Kristan Vingrys Tue, 06 Jan 2009 03:59:06 +0000 http://wordpress.org/?v=2.2 By: Dean Cornish http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/#comment-279 Dean Cornish Sun, 10 Feb 2008 22:40:56 +0000 http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/#comment-279 Heyas Kristan, Nice write up, being an ex-Microsoftie, I typically hit them with some off the wall questions- 1) You're a tester on a project, you're partnered with a developer who doesnt want to work on the project you're both on. He wants to work on other 'cooler' stuff.. What would you do? I like this question, as you can guage whether they'll take the "I'd escalate to their boss" type response indicating a preference for heirarchical environments versus collaborative, which typically would result in an answer like "I'd take them aside and talk to them, and explain I'm only trying to help, the sooner we get this done, the sooner we can move onto other things.." type answer. 2) A problem solving task, for example a common technical task like encyrption using public keys, abstract it into a problem that needs to be solved, just to figure out if they have the ability to immerse themselves into a problem. All too frequently testers take a spoon fed approach, which is if its not a requirement identified by a business user in the forms of reams of requirements documentation, then it doesnt exist- an extreme blackbox which seems to be made prolific by certain shops in Melbourne. 3) I draw a square on a piece of paper. I tell the prospect- this is a computer screen. Using as few lines as possible- show me some tests you'd make to test it can work. Each line is made up of 2 dots, in which a line is drawn between them. I do this kind of test as a test case writing problem. I like to see how creative they get.. are they the kind that just tests that it works? or do they try and break it.. and how? Heyas Kristan,

Nice write up, being an ex-Microsoftie, I typically hit them with some off the wall questions-
1) You’re a tester on a project, you’re partnered with a developer who doesnt want to work on the project you’re both on. He wants to work on other ‘cooler’ stuff.. What would you do?

I like this question, as you can guage whether they’ll take the “I’d escalate to their boss” type response indicating a preference for heirarchical environments versus collaborative, which typically would result in an answer like “I’d take them aside and talk to them, and explain I’m only trying to help, the sooner we get this done, the sooner we can move onto other things..” type answer.

2) A problem solving task, for example a common technical task like encyrption using public keys, abstract it into a problem that needs to be solved, just to figure out if they have the ability to immerse themselves into a problem. All too frequently testers take a spoon fed approach, which is if its not a requirement identified by a business user in the forms of reams of requirements documentation, then it doesnt exist- an extreme blackbox which seems to be made prolific by certain shops in Melbourne.

3) I draw a square on a piece of paper. I tell the prospect- this is a computer screen. Using as few lines as possible- show me some tests you’d make to test it can work. Each line is made up of 2 dots, in which a line is drawn between them.
I do this kind of test as a test case writing problem. I like to see how creative they get.. are they the kind that just tests that it works? or do they try and break it.. and how?

]]> By: Richard Durnall http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/#comment-257 Richard Durnall Fri, 01 Feb 2008 07:54:05 +0000 http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/#comment-257 What does the number of production defects tell me? If I have 10 defects and the application works is that worse than 1 defect that kills the application? The metric that would interest me wouldn't be the number of prod defects or even their criticality but the impact of the defects on the value-stream (what they are costing me in existing business or new opportunities). I'm playing devils advocate as usual and I guess it depends on the type of application and it's goals. I'm starting to worry that we're getting into too much of a 'management by numbers' mentality and as a symptom I'm not sure what the right things to measure are anymore!.... What does the number of production defects tell me? If I have 10 defects and the application works is that worse than 1 defect that kills the application?
The metric that would interest me wouldn’t be the number of prod defects or even their criticality but the impact of the defects on the value-stream (what they are costing me in existing business or new opportunities). I’m playing devils advocate as usual and I guess it depends on the type of application and it’s goals. I’m starting to worry that we’re getting into too much of a ‘management by numbers’ mentality and as a symptom I’m not sure what the right things to measure are anymore!….

]]> By: Sam Chen http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/#comment-255 Sam Chen Fri, 01 Feb 2008 03:50:28 +0000 http://www.vinktank.com/test-management/how-do-you-measure-the-success-of-a-test-strategy/#comment-255 If possible, it could be a good idea to get real customer involved in UAT instead of the person who deals with customer concern. Just borrow some time from the customer and let her play around the production to see if the production fully satisfies the requirement from her perspective. It could help to get quicker feedback and remove extra communication bridges. If possible, it could be a good idea to get real customer involved in UAT instead of the person who deals with customer concern. Just borrow some time from the customer and let her play around the production to see if the production fully satisfies the requirement from her perspective. It could help to get quicker feedback and remove extra communication bridges.

]]>