Deprecated: Assigning the return value of new by reference is deprecated in /home/vinkt3/public_html/wordpress/wp-includes/cache.php on line 36

Deprecated: Assigning the return value of new by reference is deprecated in /home/vinkt3/public_html/wordpress/wp-includes/query.php on line 21

Deprecated: Assigning the return value of new by reference is deprecated in /home/vinkt3/public_html/wordpress/wp-includes/theme.php on line 507
Vinktank | How do you measure the success of a test strategy

Vinktank

Pragmatic software testing & development by Kristan Vingrys

Warning: mktime() [function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28

Warning: mktime() [function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28

How do you measure the success of a test strategy

People tend to use number of test cases, ratio of passed v failed test cases, number of defects found per test case or similar metrics as measures of how good a test strategy is.

Although these metrics can give some indication, to my mind the best metric to measure the effectiveness of all your testing effort is how many unknown issues are found in production. By unknown issues I am referring to issues that were not detected prior to go live. There is often debate about how long you measure issues in production and how subjective the measure is, is it a defect or a new requirement. But in the end it doesn’t matter. If the customers are unhappy and complaining about unknown issues then it is an indication the test strategy could be improved. Defects are obvious, testing could likely pick them up, but missing requirements can also be detected by the testing effort, especially UAT.

It can be difficult to get the metrics of issues raised in production, some companies just don’t collect them. I have found it effective to get the person who deals with customer concerns to join the team at stand-up for the first couple of weeks after go live to discuss the issues raised by customers for the previous day. If that is not possible then schedule some time to talk with them soon after the application is in production. Understanding what issues are being reported by users is invaluable when assessing the effectiveness of your test strategy.

Meta | Similar


Warning: mktime() [function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
January 30th, 2008

Tagged

  • No Tags

Freshness +


3 Comments | Ping Pong

  1. Dean Cornish
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    February 10th, 2008

    Heyas Kristan,

    Nice write up, being an ex-Microsoftie, I typically hit them with some off the wall questions-
    1) You’re a tester on a project, you’re partnered with a developer who doesnt want to work on the project you’re both on. He wants to work on other ‘cooler’ stuff.. What would you do?

    I like this question, as you can guage whether they’ll take the “I’d escalate to their boss” type response indicating a preference for heirarchical environments versus collaborative, which typically would result in an answer like “I’d take them aside and talk to them, and explain I’m only trying to help, the sooner we get this done, the sooner we can move onto other things..” type answer.

    2) A problem solving task, for example a common technical task like encyrption using public keys, abstract it into a problem that needs to be solved, just to figure out if they have the ability to immerse themselves into a problem. All too frequently testers take a spoon fed approach, which is if its not a requirement identified by a business user in the forms of reams of requirements documentation, then it doesnt exist- an extreme blackbox which seems to be made prolific by certain shops in Melbourne.

    3) I draw a square on a piece of paper. I tell the prospect- this is a computer screen. Using as few lines as possible- show me some tests you’d make to test it can work. Each line is made up of 2 dots, in which a line is drawn between them.
    I do this kind of test as a test case writing problem. I like to see how creative they get.. are they the kind that just tests that it works? or do they try and break it.. and how?

  2. Richard Durnall
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    February 1st, 2008

    What does the number of production defects tell me? If I have 10 defects and the application works is that worse than 1 defect that kills the application?
    The metric that would interest me wouldn’t be the number of prod defects or even their criticality but the impact of the defects on the value-stream (what they are costing me in existing business or new opportunities). I’m playing devils advocate as usual and I guess it depends on the type of application and it’s goals. I’m starting to worry that we’re getting into too much of a ‘management by numbers’ mentality and as a symptom I’m not sure what the right things to measure are anymore!….

  3. Sam Chen
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    January 31st, 2008

    If possible, it could be a good idea to get real customer involved in UAT instead of the person who deals with customer concern. Just borrow some time from the customer and let her play around the production to see if the production fully satisfies the requirement from her perspective. It could help to get quicker feedback and remove extra communication bridges.

Leave a Reply

Freshest comments displayed topside. Comment accordingly. Name and email are required. Track comments via
Warning: strtotime() [function.strtotime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/link-template.php on line 75

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/link-template.php on line 87
RSS.

Design by: Derek Punsalan
RSS