Deprecated: Assigning the return value of new by reference is deprecated in /home/vinkt3/public_html/wordpress/wp-includes/cache.php on line 36

Deprecated: Assigning the return value of new by reference is deprecated in /home/vinkt3/public_html/wordpress/wp-includes/query.php on line 21

Deprecated: Assigning the return value of new by reference is deprecated in /home/vinkt3/public_html/wordpress/wp-includes/theme.php on line 507
Vinktank | Testing your tests

Vinktank

Pragmatic software testing & development by Kristan Vingrys

Warning: mktime() [function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28

Warning: mktime() [function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28

Testing your tests

I get great satisfaction when I finally get an automated test to successfully run and pass, but I get more satisfaction when a passing test fails and picks up a defect. When automated tests are created they are normally created to pass and then used as a safety net to detect when a system stops behaving as it should. In the case of TDD, the tests will fail first, but start passing once the code has been developed. The team starts to put a lot of trust in the tests and an area that has a lot of automated tests which are all passing will be considered to have good quality, but how confident can you be that the tests will detect a failure if it occurs.

For example, I have a test for an add functionality which has 2 and 2 as inputs and expects 4 as a result. But if the code is written to do a multiply instead of an add the test would still pass!

One way to improve your confidence in your tests is to test them. You can do this by injecting failures into the system to test the tests and verify that they will detect failures you think they should detect. In some instances this can be done by changing the code under test or the system running the application to inject the type of failures you expect the test to detect. You may be able to change some data being used by the system or disconnect an interface to simulate the failure condition. If you cannot change the system under test then look at changing the test itself. So in the previous example, change the input from 2 and 2 to 2 and 3 with the expected result being 5. This would have picked up the difference between an add and multiply.

Automated tests are code, and with any code bugs can exist. Likely defects that occur in automated tests are that they will not detect a defect in the system that the test is expected to find. By testing your tests you have a better understanding of the types of defects the tests will detect. You can use this technique to help refactoring test code to make sure the refactoring has not had any adverse effects.

Meta | Similar


Warning: mktime() [function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
January 22nd, 2008

Tagged

  • No Tags

Freshness +


4 Comments | Ping Pong

  1. Evan Bottcher
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    January 31st, 2008

    I haven’t had a chance to try this out - but have a look at ‘heckle’.

    http://glu.ttono.us/articles/2006/12/19/tormenting-your-tests-with-heckle

    It’s very much about unit tests, but it’s a fascinating idea. Mutate the code under test, to ensure that some test fails if any logic is modified. If it works this would be much more effective than code coverage…

  2. admin
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    January 24th, 2008

    Carfield,

    The problem I am trying to describe is with tests that are written incorrectly and therefore do not go “red” when they should. This is not something you will pick up by looking at coverage because there is a test to cover the functionality. However the test is written incorrectly and therefore does not fail when it would be expected to. Just as people make mistakes when writing code (also known as bugs) people make mistakes when writing tests. To find the bugs in the tests you need to test them. Bugs in tests are generally not that the test will never pass, instead the bugs are that the test passes when it should fail.

    Kristan

  3. Carfield Yim
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    January 24th, 2008

    Usually if tests have problem will trigger red bar.

    If the the test haven’t covered some case, it is coverage problem more.

  4. Richard Durnall
    Warning: mktime() [
    function.mktime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 14

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 23

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 25

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 27

    Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/functions.php on line 28
    January 23rd, 2008

    How do I test my tests that test the tests? : )

    Isn’t this more of a case of thinking about what the right test should be? In the case above the 2 and 2 is a bad test compared to 2 and 3 (or anything else that could easily yield the correct result with an easily fudged implementation)?

    I’ve dived into TDD in the last few weeks and I’m struggling with the paradigm from the point of view of what is the right test to write and what level of abstraction do I write it at. I think that’s pretty normal though…and I agree. It is satisying when a test suddenly fails.

Leave a Reply

Freshest comments displayed topside. Comment accordingly. Name and email are required. Track comments via
Warning: strtotime() [function.strtotime]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/link-template.php on line 75

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Los_Angeles' for 'PDT/-7.0/DST' instead in /home/vinkt3/public_html/wordpress/wp-includes/link-template.php on line 87
RSS.

Design by: Derek Punsalan
RSS