From 4a62ddaa48ffc1cd4b1bd9fbea01e01e20308d65 Mon Sep 17 00:00:00 2001 From: markh Date: Mon, 30 Jan 2017 15:24:52 +0000 Subject: [PATCH] introduce unittest --- 06-edges.html | 159 ------------------------------------- 06-unittest.md | 124 +++++++++++++++++++++++++++++ 10-fixtures.html | 114 -------------------------- 10-fixtures.md | 58 ++++++++++++++ 06-edges.md => 11-edges.md | 0 5 files changed, 182 insertions(+), 273 deletions(-) delete mode 100644 06-edges.html create mode 100644 06-unittest.md delete mode 100644 10-fixtures.html rename 06-edges.md => 11-edges.md (100%) diff --git a/06-edges.html b/06-edges.html deleted file mode 100644 index ee48b53..0000000 --- a/06-edges.html +++ /dev/null @@ -1,159 +0,0 @@ - - - - - - Software Carpentry: Testing - - - - - - - - - - - -
- -
-
-
-

Testing

 -

Edge and Corner Cases

-
-
-

Learning Objectives

-
-
-
    -
  • Understand that edge cases are at the limit of the function’s behavior
    • -
  • Write a test for an edge case
    • -
  • Understand that corner cases are where two edge cases meet
    • -
  • Write a test for a corner case
    • -
-
-
-

What we saw in the tests for the mean function are called interior tests. The precise points that we tested did not matter. The mean function should have behaved as expected when it is within the valid range.

-

Edge Cases

-

The situation where the test examines either the beginning or the end of a range, but not the middle, is called an edge case. In a simple, one-dimensional problem, the two edge cases should always be tested along with at least one internal point. This ensures that you have good coverage over the range of values.

-

Anecdotally, it is important to test edges cases because this is where errors tend to arise. Qualitatively different behavior happens at boundaries. As such, they tend to have special code dedicated to them in the implementation.

- -

Consider the following simple Fibonacci function:

-
def fib(n):
-    if n == 0 or n == 1:
-        return 1
-    else:
-        return fib(n - 1) + fib(n - 2)
-

This function has two edge cases: zero and one. For these values of n, the fib() function does something special that does not apply to any other values. Such cases should be tested explicitly. A minimally sufficient test suite for this function would be:

-
from mod import fib
-
-def test_fib0():
-    # test edge 0
-    obs = fib(0)
-    assert obs == 1
-
-def test_fib1():
-    # test edge 1
-    obs = fib(1)
-    assert obs == 1
-
-def test_fib6():
-    # test internal point
-    obs = fib(6)
-    assert obs == 13)
-

Different functions will have different edge cases. Often, you need not test for cases that are outside the valid range, unless you want to test that the function fails. In the fib() function negative and noninteger values are not valid inputs. Tests for these classes of numbers serve you well if you want to make sure that the function fails as expected. Indeed, we learned in the assertions section that this is actually quite a good idea.

-
-
-

Test for Graceful Failure

-
-
-

The fib() function should probably return the Python built-in NotImplemented value for negative and noninteger values.

-
    -
  1. Create a file called test_fib.py
    2. -
  2. Copy the three tests above into that file.
    3. -
  3. Write two new tests that check for the expected return value (NotImplemented) in each case (for negative input and noninteger input respectively).
    4. -
-
-
-

Edge cases are not where the story ends, though, as we will see next.

-

Corner Cases

-

When two or more edge cases are combined, it is called a corner case. If a function is parametrized by two linear and independent variables, a test that is at the extreme of both variables is in a corner. As a demonstration, consider the case of the function (sin(x) / x) * (sin(y) / y), presented here:

-
import numpy as np
-
-def sinc2d(x, y):
-    if x == 0.0 and y == 0.0:
-        return 1.0
-    elif x == 0.0:
-        return np.sin(y) / y
-    elif y == 0.0:
-        return np.sin(x) / x
-    else:
-        return (np.sin(x) / x) * (np.sin(y) / y)
-

The function sin(x)/x is called the sinc() function. We know that at the point where x = 0, then sinc(x) == 1.0. In the code just shown, sinc2d() is a two-dimensional version of this function. When both x and y are zero, it is a corner case because it requires a special value for both variables. If either x or y but not both are zero, these are edge cases. If neither is zero, this is a regular internal point.

-

A minimal test suite for this function would include a separate test for the each of the edge cases, and an internal point. For example:

-
import numpy as np
-
-from mod import sinc2d
-
-def test_internal():
-    exp = (2.0 / np.pi) * (-2.0 / (3.0 * np.pi))
-    obs = sinc2d(np.pi / 2.0, 3.0 * np.pi / 2.0)
-    assert obs == exp
-
-def test_edge_x():
-    exp = (-2.0 / (3.0 * np.pi))
-    obs = sinc2d(0.0, 3.0 * np.pi / 2.0)
-    assert obs == exp
-
-def test_edge_y():
-    exp = (2.0 / np.pi)
-    obs = sinc2d(np.pi / 2.0, 0.0)
-    assert obs == exp
-
-
-

Write a Corner Case

-
-
-

The sinc2d example will also need a test for the corner case, where both x and y are 0.0.

-
    -
  1. Insert the sinc2d function code (above) into a file called mod.py.
    2. -
  2. Add the edge and internal case tests (above) to a test_sinc2d.py file.
    3. -
  3. Invent and implement a corner case test in that file.
    4. -
  4. Run all of the tests using py.test on the command line.
    5. -
-
-
-

Corner cases can be even trickier to find and debug than edge cases because of their increased complexity. This complexity, however, makes them even more important to explicitly test.

-

Whether internal, edge, or corner cases, we have started to build up a classification system for the tests themselves. In the following sections, we will build this system up even more based on the role that the tests have in the software architecture.

-
-
-
-
- Software Carpentry - Source - Contact - License -
-
- - - - - diff --git a/06-unittest.md b/06-unittest.md new file mode 100644 index 0000000..834951b --- /dev/null +++ b/06-unittest.md @@ -0,0 +1,124 @@ +--- +layout: page +title: Testing +subtitle: The Unittest Module +minutes: 10 +--- +> ## Learning Objectives {.objectives} +> +> - Understand how to use Unittest to write and run test cases +> - Understand that Unittest can be used for many styles of test, not just unit tests +> - +> - + +The Python standard library provides a module for writing and running tests, called `unittest`, which is documented for [Python3.6](https://docs.python.org/3.6/library/unittest.html?highlight=unittest#module-unittest) and [Python2.7](https://docs.python.org/2/library/unittest.html?highlight=unittest#module-unittest). + +This module is useful for writing all styles of test, including unit tests and functional tests. + +## Functional Tests + +Functional testing is a way of checking software to ensure that it has all the required functionality that's specified within its functional requirements. This will test many methods and may interact with dependencies like file access. + +Functional tests are often the first tests that are written as part of writing a program. They can link closely to a requirement: "This programme shall ...". + +These tests are testing overall behaviour. + +They are especially useful if code is being restructured or re-factored, to build confidence that the key behaviour is maintained during the restructuring. + +Often true unit tests, as described in the previous page, test the implementation and may require changing at the same time as the code during a refactor. + +## Unittest TestCases + +unittest is part of the Python core library; it defines a class, unittest.TestCase which provides lots of useful machinery for writing tests. We define classes which inherit from unittest.TestCase, bringing in their useful behaviour for us to apply to our tests. + +The reuse of objects in this way is a really useful programming approach. Don't worry if it is new to you. For testing with unittest, you can just think of it as a framework to fit your test cases into. + +We can rewrite the tests from the previous page, using the Unittest approach. + +~~~ {.python} +import unittest + +def mean(num_list): + try: + return sum(num_list)/len(num_list) + except ZeroDivisionError : + return 0 + except TypeError as detail : + msg = ("The algebraic mean of an non-numerical list is undefined." + " Please provide a list of numbers.") + raise TypeError(detail.__str__() + "\n" + msg) + +class TestMean(unittest.TestCase): + def test_ints(self): + num_list = [1,2,3,4,5] + obs = mean(num_list) + exp = 3 + self.assertEqual(obs, exp) + + def test_zero(self): + num_list=[0,2,4,6] + obs = mean(num_list) + exp = 3 + self.assertEqual(obs, exp) + + def test_double(self): + # This one will fail in Python 2 + num_list=[1,2,3,4] + obs = mean(num_list) + exp = 2.5 + self.assertEqual(obs, exp) + + def test_long(self): + big = 100000000 + obs = mean(range(1,big)) + exp = big/2.0 + self.assertEqual(obs, exp) + + def test_complex(self): + num_list = [2 + 3j, 3 + 4j, -32 + 2j] + obs = mean(num_list) + exp = -9 + 3j + self.assertEqual(obs, exp) + +if __name__ == '__main__': + unittest.main() +~~~ + +The naming of `class` and `def` entities is important. All classes should be named beginning `Test` and all methods should be named beginning `test-`. This enables the test runner to identify test cases to run. + +Save this code to a file, e.g. `test_mean.py`, again beginning the name with `test_`. This module can be run directly with python: + +~~~ {.bash} +python test_mean.py +~~~ + +~~~ {.output} +..... +---------------------------------------------------------------------- +Ran 5 tests in 2.053s + +OK + +~~~ + +Unittest reports failures and errors on test cases, which we can see if we run Python2, as one of our tests only passes in Python3: + +~~~ {.bash} +python2 test_mean.py +~~~ + +~~~ {.output} +.F... +====================================================================== +FAIL: test_double (__main__.TestMean) +---------------------------------------------------------------------- +Traceback (most recent call last): + File "test_mean.py", line 31, in test_double + self.assertEqual(obs, exp) +AssertionError: 2 != 2.5 + +---------------------------------------------------------------------- +Ran 5 tests in 2.433s + +FAILED (failures=1) +~~~ diff --git a/10-fixtures.html b/10-fixtures.html deleted file mode 100644 index 8578641..0000000 --- a/10-fixtures.html +++ /dev/null @@ -1,114 +0,0 @@ - - - - - - Software Carpentry: Testing - - - - - - - - - - - -
- -
-
-
-

Testing

 -

Fixtures

-
-
-

Learning Objectives

-
-
-
    -
  • Understand how test fixtures can help write tests.
    • -
-
-
-

The above example didn’t require much setup or teardown. Consider, however, the following example that could arise when comunicating with third-party programs. You have a function f() which will write a file named yes.txt to disk with the value 42 but only if a file no.txt does not exist. To truly test the function works, you would want to ensure that neither yes.txt nor no.txt existed before you ran your test. After the test, you want to clean up after yourself before the next test comes along. You could write the test, setup, and teardown functions as follows:

-
import os
-
-from mod import f
-
-def f_setup():
-    # The f_setup() function tests ensure that neither the yes.txt nor the
-    # no.txt files exist.
-    files = os.listdir('.')
-    if 'no.txt' in files:
-        os.remove('no.txt')
-    if 'yes.txt' in files:
-        os.remove('yes.txt')
-
-def f_teardown():
-    # The f_teardown() function removes the yes.txt file, if it was created.
-    files = os.listdir('.')
-    if 'yes.txt' in files:
-        os.remove('yes.txt')
-
-def test_f():
-    # The first action of test_f() is to make sure the file system is clean.
-    f_setup()
-    exp = 42
-    f()
-    with open('yes.txt', 'r') as fhandle:
-        obs = int(fhandle.read())
-    assert obs == exp
-    # The last action of test_f() is to clean up after itself.
-    f_teardown()
-

The above implementation of setup and teardown is usually fine. However, it does not guarantee that the f_setup() and the f_teardown() functions will be called. This is becaue an unexpected error anywhere in the body of f() or test_f() will cause the test to abort before the teardown function is reached.

-

These setup and teardown behaviors are needed when test fixtures must be created. A fixture is any environmental state or object that is required for the test to successfully run.

-

As above, a function that is executed before the test to prepare the fixture is called a setup function. One that is executed to mop-up side effects after a test is run is called a teardown function. By giving our setup and teardown functions special names pytest will ensure that they are run before and after our test function regardless of what happens in the test function. Those special names are setup_function and teardown_function, and each needs to take a single argument: the test function being run (in this case we will not use the argument).

-
import os
-
-from mod import f
-
-def setup_function(func):
-    # The setup_function() function tests ensure that neither the yes.txt nor the
-    # no.txt files exist.
-    files = os.listdir('.')
-    if 'no.txt' in files:
-        os.remove('no.txt')
-    if 'yes.txt' in files:
-        os.remove('yes.txt')
-
-def teardown_function(func):
-    # The f_teardown() function removes the yes.txt file, if it was created.
-    files = os.listdir('.')
-    if 'yes.txt' in files:
-        os.remove('yes.txt')
-
-def test_f():
-    exp = 42
-    f()
-    with open('yes.txt', 'r') as fhandle:
-        obs = int(fhandle.read())
-    assert obs == exp
-

The setup and teardown functions make our test simpler and the teardown function is guaranteed to be run even if an exception happens in our test. In addition, the setup and teardown functions will be automatically called for every test in a given file so that each begins and ends with clean state. (Pytest has its own neat fixture system that we won’t cover here.)

-
-
-
-
- Software Carpentry - Source - Contact - License -
-
- - - - - diff --git a/10-fixtures.md b/10-fixtures.md index 18bd440..6dd630b 100644 --- a/10-fixtures.md +++ b/10-fixtures.md @@ -104,3 +104,61 @@ In addition, the setup and teardown functions will be automatically called for _every_ test in a given file so that each begins and ends with clean state. (Pytest has its own neat [fixture system](http://pytest.org/latest/fixture.html#fixture) that we won't cover here.) + + +## Unittest + +The `unittest` module also provides this functionality for us with the `setUp` and `tearDown` methods that allow you to define instructions that will be executed before and after each test method (note the capitalisation of these names, which is important). + +~~~ {.python} +import os +import unittest + +def f(): + files = os.listdir('.') + if 'no.txt' not in files: + with open('yes.txt', 'w') as yesfile: + yesfile.write('42') + +class TestMod(unittest.TestCase): + def setUp(self): + # The setUp() function tests ensure that neither the yes.txt nor the + # no.txt files exist. + # unittest will run this before each test case is run. + files = os.listdir('.') + if 'no.txt' in files: + os.remove('no.txt') + if 'yes.txt' in files: + os.remove('yes.txt') + + def tearDown(self): + # The tearDown() function removes the yes.txt file, if it was created. + # unittest will run this after each test case + files = os.listdir('.') + if 'yes.txt' in files: + os.remove('yes.txt') + + def test_f(self): + # The first action of test_f() is to make sure the file system is clean. + exp = 42 + f() + with open('yes.txt', 'r') as fhandle: + obs = int(fhandle.read()) + self.assertEqual(obs, exp) + +if __name__ == '__main__': + unittest.main() +~~~ + +~~~ {.bash} +python test_setup_teardown.py +~~~ + +~~~ {.output} +. +---------------------------------------------------------------------- +Ran 1 test in 0.000s + +OK + +~~~ diff --git a/06-edges.md b/11-edges.md similarity index 100% rename from 06-edges.md rename to 11-edges.md