Testing¶
cloud-init has both unit tests and integration tests. Unit tests can
be found in-tree alongside the source code, as well as
at tests/unittests
. Integration tests can be found at
tests/integration_tests
. Documentation specifically for integration
tests can be found on the Integration Testing page, but
the guidelines specified below apply to both types of tests.
cloud-init uses pytest to run its tests, and has tests written both
as unittest.TestCase
sub-classes and as un-subclassed pytest tests.
Guidelines¶
The following guidelines should be followed.
Test Layout¶
- For ease of organisation and greater accessibility for developers not
familiar with pytest, all cloud-init unit tests must be contained
within test classes
- Put another way, module-level test functions should not be used
- As all tests are contained within classes, it is acceptable to mix
TestCase
test classes and pytest test classes within the same test file- These can be easily distinguished by their definition: pytest
classes will not use inheritance at all (e.g.
TestGetPackageMirrorInfo), whereas
TestCase
classes will subclass (indirectly) fromTestCase
(e.g. TestPrependBaseCommands)
- These can be easily distinguished by their definition: pytest
classes will not use inheritance at all (e.g.
TestGetPackageMirrorInfo), whereas
pytest
Tests¶
- pytest test classes should use pytest fixtures to share functionality instead of inheritance
- pytest tests should use bare
assert
statements, to take advantage of pytest’s assertion introspection- For
==
and other commutative assertions, the expected value should be placed before the value under test:assert expected_value == function_under_test()
- For
pytest
Version Gotchas¶
As we still support Ubuntu 16.04 (Xenial Xerus), we can only use pytest features that are available in v2.8.7. This is an inexhaustive list of ways in which this may catch you out:
- Support for using
yield
inpytest.fixture
functions was only introduced in pytest 3.0. Such functions must instead use thepytest.yield_fixture
decorator. - Only the following built-in fixtures are available [1]:
cache
capfd
caplog
(provided bypython3-pytest-catchlog
on xenial)capsys
monkeypatch
pytestconfig
record_xml_property
recwarn
tmpdir_factory
tmpdir
- On xenial, the objects returned by the
tmpdir
fixture cannot be used where paths are required; they are rejected as invalid paths. You must instead use their.strpath
attribute.- For example, instead of
util.write_file(tmpdir.join("some_file"), ...)
, you should writeutil.write_file(tmpdir.join("some_file").strpath, ...)
.
- For example, instead of
- The pytest.param function cannot be used. It was introduced in pytest 3.1, which means it is not available on xenial. The more limited mechanism it replaced was removed in pytest 4.0, so is not available in focal or later. The only available alternatives are to write mark-requiring test instances as completely separate tests, without utilising parameterisation, or to apply the mark to the entire parameterized test (and therefore every test instance).
Mocking and Assertions¶
Variables/parameter names for
Mock
orMagicMock
instances should start withm_
to clearly distinguish them from non-mock variables- For example,
m_readurl
(which would be a mock forreadurl
)
- For example,
The
assert_*
methods that are available onMock
andMagicMock
objects should be avoided, as typos in these method names may not raiseAttributeError
(and so can cause tests to silently pass). An important exception: if aMock
is autospecced then misspelled assertion methods will raise anAttributeError
, so these assertion methods may be used on autospeccedMock
objects.For non-autospecced
Mock
s, these substitutions can be used (m
is assumed to be aMock
):m.assert_any_call(*args, **kwargs)
=>assert mock.call(*args, **kwargs) in m.call_args_list
m.assert_called()
=>assert 0 != m.call_count
m.assert_called_once()
=>assert 1 == m.call_count
m.assert_called_once_with(*args, **kwargs)
=>assert [mock.call(*args, **kwargs)] == m.call_args_list
m.assert_called_with(*args, **kwargs)
=>assert mock.call(*args, **kwargs) == m.call_args_list[-1]
m.assert_has_calls(call_list, any_order=True)
=>for call in call_list: assert call in m.call_args_list
m.assert_has_calls(...)
andm.assert_has_calls(..., any_order=False)
are not easily replicated in a single statement, so their use when appropriate is acceptable.
m.assert_not_called()
=>assert 0 == m.call_count
When there are multiple patch calls in a test file for the module it is testing, it may be desirable to capture the shared string prefix for these patch calls in a module-level variable. If used, such variables should be named
M_PATH
or, for datasource tests,DS_PATH
.
Test Argument Ordering¶
- Test arguments should be ordered as follows:
mock.patch
arguments. When used as a decorator,mock.patch
partially applies its generatedMock
object as the first argument, so these arguments must go first.pytest.mark.parametrize
arguments, in the order specified to theparametrize
decorator. These arguments are also provided by a decorator, so it’s natural that they sit next to themock.patch
arguments.- Fixture arguments, alphabetically. These are not provided by a decorator, so they are last, and their order has no defined meaning, so we default to alphabetical.
- It follows from this ordering of test arguments (so that we retain
the property that arguments left-to-right correspond to decorators
bottom-to-top) that test decorators should be ordered as follows:
pytest.mark.parametrize
mock.patch
[1] | This list of fixtures (with markup) can be reproduced by running: py.test-3 --fixtures -q | grep "^[^ -]" | grep -v '\(no\|capturelog\)' | sort | sed 's/.*/* ``\0``/'
in a xenial lxd container with python3-pytest-catchlog installed. |