Testing

Cloud-init has both unit tests and integration tests. Unit tests can be found at tests/unittests. Integration tests can be found at tests/integration_tests. Documentation specifically for integration tests can be found on the Integration testing page, but the guidelines specified below apply to both types of tests.

Cloud-init uses pytest to run its tests, and has tests written both as unittest.TestCase sub-classes and as un-subclassed pytest tests.

Guidelines

The following guidelines should be followed.

Test layout

  • For ease of organisation and greater accessibility for developers unfamiliar with pytest, all cloud-init unit tests must be contained within test classes. In other words, module-level test functions should not be used.

  • Since all tests are contained within classes, it is acceptable to mix TestCase test classes and pytest test classes within the same test file.

    • These can be easily distinguished by their definition: pytest classes will not use inheritance at all (e.g., TestGetPackageMirrorInfo), whereas TestCase classes will subclass (indirectly) from TestCase (e.g., TestPrependBaseCommands).

  • Unit tests and integration tests are located under cloud-init/tests.

    • For consistency, unit test files should have a matching name and directory location under tests/unittests.

    • E.g., the expected test file for code in cloudinit/path/to/file.py is tests/unittests/path/to/test_file.py.

pytest tests

  • pytest test classes should use pytest fixtures to share functionality instead of inheritance.

  • pytest tests should use bare assert statements, to take advantage of pytest’s assertion introspection.

pytest version “gotchas”

As we still support Ubuntu 18.04 (Bionic Beaver), we can only use pytest features that are available in v3.3.2. This is an inexhaustive list of ways in which this may catch you out:

  • Only the following built-in fixtures are available [1]:

    • cache

    • capfd

    • capfdbinary

    • caplog

    • capsys

    • capsysbinary

    • doctest_namespace

    • monkeypatch

    • pytestconfig

    • record_xml_property

    • recwarn

    • tmpdir_factory

    • tmpdir

Mocking and assertions

  • Variables/parameter names for Mock or MagicMock instances should start with m_ to clearly distinguish them from non-mock variables. For example, m_readurl (which would be a mock for readurl).

  • The assert_* methods that are available on Mock and MagicMock objects should be avoided, as typos in these method names may not raise AttributeError (and so can cause tests to silently pass).

    • An important exception: if a Mock is autospecced then misspelled assertion methods will raise an AttributeError, so these assertion methods may be used on autospecced Mock objects.

  • For a non-autospecced Mock, these substitutions can be used (m is assumed to be a Mock):

    • m.assert_any_call(*args, **kwargs) => assert mock.call(*args, **kwargs) in m.call_args_list

    • m.assert_called() => assert 0 != m.call_count

    • m.assert_called_once() => assert 1 == m.call_count

    • m.assert_called_once_with(*args, **kwargs) => assert [mock.call(*args, **kwargs)] == m.call_args_list

    • m.assert_called_with(*args, **kwargs) => assert mock.call(*args, **kwargs) == m.call_args_list[-1]

    • m.assert_has_calls(call_list, any_order=True) => for call in call_list: assert call in m.call_args_list

      • m.assert_has_calls(...) and m.assert_has_calls(..., any_order=False) are not easily replicated in a single statement, so their use when appropriate is acceptable.

    • m.assert_not_called() => assert 0 == m.call_count

  • When there are multiple patch calls in a test file for the module it is testing, it may be desirable to capture the shared string prefix for these patch calls in a module-level variable. If used, such variables should be named M_PATH or, for datasource tests, DS_PATH.

Test argument ordering

  • Test arguments should be ordered as follows:

    • mock.patch arguments. When used as a decorator, mock.patch partially applies its generated Mock object as the first argument, so these arguments must go first.

    • pytest.mark.parametrize arguments, in the order specified to the parametrize decorator. These arguments are also provided by a decorator, so it’s natural that they sit next to the mock.patch arguments.

    • Fixture arguments, alphabetically. These are not provided by a decorator, so they are last, and their order has no defined meaning, so we default to alphabetical.

  • It follows from this ordering of test arguments (so that we retain the property that arguments left-to-right correspond to decorators bottom-to-top) that test decorators should be ordered as follows:

    • pytest.mark.parametrize

    • mock.patch