Testing

Cloud-init has both unit tests and integration tests. Unit tests can be found at tests/unittests. Integration tests can be found at tests/integration_tests. Documentation specifically for integration tests can be found on the Integration testing page, but the guidelines specified below apply to both types of tests.

Cloud-init uses pytest to run its tests, and has tests written both as unittest.TestCase sub-classes and as un-subclassed pytest tests.

Guidelines

The following guidelines should be followed.

Test layout

• For ease of organisation and greater accessibility for developers unfamiliar with pytest, all cloud-init unit tests must be contained within test classes. In other words, module-level test functions should not be used.

• Since all tests are contained within classes, it is acceptable to mix TestCase test classes and pytest test classes within the same test file.

  • These can be easily distinguished by their definition: pytest classes will not use inheritance at all (e.g., TestGetPackageMirrorInfo), whereas TestCase classes will subclass (indirectly) from TestCase (e.g., TestPrependBaseCommands).

• Unit tests and integration tests are located under cloud-init/tests.

  • For consistency, unit test files should have a matching name and directory location under tests/unittests.

  • E.g., the expected test file for code in cloudinit/path/to/file.py is tests/unittests/path/to/test_file.py.

pytest tests

• pytest test classes should use pytest fixtures to share functionality instead of inheritance.

• pytest tests should use bare assert statements, to take advantage of pytest’s assertion introspection.

Dependency versions

Cloud-init supports a range of versions for each of its test dependencies, as well as runtime dependencies. If you are unsure whether a specific feature is supported for a particular dependency, check the lowest-supported environment in tox.ini. This can be run using tox -e lowest-supported. This runs as a Github Actions job when a pull request is submitted or updated.

Mocking and assertions

• Variables/parameter names for Mock or MagicMock instances should start with m_ to clearly distinguish them from non-mock variables. For example, m_readurl (which would be a mock for readurl).

• The assert_* methods that are available on Mock and MagicMock objects should be avoided, as typos in these method names may not raise AttributeError (and so can cause tests to silently pass).

  • An important exception: if a Mock is autospecced then misspelled assertion methods will raise an AttributeError, so these assertion methods may be used on autospecced Mock objects.

• For a non-autospecced Mock, these substitutions can be used (m is assumed to be a Mock):

  • m.assert_any_call(*args, **kwargs) => assert mock.call(*args, **kwargs) in m.call_args_list

  • m.assert_called() => assert 0 != m.call_count

  • m.assert_called_once() => assert 1 == m.call_count

  • m.assert_called_once_with(*args, **kwargs) => assert [mock.call(*args, **kwargs)] == m.call_args_list

  • m.assert_called_with(*args, **kwargs) => assert mock.call(*args, **kwargs) == m.call_args_list[-1]

  • m.assert_has_calls(call_list, any_order=True) => for call in call_list: assert call in m.call_args_list

    • m.assert_has_calls(...) and m.assert_has_calls(..., any_order=False) are not easily replicated in a single statement, so their use when appropriate is acceptable.

  • m.assert_not_called() => assert 0 == m.call_count

• When there are multiple patch calls in a test file for the module it is testing, it may be desirable to capture the shared string prefix for these patch calls in a module-level variable. If used, such variables should be named M_PATH or, for datasource tests, DS_PATH.

Test argument ordering

• Test arguments should be ordered as follows:

  • mock.patch arguments. When used as a decorator, mock.patch partially applies its generated Mock object as the first argument, so these arguments must go first.

  • pytest.mark.parametrize arguments, in the order specified to the parametrize decorator. These arguments are also provided by a decorator, so it’s natural that they sit next to the mock.patch arguments.

  • Fixture arguments, alphabetically. These are not provided by a decorator, so they are last, and their order has no defined meaning, so we default to alphabetical.

• It follows from this ordering of test arguments (so that we retain the property that arguments left-to-right correspond to decorators bottom-to-top) that test decorators should be ordered as follows:

  • pytest.mark.parametrize

  • mock.patch