Integration Testing

Overview

Integration tests are written using pytest and are located at tests/integration_tests. General design principles laid out in Testing should be followed for integration tests.

Setup is accomplished via a set of fixtures located in tests/integration_tests/conftest.py.

Image Selection

Each integration testing run uses a single image as its basis. This image is configured using the OS_IMAGE variable; see Configuration for details of how configuration works.

OS_IMAGE can take two types of value: an Ubuntu series name (e.g. “focal”), or an image specification. If an Ubuntu series name is given, then the most recent image for that series on the target cloud will be used. For other use cases, an image specification is used.

In its simplest form, an image specification can simply be a cloud’s image ID (e.g. “ami-deadbeef”, “ubuntu:focal”). In this case, the image so-identified will be used as the basis for this testing run.

This has a drawback, however: as we do not know what OS or release is within the image, the integration testing framework will run all tests against the image in question. If it’s a RHEL8 image, then we would expect Ubuntu-specific tests to fail (and vice versa).

To address this, a full image specification can be given. This is of the form: <image_id>[::<os>[::<release]] where image_id is a cloud’s image ID, os is the OS name, and release is the OS release name. So, for example, Ubuntu 18.04 (Bionic Beaver) on LXD is ubuntu:bionic::ubuntu::bionic or RHEL 8 on Amazon is ami-justanexample::rhel::8. When a full specification is given, only tests which are intended for use on that OS and release will be executed.

Image Setup

Image setup occurs once when a test session begins and is implemented via fixture. Image setup roughly follows these steps:

• Launch an instance on the specified test platform
• Install the version of cloud-init under test
• Run cloud-init clean on the instance so subsequent boots resemble out of the box behavior
• Take a snapshot of the instance to be used as a new image from which new instances can be launched

Test Setup

Test setup occurs between image setup and test execution. Test setup is implemented via one of the client fixtures. When a client fixture is used, a test instance from which to run tests is launched prior to test execution and torn down after.

Test Definition

Tests are defined like any other pytest test. The user_data mark can be used to supply the cloud-config user data. Platform specific marks can be used to limit tests to particular platforms. The client fixture can be used to interact with the launched test instance.

A basic example:

USER_DATA = """#cloud-config
bootcmd:
- echo 'hello config!' > /tmp/user_data.txt"""


class TestSimple:
    @pytest.mark.user_data(USER_DATA)
    @pytest.mark.ec2
    def test_simple(self, client):
        print(client.exec('cloud-init -v'))

Test Execution

Test execution happens via pytest. To run all integration tests, you would run:

pytest tests/integration_tests/

Configuration

All possible configuration values are defined in tests/integration_tests/integration_settings.py. Defaults can be overridden by supplying values in tests/integration_tests/user_settings.py or by providing an environment variable of the same name prepended with CLOUD_INIT_. For example, to set the PLATFORM setting:

CLOUD_INIT_PLATFORM='ec2' pytest tests/integration_tests/