Autopkgtest - Defining tests for Click packages

This document describes how the autopkgtest tester core (the program adt-run) interprets and executes tests found in Click packages.

Overview

A click package's manifest defines tests and their metadata in the "x-test" dictionary. Each entry's key is the test name, and the value is a test executable or a dictionary that specifies the test command, dependencies, and restrictions. For example, a manifest might look like this:

{
    "name": "testclick",
    "title": "autopkgtest test click app",
    [... other manifest entries ...]
    "x-test": {
        "test1": "tests/simple",

        "test2": {
            "path": "tests/test-ui",
            "depends": ["ubuntu-ui-toolkit-autopilot"],
            "restrictions": ["allow-stderr"],
        },

        "test3": {
            "command": "echo hello"
        }
    }
}

The complete manifest usually gets generated during building and thus is read from the .click binary package. However, the tests are run from the click package source as usually they are not shipped in the binary. Thus at the moment you always have to specify the click source directory and the built .click binary package (in that order) to adt-run.

The tests are run with the corresponding click package being installed into the virtualization server. Note that this may fail if the testbed does not have "click" itself or the click package's requested framework installed.

The cwd of each test is guaranteed to be the root of the click source package, which will have been unpacked but not built. However note that the tests must test the installed version of the click package. Tests may not modify the source tree (and may not have write access to it).

During execution of the test, the environment variable $ADTTMP will point to a directory for the execution of this particular test, which starts empty and will be deleted afterwards (so there is no need for the test to clean up files left there).

If tests want to create artifacts which are useful to attach to test results, such as additional log files or screenshots, they can put them into the directory specified by the $ADT_ARTIFACTS environment variable. When using the --output-dir option, they will be copied into outputdir/artifacts/.

Control fields

The test fields which may appear in each test definition dictionary are:

path
(string) Specifies a path to an executable relative to the click source tree which runs the test.
command
(string) Shell command which is passed verbatim to bash
autopilot_module
(string) Python module name in tests/autopilot which is an Autopilot test. See below for details.

Exactly one of path, command, or autopilot_module must be given for each test. All other fields below are optional, and assumed to be empty lists of not present.

depends
(list of strings) Debian package dependencies. Every list entry is a single package name or |-separated alternatives, and may contain the usual Debian version and/or architecture restrictions. Note that with testbeds which are system image based you cannot install additional Debian packages, so for these this can merely ensure that the test dependencies are already installed.
restrictions
(list of strings) List of test restrictions as defined in README.package-tests.rst.
features
(list of strings) List of test features as defined in README.package-tests.rst.
classes
(list of strings) List of test class names, see README.package-tests.rst.

Simple test specification

If a test does not need any additional dependencies, restrictions etc., and just consists of a path to an executable, it can be specified as

"testname": "tests/testscript"

which is equivalent to

"testname": {
    "path": "tests/testscript"
}

Autopilot tests

For autopilot tests you can use the autopilot_module test field, which will be interpreted as a Python module name under tests/autopilot. An appropriate test command and default dependencies are automatically provided. Thus a test description

"testname": {
    "autopilot_module": "foo_tests",
    "depends": ["python3-dateutil"],
    "restrictions": ["allow-stderr"]
}

expands to

"testname": {
    "command": "PYTHONPATH=tests/autopilot:$PYTHONPATH python3 -m autopilot.run run foo_tests",
    "depends": ["ubuntu-ui-toolkit-autopilot", "autopilot-touch", "python3-dateutil"],
    "restrictions": ["allow-stderr"]
}

As a special case, if the test name is "autopilot" and the value is a single identifier, it is interpreted as an autopilot_module. Thus the description

"autopilot": "foo_tests"

is equivalent to

"autopilot": {
    "autopilot_module": "foo_tests"
}

which further expands to a complete description like above.