Autopkgtest - Running tests

This document gives an overview how to run tests with autopkgtest. It does not cover each detail, please consult the individual manpages like autopkgtest(1), autopkgtest-schroot(1), etc. for all available options.

Ingredients for Debian packages:

Ingredients for Click packages:

Finally you need a virtualization server, which creates the environment in which the test runs. Depending on how intrusive the test is this can provide various degrees of isolation, from "run on my local system" (fastest, but unsafe) to "run in a temporary virtual machine" (slowest, but highest possible isolation). These are described in detail below.

autopkgtest

The autopkgtest program is the main program to run tests which gets all these ingredients as arguments, in the following form:

autopkgtest [options] <source package> [<binary package> ...] -- <virt-server> [<virt-server options>]

Specifying tests and packages

All possible options are explained in the autopkgtest(1) manpage. This section shows the most common scenarios, with using "mysrc" as source package name. Note that specifying the virtualization server and its options is handled in the following section, and it is independent of specifying tests and packages, so it is merely abbreviated as virt-server here.

Output

Unless you specify some options, autopkgtest just writes the logging, test outputs, and test results to stdout/stderr and exits with code 0 on success, or some non-zero code if there were skipped or failed tests or problems with the virt-server. (See autopkgtest(1) for defined codes).

For getting output files you have three choices:

You can also combine these.

Virtualization server

schroot

autopkgtest ... -- schroot schroot-name

Run tests in the specified schroot. You can use mk-sbuild(1) to conveniently create schroots, and run this as normal user if you configured schroot accordingly.

This server is the fastest available that provides "proper" file system isolation and revert, but it does not provide enough isolation for tests that need to start services, reconfigure the network, or open TCP ports which are already open at the host. If your test does not need to do these things this is the recommended server, as schroots are also useful for other tasks like building packages with sbuild.

See autopkgtest-schroot(1) manpage.

LXC

autopkgtest ... -- lxc container-name

Run tests in the specified LXC container. Containers provide full service and network isolation, but tests or packages cannot change the kernel or hardware configuration. If your test does not need that, this is the recommended server as it is faster than QEMU and works on all Linux architectures.

container-name will be cloned or be called with a temporary overlay file system if you specify the -e (--ephemeral) option, thus it will never be modified and you can run several tests in parallel safely. Unless your test or architecture or RAM availability doesn't work with overlayfs, using -e is highly recommended for better performance.

If your user can get root privileges with sudo, you can call autopkgtest as your normal user and specify -s (--sudo) so that the container can be started as root.

See autopkgtest-virt-lxc(1) manpage. This also explains how to build containers.

QEMU

autopkgtest ... -- qemu path/to/image

Run tests with QEMU/KVM using the specified image. The image will be run with a temporary overlay file system, thus it will never be modified and you can run several tests in parallel safely.

If your test needs a full machine including kernel/hardware access, this is the recommended runner; it provides complete system isolation, revert and breaks-testbed capabilities. But it is also the one with the biggest overhead and only works well on architectures with KVM acceleration (i. e. mostly x86).

See autopkgtest-virt-qemu(1) manpage. This also explains how to build suitable images, and the requirements of the guest.

null

autopkgtest ... -- null

This does not do any virtualization, but runs tests straight on the host. Beware that this will leave some clutter on your system (installed test or build dependency packages, configuration changes that the tests might make, etc.). It is not able to run tests with the "breaks-testbed" restriction. See autopkgtest-virt-null(1) manpage.

chroot

autopkgtest ... -- chroot /path/to/chroot

Run tests in the specified chroot. You need to call autopkgtest as root for this. There is no automatic cleanup or revert for the chroot, so unless you can provide this by some other means, don't use this.

ssh

autopkgtest ... -- ssh -l joe -H testhost.example.com

This is a generic runner for an externally set up testbed which assumes nothing else than a working ssh connection. This can call a "setup script" to create/configure a testbed (such as spinning up a cloud VM with nova or setting up SSH on a phone through ADB). See the manpage for details. autopkgtest ships setup scripts for an adb host (mostly for Ubuntu Touch), for nova (for cloud instances) and for MaaS currently; see their comment headers in /usr/share/autopkgtest/ssh-setup/.