Test-Driven Development in Debian - BoF
=======================================

Questions
---------

- what can we do to improve package testing in Debian?

- which archive-wide packaging testing initiatives are you aware of in Debian?

 * Pere has done some archive-wide rebuilds (apt-get/aptitude), upgrades (including Gnome, XFCE, KDE desktops), …
 
 http://people.skolelinux.org/pere/blog/Lenny__Squeeze_upgrades__removals_by_apt_and_aptitude.html
 
 http://lintian.debian.org/~pere/
 
 * Rleigh also did some to test the various build-{arch,indep} strategies (and the breakeability of the archive).
 
 * "of course", Lucas' frequent archive-wide rebuilds.

- how do you test your packages?

 * Using unstable users. :D
 * Build in clean chroot (sbuild, …)
 * Check it with Lintian
 * piuparts?
 * Checklist verified manually.
 * Scripts in virtual machines.
 * dput'ting to a private repo, test upgrade, removal and install from a remote repository.
 * use debian/pbuilder-test.
 * extensive upstream tests (which I wrote myself)
 * continous integration where commit->push triggers autobuilds in Jenkins
 * build-time testing while building across wide-range of backports (invaluable information for upstream)

- do you have packages with (extensive) upstream test suites?

 * pyside has ~300 tests, with too many failing on weird arches
 * libdbi-drivers has a test suite which needs MySQL and postgress to be
 running and it was quite tricky to have them (mysql & pgsql) running in a
 build environment. (have a look in the source package if you want to see
 how it has been done.) It'd be great to have it standardised and reusable
 in a helper tool.
 * Openstack compute (nova) has loads of tests
 * haveged has test suites, detected and run automatically by dh 7
 * ruby packages (both the interpreters and libraries) have quite large test suites
 * Scilab packages has almost ~3000 tests (numerical computing, graphics, etc).
 * Majority of python-* packages maintained by NeuroDebian team (never counted number of the tests)
 * Shadow has a testsuite with currently about 800 tests (in the repository: http://anonscm.debian.org/viewvc/pkg-shadow/debian/trunk/tests/, not in the package). It is a good candidate for a "breaks-computer" testsuite (or at least change your root password, and do other nasty thing with your lovely config files). The test suite also supports code coverage (http://pkg-shadow.alioth.debian.org/coverage/ updated from time to time).

- if yes, do you run those test suites at package build-time?

 * yes ;-)

- if not, why not?
  - e.g. they are not flexible enough to be run at build-time
  * too time/memory hungry
  * require Internet access
  * require X
  * (shadow) I always assumed buildd maintainers would not like to get their passwd/shadow/PAM... files changed.

- are you aware of testing-related initiatives that other distribution are doing
  and that Debian can adopt?

  * yes -- there is quite a few in Ubuntu-land (see http://lists.debian.org/debian-devel/2011/02/msg00025.html)

- how can we leverage Debian packaging testing to benefit upstreams?
  - e.g. we have way more architectures than avg upstream
  * we can be ahead of upstream with testing on sid, thus preventing
    upcoming issues
  * we can test across a range of backports (Debian-/Ubuntu-releases)
    thus preventing upstream problems on their user's system
    (when not installed from Debian apt repository)
  
- how can we collect system tests collaboratively?

  * Add tests to LTP?

- how can we get package maintainers to pay attention to test failures?

  * File RC bugs

- what should the behavior of the test system when a test is freezing the test suite ?
  * For thoses tests which might freeze, I run under expect.

* what about graphics tests (with OpenGL in the worst case scenario) ? xvfb-run ?

See suggested tools in the Minutes.

Minutes
-------

autopkgtest
-----------

http://dep.debian.net/deps/dep8/
(Currently no packages implement DEP8)

build-time packages testing
---------------------------

 * dh v7 and dh_auto_test
 * Build-time tests are disable'able with the "nocheck" option in
   DEB_BUILD_OPTIONS, see Debian Policy 4.9.1.

Common "tests" names/targets lintian "should" check for:
 - make [check|test]
 - t/runtests (for Perl)?
 - ?
 ^ AFAIK dh7 looks for those already.
   Not everyone uses dh7, so there is still a good reason for finding these
   test suites and check if they are run. ACK.
 - Open question would be "what bug severity is suitable for packages that don't
   run their test suites"
 
 Liw has created prototypes of tools to test bootloaders or X stuff, vmdebootstrap and systest: 
 http://blog.liw.fi/tag/tddd/
 
 tuitest: a tool to create and run automated tests of text user interfaces
 http://synflood.at/programs.html
 
Sikuli <http://sikuli.org/> (in Wheezy) can do automated testing
of graphical interfaces using image recognition techniques. It can
probably used with virtualization do integration tests of the
installation system and the basic desktop.

Common gotcha's
---------------

* Testing Python applications using matplotlib: might require seting up $HOME and