Squashing Release-Critical Bugs in Debian: A Primer

A running theme in the Debian release team's missives to the rest of the project is the perpetual need for attention to bugs that have release-critical (RC) severity. These bugs are always in need of attention both from package maintainers and from interested others. However, while package maintainers are generally expected to know how to handle RC bugs on their own packages, it can be daunting for a non-maintainer to find and tackle an RC bug that is appropriate to their skill level.

The purpose of this document is to make the case, through the presentation of several simple strategies for choosing and approaching release-critical bugs, that RC bugs can be assailable by individuals with only a moderate background in shell or programming languages and a healthy understanding of Debian packaging practices.

Picking RC bugs

The first step in picking an RC bug to work on is to rule out certain bugs that you shouldn't work on. You should generally avoid:

Since this is already quite a list of criteria to sort through, it helps to have some tools to do some of the work for us. bugs.debian.org holds the more or less canonical list of release-critical bugs for the release, and is updated every 6 hours. However, the list at bts.turmzimmer.net is updated more frequently and is also much more flexible, letting us easily exclude many of the bugs we want to avoid: bugs with patches, bugs that only affect unstable, and new bugs. The page also gives you an option to exclude bugs fixed in delayed uploads, but only by also excluding packages that have been hinted for removal from testing by the release team; and it lets you exclude claimed bugs, but only by also excluding orphaned packages. As a result, neither of these buttons are really recommended; instead, the color-coding of the bug list will help you skip over delayed or claimed bugs. You may also see bugs tagged pending, meaning the maintainer has promised to upload the fix soon; but actions speak louder than words, so if these bugs don't have patches yet in the BTS, they're probably worth working on.

At most points in the release cycle, eliminating all of the bugs mentioned above will still leave you with a sizeable number of bugs to choose from — still a difficult task. At this point, the best strategy for picking bugs to work on is a simple one: look for bugs in packages you use and are already familiar with! The rc-alert script in the devscripts package can show you a list of RC bugs in packages you have installed on your system; but even if you don't see any bugs in packages you know, please consider helping with the bugs on this page — they're still important for the release process, even if you don't directly need those packages to be fixed.

If you can't find any bugs on this list that you think you're able to tackle, consider looking at some of the younger bugs or bugs that only affect unstable before giving up. If you're going to look at bugs in unstable, however, please only look at bugs in packages that you use; a well-intentioned NMU on a package that should actually be orphaned and/or removed from the archive can be worse than letting the package fester in unstable. If a package is only present in unstable and it has RC bugs, this often means nobody cares about the package, and we're better off not shipping it in a stable release.

If all else fails in your search for an RC bug to squash, try asking a Debian Developer, preferably one who knows your skill level, for suggestions of bugs to work on. The #debian-bugs channel on irc.debian.org is always open, and you can often find developers there who would be happy to help match you with bugs.

Common classes of RC bugs

Since a large number of release-critical bugs fall into certain common categories, it's useful at this point to discuss those categories and some strategies for handling them, to help you further filter the list of bugs you may want to work on.

FTBFS (fails to build from source) bugs

This class forms a large percentage of all RC bugs. They are usually detected in unstable, since failing to build usually blocks the package from reaching testing, in which case they wouldn't show up on the turmzimmer.net view we're using; but as the release progresses, various developers do run automated rebuild tests of the testing suite and file bugs about build failures in testing. In addition, some build failures are caused by changes in other packages; a perfectly working package in testing can get an RC bug because one of the packages it build-depends on has disappeared. As a result, we can usually count on some FTBFS bugs affecting testing at any given time.

Even if they're not always easy to fix, these bugs are easy to get started on: you know that there's a build failure, so you start by making sure the bug is reproducible. If the build failure is on an autobuilder, you can use buildd.debian.org to find full build logs, even if it's for an architecture that you don't have access to. If the failure was not from an autobuilder, the bug report should include information on how to reproduce it; and once you can reproduce it, you should be able to start debugging it.

Reproducing a FTBFS can sometimes be a challenge; the following strategies may help you when you run into a build failure that you can't reproduce.

Some packages uploaded to the archive may have built on the developer's machine but nowhere else. This is usually caused by missing build dependencies, and is one of the easiest kind of FTBFS bugs to fix. Most missing build dependencies can be guessed from the error message with some healthy use of apt-cache search and a bit of trial and error. When specific files are reported as missing, you can use apt-file or the content search of packages.debian.org to find out which package(s) contain them.

Sometimes packages seem to miss build dependencies, but only on some machines, whereas on others they build fine. You may have encountered a timeskew issue. This happens when a file that is generated from another one is included in the source of the package (e.g. configure, generated by autoconf, or the C files generated by bison/yacc) and the Debian diff contains patches to both of them. Since dpkg doesn't preserve the order of modification dates when patching files during the extraction of a source package, the make program used to build the package may get confused whether it has to regenerate these files. See /usr/share/doc/autotools-dev/README.Debian.gz from the autotools-dev package for a more exhaustive explanation and common workarounds.

Other packages will have arch-specific build errors. Even if you don't have hardware of that architecture, the build logs may tell you enough to debug the problem. Don't be afraid to try! You may not understand all the error messages, but looking at the source (many of these errors will give you a line number to work with) and asking your friendly neighborhood web search engine may give you enough to try a few guesses. Who knows, you may just learn a new programming language in the process. ;)

Still other packages might have once built, but no longer build anywhere. There's no one-size-fits-all recipe for fixing these, but these also often require updated build dependencies.

Install/uninstall failures, or other policy-violating weirdness

Maintainer script bugs and policy compliance issues are some of the most straightforward bugs to work on.

Security bugs

Don't be afraid of these bugs just because they say "security". Even if you don't know the language used to implement the package, it may be possible to help with security bugs by tracking down information. Many security bugs start out with little more information than a CVE ID (CAN-*) when they reach the BTS — with no real explanation of what, or where, the bug is. Some web research can help track this information down for the maintainer: sometimes in the form of text explanations of the security bug, sometimes in the form of actual patches published by other vendors (or by upstream). The truly audacious may be willing to research such security bugs in upstream's revision control systems as well (works best if you understand the programming language well enough to recognize a possible security fix when you see one).

The Developer's reference also includes a section about helping with security bugs.

Testing-only bugs

Bugs that are tagged "testing" might not seem like good candidates to work on, but when a bug is tagged testing, it means that the package in unstable is not broken — but that either that package hasn't reached testing (in which case these are often bugs that were closed, then reopened with the "testing" tag), or some other package needed to fix the bug hasn't reached testing. Sometimes the only thing needed is time, but sometimes there are other problems keeping the fixed package from hitting testing — finding these problems, and pointing them out to the maintainer (and submitting patches!) can be a big help.

Note that many bugs that are tagged "testing" will also be tagged "patch", so you will want to un-exclude patched bugs from the web view when looking for these.

To find out why a package isn't in testing yet, find the name of the source package using

apt-cache showsrc <package> | grep ^Package

Then use the grep-excuses tool from the devscripts package:

grep-excuses <sourcepackagename>

You can also find this information in many of the web pages about package status, including http://bjorn.haxx.se/debian/testing.pl and http://packages.qa.debian.org/.

Sometimes, the package will be held out by missing binary builds of the package; sometimes it will be held out by other release-critical bugs in the package that are specific to unstable; and sometimes it will be held out by issues in packages that it depends on (i.e., usually one of the other two kinds of issues above). Linking one of these problems to the unfixed-in-testing bug can be a help to the maintainer, even if it means you have to file a new release-critical bug report in the process.

Here is example output of grep-excuses, showing a package that is being held out of testing:

$ grep-excuses asterisk
asterisk (1:1.0.2-2 to 1:1.0.3-2)
    Maintainer: Debian VoIP Team 
    2 days old (needed 2 days)
    out of date on ia64: asterisk, asterisk-gtk-console, asterisk-h323 (from 1:1.0.2-3.1)
    out of date on mips: asterisk, asterisk-gtk-console, asterisk-h323 (from 1:1.0.2-3.1)
    out of date on mipsel: asterisk, asterisk-gtk-console, asterisk-h323 (from 1:1.0.2-3)
    Not considered

A detailed explanation of this output can be found at http://www.debian.org/devel/testing.

Information about the status of out-of-date binaries can also be found at http://people.debian.org/~igloo/status.php.

Following up on a bug report

Once you think you've squashed a bug — which usually means you've reproduced it in a controlled environment, made a change to the package, and confirmed that the bug went away — you're ready to submit a patch to the BTS. For a guide to creating patches for Debian packages, please see The Debian-Women Wiki.

Once you have your patch, send it to the bug report by emailing it as an attachment to ######@bugs.debian.org, where ###### is the number of your bug. You should also send a mail to control@bugs.debian.org with the text "tags ###### +patch" in the body, so that other people can see that the bug is patched without reading through the whole bug log. You may find the bts command from the devscripts package useful for sending commands to the BTS.

Some bugs are harder than others, of course, and you may not succeed in fixing the bug on your own. That's ok; if you want to stop, or you get stuck, you're not doing anything the rest of us haven't done before. But if you've been working with the bug for a while, please let others learn from your efforts by sending a follow-up message to the BTS. For instance, if it took you a while to figure out how to reproduce it because the info in the original bug wasn't clear, sending more info to the bug report about how to reproduce the bug is helpful. Information about cases where you could not reproduce the bug are also helpful, as well as information about things you tried that you thought should have fixed the bug but didn't. Sometimes it takes four or five people trying to reproduce the bug before someone comes up with the right combination, so don't think your contributions are insignificant. Every bit closer we get to fixing a release-critical bug is a little bit closer we get to releasing.


This document was written by Steve Langasek <vorlon@debian.org>, and all bugs are his responsibility. Thanks are owed to everyone who attended the on-line seminar that served as the prototype for this webpage, for their questions and participation; and also in particular to Helen Faulkner, Matt Zimmerman, Jeroen van Wolffelaar, Alexander Schmehl, Dafydd Harries, and Frank Lichtenheld for thinking of things I didn't and helping to make this document "a little bit closer to release-worthy" than it would be otherwise.