For quite a while, Lintian has been able to create performance logs (--perf-debug --perf-output perf.log) that help diagnose where lintian spends most of its runtime.  I decided to make lintian output these logs on lintian.debian.org to help us spot performance issues, though I have not been very good at analysing them regularly.

At the beginning of the month, I finally got around to look a bit into one of them.  My findings on IRC triggered Mattia Rizzolo to create this following graph.  It shows the accumulated runtime of each check/collection measured in seconds.  Find these findings, we set out to solve some of the issues.  This lead to the following changes in 2.5.33 (in no particular order):

• Increased buffer size in check/cruft.pm to reduce overhead [bc8b3e5] (S)
• Reduced overhead in strings(1) extraction [b058fef] (P)
• Reduced overhead in spell-checking [b824170] (S)
  • Also improves the performance of spellintian!
• Removed a high overhead check that did not work [2c7b922] (P)

Legend: S: run single threaded (1:1 performance improvement).  P: run in parallel.

Overall, I doubt the changes will give a revolutionary change in speed, but it should improve the 3rd, 4th and 5th slowest parts in Lintian.

Beyond runtime performance, we got a few memory optimisations in the pipeline for Lintian 2.5.34:

• Remove member from "non-dir" nodes in the Lintian path graph (2%) [6365635]
• Remove two fields at the price of computing them as needed (~5%) [a696197 + 8dacc8e]
• Merge 4 fields into 1 (~8%) [5d49cd2 + fb074e4]
• Share some memory between package-based caches (18%) [ffc7174]

Combined these 6 commits reduce memory consumption in caches by ~33% compared to 2.5.33, when lintian processes itself.  In absolute numbers, we are talking about a drop from 12.53MB to 8.48MB.  The mileages can certainly vary depending on the package (mscgen "only" saw an ~25% improvement).  Nevertheless, I was happy to list #715035 as being closed in 2.5.34. :)

Debian now have over 22 000 source packages and 45 500 binary packages.  To counter that, the FTP masters and I have created a dak tool to automatically remove packages from unstable!  This is also much more efficient than only removing them from testing! :)

The primary goal of the auto-decrufter is to remove a regular manual work flow from the FTP masters.  Namely, the removal of the common cases of cruft, such as "Not Built from Source" (NBS) and "Newer Version In Unstable" (NVIU).  With the auto-decrufter in place, such cruft will be automatically removed when there are no reverse dependencies left on any architecture and nothing Build-Depends on it any more.

Despite the implication in the "opening" of this post, this will in fact not substantially reduce the numbers of packages in unstable. :) Nevertheless, it is still very useful for the FTP masters, the release team and packaging Debian contributors.

The reason why the release team benefits greatly from this tool, is that almost every transition generates one piece of "NBS"-cruft.  Said piece of cruft currently must be  removed from unstable before the transition can progress into its final phase.  Until recently that removal has been 100% manual and done by the FTP masters.

The restrictions on auto-decrufter means that we will still need manual decrufts. Notably, the release team will often complete transitions even when some reverse dependencies remain on non-release architectures.  Nevertheless, it is definitely an improvement.

Omelettes and eggs: As an old saying goes "You cannot make an omelette without breaking eggs".  Less so when the only "test suite" is production.  So here are some of the "broken eggs" caused by implementation of the auto-decrufter:

• About 30 minutes of "dak rm" (without --no-action) would unconditionally crash.
• A broken dinstall when "dak auto-decruft" was run without "--dry-run" for the first time.
• A boolean condition inversion causing removals to remove the "override" for partial removals (and retain it for "full" removals).
  • Side-effect, this broke Britney a couple of times because dak now produced some "unexpected" Packages files for unstable.
• Not to mention the "single digit bug closure" bug.

Of the 3, the boolean inversion was no doubt the worst.  By the time we had it fixed, at least 50 (unique) binary packages had lost their "override".  Fortunately, it was possible to locate these issues using a database query and they have now been fixed.

Before I write any more non-trivial patches for dak, I will probably invest some time setting up a basic test framework for dak first.

During the "intermission" on the day of the Jessie release, Julien, Ivo, AJ and I spent some time improving Britney2.  Due to said intermission, we can proudly say that from the very first run for Stretch, Britney2 was:

• running on Python3k.  Kudos to Julien and Ivo.
• doing consistency checks of identical packages present in two more suites.  Kudos to AJ.
  • Said checking is mostly useful for catching silly mistakes in our test suite.  A very welcome change, as data inconsistency plus hash randomisation (default in Python3k) caused some of the tests to fail sporadically.
  • Also, many thanks to AJ for providing a patch for #709460.  Sadly, I do not remember if we managed to merge that prior to the first Stretch run.
• outputting some statistics about the "package graph" and some performance counters from its installability tester.

The performance counters are mostly interesting for me, when I mess with the installability tester.  A couple of backtrack related numbers from the Britney run early today:

• 77 078 times, Britney would create a "full restore point" and recurse.
  • In 10 of those cases, she would reject the guess, backtrack back to the restore point and move on to the next guess.
  • In the remaining 77 068 times, she would accept the candidate (and thereby solve the query).
    • NB: This number is not directly visible and has to be computed manually.  It is possible for Britney to do multiple "accept"-recursions for the same query.
• 52 times, she would have exhausted all but one option.  In this case, she simply goes "all-in" and skips the restore point.
• 54 618 times, she would accept the guess using a partial restore point without needing to recurse.
• An (sadly) uncounted number of times, she would reject the guess using a partial restore point without needing to recurse.

Furthermore, about 82% of the  ~577 000 times Britney called "is_installable" in this run, the installability tester answered with a cached result.  I guess it was a trivial run. :)

It was quite an experience to partake in the Jessie release - and also a rather long "Saturday".  This post is mostly a time line of how I spent my release day with doing the actual release.  I have glossed over some details - the post is long enough without these. :)

We started out at 8 (UTC) with a final "dinstall" run, which took nearly 2 hours.  It was going to take longer, but we decided to skip the synchronisation to "coccia.debian.org" (the server hosting the DD-accessible mirror of release.debian.org).

The release itself started with the FTP masters renaming the aliases of Squeeze, Wheezy and Jessie to oldoldstable, oldstable and stable respectively.   While they worked, the release team reviewed and double checked their work.  After an hour (~11), the FTP masters reported that the stable releases were ready for the final review and the SRMs signed the relevant "Release" files.

Then the FTP masters pushed the stable releases to our CD build server, where Steve McIntyre started building the installation images.  While Steve started with the CDs, the FTP masters and the release team continued with creating a suite for Stretch.  On the FTP/release side, we finished shortly before 12:30.  At this point, our last ETA from Steve suggested that the installation media would take another 11 and a half hours to complete.  We could have opened for mirror synchronisation then, but we decided to wait for the installation media.

At 12:30, there was a long "intermission" for the release team in the release process.  That was an excellent time to improve some of our tools, but that is for another post. :)

We slowly started to resume around 22:20, where we tried to figure out when to open for the mirror synchronisation to time it with the installation media.  We agreed to start the mirror sync at 23:00 despite the installation media not being completely done then.  They followed half an hour later, when Steve reported that the last CD was complete.

At this point, "all" that was left was to update the website and send out the press announcement.  Sadly, we were hit by some (minor) issues then.  First, I had underestimated the work involved in updating the website. Secondly, we had no one online at the time to trigger an "out of band" rebuild of the website.  Steve and I spent an hour and a half solving website issues (like arm64 and ppc64el not being listed as a part of the release).  Unsurprisingly, I decided to expand our the "release checklist" to be slightly more verbose on this particular topic.

My "Saturday" had passed its 16th hour, when I thought we had fixed all the website issues (of course, I would be wrong) and we would now just be waiting for the an automatic rebuild.  I was tempted to just punt it and go to bed, when Paul Wise rejoined us at about 01:25.  He quickly got up to speed and offered to take care of the rest.  An offer I thankfully accepted and I checked out 15 minutes later at 01:40 UTC.

That more or less covers the Jessie release day from my PoV.  After a bit of reflection inside the release team, we have found several points where we can improve the process.  This part certainly deserves its own post as well, which will also give us some time to flesh out some of the ideas a bit more. :)

Indeed, we settled on a release date for Jessie - and pretty quick too.  I sent out a poll on the 28th of March and yesterday, it was clear that the 25th of April was our release date. :)

With that said, we still have some items left that needs to be done.

• Finishing the release notes.  This is mostly pending myself and a few others.
• Translation of the release-notes.  I sent out a heads up earlier today about what sections I believe to be done.
• The d-i team got another release planned as well.
• All the RC bugs you can manage to fix before the 18th of April. :)

Earlier today, I posted a mail to debian-devel about how approximately 25 RC bugs affecting Jessie have been unblocked.  As mentioned, I planned to age some of them.  The expected result is that about 18 of them will migrate tonight and the remaining 7 of them will migrate tomorrow night.

After that, there are no more RC bugs waiting for the RT to unblock them!  The only remaining item on the list is cgmanager, for which we are requesting a t-p-u (maintainer already contacted about it).  If you want a release sooner, please have a look at the list of remaining RC bugs or/and start testing upgrades.

In other news, the glibc regression got fixed.  The new version of glibc has already been approved by us.  It is now waiting for the debian-installer team to testing it and approve it.

I had the mixed pleasure of doing a partial rewrite of lintian's reporting framework.  It started as a problem with generating the graphs, which turned out to be "not enough memory". On the plus side, I am actually quite pleased with the end result.  I managed to "scope-creep" myself quite a bit and I ended up getting rid of a lot of old issues.

The major changes in summary:

• A lot of logic was moved out of harness, meaning it is now closer to becoming a simple "dumb" task scheduler.  With the logic being moved out in separate processes, harness now hogs vastly less memory that I cannot convince perl to release to the OS.  On lilburn.debian.org "vastly less" is on the order of reducing "700ish MB" to "32 MB".
• All important metadata was moved into the "harness state-cache", which is a simple YAML file. This means that "Lintian laboratory" is no longer a data store. This change causes a lot of very positive side effects.
• With all metadata now stored in a single file, we can now do atomic updates of the data store. That said, this change itself does not enable us to run multiple lintian's in parallel.
• As the lintian laboratory is no longer a data store, we can now do our processing in "throw away laboratories" like the regular lintian user does.  As the permanent laboratory is the primary source of failure, this removes an entire class of possible problems.

There are also some nice minor "features":

• Packages can now be "up to date" in the generated reports.  Previously, they would always be listed as "out of date" even if they were up to date.  This is the only end user/website-visitor visible change in all of this (besides the graphs are now working again \o/).
• The size of the harness work list is no longer based on the number of changes to the archive.
• The size of the harness work list can now be changed with a command line option and is no longer hard coded to 1024.  However, the "time limit" remains hard coded for now.
• The "full run" (and "clean run") now simply marks everything "out-of-date" and processes its (new) backlog over the next (many) harness runs.  Accordingly, a full-run no longer causes lintian to run 5-7 days on lilburn.d.o before getting an update to the website.  Instead we now get incremental updates.
• The "harness.log" now features status updates from lintian as they happen with "processed X successfully" or "error processing Y" plus a little wall time benchmark.  With this little feature I filed no less than 3 bugs against lintian - 2 of which are fixed in git.  The last remains unfixed but can only be triggered in Debian stable.
• It is now possible with throw-away labs to terminate the lintian part of a reporting run early with minimal lost processing.  Since the lintian-harness is regular fed status updates from lintian, we can now mark successfully completed entries as done even if lintian does not complete its work list.  Caveat: There may be minor inaccuracies in the generated report for the particular package lintian was processing when it was interrupted.  This will fix itself when the package is reprocessed again.
• It is now vastly easier to collect new meta data to be used in the reports.  Previously, they had to be included in the laboratory and extracted from there.  Now, we just have to fit it into a YAML file.  In fact, I have been considering to add the "wall time" and make a "top X slowest" page.
• It is now possible to generate the html pages with only a "state-cache" and the "lintian.log" file.  Previously, it also required a populated lintian laboratory.

As you can probably tell, I am quite pleased with the end result.  The reporting framework lacks behind in development, since it just "sits there and takes care of itself".  Also with the complete lack of testing, it also suffers from the "if it is not broken, then do not fix it" paradigm (because we will not notice if we broke until it is too late).

Of course, I managed to break the setup a couple of times in the process.  However, a bonus feature of the reporting setup is that if you break it, it simply leaves an outdated report on the website.

Anyway, enjoy. :)

Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet.  Even if we were, it would be unlikely that said date would be in January.  Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release.  If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages.  This was a major issue for Wheezy.  So far I have only done some minor prodding here.  However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover.  As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left.  E.g. #759530.  There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression.  Most of these have been resolved now.  There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

In the last 24 hours, the number of RC bugs currently affecting Jessie was reduced to just under half of the same number for Wheezy.

There are still a lot of bugs to be fixed, but please keep up the good work. :)

"Page 1 of 5"

To be honest, I do not know how many mails it shows "per page" (though I assume it is a fixed number).  So for comparison, I found the month on debian-devel@l.d.o with the  highest volume in the past two years:  May 2013 with "Page 1 of 4".

I hope you will please forgive us, if we are a bit terse in our replies or slow to reply.  We simply got a lot to deal with.  :)

At the start of this week, I wrote that we had 58 open unblock requests open (of which 25 were tagged moreinfo).  Thanks to an extra effort from the Release Team, we now down to 25 open unblocks - of which 18 are tagged moreinfo.

We have now resolved 442 unblock requests (out of a total of 467).  The rate has also declined to an average of ~18 new unblock requests a day (over 26 days) and our closing rated increased to ~17.

With all of this awesomeness, some of us are now more than ready to have a well-deserved weekend to recharge our batteries.  Meanwhile, feel free to keep the RC bug fixes flowing into unstable.

The release team receives an extreme amount of unblock requests right now.  For the past 22 days[1], we have been receiving no less than 408 unblock/ageing requests.  That is an average of ~18.5/day.  In the same period, the release team have closed 350 unblocks requests, averaging 15.9/day.

This number does not account for number of unblocks, we add without a request, when we happen to spot when we look at the list of RC bugs[2]. Nor does it account for unblock requests currently tagged "moreinfo", of which there are currently 25.

All in all, it has been 3 intensive weeks for the release team.  I am truly proud of my fellow team members for keeping up with this for so long!  Also a thanks to the non-RT members, who help us by triaging and reviewing the unblock requests!  It is much appreciated. :)

Random bonus info:

• d (our diffing tool) finally got colordiff support during the Release Sprint last week.  Prior to that, we got black'n'white diffs!
  • ssh coccia.debian.org -t /srv/release.debian.org/tools/scripts/d
  • Though coccia.debian.org do not have colordiff installed right now.  I have filed a request to have it installed.
• The release team have about 132 (active) unblock hints deployed right now in our hint files.

[1] We started receiving some in the 10 days before the freeze as people realised that their uploads would need an unblock to make it into Jessie.

[2] Related topics: "what is adsb?" (the answer being: Our top hinter for Wheezy)

The release team are doing a sprint right now up the mini-DebConf in Cambridge, kindly hosted by ARM.

We have a fairly large agenda of around 10 items, ranging from "simple" things like determine the name for the next release to reviewing the list of RC bugs affecting Jessie.

Personally, I am very pleased with our progress.  We managed to finish 8 of items on the first day.  Furthermore, we spent several hours on the RC bug list and keeping up with the unblock requests.

There is also live status of the release team, courtesy of Jonathan Wiltshire.  Mind you it is manually updated.

We will announce the results of our sprint Sunday morning in our Release update talk.  The announcement will also be posted to debian-devel-announce like our freeze announcement.

Update: fix link to the live status

The upcoming version of Lintian will feature a new set of API that attempts to promote safer code. It is hardly a "ground-breaking discovery", just a much needed feature.

The primary reason for this API is that writing safe and correct code is simply too complicated that people get it wrong (including yours truly on occasion).   The second reason is that I feel it is a waste having to repeat myself when reviewing patches for Lintian.

Fortunately, the kind of issues this kind of mistake creates are usually minor information leaks, often with no chance of exploiting it remotely without the owner reviewing the output first[0].

Part of the complexity of writing correct code originates from the fact that Lintian must assume Debian packages to be hostile until otherwise proven [1]. Consider a simplified case where we want to read a file (e.g. the copyright file):

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  # BAD: This is an example of doing it wrong
  open(my $fd, '<', $info->unpacked($filename));
  ...;
  close($fd);
  return;
}

This has two trivial vulnerabilities [2].

1. Any part of the path (usr,usr/share, ...) can be a symlink to "somewhere else" like /

   • Problem: Access to potentially any file on the system with the credentials of the user running Lintian.  But even then, Lintian generally never write to those files and the user has to (usually manually) disclose the report before any information leak can be completed.

2. The target path can point to a non-file.

   • Problem: Minor inconvenience by DoS of Lintian.  Examples include a named pipe, where Lintian will get stuck until a signal kills it.

Of course, we can do this right [3]:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
use Lintian::Util qw(is_ancestor_of);
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $root = $info->unpacked
  my $path = $info->unpacked($filename);
  if ( -f $path and is_ancestor_of($root, $path)) {
    open(my $fd, '<', $path);
    ...;
    close($fd);
  }
  return;
}

Where "is_ancestor_of" is the only available utility to assist you currently.  It hides away some 10-12 lines of code to resolve the two paths and correctly asserting that $path is (an ancestor of) $root.  Prior to Lintian 2.5.12, you would have to do that ancestor check by hand in each and every check [4].

In the new version, the correct code would look something like this:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $path = $info->index_resolved_path($filename);
  if ($path and $path->is_open_ok) {
    my $fd = $path->open;
    ...;
    close($fd);
  }
  return;
}

Now, you may wonder how that promotes safer code.  At first glance, the checking code is not a lot simpler than the previous "correct" example.  However, the new code has the advantage of being safer even if you forget the checks.  The reasons are:

1. The return value is entirely based on the "file index" of the package (think: tar vtf data.tar.gz).  At no point does it use the file system to resolve the path.  Whether your malicious package trigger an undef warning based on the return value of index_resolved_path leaks nothing about the host machine.
   1. However, it does take safe symlinks into account and resolves them for you.  If you ask for 'foo/bar' and 'foo' is a symlink to 'baz' and 'baz/bar' exists in the package, you will get 'baz/bar'.  If 'baz/bar' happens to be a symlink, then it is resolved as well.
   2. Bonus: You are much more likely to trigger the undef warning during regular testing, since it also happens if the file is simply missing.
2. If you attempt to call "$path->open" without calling "$path->is_open_ok" first, Lintian can now validate the call for you and stop it on unsafe actions.

It also has the advantage of centralising the code for asserting safe access, so bugs in it only needs to be fixed in one place.  Of course, it is still possible to write unsafe code.  But at least, the new API is safer by default and (hopefully) more convenient to use.

--

As mentioned by Raphaël Hertzog, I have been spending some time on improving Britney2.  Just the other day I submitted a second branch for review that I expect to merge early next week.  I also got another set patches coming up soon.  Currently, none of them are really user visible, so unless you are hosting your own version of Britney, these patches are probably not all that interesting to you.

The highlights:

1. Reduce the need for backtracking by finding semantically equivalent packages.
2. Avoid needing to set up a backtrack point in some cases.
   • This has the side-effect of eliminating some O(e^n) runtime cases.
3. Optimise "installability" testing of packages affected by a hinted migration.
   • This has the side-effect of avoiding some O(e^n) runtime cases when the "post-hint" state does not trigger said behaviour.
   • There is a follow-up patch for this one coming in the third series to fix a possible bug for a corner-case (causing a valid hint to be incorrectly rejected when it removed an "uninstallable" package).
4. Reduce the number of affected packages to test when migrating items by using knowledge about semantically equivalent packages.
   • In some cases, Britney can now do "free" migrations when all binaries being updated replace semantically equivalent packages.
5. (Merge pending) Avoid many redundant calls to "sort_actions()", which exhibits at least O(n^2) runtime in some cases.
   • For the dataset Raphaël submitted, this patch shaves off over 30 minutes runtime.  In the particular case, each call to sort_actions takes 3+ minutes and it was called at least 10 times, where it was not needed.
   • That said, sort_actions have a vastly lower runtime in the runs for Debian (and presumably also Ubuntu, since no one complained from their side so far).

The results so far:

After the first patch series was merged, the Kali dataset (from Raphaël) could be processed in "only" ~2 hours. With the second patch series merged, the dataset will drop by another 30-50 minutes (most of which are thanks to the change mentioned in highlight #5).

The third patch series currently do not have any mention-worthy performance related changes.  It will probably be limited to bug fixes and some refactoring.

Reflections:

The 3 first highlights only affects the "new" installability tester meaning that the Britney2 instances at Ubuntu and Tanglu should be mostly unaffected by the O(n^2) runtime.  Although those cases will probably just fail with several "AIEEE"s. :) The 5th highlight should equally interesting to all Britney2 instances though.

For me, the most interesting part is that we have never observed the O(n^2) behaviour in a daily "sid -> testing" run.  The dataset from Raphaël was basically a "stable -> testing/sid" run, which is a case I do not think we have ever done before.  Despite our current updates, there is still room for improvements on that particular use case.

In particular, I was a bit disheartened at how poorly our auto hinter(s) performed on this dataset.  Combined they only assisted with the migration of something like 28 "items".  For comparison, the "main run" migrated ~7100 "items" and 9220 items were unable to migrate. Furthermore, the "Original" auto hinter spend the better part of 15 minutes computing hints  - at least it results in 10 "items" migrating.

Links to the patches:

• First series (already merged)
• Second series (pending, to be merged Monday, the 11th of August)
• Third series (under development. Warning: the branch will be rebased if/as needed)

The next version of Lintian, 2.5.22, is long overdue - mostly because 2.5.21 FTBFS in sid. Besides fixing test failures, 2.5.22 also fixes:

• False-negative for most/all JAR related tags caused by file(1) changing its output for these files.
• A performance regression in checks/cruft.pm. For large source packages, this could severely affect performance of Lintian.  This is #738342.
• A regression where lintian would not parse options appearing after file arguments.
• Some false-positives in the "GFDL with invariants" check.

Besides bug fixes, we also added a couple of new features/nice changes:

• The --color option now defaults to "auto" instead of "never".
• The homepage now links to the generated API-docs and thus, also the developer tutorial.
  • Patches extending/improving either/both are more than welcome.
• The test-suite runner now caches test artifacts meaning that subsequent test runs become vastly faster.
• The test-suite runner can now run tests under "Devel::Cover", so we can now collect statistics about how much of the code is covered by the test suite.
• The "html_reports" tool (used e.g. on lintian.d.o) has been optimised. In particular, it only calls gnuplot twice, instead of once per data file (of which there was basically one per tag).
• The html-pages generated by the "html_reports" tool now uses "content"-based names for resources (e.g. images and the .css file). This allows them to be cached vastly longer, since any content changes will rename the resource and force browser to refetch them.
  • All resources handled this way are now stored in "resources/".  As a side-effect, this means that images files are moved from "images/" to "resources/".
  • A notable exception to this is the auto-generated (.svg) graphs.

We just need to do some final tweaks and get the tag descriptions reviewed before we are ready to release the next version.

Today, we held an one-hour IRC-meeting to debate the status of the current architectures in sid.  During that one hour, we brought up all 13 architectures.  We made a decision on 9 of the architectures.  The remaining 4 will be revisited within 2 months.  As for the actual results, we will announce these on debian-devel-announce with our next bits mail.  I suspect the announcement to happen within a couple of days.

I am quite pleased that we managed to keep the meeting to one hour.  Especially considering that it was planned less than a week ago.  One of the things, that I believe helped us, was that we had summarised the status for each architecture prior to the meeting.  In my view, the summary allowed us to finish at least half of the architectures within their allocated 5 minutes.  In fact, we had 20 minutes for the last 2 architectures - partly because amd64 + i386 combined took about a minute.

I am very pleased to say that RC bug counter for Jessie finally dropped below 500 RC bugs.  For comparison, we broke the 700 RC bug curve in the start of November, so that is about 200 RC bugs in about 7-8 weeks (or about 25-28 RC bugs per week).

Today, we have about 162 RC bugs on the auto-removal list.  Though, I suspect many of the affected packages have reverse dependencies, so their removal from testing may be up 30 days away.  Nevertheless, by this time in January, we could be looking at no more than 350 RC bugs left for Jessie.

Yesterday, I have managed to finish up an old pet peeve of mine.  I wrote a series of patches to have harness reprocess all packages, which were last checked by an older version of Lintian than the current.  It is not completed to the level I would like, but it is good enough for now.

The implementation works by processing up to 1024 groups after each daily run.  It also have a timer to skip this, if the run has taken more than 4 hours.  Packages to be reprocessed are sorted by the version they were last processed - should a new version of Lintian be upload before all packages have been reprocessed, the oldest results are refreshed first.  :)

A rough estimate suggests than in about 3-4 weeks, the entire archive will have been reprocessed with Lintian 2.5.20.  It is by no means faster than a full run (which is about a week), but it is automatic and removes the "human" factor in keeping Lintian results up to date.

I have decided to step down as main maintainer of Lintian and pass the baton to Bastien Roucariès.  This is actually "fairly" old news, since I announced this almost a month ago.  However, I was not very vocal about it until now (so do not be surprised if you had not heard of this before).

In the past month, I never once regretted having stepped down.  If anything, I should probably have done it a bit earlier.  Beyond the lack of motivation, I also realised that I had become an "all talk and no do"-maintainer.  The kind that I have been advocating against publicly.  This may sound weird to a lot of people, who knows me as "the Lintian guy" or "Mr. Lintian" (or whatever Lintian-related nickname people gave me).  But the fact is that Bastien has done more for Lintian in the past month than I have in the past two.

Despite stepping down as main developer, I have not left Lintian completely.  I am still around for helping/explaining, uploading and keeping lintian.debian.org running.

It looks like we have dropped (or will drop) below 700 RC bugs affecting Jessie today[1].  :)

On a related note, I got my eye on libfso-glib + fso-gsmd, samba + -samba4, mono and haskell.  By my count, each of the four groups will (once they migrate) bring at least 4 RC bug fixes with them to testing.

Besides that, we still have ~97 RC bugs in the autoremoval queue[2], so we could be looking at 600 RC bugs in a week or two.

Please keep up the steady flow of RC bug fixes; but also, please make sure your packages will actually migrate.  If you need help interpreting the excuses page, do not hesistate to ask - I will not bite.

If you want to help, please consider having a look at the "cleaner" or the "Release Team"-view on the UDD bug tracker.  Sadly not everything there contains actionable problems[3].

[1] http://bugs.debian.org/release-critical/ says 700 while the IRC bot in #debian-devel-changes says we are at 692.

[2] http://udd.debian.org/bugs.cgi?release=jessie_and_sid&merged=ign&autoremovals=only&fnewerval=7&flastmodval=7&rc=1&sortby=id&sorto=asc

dd-list-like: http://udd.debian.org/cgi-bin/autoremovals.cgi

[3] E.g. the kfreebsd-8 bugs appear because kfreebsd-8 is listed in "Built-Using" for d-i, but the package has already been removed from testing.  So there is nothing to do with that (except possibly waiting for a new version of d-i).

Early this month, the first batch of packages were automatically removed from testing due to RC bugs.  Personally, I find that the results stand out pretty well on the RC bug graph[1].  There has been a very rapid decline in the number of RC bugs affecting testing - a week or two ago, we had over 1000 RC bugs affecting testing and now it has dropped to something like 760.

There has also been quite a few RC bug fixes in this time period.  I spent a bit of time finding and aging some of the RC bug fixes in sid, so they would migrate to testing faster[2].  On the first batch, we saw something like 40 RC bugs disappear and we still have more RC bugs fixes in the sid->testing pipeline.

While looking for packages I could age, I also noticed that we have quite a few RC bug fixes blocked by "out of date" binaries (which is usually either build issues or uncoordinated transitions).  I have been filing a couple of RC bugs for these, but it does not scale do this kind of work manually.  If you have uploaded a package recently, please take the 5 minutes to check that it actually built and it has no "out of date" binaries issues in its excuses.

[1] http://bugs.debian.org/release-critical/

[2] The aging was equivalent to an urgency=medium, so reduced the requirement from 10 days to 5 days.

Today, I released Lintian 2.5.19.  It is mostly a bug-fix release, but it also features a new tag.  The new tag, homepage-in-binary-package, is emitted for packages where the source package has no Homepage-field, but one of its binaries do.  The problem is that some tools (e.g. the PTS) only processes the data from the .dsc files (or Sources indices from the mirrors) and would in these cases miss the Homepage of the page.

We now also carry a work around for a bug in tool, t1disasm, where it ends up in an "infinite recursion" (i.e. until the C call stack overflows). That issue is filed as #724571.  The work around is to prevent Lintian from exiting with a failure (which causes packages to be auto-rejected when uploaded to Debian archive).  These problems are currently "reduced" to a warning on stderr.

In other (Lintian related news), the full run was completed yesterday after only 4 days (rather than the 8, I expected).  While that is a pleasant surprise, I cannot find any changes in Lintian since the last run that would explain this difference.  My current conjecture is that our last "pseudo" full run (when we added amd64) included more processing since the amd64 packages would not have any pre-collected information.

With the "follow up" run to catch packages uploaded after the full run, Lintian spent a little more than 3 and half days (~86.5 hours) processing about 20700 package groups.  This gives an average of about 240 groups an hour (or ~4 a minute).  Though, remember these numbers include some data having been pre-collected from previous run on the majority of packages.

Today I had to pleasure of releasing Lintian 2.5.18.1.  It is a minor bug fix release on top of Lintian  2.5.18 (which I also released today...).  When I was about to deploy the new version of Lintian on lintian.debian.org, I noticed a couple of errors in the log file that smelled of bugs.  Indeed, when I investigated them I found two bugs, which were both introduced in Lintian 2.5.17.

Due to another bug in 2.5.17 and the some of the false-positives fixes, we are now doing a "full run" on lintian.debian.org.  After having run for 7 hours, Lintian passed "cipux".  My rough estimate suggests that it will take 8 days at the current rate.  For comparision, when we added amd64, Lintian spent 5 days processing all the new binaries (plus their source and the old binaries built from those source packages).

Of course estimate is rather rough.  In particular, I am not sure that the "heavy duty groups" (e.g. linux and libreoffice) are evenly spread out among the letters of the alphabeth.  Anyhow, there will be no updates on lintian.debian.org for the next week or so.