I have just uploaded Lintian 2.5.18 to unstable. While fixing 22 bugs, it only features 5 new tags.
- debian-changelog-has-wrong-weekday
- debian-rules-missing-good-practice-target-dfsg
- empty-udeb-package
- file-name-in-PATH-is-not-ASCII
- misplaced-extra-member-in-deb
The release also include fixes to some false-positives, such as "python:any" dependencies triggering python-script-but-no-python-dep, a rewritten README file. We also included a patch to make Lintian accept the "Original-Maintainer" field by default for non-Debian vendors (even if they do not have a profile and Lintian ends up loading the "debian/main" profile).
We also added support for running Lintian directly from a git checkout or source tree without setting LINTIAN_ROOT (or passing --root). Since that was the primary use-case for --root that option has now been deprecated. I also had lintian and lintian-info require the "--include-dir" and "--[no-]user-dir" options as the first if given at all.
I would like to thank Bastien Roucariès, Gaudenz Steinlin, Gunnar Wolf, Jérémy Bobbio and Lucas Nussbaum for contributing to Lintian and the many who submitted reports or suggestions for improvements. I would also like to thank Brian "hugmeir" Fraser, who assisted me in identifying and working around a bug in Perl's "glob" function when run under threads (filed upstream as RT#119897).
A couple of days ago, the upstream maintainer of autodie, Paul Jamieson Fenwick (PJF), released autodie 2.21. It includes all of the performance improvements I had proposed in "Optimizing autodie" (plus a few extra ones authored after I wrote the post) and a small fix to one of the tests (due to changes in Carp).
As mentioned in my earlier post, these improvements gives about a factor 4 load time improvement in most cases. For the sake of it, I have decided to use Lintian as an example of how good this improvement is. First off, Lintian loads uses autodie in about 75 unique files in the current git master branch.
$ grep -lr 'use autodie' lib/ checks/ collection/ frontend/lintian | wc -l 75
As a simple benchmark, lets consider the following Perl one-liner:
$ /usr/bin/time -f "%es %Us %Ss" perl -Mautodie \
-e 'print $autodie::VERSION, " => ";' \
-e 'eval join("\n", map {"package X$_; use autodie;"} (1..74));'
A quick-n-dirty comparison gives:
2.20 => 0.91s 0.90s 0.01s 2.21 => 0.15s 0.15s 0.00s # NB: the times are "wall, user and sys" in that order, in case you # are too lazy to read time(1). :)
So, 2.21 is now closer to a factor 6 rather than the original factor 4. I have to admit that the timings seem to be "very machine dependent" as my laptop shows a factor 7 improvement (from ~1.39 to ~0.22s).
With at least 370 tests cases running in the Lintian testsuite, I can look forward to a reduction in "user time" on about 4 minutes.
In my previous post, I explained about the "age" excuse in Britney. In this post, I will cover the slightly more tricky "out of date" excuse. A simple example of excuse is:
out of date on mipsel: blinken (from 4:4.8.4-1)
In this simple case, Britney simply states that the binary "blinken" on mipsel has not yet been rebuilt.
Generally, the presence of an excuse like this is sufficient to stop testing migration. The exception to this rule is if the architecture has been marked as "not keeping up". At the moment we do not have any such architectures.
The tricky part of an "out of date"-excuse is that Britney simply identifies a symptom and not a cause. In the excuse above, it is not possible to determine if the build failed on that architecture or simply has not finished yet.
But the part that probably confuses the most, is when Britney lists binaries no longer built from that source:
out of date on i386: libfsoframework0 (from 0.11.0-1.1)
This case actually happens quite often and is (usually) a sign of a transition has been started. The part that confuses many is that the "build-state" of their package will be "Installed" (as in "It was built successfully and has been uploaded to the archive"). From there they tend to conclude "Britney is broken", where "The excuse information is a bit unhelpful" might be more accurate.
Caveats:
- If the source package has not yet been built on that architecture, these old binaries are interleaved into the "regular" list of "out of date"-binaries.
- The list of excuses are updated twice a day (and there is also a delay between its update and the PTS picking it up). Thus, if the "build-state" of your package for that architecture has been "Installed" for less than 24 hours, you may just want double check before jumping to conclusions.
The problem is that the binary still has reverse dependencies in unstable, so it cannot migrate to testing. When the last reverse dependency has been updated, the FTP masters will usually semi-automatically "decruft" your package and the excuse will disappear.
On the other hand, if the binary still has reverse dependencies, you probably started a transition. In that case, your package will be stuck in unstable until all the reverse dependencies are ready (or, at least, rebuilt to use the replacement package). Generally, it is good if the Release Team was involved before you did this (see "this page").
In summary, if your package has "Out of date on [architecture]" in its excuses it can be one (or more) of:
- Your package is not built yet on the architecture. This is a transient problem, wait for the build servers to build your package.
- Your package FTBFS on that architecture. This is usually a permanent problem where you need to investigate why it happened and how to solve it. Sadly, often it can take weeks before someone files a bug against your package for this build failure, so being proactive might be worth your time.
- The binary is no longer built binary. This is usually a transition. If the binary packages listed has no reverse dependencies in sid and testing, then just wait for the FTP masters to decruft your package.
Every package maintainer has probably looked at the "Testing migration" section of their package's PTS page and wondered: "Okay? What the hell does that mean?". Or perhaps "Is that something I should do something about?". Lets have a look at an example "excuse" (from the PTS)[1]:
excuses: * Maintainer: Debian Python Modules Team * Too young, only 8 of 10 days old * Updating python-scipy fixes old bugs: #691254, #707315 * Not considered
The first thing to look for is the phrase "Valid candidate". If that appears, Britney will attempt to migrate your package to testing. Note the migration may fail, but we can come back to that (in a) later (post).
On the other hand, if "Not considered" appears, then Britney believes your package is not ready to migrate yet. At this point, the excuse will contain 1 or more reasons explaining why Britney thinks your package should not migrate.
The most common problem is probably "age". In the example above, that is indeed the problem denoted by the "Too young, only 8 of 10 days old". Age is incremented on the "evening run" (i.e. the 10 pm UTC run). The excuse of that run will include the up to date age. So when you see:
Too young, only 9 of 10 days old
It means that the age criteria will be satisfied in the next "evening run". If the age requirement is satisfied, the entry will look like:
11 days old (needed 10 days)
If there is a "Not considered", it means there is something else you need to fix to make your package migrate. Note that the age is based on the day the package was uploaded to sid. If the package migrates to testing and is later removed from testing again, the age will usually be "ridiculously" high. This is because all the days spent the package spent in testing is also included in the age.
[1] It looks similar on the "real" excuses page, except the "excuses:" header is actually the name of the source. Is also includes information about the versions involved (i.e. "[from-version] to [to-version]").
We finally managed to deal with one of the major runtime bottlenecks in Lintian. Previously, our usage of file(1) would sometimes spend a long time classifying various text files. In Lintian 2.5.14, we applied a series of patches that allowed us to disable file(1)'s "ascii" test. This means that Lintian will now see all text files as "data", which is somewhat uninformative. However, it turns
N: [...] file-info for source:linux/3.2.20-1 done (120.350s)
into
N: [...] file-info for source:linux/3.2.20-1 done (20.984s)
In the same version, we also had Lintian create fewer empty files and directories in its laboratory. For the normal user, this change has hardly any practical effect. But for lintian.d.o, this reduces the minimum inodes consumed for binary entries with 8 or so.
For Lintian 2.5.15, we have may end up reducing the memory requirements of Lintian a bit. So far, we got two changes applied that is taking out at least 12MB of RAM usage on the linux source package and 21 of its binaries. Furthermore, Lintian will (if Devel::Size is available) be able to tell how much memory it is using for its caches when run with "-dddd".
After converting Lintian to using autodie, I was a bit trouble by the effects it had on Lintian's (start-up) performance. This overhead accumulated to at least a 1s when checking packages. Indeed, I am not the first to complain about this problem.
For those who have not heard of it, autodie is a Perl "pragma" that basically alters subs and built-ins from a "check return value for errors"-sub (e.g. like C) to "throws exception on error"-sub (e.g. like Python). So, you can replace "open(...) or die" with "open(...)" when autodie is in effect.
autodie achieves this by exporting a little sub that wraps the target sub (or built-in). This little wrapper is compiled[1] into the importing package. The problem with this solution is that it ends up compiling these subs once per importing package.
Secondly, all of these wrappers are wrapped in something called a "leak guard". As autodie is a "lexical pragma" it is not permitted to leak "across" file boundaries. The leak guards take care of that by doing a runtime check of the caller and reverting to the original sub/built-in if a leak occurred. These leak guards were also compiled into the importing package.
It is probably not much of a surprise to anyone that autodie spent quite a bit of its runtime compiling these wrappers and their leak guards. Together with upstream, I have addressed some of these problems.
The first optimization was to allow some wrappers for built-ins to be reused that does not take "glob" or bareword arguments[2]. Secondly, we started using closures to generate the leak guards. Combined these two effectively cut the load time in half for subsequent loads of autodie, though it did have a negative effect on the call overhead for these wrappers (~1s for 1M calls). That landed in autodie v2.18.
The extra runtime overhead of v2.18 was basically caused by the leak guard being split into two (a small closure, which called the "real" leak guard with an extra argument). We managed to merge these two parts into one without (too much) loss of code maintainability. With that, the call overhead of these wrappers dropped by ~3s (for 1M calls) and thus the wrappers in autodie v2.20 were even faster than the ones in v2.17.
Beyond these improvements, I have sent upstream several patches (still pending review) that will improve the performance of autodie even further. If merged, we are looking at another 1.5s (per 1M calls) removed and almost a factor 4 reduction on the load time of autodie.
This factor 4 comes from lazily compiling the wrapper on demand, so some of the price is moved to the call overhead. That said, it is a net gain since it is highly unlikely that every module using autodie will use every single wrapped built-in (how often do you use both "read" and "sysread" in the same module)?
[1] It is compiled via a call to eval using some template code.
[2] For the rest of the built-ins, it could probably just use Symbol::qualify to qualify bareword filehandles. Perhaps that will come in a future version of autodie. :)
During the Wheezy freeze, the Debian release team deployed 3254 hints[1]. This number may include some duplicates (i.e. where two members of the team hinted the same package), it certainly does not include a lot of rejected requests .
The top hinter was *drum roll*... Adam, who did 1799 hints(That is 55% of all hints during the freeze). For comparison, the second and third runner ups added together did 1023 hints (or 31.4%). Put in a different way, on average Julien Cristau and I would both add about 1.5 hints each day and Adam would on his own add 5.6 hints a day.
Of course, this is not intended to diminish the work other the rest of the team. Reviewing and unblocking packages is not all there is to a release. Particularly, a great thanks to Cyril Brulebois for his hard work on the Debian Installer (without which Debian Wheezy could not be released at all).
Enjoy!
[1] Determined by:
egrep -c 'done 201(3|2-?(07|08|09|10|11|12)) $HINT_FILE
It does not count hints, but the little "header" we add above our hints. One header can apply to multiple hints (which is good, because udeb unblocks are usually paired with a regular unblock and we don't want to count them as two separate hints).
About 5-6 weeks ago, I wrote about the Wheezy release progress, so it is about time for another update. According to UDD, we are down to 204 RC bugs (down from 249, since my last post). It is not quite the 2.4 RC bugs per day - actually it is about 1.1 RC bug a day.
Unfortunately, we do not appear to be fixing bugs faster than we are reporting them at the moment. If you look at Richard Hartmann's post from last week, then we had 206 RC bugs left. Even worse, we appear to have regressed between week 7 and 8 of this year (194 to 206). If you want the Wheezy release to happen soon, please consider helping us by providing bug fixes that comply with the Wheezy freeze policy.
If the pace of RC bug fixes do not pick up, the alternative is that the release team "deals" with the bugs. Note that "deals" generally falls into one of 2 categories. Either we defer/ignore the problem for Wheezy[1] or we remove affected packages from Wheezy/testing. Particularly, if we have to remove packages, they may take reverse dependencies with them as collateral damage.
I do not like these tools anymore than you do. But if the RC bugs fixes are not coming in, it is the only two tools we have left.
[1] Meaning that at best the bug fix will occur at Wheezy point release... at worst, not at all.
In January, I did a "TV-shop ad"-style post on a little script called "lintian-overrider" and it prompted Simon to ask:
That’s a great tool, but don’t you fear it makes unjustified overrides too easy ?
In my experience, people sometimes have issues writing overrides (justified or not). In fact it sometimes leaves them really frustrated with Lintian. If that frustration eventually leads to them reject Lintian, then we are doing the project a disservice. I will quote Russ Allbery as I believe he covered when he wrote:
I care most about all of the regular Debian developers [...] continuing to use Lintian, so that Lintian can stay as effective as it is now at getting people to make archive-wide changes. [...] This only works if we can get nearly everyone uploading packages to run Lintian all the time.
If a handful of people adds overrides for tags they should not have, then we can fix that (e.g. by filing a bug against their packages). But it is unlikely that we will ever make them run Lintian again once they have boycott it.
In December, I wrote a post on the progress of the Wheezy release. Back then, we had 348 RC bugs in testing and today there are about 249 left. A bit of simple math put us at 99 RC bugs fixed since last and an average of about 2.4 RC bugs fixed per day (up from 1.8). Assuming a constant rate, we would be able to release in about 100 days or 3 months (plus 1 week or two) from now. If you want the release earlier than that, fix RC bugs even faster! :)
Though, the data we are looking might be inaccurate. We filed a bug against UDD, which claims that #639407 affects Wheezy while it is in fact fixed as far as the BTS is concerned. If UDD is only wrong in this direction, we can hope for a positive surprise when the UDD bug is fixed. On the other hand, we may also be unpleasently surprised if not.
While talking about the UDD, I would like to mention a new feature its bug search script. It is now possible to filter based on whether or not a package is unblocked. If you take this into account (and assume that all unblocked packages will migrate in a timely manner), our RC bug count drops to about 219. It also gives us a much better view of how many packages that could actually use an unblock. which is now down to about 61.
In case you have been wondering what is up with the "removal of RC buggy leaf packages". We have still been looking at those, but for the last couple of times there have been at most one candidate for a given run (obviously not the same package each time). It felt a bit excessive to send an email to debian-devel for just one RC bug.
Have you ever tried to add a Lintian override only to get it wrong? Fret not, with the "Lintian-overrider 2000" such are problems of the past! Simply feed the tag emitted by Lintian to the Lintian-overrider 2000 and it will show you the correct format for the override plus the file to put said overide in. Furthermore, it will show you variants that you may (or may not) want to use instead.
$ echo "W: login: setuid-binary bin/su 4755 root/root" | \
lintian-overrider --alternative-forms
--8<-- debian/login.lintian-overrides --8<--
# If you want to override all (present and future) variants
# of this tag, use:
# setuid-binary
setuid-binary bin/su 4755 root/root
# Alternative forms...
# login: setuid-binary bin/su 4755 root/root
# login binary: setuid-binary bin/su 4755 root/root
# For architecture specific overrides, use one of:
# login [i386-any amd64-any other-archs] binary: setuid-binary bin/su 4755 root/root
# login [!i386-any !amd64-any !other-archs] binary: setuid-binary bin/su 4755 root/root
--8<-- End of debian/login.lintian-overrides --8<--
No more fiddling with that stupid syntax. Just feed it to the Lintian-overrider 2000 and instantly you will get the overrides you want!
If you sometimes make a copy-waste mistake or just feel life is too short manually update those files, the Lintian-overrider 2000 is the tool for you! With its --source-dir command line option, your lintian overrrides are updated automatically!
The Lintian-overrider 2000 can also automatically maintain your Lintian overrides for you! It is simple, just do:
$ lintian -o <path/to/your/changes-file.changes> | \
lintian-overrider --there-are-no-issues --source-dir <path/to/unpacked/source-tree>
Alioth is ready to take your git clone (or HTTP GET) request, so go order your copy now!
</tv-shop-ad>
In 2011, I wrote about how small files could consume a lot of space. I meant to do a follow-up on the savings but I forgot about it until now.
In 2.5.7, we started compressing some of the collected data files. Some of these are ridiculously compressable (#664794). Even better, compressing them is sometimes faster than writing them directly to the disk, so in some cases it is a pure win/win. For lintian.d.o, we also see a vast size reduction in overall size of the laboratory.
I have taken a few samples occasionally. The samples were done with du(1):
$ du -csh [--apparent-size] laboratory/*
| Version/date | du -csh | --apparent-size |
| N/A - around 20 Mar 2012 (#664794) | 16G | 13G |
| 2.5.6 (Fri Apr 27 2012) | 14GB | N/A |
| 2.5.6 (Mon Jun 04 2012)) | N/A | 12G |
| 2.5.10.2 (Fri Sep 21 2012) | 12G | 8.3G |
| 2.5.11 (Wed Jan 2 2013) | 10G | 6.1G |
And the most awesome part of this? The comparison is quite biased against the 2.5.11 entry, which is the only entry to also process experimental (approx. 10% extra packages). Some of the early entries (2.5.6 and "older") might also have suffered from the "too many links" issue[1]. I only wish I had been better at collecting data points, so I could have made a proper graph of it. :)
It sounds almost too good to be true, but if you look at the size of one of the linux-image packages[2], the space usage dropped from 27M to 15M between 2.5.5 to 2.5.9. Currently it is squeezed down to 14M (tested with head of the git master branch).
[1] I believe is about 5-10% less binary packages processed for those runs.
[2] linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb
Wheezy has been frozen for 5-and-(almost-)a-half months now.
Last month, Neil sent a mail to d-d-a stating that there were 403 RC bugs left and today UDD claims there are 348 left. My "pre-coffee^Wtea" math gets this to an average of approximately 1.8 RC bugs a day.
This average is only the change in "total" RC bugs between the 8th of Nov and today. It does not account for RC bugs being filed and fixed between these two days.
However, there is another number that worries me. Namely the number of RC bugs fixed in sid, but not Wheezy. Today it is 139. According to Richard's weekly updates, that number has generally remaining in the interval 143-148 for 6 weeks (except week 45 where it went down to 134).
Those 139 are generally "just" an unblock (and a couple of days) from reaching Wheezy (or a tpu upload). It does also include "unblocked, but not old enough to migrate" packages. Unfortunately the 112 "unread" emails in my d-release inbox suggests most of these 139 still needs attention.
Most of these "unread" emails are (unfortunately) unblock/tpu requests that none of us (to my knowledge) has had time to respond to. So, if you cannot find an RC bug to fix, I hope you will consider doing a bit of peer review and help us bring down the number of unanswered unblock requests.
DDs can access the same tools we use (See Neil's mail). Otherwise, the diff is just a:
$ dget -d http://.../pkg_testing-version_arch.dsc
$ dget -d http://.../pkg_sid-version_arch.dsc
$ debdiff pkg_testing-version_arch.dsc pkg_sid-version_arch.dsc
$ LINTIAN_ROOT=. perl lintian-ng ../lintian_2.5.10.dsc Successfully unpacked 1 packages Did you know that source:lintian/2.5.10 contains 3745 files or/and directories (excl. root dir) This useless information was brought to you by Lintian-NG $ git describe 2.5.10-61-g7670427 $ wc -l lintian-ng 90 lintian-ng
This includes everything from creating a temporary lab, adding the packages to it and running all of Lintian's collections on it (parallelised, of cource).
Writing this script yesterday would probably have required 250-300 lines of code to achieve the same. That said, 90 lines are still a bit much to copy/waste into this blog, so I won't do a "$ cat lintian-ng". :)
If you are interested in how it looks you can (for now) fetch it from http://people.debian.org/~nthykier/lintian-ng/lintian-ng
Oh yeah, "files or/and directories" is actually any (extractable) entry in the tarball...
Thanks to a heads up from Bastian Blank, I learned that Lintian 2.5.7 and 2.5.8 were horribly slow on the Linux binaries. Bastian had already identified the issue and 2.5.9 fixed the performance regression.
But in light of that, I decided to have a look at a couple of other bottlenecks. First, I added a simple benchmark support to Lintian 2.5.10 (enabled with -dd) that prints the approximate run time of a given collection. As an example, when running lintian -dd on lintian 2.5.10, you can see something like:
N: Collecting info: unpacked for source:lintian/2.5.10 ... [...] N: Collection script unpacked for source:lintian/2.5.10 done (0.699s)
When done on linux-image, the slowest 3 things with 2.5.10 are (in order of appearance):
[...] N: Collection script strings for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (12.333s) N: Collection script objdump-info for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (15.915s) [...] N: Finished check: binaries (5.911s) [...]
(The mileage (and order) probably will vary a bit.)
The version of linux-image I have been testing (3.2.20-1, amd64) has over 2800 ELF binaries (kernel modules). That makes the runtime of strings and objdump-info much more dominating than in "your average package". For the fun of it - I have done a small informal benchmark of various Lintian versions on the binary.
I have used the command line:
# time is the bash shell built-in and not /usr/bin/time $ time lintian -EvIL +pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null # This was used with only versions that did not accept -L +pedantic $ time lintian -EvI --pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null
With older versions of Lintian (<= 2.5.3) Perl starts to emit warnings; these have been manually filtered out. I used lintian from the git repository (i.e. I didn't install the packages, but checked out the relevant git tags). I had libperlio-gzip-perl installed (affects the 2.5.10 run).
Most results are only from a single run, though I ran it twice on the first version (hoping my kernel would cache the deb for the next run). The results are:
2.5.10
real 0m28.836s
user 0m36.982s
sys 0m3.280s
2.5.9
real 1m9.378s
user 0m33.702s
sys 0m11.177s
2.5.8
real 4m54.492s
user 4m0.631s
sys 0m30.466s
2.5.7 (not tested, but probably about same as 2.5.8)
2.5.{0..6}
real 1m20s - 1m22s
user 0m19.0s - 0m20.7s
sys 0m5.1s - 0m5.6s
I think Bastian's complaint was warranted for 2.5.{7,8}. :)
While it would have been easy to attribute the performance gain in 2.5.10 on the new parallelization improvements, it is simply not the case. These improvements only apply to running collections when checking multiple packages. On my machine, the parallelization limit for a package is effectively determined by the dependencies between the collections on my machine.
Instead the improvements comes from reducing the number of system(3) (or fork+exec) calls Lintian does. Mostly through using xargs more, even if it meant slightly more complex code. But also, libperlio-gzip-perl shaved off a couple of seconds on "binaries" check.
But as I said, linux-image is "not your average package". Most of the improvements mentioned here are hardly visible on other packages. So let's have a look at some more other bottlenecks. In my experience the following are the "worst offenders":
- unpacked (collection)
- Seen on wesnoth-1.9 source. Here the problem seems to be tar+bzip2, so there is not really a lot to do (on the Lintian side). Though feel free to prove me wrong. :)
- file-info (collection)
- Seen in eclipse/eclipse-cdt source. file(1) appears to spend a lot of time classifying some source files. For eclipse-cdt, I experience an approx. 10 second speed up (from 40s to 30s) if file are recompiled with -O2. (That would be #659355). However, even if file is compiled with -O2, the file-info collection is still the dominating factor.
- manpages (check)
- Running man on manpages can be a dominating factor in certain doc packages. This is #677874 and suggestions for fixing it are more than welcome.
But enough Lintian for now... time to fix some RC bugs!
I have been avoiding #629247 for quite a while. Not because I think we couldn't use a better shell parser, but because I dreaded having to write the parser. Of course, #629247 blocks about 16 bugs and that number will only increase, so "someone" has to solve it eventually... Unfortunately, that "someone" is likely to be "me". So...
I managed to scrabble down the following Perl snippet. It does a decent job at getting lines split into "words" (which may or may not contain spaces, newlines, quotes etc.). It currently tokenizes the "<<EOF"-constructs (heredocs?). Also it does not allow one to distinguish between "EOF" and " EOF" (the former ends the heredoc, the latter doesn't.).
Other defects includes that it does not tokenize all operators (like ">&"). Probably all I need is a list of them and all the "special cases" (Example: ">&" can optionally take numbers on both sides, like ">&2" or "2>&1").
It does not always appear to terminate (I think EOF + unclosed quote triggers this). If you try it out and notice something funny, please let me know.
You can also find an older version of it in the bug #629247 and the output it produced at that time (that version used " instead of - as token marker).
#!/usr/bin/perl
use strict;
use warnings;
use Text::ParseWords qw(quotewords);
my $opregex;
{
my $tmp = join( "|", map { quotemeta $_ } qw (&& || | ; ));
# Match & but not >& or <&
# - Actually, it should eventually match those, but not right now.
$tmp .= '|(?<![\>\<])\&';
$opregex = qr/$tmp/ox;
}
my @tokens = ();
my $lno;
while (my $line = <>) {
chomp $line;
next if $line =~ m/^\s*(?:\#|$)/;
$lno = $. unless defined $lno;
while ($line =~ s,\\$,,) {
$line .= "\n" . <>;
chomp $line;
}
$line =~ s/^\s++//;
$line =~ s/\s++$//;
# Ignore empty lines (again, via "$empty \ $empty"-constructs)
next if $line =~ m/^\s*(?:\#|$)/;
my @it = quotewords ($opregex, 'delimiters', $line);
if (!@it) {
# This happens if the line has unbalanced quotes, so pop another
# line and redo the loop.
$line .= "\n" . <>;
redo;
}
foreach my $orig (@it) {
my @l;
$orig =~ s,",\\\\",g;
@l = quotewords (qr/\s++/, 1, $orig);
pop @l unless defined $l[-1] && $l[-1] ne '';
shift @l if $l[0] eq '';
push @tokens, map { s,\\\\",",g; $_ } @l;
}
print "Line $lno: -" . join ("- -", map { s/\n/\\n/g; $_ } @tokens ) . "-\n";
@tokens = ();
$lno = undef;
}
Here is a little example script and the "tokenization" of that script (no, the example script is not supposed to be useful).
$ cat test
#!/bin/sh
for p in *; do
if [ -d "$p" ];then continue;elif
[ -f "$p" ]
then echo "$p is a file";fi
done
$ ./test.pl test
Line 3: -for- -p- -in- -*- -;- -do-
Line 4: -if- -[- --d- -"$p"- -]- -;- -then- -continue- -;- -elif-
Line 5: -[- --f- -"$p"- -]-
Line 6: -then- -echo- -"$p is a file"- -;- -fi-
Line 7: -done-
The new version of Lintian (2.5.8) can pretty much be summed up as:
Its like 2.5.7, only with less false positives and no FTBFS.
Especially people annoyed by the hardening flags will hopefully find that 2.5.8 greatly reduces the number of false positives. I believe this is best demonstrated with an example:
$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb | wc -l 2.5.7 94 [...] $ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb | wc -l 2.5.8 4
However, nothing comes for free. We dropped hardening-no-stackprotector from the default profile (and demoted it to an "I" tag). For hardening-no-fortify-functions we made a "false positive -> false negative" trade-off by ignoring binaries if their only unprotected function is memcpy. For more information, please refer to #673112.
hardening-no-stackprotector is still available and can be used via the debian/extra-hardening profile (or via the --tags argument). For the 2.5.7 behaviour of hardening-no-fortify-functions, you have to use hardening-check directly.
But 2.5.7 had other changes besides the myriad of false-positives:
- Around 19 redundant tags were removed.
A consequence of this is that we no longer warn if you use things like "dpkg --assert-working-epoch" or your postinst creates a "usr/doc transition symlink".
- The last of the Lintian's (perl) modules have been put under the Lintian name space.
With all the Lintian modules under the Lintian namespace, we can install them in the standard perl @INC path. Admittedly, I am not certain we are ready to commit to the current API in these modules, which is one of the reasons why they are still installed in /usr/share/lintian. But in a couple of releases things may look differently. :)
- Lazy loading of data files
Lintain 2.5.7 ships over 50 data files of various kinds (usually white- or blacklists of some kind). Before 2.5.7, all data files with a few exceptions would be loaded eagerly (i.e. as soon as the check was run for the first time). A few data files had been special cased with "manual laziness".
In 2.5.7, Lintian::Data was updated to lazily load the data. So depending on the packages being checked, Lintian may now load fewer data files.
- Create proper data files for tables embedded within checks.
While this is not something exclusive to 2.5.7, we have separated quite a few checks from their data tables by now. My personal favorites are the table of known interpreters and (from 2.5.8) the table of known versioned interpreters.
- Support for Vendor specific data files.
I am certainly biased here. But this is probably the most awesome feature in Lintian 2.5.7. It is now possible for vendors to extend or simply "shadow" the core Lintian data files. Allow me to demonstrate how this can be used:
$ lintian --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc E: a2ps source: build-depends-on-obsolete-package build-depends: dpatch $ cat ~/.lintian/profiles/local/main.profile Profile: local/main Extends: debian/main $ cat ~/.lintian/vendors/local/main/data/fields/obsolete-packages @include-parent @delete dpatch bison $ lintian --profile local/main --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc E: a2ps source: build-depends-on-obsolete-package build-depends: bison $
It is a toy example, but I believe it is a good demonstration of the feature.
The full documentation of Vendor specific data files can be found in the Lintian User manual (in lintian/2.5.7 or newer). It will also be on lintian.debian.org when we find time to update lintian there. :)
Credit where it is due and I believe it is due for Jakub Adam for packaging eclipse packages. If you use any of the eclipse packages provided the apt repositories for Wheezy or sid, it is very likely you have Jakub Adam to thank for it.
I also believe that Miguel Landaeta and James Page deserve praise for their work. Miguel is to thank for the removal of libservlet2.4-java and updating its reverse dependencies - not in that order ;-). James Page, on the other hand, has been introducing and updating a lot of packages, noticeably the jenkins packages.
Thank you and keep up the good work.
If you want to enable all Lintian tags, just remember the phrase:
Some sponsors are "evil and pedantic"
Or on the command-line:
$ lintian -EvIL +pedantic ...
It works for Lintian 2.5.5 (and newer), which handles "pedantic" like other severities. If you need help understanding the tags, you can add an extra "i" (-i) to "evil". That being said, remember that experimental (-E) and pedantic (-L +pedantic) tags are what they are for a reason. Also quite a few people will probably find verbose (-v) too noisy. However, leaving any of them out would have ruined the mnemonic. :)
Of course, you can also ask Lintian to enable all tags via your lintianrc file. Here is a quick-start:
display-info = yes # or no display-experimental = yes # or no pedantic = yes # or no verbose = yes # or no
For the past 5 days, I have worked on replacing a part of Britney (our beloved testing migration tool). The part I am trying replacing is her "installability tester", which is probably one of the most used parts of Britney.
I know premature optimization is the root of all evil, but I felt compelled to be a bit "evil" this time. The reason is that a naïve algorithm could easily spend decades on computing the result of our most complete tests (each consists of roughly 2 * 35 000 packages).
Within a couple of days, I felt I had a reasonable candidate to use on the large tests. On the large tests, it ran with a noticeable speed regression and it produced one different result. The result difference was about 20-40 source packages and their binaries.
You may have noticed I used "difference" and not "regression". In the given test, our current implementation at some point gives up with an "AIEEE" error. So I decided to turn my attention to the speed regression first.
As any "good" programmer, I decided to use a profiler (cProfile) to determine my bottle-necks. This turned out to be a good idea, as the bottleneck was not quite where I thought it was. I wish I could say I just patched it out the issue, but... unfortunately not.
I played around with various changes such as using "implied" for loops rather than explicit ones etc. Some times I would manage to cut the runtime in half only to be unable to reproduce it later. Yesterday I finally realized what caused this. I was working in 3 different chroots, two 64-bit sid chroots[1] and a 32-bit stable chroot.
Turns out that my replacement is only slow in the stable chroot. As soon as I moved to a sid chroot, the runtime was more or less cut in half. I have not bothered to check if it is the 32 vs 64 bit, the stable vs sid part or maybe the python profile in stable is just slow[2]. I simply moved to my sid chroots for now and continued working.
With that taken care of, I figured I would remind myself of what I was up against. On the largest test I have available, I profiled my implementation to around 4:30 - 5:00 minutes. Regardless of my changes, I always got times in that range. I did manage to patch a bug in my code that reduced by only diff to 10 source packages (plus their binaries) at the price of 30 seconds.
So I was back to 5 minutes according to the profiler, but I noticed that my test runner disagreed. According to my test runner my implementation had a runtime of approx. 3 minutes and 40 seconds. For comparison my test runner puts the original implementation at 3 minutes and 20 seconds for that test.
In short, I have spent most of my day trying to optimize my code to compensate for the +20% overhead the profiler introduced.
Admittedly, my replacement still needs some speed improvement on some other tests where it still has a factor 2 runtime regression. I also need to fix the last diff, which now I suspect is in my replacement.
[1] In case you were wondering, yes those chroots are on two different machines.
[2] Of course, it could be a combination of that as well... Anyhow, I am too lazy to research it at the moment.
While looking at the dpkg Breaks-field[0]...
$ aptitude show dpkg Package: dpkg [...] Version: 1.16.1.2 [...] Breaks: apt (< 0.7.7), aptitude (< 0.4.7-1), dpkg-dev (< 1.15.8), libdpkg-perl (< 1.15.8), pinfo (< 0.6.9-3.1), tkinfo (< 2.8-3.1)
... it occurred to me that most (all?) of these relations were irrelevant to Britney when she migrated dpkg 1.16.1.2 to testing. Right now, at least the APT relations are only relevant if you are doing something like a distribution upgrade from Lenny/Squeeze to Wheezy. Similarly, the version constraints in a lot of dependency relations (e.g. "libc6 (>= 2.11)") are satisfied in testing and unstable at the same time.
Removing the version constrains on dependencies is a rather minor thing as it is basically just a minor constant time optimization on each dependency check. However, removing an entire clause slightly reduces the "problem size" a bit. Particularly, the Conflicts/Breaks relations tends to be expensive for us.
The first task was to identify the relations that can (or cannot) be simplified. In a Britney run we at most 4 versions of the same package per architecture, though usually only 1 or 2[1]. I devised a small set of rules to simplify the relations. These rules are applied to each package in the relation (atomic proposition, if you will).
- If the relation is versioned and it involves a virtual package in any suite, then do not change the relation in any way. Rationale: A virtual package cannot satisfy any versioned relation (Policy Manual §7.5)
- If the relation is a dependency (i.e. Pre-Depends or Depends) and the package is not available in any suite, then do not change the relation in any way.
- If the relation is versioned and the relation is satisfied in all suites (where the "relationed-on" package is available), then remove the version constrain. Rationale: If all (present) versions satisfy the relation, then version constrain does not change the semantics.
- If the relation is a conflict (i.e. Conflicts or Breaks) and relation is unsatisfiable in all suites, then remove the relation. Rationale: If none of the (present) versions satisfy the conflict-relation, then there is no conflict[2].
The rules are rather conservative in some cases and there is room for improvements. However, one has to remember that removing too few relations costs a bit in runtime, removing too many breaks testing... and possibly a lot. Obviously, I prefer the former to the latter (especially because I will be a part of the "clean up"-crew).
I tested my implementation of those four rules above against the current master branch. In short, it produces the same result as the master branch in all the tests so far. In the hand-made test-suite, the tests generally do not have any superfluous relations. Thus, it is slightly slower, though usually within 0.1 seconds of the master branch.
On the other hand, in the live-data samples I have collected so far it does vastly better. For the longest run (sample from 2011-12-13), it reduces the runtime with ~70 seconds (from ~215 to ~145 seconds). In the other runs, it reduces total runtime with ~35 and ~2 seconds, respectively. In these samples, only the amd64 and i386 packages are considered (and human hints are ignored).
For those interested, the code is available in my branch. :)
[0] There is perfectly valid reason for doing that.
I might get back to that in a later post.
[1] One in unstable, testing, testing-proposed-updates and proposed-updates. The latter may seem a bit weird, but... [0]
[2] This is the rule that prune relations like the one in the dpkg Breaks-field.
Edit: 2011-01/09, clarified that we have at most 4 versions per architecture.
About 26-28 hours ago in #debian-release on IRC:
<nthykier> damn, a britney run in 5 minutes
<adsb> they happen
<adsb> you've been spoilt by never seeing b1 at her "finest" *cough*
<aba> you mean, running for more than a day?
<Ganneff> adsb wants night-long runs?
<aba> I can remember runs where we had to block certain packages to
make sure the run could actually end *sometimes*
<adsb> I'm quite happy with just the memory of that sort of run,
thanks :P
<nthykier> I don't mind being spoiled if it stays at 5 minutes :P
I took the liberty of collecting the resulting data for the Britney test suite. In its reduced state[1] in runs in 30 seconds on my machine. It is already my favourite live data sample in the test suite. :D
[1] Only i386 and amd64 are considered, manual hints are ignored etc.
Lately, I have been working more on release stuff. It all started when Julien convinced me to do the gpsd transition. It was a small, simple transition though I had to bounce obdgpslogger from testing (#648495). After that I picked up the zita-convovler transition (finished yesterday) and gssdp/gupnp{,-igd} (currently blocked by #653131 and #652783). I also got the mono, libindicator+libdbusmenu+libindicate and the libarchive transitions on my to-do list.
The mono transition is going to be most interesting and challenging of these. It is a bit above 100 packages and the binNMU order (for the 30ish packages that are not arch:all) is non-trivial. Thankfully, Iain Laney appears to have that part covered and will be helping me get it right.
I am also very happy with Britney2's transition assistance. Unlike her retired older sister, Britney2 "smooth updates" libraries, which allows us to break the transition into smaller steps.
Normally when Britney2 migrates a source package, she will throw out all the binaries from the old (version of the) source package. Then she moves the new source package and its binaries into testing. But in a smooth update, she will keep the old library binary packages around (if they are still in use).
In a concrete example, during the zita-convolver transition, we transitioned from libzita-convolver2 to libzita-convolver3. On the 24th of December[1], Britney migrated zita-convolver 3.1.0-1 to testing with libzita-convolver3, but kept libzita-convolver2 in testing as well. This is because ir.lv2 was not ready to migrate at that time.
With Britney1 zita-convolver 3.1.0-1 would have had to wait until all of its reverse dependencies were ready to migrate. For a small transition like zita-convolver (with 3 or so reverse dependencies), it would have been easy. But having to "keep" 100+ packages "migration ready" for the mono transition... that is where handling a transition becomes an art.
I may still need some hinting to finish the mono transition, but most likely it will be a lot easier than it would have been with Britney1. :)
[1] The PTS says the 25th. This is because it uses the day it receives the "migration"-email from trille, which was sent the day after.
We have been working on adding build-arch support for about a week now and I figured a little status update would be in order. :)
According to UDD, we had 506 packages to fix when we started and after todays update it has dropped to 485. It is a little less than the "4 packages a day" needed to ensure that they are all fixed in Wheezy, but I think it has been a good start. :)
On a related note, I am very pleased to see the progress on the general build-arch fixes. According to the statistics collected by Lintian on lintian.d.o[1], maintainers in Debian has fixed a total of 67 packages since we started. To put that into perspective, we fixed about 36 packages the week before that.
[1] Currently only DD-accessible on lintian.debian.org
/srv/lintian.debian.org/history/tags/debian-rules-missing-recommended-target.dat
The other day, I was asked if it was really possible to get build-arch support in Wheezy (as in, buildds using build-arch) by adding these optional targets.
Lets have a look at the data that is available to us:
At least ~500 packages must be fixed (the "reduced set")
The Wheezy freeze is expected in June (as I recall, I may be off)
- NMUs can take quite a while, so lets reduce it to May.
To simplify my calculation I will assume we can fix packages over a course of 150 days (which is 5 months of ~30 days or every day in Dec to April). So 500/150 = 10/3 =~ 3.3 packages from the reduced set should be fixed every day. Erring on the side of caution, we should make that 4 packages every day.
So if we fix 4 packages from the reduced set every day, we will definitely fix all of them before Wheezy. But the reduced set are only the source packages that could possibly "benefit" from having a build-arch target (it builds both arch:all and non-arch:all packages). There could (and probably will) still be sources building non-arch:all packages without a build-arch target. Furthermore, with the rate of 4 a day we will only have a month to get dpkg and buildd support...
In short, no. I do not expect us to get archive-wide build-arch support on buildds for Wheezy. But I will do my best to ensure that option #4 "flip the switch" becomes very attractive early in the "Wheezy + 1" development schedule.
I hope you will join us in this endeavour. Most of the time it is a trivial 3-4 line fix and often you can even throw in some hardening flags to spice it up a bit. The easiest way to help is to fix your (team's) packages listed in the "reduced set". Once you are done with that you can look at the "rest" of your (team's) packages (see the full dd-list).
The most important thing to remember is that Build-Depends-Indep is still broken! That is, you cannot rely on Build-Depends-Indep being installed on a buildd in the "build" target. So any existing workarounds have to stay for now (i.e. check for certain commands or deferring indep till binary-indep is called). If you are in doubt about how to fix a certain package feel free to ask for help.
Note: I pull the data directly from lintian.d.o (full set) and UDD (reduced set). I try to remember to refresh it daily. The full set is basically (z)grep | cut | uniq on the lintian log, the reduced set is found using the UDD query from this script (based on a query done by Jakub Wilk). Since all data is (in)directly based on lintian.d.o only packages that are in sid are considered.