%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% DebConf5: Debian-Kernel Team Overview and Status
%
% May 2005
% Copyright (C) 2005 dann frazier <dannf@debian.org>
% Copyright (C) 2005 Horms <horms@verge.net.au>
%
% This program is free software; you can redistribute it and/or
% modify it under the terms of the GNU General Public License as
% published by the Free Software Foundation; either version 2 of the
% License, or (at your option) any later version.
%
% This program is distributed in the hope that it will be useful, but
% WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
% General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with this program; if not, write to the Free Software
% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
% 02111-1307  USA
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass{article}
\usepackage{graphicx}
\usepackage{times}

\title{
	Debian-Kernel Team Overview and Status\\
	DebConf 5
} 

\author{
  dann frazier\\
  dannf@debian.org\\
  dannf@hp.com\\
  \\
  Simon Horman\\
  horms@debian.org
}

% Deleting this command produces today's date.
%\date{
%	29 May 2005
%}   

\usepackage{hyperref}
\begin{document}           

\maketitle 

%\begin{abstract}

%Abstract Needed.

%\end{abstract}

\section{A Brief History} \label{sec:history}
In the beginning, or at least as far back as our changelogs go, Herbert Xu
maintained the Debian kernel-source package:

\begin{verbatim}
kernel-source-2.0.29 (2.0.29-6) frozen unstable; urgency=medium

  * Merged Debian changes from 2.0.30.

 -- Herbert Xu <herbert@debian.org>  Sun, 25 May 1997 22:26:30 +1000
\end{verbatim}

Among his other Debian accomplishments, he also maintained the i386 and alpha kernel-image packages, and was the original author of Debian's initrd-tools.

Maintaining these packages was (and still is) a time-intensive task. Herbert 
was, in my experience, a maintainer who was very responsive to bug reports and 
e-mail, and excelled at keeping up with security issues and new upstream 
releases.

Since Herbert maintained the i386 and alpha kernel-image packages, builds for 
those architectures closely tracked new kernel-source releases. However, each 
of the other architectures had its own maintainer. Those maintainers may or may
not use a kernel-source package as their base. Those that did use kernel-source
did not all stay synchronized on the same version.

As an example, let's take a look at the woody release. There are 10
kernel-source packages in 3.0r5. An audit of the source packages that are
included in 3.0r5 show 31 that produce kernel-image packages. Of those, 3 do
not build-depend on any kernel-source package. Instead, these 3 packages
provide their own linux source. Therefore, the security team would need to
port a security patch to 13 source trees and rebuild 31 packages in order to
fix a vulnerability in the woody distribution! And this is ignoring things like
ABI changes which may require rebuilds of module packages, and other packages
that include their own kernel binaries.

To attempt to solve this and other coordination issues, the creation of a
\href{http://lists.debian.org/debian-kernel}{debian-kernel list} was
\href{http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=218923&archive=yes}{requested}
by Francesco Paolo Lovergine and Sven Luther.

Herbert
\href{http://lists.debian.org/debian-devel/2004/05/msg00276.html}{resigned}
from Debian on May 19, 2004. Andres Salomon
\href{http://lists.debian.org/debian-devel/2004/05/msg00288.html}{suggested}
that a kernel team be created, citing the gnome team as an example of
successful team package maintenance. Herbert handed over the responsibility of
finding a new maintainer for the kernel packages to Martin Michlmayr, the
presiding Debian Project Leader at the time. Martin's
\href{http://lists.debian.org/debian-devel/2004/05/msg01246.html}{response} was
indeed an attempt to assemble a kernel team. Later that month,
William Lee Irwin III
\href{http://lists.debian.org/debian-devel/2004/05/msg01441.html}{announced}
his intent to NMU Herbert's kernel packages on behalf of the newly formed
debian-kernel team.

Since then, much has been accomplished. I've captured some of the more notable milestones here.

\begin{itemize}
\item 2003.11.03: debian-kernel list requested
\item 2004.05.19: Herbert Xu resigns
\item 2004.05.22: William Lee Irwin III announces his intent to NMU Herbert's
package, changing the Maintainer field to the kernel team
\item 2004.05.23: Christoph Hellwig splits the monolithic Debian kernel patch
into functional pieces
\item 2004.06.15: First kernel-source upload by the kernel team (2.6.6-2)
\item 2004.06.18: kernel project started on alioth
\item 2004.06.21: initrd-tools added to svn
\item 2004.06.23: Jeff Bailey sets up DebianKernel wiki
\item 2004.07.12: source, ia64, powerpc and sparc in svn
\item 2004.07.15: i386 added to svn
\item 2004.07.27: amd64 added to svn
\item 2004.07.28: mips and alpha added to svn
\item 2004.08.09: hppa added to svn
\item 2004.08.13: upstream 2.6.8 released
\item 2004.08.14: kernel-source-2.6.8 uploaded
\item 2004.08.17: s390 added to svn
\item 2004.08.25: 2.4.27 chosen as the 2.4 kernel for sarge
\item 2004.10.10: linux-kernel-di (0.1) released
\item 2004.10.20: m68k added to svn
\item 2004.11.02: kernel-source-2.6.9 ACCEPTED
\item 2004.12.05: kernel changes ABI and breaks d-i for the first time
\item 2005.01.08: kernel-source-2.6.10 ACCEPTED
\item 2005.02.02: first formal debian-kernel irc meeting; 2.6.8 vs. 2.6.10 in
sarge
\item 2005.03.19: Andres Salomon commits kernel-source-nonfree-2.6.11-2.6.11
\item 2005.03.26: kernel-source-2.6.11 ACCEPTED
\item 2005.04.12: irc meeting, deciding that sarge kernels are now frozen
\end{itemize}

\section{Who We are and What We Do}
The Debian Kernel Team is made up of a loose collection of Debian Developers
and non-Debian Developers.  We maintain a number of kernel and kernel-related
packages.  This includes kernel-source-*, cramfs, initrd-tools, mkvmlinuz and
the kernel-images for most architectures.  We currently only maintain Linux
kernels.

Historically, architectures that were maintained outside of the kernel.org
tree would provide a kernel-patch package for their architecture, and apply
that on top of the latest kernel-source package when preparing an image build.
With the creation of the kernel team, this practice has become discouraged.
It is our goal to build as many architectures as possible from a common source
tree.  This requires more active collaboration between porters, but has the
benefit of additional peer review and avoids surprising conflicts between the
kernel-source and the architecture patches later.


Like other debian teams, we communicate primarily via
\href{mailto:debian-kernel@lists.debian.org}{e-mail} and
\href{irc://irc.debian.org/#debian-kernel}{irc}.  Though we have had two
formally scheduled IRC meetings, we do not hold them regularly.

The kernel team uses svn.debian.org for source control.

\section{Kernel Security}
The Debian Kernel Team owns security support for Debian kernels in unstable and
testing. Security fixes to kernels in stable are managed by the Debian Security
Team. Like with any other Debian packages, users should not assume that all
known security issues have been fixed in the unstable and testing versions.
Less severe issues may be committed into svn but not uploaded until some more
significant event triggers a release. Fixes that change the ABI maybe delayed
to avoid breaking an existing release of DebianInstaller.

\subsection{Security Maintenance}
The Debian Kernel Team is notified of security vulnerabilities and fixes
through multiple channels. The testing-security team tracks known security
vulnerabilities in the whole of Debian/testing.  The Ubuntu kernel team shares
patches with us, and other issues come in through the bug tracking system.  The
Debian Kernel team doesn't currently monitor closed lists such as vendor-sec,
so security fixes are not normally made available in unstable immediately after
they become public.

Security is high-maintenance for the Linux kernel - a quick count of the
changelog entries in kernel-source-2.6.8 that are clearly
identified as security yields 55; and this is just one of the three
kernel-source trees currently maintained by the Debian Kernel Team.  The
kernel team, principally Simon Horman and Andres Salomon, have been very
diligent about getting vulnerabilities fixed as they come in.

\subsection{Kernel Team/Security Team Maintenance Hand-off}
Sarge will be the first stable release since the kernel team came into
existence. As such, the hand-off process between the kernel and security teams
is still being defined. The biggest issue is how we manage security maintenance
during the freeze. Within this period, the
security team hasn't taken over maintenance yet, but the kernel team is no
longer able to make changes that will automatically propagate into testing.

During the freeze, the kernel team is still actively tracking security updates
for unstable. Also, the kernels in testing more closely resemble the kernels in
unstable than the kernels in stable. These arguments suggest that the kernel
team is the logical maintenance unit during this period, and this will
hopefully save some work for the security team once the release occurs. The
current procedure is to continue uploading security fixes to unstable. This
makes the packages available in an environment that is likely to receive user
testing. Once sarge is released, these packages can be used as the basis for
security uploads.

\section{Bug Tracking}
Due to the multiple layers in the existing kernel build process, a user
discovered bug may exist in any one of a number of packages.  Most bugs should
eventually be assigned to the associated kernel-source package.  Bugs
related to the kernel config should remain with the kernel-image package.

Since new package names are used when we move to a new upstream release, we
often need to reassign bugs against older packages.  These bugs get reassigned
to the pseudo-package ``kernel'' so that they aren't left behind and forgotten.

Andreas Barth put together a web page to track kernel bugs in sarge with a
severity of important or greater:
\href{http://bts.turmzimmer.net/kernel.php}{http://bts.turmzimmer.net/kernel.php}

\section{Debian Patch Set}
The Debian Kernel team aims to provide kernels for our users that are
functional, free, and secure, while minimizing our divergence from upstream.

\subsection{Where Can I Find the Debian Patches?}
Despite attempting to stay close to upstream, Debian applies a number of
patches to the tree for one reason or another.  These patches are contained
in the kernel-patch-debian-VERSION package.

\begin{verbatim}
# In the kernel-source-2.4.27 directory unpacked previously
# Patch back to pristine source
/usr/src/kernel-patches/all/2.4.27/unpatch/debian
# Patch the tree back up to the latest debian version
/usr/src/kernel-patches/all/2.4.27/apply/debian
# Patch to an arbitrary release
/usr/src/kernel-patches/all/2.4.27/apply/debian 2.4.27-7
\end{verbatim}

kernel-patch-debian-VERSION installs patches under
\texttt{/usr/src/kernel-patches/all/2.4.27/debian/}.  Most patches include a
comment at the top, describing briefly where they are from, what they do,
and their status with respect to upstream.

For example:
\begin{verbatim}
$ bzcat /usr/src/kernel-patches/all/2.6.8/debian/ia64_cyclone_nosmp.dpatch.bz2  | head
#! /bin/sh -e
## <PATCHNAME>.dpatch by <PATCH_AUTHOR@EMAI>
##
## All lines beginning with `## DP:' are a description of the patch.
## DP: Description: fix cyclone build on IA64 for UP
## DP: Patch author: Jesse Barnes <jbarnes@sgi.com>
## DP: Upstream status: submitted

. $(dirname $0)/DPATCH
\end{verbatim}

And the meta-files in the \texttt{series//} subdirectory
indicate which patch went into which version.

\begin{itemize}
\item{}
A line beginning with a \texttt{+} denotes an added patch

\item{}
A line beginning with a \texttt{-} denotes an removed patch

\item{}
A line beginning with a \texttt{X} denotes removed file, \\
usually to make the tree DFSG free.
\end{itemize}
 
\subsection{Patch Acceptance Guidelines}
Patches the kernel team like:

\begin{itemize}
\item Security Fixes
\item Driver Fixes
\item Stability Fixes
\end{itemize}

... actually, more or less any type of fix

Patches the kernel team generally reject:

\begin{itemize}
\item New features
\item Out-of Tree Drivers
\item My favourite patch-set
\end{itemize}

The best way to get a patch into the Debian kernel is to get it accepted
upstream first. Not only does this signify a certain level of quality, but it
gives us a guarantee that we will be able to drop this patch from our
Debian-specific tree at some point.

\section{Kernel ABI Changes} \label{sec:abi}
Occasionally, and usually due to a security issue, a change is accepted into
a kernel package that changes its ABI\footnote{The upstream ABI interface
policy does not discourage ABI changes.  In fact, changing an interface to
resolve a security issue can have positive characteristics.  Quoting from
Documentation/stable\_api\_nonsense.txt: ``Security issues are also a very
important for Linux (SIC).  When a security issue is found, it is fixed in a very
short amount of time.  A number of times this has caused internal kernel
interfaces to be reworked to prevent the security problem from occurring.  When
this happens, all drivers that use the interfaces were also fixed at the
same time, ensuring that the security problem was fixed and could not
come back at some future time accidentally.  If the internal interfaces
were not allowed to change, fixing this kind of security problem and
insuring that it could not happen again would not be possible.''}
The ABI change is represented by a change in the SONAME portion of the
kernel-image package name.  For example,
in \texttt{kernel-image-2.6.8-3-686-smp}, the SONAME is \texttt{3}.  Changing
the SONAME has a significant impact, even outside of the kernel team.
Obviously, any kernel-module packages need to be recompiled against the updated
kernel-headers.  

Since a change in the SONAME changes the package name, apt will not
automatically upgrade a system to the new kernel.  In order for users to track
the latest available kernel (meaning the latest kernel security fixes in the
stable distribution), installing one of the kernel meta packages is strongly
recommended.  For example, \texttt{kernel-image-2.6-k7} currently depends upon
kernel-image-2.6.8-2-k7.  If the kernel ABI should be incremented, a newer
version will be made available with a dependency on kernel-image-2.6.8-3-k7.

Changes to the kernel ABI also have an impact on the installer, which will be
discussed in the next section.


\section{Working with the Debian Installer Team}
The Debian Installer architecture is very modular.  The highly granular
components (a.k.a., udebs) and their well defined interfaces provide a powerful
mechanism for developers who wish to add or remove features, port the installer
to new hardware, or provide new installer features.

To illustrate this, let's take a look at the components that are required to
permit a user to install Debian onto an XFS filesystem.

\begin{itemize}
\item{partman-xfs} - partman is Debian Installer's partitioning and filesystem
engine.  partman-xfs plugs into partman and provides methods for selecting,
creating, and mounting an XFS partition.  The partman-xfs udeb has dependencies
upon the xfsprogs and xfs-modules udebs.
\item{xfsprogs-udeb} - This udeb provides the mkfs.xfs binary.
\item{xfs-modules} - The udebs that provide this virtual packages provide the
linux kernel modules necessary for XFS support.
\end{itemize}

This level of granularity makes it possible to build a tiny installer by
stripping out all of the components that you know you're users will not need.
An installer can be built that is limited to supporting reiserfs on IDE
devices, simply by removing all of the unnecessary modules from the build.  If
additional components are needed, they can be pulled from a network repository
(assuming the components necessary for accessing the network were not excluded,
of course!).

Like many udebs, kernel modules udebs were originally built from the same
source package as their deb counterparts.  However, since kernel-images have
a different source package for each architecture, creating these udebs created
redundant work with inconsistent results.  This is a class of problem that
Joey Hess excels at solving.

Joey's solution was to create the kernel-wedge and linux-kernel-di packages.
linux-kernel-di creates the necessary udebs by declaring a build-dependency
upon the appropriate kernel-image, and copying its modules into the appropriate
udebs. A separate linux-kernel-di source package exists for each architecture
and kernel version.  However, most of the required code lives in kernel-wedge,
and is shared by all linux-kernel-di packages.

The number of indirections between a build of the Debian Installer and the
actual source code used to create its kernel has led to a couple of interesting
issues:

\subsection{Kernel ABI Dependency}
The Debian Installer may pull kernel module udebs
from multiple locations during an installation, and these modules must be
compatible with the runtime kernel of the installer.  As mentioned in
section \ref{sec:abi}, occasionally a change is included in a kernel build that
breaks the module ABI.  If the kernel module udebs corresponding to the kernel
used in a given build of Debian Installer are superseded by modules conforming
to a different ABI within a release, then certain classes of network installs
will break.

\subsection{Version Skew and GPL Compliance}
Each link in the chain of packages
that produce kernel udebs has its own lifecycle.  Consider the following chain
of events:
\begin{enumerate}
\item kernel-image-2.6.8-ia64 (2.6.8-1) is built against kernel-tree-2.6.8-1
\item linux-kernel-di-ia64-2.6 (1.0) is built against the above kernel-image
\item All of the above packages propagate into testing
\item kernel-image-2.6.8-ia64 (2.6.8-2) is built against kernel-tree-2.6.8-2
\item kernel-image-2.6.8-ia64 (2.6.8-2) propagates into testing
\end{enumerate}

At this point, the source code to recreate the Debian Installer build would no
longer exist in testing.  Because of the
\href{http://wiki.debian.net/?DebianKernelTree}{``rollback'' facility} in
kernel-tree, it is believed that the GPL terms are
satisfied if a new kernel-tree propagates into testing before kernel-image is
rebuilt against it.  However, kernel-image does not provide a similar feature.
Thanks to James Troup and the release team, this issue was resolved with the
creation of a special sarge-r0 distribution containing packages used to build
Debian Installer.

\section{In the Works}
\subsection{Architecture Synchronization}
Earlier \ref{sec:history}, I noted that a major problem with the woody release
was the large number of different kernel-source packages used to build
kernel-images.  Minimizing the number of active kernel-source packages is one
of a number of architecture synchronization issues the kernel team is
attempting to resolve.  A daily-generated page provides a window into how
closely synchronized architectures are on a given kernel-tree release:
\href{http://people.debian.org/~dannf/kernel-stats/kernel-avail.html}{http://people.debian.org/~dannf/kernel-stats/kernel-avail.html}.  In the future, Dann
plans to update this page to track linux-kernel-di packages as well.

kernel-source skew isn't the only architecture synchronization issue we face.
Other issues include:

\begin{itemize}
\item{Consistent Kernel Configs} - Each architecture maintains kernel config
files independently.  It would be nice if all architectures had a common
feature set and the same things were built as modules on all architectures,
where feasible.
\item{Build System Bugs} - Many of the kernel-image packages diverged from the
same code base.  When a bug in the build system is discovered it may be fixed
within one of the packages, only to pop up on a different architecture later.
Structural changes, such as the introduction of the kernel-tree concept or
the building of udebs for Debian Installer, usually take a long time to
complete (if ever), since the changes must be made to multiple packages.
\texttt{kernel-package} provides a code base that is shared among multiple
architectures, but more commonality is desired.
\end{itemize}

These are some of the reasons the kernel team is looking to create a unified
source package that builds kernel packages for multiple architectures.

\subsection{A Common Source Package}
At the time of this writing, Jurig Smakov has announced a working
prototype of a common kernel-source-2.6.11 package that is capable of building
i386 packages.  The 2.6.12 timeframe is being suggested as a goal for a first
upload of this package.

\subsection{Moving drivers to non-free}
With 2.6.11, the kernel team has started the process of moving drivers which
contain binary blobs to the non-free section.  These drivers have been removed
from main, and Andres Salomon has created a kernel-nonfree-modules package.
However, many of these firmware blobs exist in source files with only a GPL
boilerplate.  Since these firmware blobs are not provided in ``the preferred
form of the work for making modifications to it'', as required by the GPL,
Debian cannot legally redistribute these files.  Sven Luther and other
members of the kernel team are working to contact the associated copyright
holders, requesting they modify the license to explicitly grant redistribution
rights of this data in ``blob'' form.  At the time of this writing, this
process is beginning to produce some results.  Both QLogic and Broadcom (tg3
driver) have provided promising responses.

The current status of this process can be found at: 
\href{http://wiki.debian.net/?KernelFirmwareLicensing}{http://wiki.debian.net/?KernelFirmwareLicensing}.

\subsection{initramfs}
initramfs is a feature new to 2.6 linux kernels, and aims to solve many of
the problems associated with initial ramdisks (initrds).

Jeff Bailey has been working on an initramfs-based replacement for
\texttt{initrd-tools}.

Quoting from \href{http://www.ubuntulinux.org/wiki/Initramfs}{http://www.ubuntulinux.org/wiki/Initramfs}, some of the benefits of initramfs include:
\begin{itemize}
\item CPIO archive, so no filesystems at all are needed in kernel. The archive
is simply unpacked into a ram disk.
\item This unpacking happens before do\_basic\_setup is called. This means that
firmware files are available before in-kernel drivers load.
\item The userspace init is called instead of prepare\_namespace. All finding
of the root device, and md setup happens in userspace.
\item An initramfs can be built into the kernel directly by adding it to the
ELF archive under the section name .init.ramfs
\item initramfs' can be stacked. Providing an initramfs to the kernel using the
traditional initrd mechanisms causes it to be unpacked along side the
initramfs' that are built into the kernel.
\item All magic naming of the root device goes away. Integrating udev into the
initramfs means that the exact same view of the /dev tree can be used
throughout the boot sequence. This should solve the majority of the SATA
failures that are seen where an install can succeed, but the initrd cannot
boot.
\end{itemize}

By including hotplug-ng and udev, we will be able to perform device
autodetection from within the initramfs; however it is unclear if Debian users
will be able to handle the memory footprint required to have all possible
modules available at initramfs time.  \texttt{initrd-tools} tries to solve
this problem by detecting the necessary modules at initrd creation time, but
this process is error-prone.

Jeff has also been able to reduce the size of an initramfs when compared to a
typical Debian initrd by using klibc instead of glibc.

Unfortunately, not all Debian architectures have a 2.6 kernel, so it will be
some time before we can do a universal switch to an initramfs process.

\subsection{Automated/Nightly Testing}
Automated testing from source control is a goal for the kernel team, but
development is in the early stages.  Maybe we'll see some progress here at
DebConf5!

\section{How Can I Help?}
Bug reports are always welcome - the more information you can provide, the
better.  Try to do as much investigation as you can prior to reporting a bug.
\begin{itemize}
\item STFW :)  Check LKML/Google to see if the bug you found has already been
discussed.
\item Test the latest upstream version of the kernel to see if your bug still
exists.
\item If the bug came about after an upgrade, try to narrow down the version
in which it was introduced.
\item Take the time to make sure someone else hasn't already filed your bug by
\href{http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=maint&data=kernel-team%40lists.debian.org&archive=no}{searching the bug tracking system for bugs owned
by kernel-team@lists.debian.org}.
\end{itemize}

\end{document}             % End of document.
