| Author: | Sergio Talens-Oliag |
|---|---|
| email: | sto@debian.org |
| Date: | 2005-06-30 14:15:20 +0200 (dj, 30 jun 2005) |
There is a lot of people working on Custom Debian Distributions, but it seems that everyone is reinventing the wheel to do it.
This document proposes the design of a CDD Toolkit that tries to standardize the way developers define their CDD and provides tools to distribute, install, update and manage the customized system.
This toolkit is being developed on the current CDD Project on alioth and discussed on the debian-custom@lists.debian.org mailing list.
The basic idea is that each CDD can be distributed in source or in binary form using a single deb package that contains a description of the CDD and all the metadata needed to install it (or to generate an installer for it).
For this toolkit, a CDD is defined as a set of tasks that can be used to install and configure packages on the system for a given purpose.
A task definition has to include a list of packages (or CDD tasks) that have to be installed to provide the task's functionality and the information needed to configure those packages or, at least, give pointers to that information. Besides the needed dependencies, a task can also recommend and suggest other packages and tasks.
The CDD Description is written using a format similar to the Debian Control File, and is distibuted as one file (or on multiple ones, if we use the include feature). The file declares one or more tasks and the all information needed to configure, update and remove the packages related to them.
The description and any aditional files needed to install the different tasks will be distributed on the binary CDD debian package (the source file will contain only the description and related data, but some files distributed on the binary will be generated by the CDD tookit tools).
Currently the configuration data that can be included and distributed includes preseeding (for the debian-installer and the system's debconf), scripts that have to be executed to generate configuration files after package installation (that scripts are only used to generate modified configuration files that can't be generated using debconf) and task control scripts, that is, scripts that have to be executed before and after task installation or removal (like the package control scripts).
Other configuration data like special user menus can also be included on the CDD metadata, but for other things (like branding), proper debian packages are probably a better idea and can be included on the task dependencies.
The system is divided into two sets of tools:
The CDD definition is written using a control file with a syntax similar to the one used for Debian Control files.
The control file consists of one or more paragraphs. The paragraphs are separated by blank lines.
Each paragraph consists of a series of data fields; each field consists of the field name, followed by a colon and then the data/value associated with that field. It ends at the end of the line. Horizontal whitespace (spaces and tabs) may occur immediately before or after the value and is ignored there; it is conventional to put a single space after the colon.
Some fields' values may span several lines; in this case each continuation line must start with a space or a tab. Any trailing spaces or tabs at the end of individual lines of a field value are ignored.
Except when stated, only a single line of data is allowed and whitespace is not significant in a field body. Whitespace must not appear inside names (of packages, architectures, files or anything else) or version numbers, or between the characters of multi-character version relationships.
Field names are not case-sensitive, but it is usual to capitalize the field names using mixed case as shown on the fields description below.
Blank lines, or lines consisting only of spaces and tabs, are not allowed within field values or between fields - that would mean a new paragraph.
Include: the value given to this field is the path to a file and it's occurrence is replaced by the contents of the referenced file. Note that path can be absolute or relative to the current control file path. This field has been introduced to allow developers to use different directory layouts and to be able to reuse task definitions.
Task (required): Name of a task; always is written on the first line of a paragraph and all the fields of its paragraph and the following ones apply to this task (until a new task declaration is found). Note that the the same task name can be present more than once, for the second and following occurrences the fields are overridden or appended to the original task definition.
Description (optional): short description of the task, it is useful in general and is needed to build metapackages based on our tasks (if the field is not present we generate a simple description using the task name).
Architecture (optional): defaults to all, but can be used to include a list of supported architectures.
Section (optional): section given to the task (misc by default).
Depends, Recommends, Suggests (optional): At least one of them is required, they list the packages we want to include and can be present multiple times
The name of the field will imply the same as on the control file, and will have different effects depending on the target system (i.e. for tasksel recommends and suggests can be put on different tasks to be able to install only the basic dependencies).
Conflicts (optional): This should not be needed, but maybe is interesting to be able to declare explicit conflicts for important and standard packages, mainly to avoid it's installation on the initial base-installer run.
Task-Depends, Task-Recommends, Task-Suggests and Task-Conflicts (optional): Tasks are not packages, to handle the relationship between them we use those fields instead of the standard ones. Note that the contents of those fields are simple lists of tasks names, there are no virtual tasks, or'ed dependencies or versioned dependencies. The virtual tasks make no sense (tasks are already virtual), nor the or'ed dependencies (we can build a task with or'ed package dependencies) and the versioning is unnecesary (the tasks have no version number and the description contains all the tasks, if we need two versions of a task they need to have two different names).
Task-Relevance (optional): Number that declares the relevance given to the task on the tasksel .desc files.
Task-Scripts (optional): Name of a file or a directory containing configuration scripts for the task. The files included on a directory are executed in order and the field can appear more than once.
The scripts take parameters similar to those defined for postinstall and preinstall scripts; they must support the keywords post-install, pre-update, post-update, pre-remove and post-remove as first parameter (note that we don't use purge, as the remove stage has to leave the package without customizations).
Task-Scripts-Depends (optional): tools needed to execute the configuration scripts (i.e. cfengine, python, ... or whatever tools the developer uses on them).
Note that the tools needed by the scripts of an installed CDD will not be removed by the cddtool (i.e. if a task that includes the tools is removed) and if the tools are removed by hand they will will be reinstalled by the cddtool when trying to use the scripts that need them.
Install-Task (optional): boolean value that indicates if a task has to be installable from the debian-installer, that is, if it has to be shown on the cdd description package debconf questions.
Meta-Task (optional): boolean value that indicates if a task should be considered a metapackage or not. The flag is used when generating metapackages to know if the task dependencies have to be expanded or not.
Menu (optional): Name of a file or directory containing menu definitions for the task. If it appears more than once the menu the values all used. Note that a package can have a related menu that will be appended to the task one.
Package (optional): field that will be used to add meta-information related to packages included on the task, declaring that a set of fields like Cfg-Scripts, Debconf-Preseed, Pkg-Relevance, etc. are related to the given package. This field will always be on the first line of a paragraph and the values associated with it have to be on the same paragraph.
Pkg-Relevance (optional): A number that indicates the relevance of the packages the task depends on; not all packages of a task have the same relevance, if it's value is changed the packages declared afterwards have the given relevance. Note that this attribute is given to the packages and used mainly to choose how to put packages on CD if the CDD installation needs more than one.
Installer-Depends (optional): list of additional packages to include on the debian-installer.
Installer-Conflicts (optional): list of packages not to include on the debian-installer.
Installer-Package (optional): The same as Package, but applied to debian-installer packages (udeb).
Installer-Pressed (optional): Name of a file or directory containing debconf answers for the debian-installer. It can appear more than once.
Debconf-Preseed (optional): Name of a file or directory containing debconf answers for an installed system. It can appear more that once.
Base-Config (optional): Name of a file or directory containing files to be installed under /usr/lib/base-config/; note that if the directory contains subdirectories those will also be copied to /usr/lib/base-config/ (i.e. a menu/ subdirectory will be copied to /usr/lib/base-config/menu/).
Base-Config-Menu (optional): Name of a file or directory containing files to be installed under /usr/lib/base-config/menu/. If the developer uses a directory for all the base-config stuff that won't be needed.
Cfg-Scripts (optional): Name of a file or a directory containing configuration scripts for the task. If can appear more than once and inside a directory all scripts are included to be executed in order.
Note that the configuration scripts will be used against the installed system and will generate files that will be stored under the /var/lib/cddtk directory.
The scripts will get the TASK name as first argument and an ACTION and an optional list of ACTION_ARGUMENTS:
ACTION will be one of the following:
Cfg-Script-Depends (optional): tools needed to execute the configuration scripts (i.e. cfengine, python, ... or whatever tools the developer uses on them). Note that the tools needed by the scripts of an installed CDD will not be removed by the cdd-tool (i.e. if a task that includes the tools is removed) and if the tools are removed by hand they will will be reinstalled by the cdd-tool when trying to use the scripts that need them.
For fields not defined here we will use the prefix X- and they will be ignored by the cdd-tool system.
The control file can include comments using the # symbol at the beginning of a line (ONLY).
The idea is to have a single tool that acts as a fronted to all the CDD utilities, the syntax would be something like the following:
cddtool OPTIONS COMMAND CMND_ARGS
Where OPTIONS is a list of options common to all tools, like:
--version --help --debug --verbose --conf=FILENAME
COMMAND is one of:
build: builds a CDD binary package.
get: downloads all packages needed for the CDD (similar to debpartial-mirror)
install: installs all packages related to a CDD task applying preseeding and executing configuration scripts.
The command applies the preseeding, reconfigure the already installed packages (see the reconfigure command) and install the missing packages.
update: updates all packages related to the CDD keeping customizations without making questions to the user. To do an update first we get the latest version of the CDD task, apply the new preseedings, upgrade the packages and reaply the configuration changes.
reconfigure: reconfigures all the packages included on a CDD task, reaplying original task preseedings before calling the reconfigure scripts.
And the CMND_ARGS usually include optional SUBCOMMANDS, the name of the CDD and a TASK_LIST to work with:
cddtool OPTIONS COMMAND CDDNAME TASK_LIST
Note that the tool is work in progress and possibly there will be additional commands and a configuration file (not already defined).
Into this secction we will discuss the programs we plan to use to support the installation of custom distributions.
The tools are related to the different installation systems:
This proposal talks about Task and Package paragraphs, but the idea is that each task is one entity, so debconf preseeds and configuration scripts are used for all the packages of the task. It also says that the cddtool could be used to install, update and remove packages, but there is no indication of how that could be integrated with the usual Debian tools (that is, with apt).
Currenlty CDD handle other package's configuration files using two techniques:
The preferred system is the first one, but it is not applicable in a lot of situations or breaks used after upgrades.
The second method is not very clean and makes upgrades noisy for the user, as dpkg tells him that files that he does not know how to handle have been modified.
We propose a third mechanism that could be useful to install and remove customizations without interference with the Debian packages, at least in controlled environments.
The idea is that we could support CDD updates using a common system for the cfg-scripts described on the proposal and installing hooks for apt with the cddtool (or, better, with a specialized cddtk-apt written for the shell).
To do it we will install the following file:
DPkg::Post-Invoke { "test -x /usr/sbin/cddtk-apt && /usr/sbin/cddtk-apt post-invoke"; };
DPkg::Pre-Install-Pkgs { "/usr/sbin/cddtk-apt pre-install-pkgs"; };
DPkg::Tools::Options::/usr/bin/cddtk-apt::Version "2";
as /etc/apt/apt.conf.d/10cddtk.
The Pre-Install-Pkgs script is called when the packages are already on the local filesystem, but before calling dpkg to unpack and configure them (the DPkg::Tools::Options is used to get a list of the actions that are going to be done on the standard input of the cddtool invocation).
The Post-Invoke script is called after all packages are installed and configured.
The idea is to use those scripts to disable non-policy compliant customizations while installing or upgrading packages and reaply the customizations after installation.
Note that this system's usage has to be kept to a minimum (i.e. we should use it only to enable the inclusion of external configuration files when possible), and that all files related to a customization that are not going to conflict with the normal system should be handled using normal debian packages (the main custom package or packages linked to a task).
See the cddtk-apt document (included on the cddtk-runtime package) to see how the tools are going to be implemented.
To integrate our system with the debian-installer we could generate tasksel tasks or metapackages, but we also want to be able to have a very simple debian-installer, so we have defined a way to be able to select the cdd-tasks at the very begining of the installation process, to be able to choose them since the beging and preseed the debian-installer accordingly.
To build a really custom distribution almost all derived distributions apply a lot of changes to the debian packages; with some help of the Debian Developers that could be avoided changing the way the packages read configuration files (i.e. not installing conffiles if they are complicated or can't be generated by debconf for the general case), adding debconf support to them (possibly using very low priority questions) or simply adding some kind of support for branding (i.e. using a standard file that is in fact a link handled by update-alternatives or a similar system).
Once we have a common cdd toolkit there will probably be more custom distributions and we will have a better position to ask all debian developers to help us make debian more customizable.