============================================================================== # README.sources for debian-reference (Version 2 series) # # Sat Sep 12 15:29:21 UTC 2009 # # Osamu Aoki, osamu at debian dot org # ============================================================================== I will touch on following topics. * Quick guide * Document style * Basics of build process * What is @@@ in asciidoc * What is @-@ in asciidoc and raw XML * Making copy of SVN contents * Updating contents in English XML source * Fixing typo in English XML source * Updating remote data contents * Creating Translation PO for a new language * Updating Translation PO * Updating for new release (squeeze --> squeeze+1) * URL used for translation and its initialization * Utilities for spell check * Utilities of PO (split and merge and uniq PO) * VCS locations and history == Quick guide == Please note that for http://www.debian.org/ web site publishing, XML/PO files are used to build pages. Before committing, please do this. * Update from SVN to get the latest source: "svn co" or "svn up" * Fix fuzzy in PO using via poedit GUI etc. * Update XML/PO files. * If updated English ascoodoc: "make xml" * If updated translation PO only: "make po" * Fix fuzzy in PO using via poedit GUI etc. * Generate HTML to check result. * If updated English ascoodoc: "make all" * If updated translation PO only: "make po-test" Please note old translated documents are in SGML and its content is very loosely related to current English content. In order to recycle as much old content, I usually use followings: * "poedit" to edit debian-reference.??.po file to be updated. * browser to display debian-reference.??.html from sage/lenny package * copy-and-paste to po file (via poedit GUI) * add XML tags as needed There are few utilities scripts I use to maintain this package. Please copy them to from utils/ directory to directory included in $PATH such as /usr/local/bin/. == Document style == Besides things mentioned in "About this document" in Preface, I used IBM Style Guide (IBM_Style_27Apr09.pdf) from IBM site: URL: http://www.ibm.com/software/ucd/files/IBM_Style_27Apr09.pdf ( You may wish to check newer version linked from: URL: http://www.ibm.com/software/ucd/otherresources.html ) Currently, there is big gap. I will slowly polish this document to be compliant as possible. There are exceptions. * I use Latin acronyms: e.g., etc., i.e., read/write * I use lines starting with number == Basics of build process == You must have lenny system with following packages installed: * subversion * pbuilder * asciidoc (only if you update English) * libxml2-utils * devscripts * wget * dctrl-tools * Files in debian/control Build-Depends Basic flow of build process can be summarized as: all the asciidoc/*.txt files and remotely fetched files by wget :(asciidoc @ lenny version) : :(build from asciidoc by : : "make xml" or "make all") : v v >>> debian-reference.raw.xml --> url.ent and other *.ent (source XML) :(build from raw.xml by "make po" or "make po-test") v debian-reference.en.xml --> all English final formats (HTML, text) debian-reference.en.xmlt \ (This removes untranslating part) (XML file) | \ \ v \ \ debian-reference.pot +--+-> debian-reference.ja.xml (POT file) | / | v / v >>> debian-reference.ja.po all Japanese final formats (HTML, text) (PO file) Please note po files are generated from English debian-reference.en.xmlt. This debian-reference.en.xmlt is an XML file very similar to debian-reference.en.xml which is also XML file. Untranslatable contents in debian-reference.en.xml is replaced by "DUMMY" to reduce clutter in PO file. Please note actual build process starts from debian-reference.raw.xml and debian-reference.ja.po files. These are "REAL" source files as far as package build process and www.debian.org web page building process by DDP are concerned. Asciidoc format is used as a helper to make debian-reference.raw.xml source with minimum typing. This may be dropped once squeeze is out. After squeeze release, debian-reference.raw.xml will be the source. Due to asciidoc conversion logic differences, we must use lenny version of asciidoc for this source. Otherwise you may get different output to the raw.xml and break all the po files. Please note Makefile has few key parameters which affects build process. BUILD_MODE := xml (normal-build-from-raw-xml mode, default) BUILD_MODE := asciidoc (build-from-asciidoc mode, use this via "all" "xml" target in lenny) PDIR := something for normal build situation PDIR := ddp-svn for /org/www.debian.org/ddp-svn/trunk/quick-reference PUBLISHDIR := build result file location LANGALL := language list for all LANGPO := language list for PO based translation LANGSGML := language list for SGML based old translation (calculated) Please note following targets: make all Update source from asciidoc (if newer) Results are in $(PUBLISHDIR) Do not require remote connection if you already have done so. make xml Create XML/PO source files from asciidoc (always force) make po-test Results are in $(PUBLISHDIR) Build html etc using current raw.xml and po files make po Update PO files from raw.xml make publish Used by DDP for www.debian.org Do not require remote connection nor asciidoc make html Create html files make txt Create txt files make css Create css files make clean Clean source to make it ready for packaging make distclean Clean source to make it ready to be updated remotely make update Update source with remote data. == What is @@@ in asciidoc == You see quite a bit of @@@ in asciidoc source. That is placed there when idiosyncrasy of asciidoc hit me. Asciidoc to raw.xml conversion result is filtered with sed script bin/replace to get right characters into a resulting XML file (debian-reference.en.raw) without worrying about asciidoc converting to something funny. For example: @@@sq@@@ is converted to ' These conversion occur right after asciidoc conversion but before storing to raw.xml files. So this is purely work around for asciidoc idiosyncrasies. == What is @-@ in asciidoc and raw XML == You see quite a bit of @-@ in asciidoc and generated XML source. That is placed there when auto-generated data replaces them as entity in the final form (*.xml, *.po, *.html, ...). For example: @-@foo@-@ is converted to &foo; @-@popcon1@-@ is converted to popcon value and URL link of package in column 1. @-@psize1@-@ is converted to size value and URL link of package in column 1. @-@popcon2@-@ is converted to popcon value and URL link of package in column 2. @-@psize2@-@ is converted to size value and URL link of package in column 2. These survive asciidoc conversion but are transformed from raw.xml to en.xml. == Making copy of SVN contents == If you just need readonly access to SVN contents, you can do this with: $ mkdir -p path/to/use/ $ cd path/to/use/ $ svn co svn://svn.debian.org/svn/ddp/manuals/trunk/quick-reference qref If you need read/write access, please follow me ... DR is part of Debian Documentation Project (DDP) at http://www.debian.org/doc/ddp . DDP uses alioth service. If you do not have alioth account (i.e. non DD), you can set up by clicking "New account" at the top right corner. You can join DDP by clicking [Request to join] at http://alioth.debian.org/projects/ddp/ and send your request to join. You should send short reminder to debian-doc@lists.debian.org . Then you can get alioth account activated for DDP. Please use English page etc. (Otherwise we get non-English message) You should set up SSH (passwordless login). Then: $ mkdir -p path/to/use/ $ cd path/to/use/ $ svn co svn+ssh://svn.debian.org/svn/ddp/manuals/trunk/qick-reference qref == Updating contents in English XML source == Typical work flow of creating latest XML source from asciidoc/po4a is: * Get latest source with "svn up" to sync with latest upstream. * Edit "asciidoc/*.txt" files to update English contents if needed. * Run "make xml". --> It runs msgmerge for you. * Edit "po4a/debian-reference.??.po" to update translation if needed. --> remove "fuzzy" while fixing translation. * Run "make all" for test build html, --> If error, back to Edit "asciidoc/*.txt" files. * Save updated XML source with "svn ci". * Clean source as needed with "make clean". * Test build package using "debuild" and "pdebuild". For editing "po4a/debian-reference.??.po", use specialized po file editor such as poedit for ease. == Fixing typo in English XML source == If you find typo in English and you know its fix should not affects translation, you can do following: * Get latest source with "svn up". * Fix source as needed for merge errors. * Run "make xml" to be sure. via poedit GUI--> It runs msgmerge for you. * Save updated XML source with "svn ci" to sync with upstream. * Edit "asciidoc/*.txt" files to typo English contents if needed. * Run "make typo". --> It runs msguntypot for you. * Run "make all" for test build html --> If error, back to Edit "asciidoc/*.txt" files. * Save updated XML source with "svn ci". == Updating remote data contents == You need remote connection to update popcon, package size, and other remotely fetched data. * Get latest source with "svn up". * Run "make distclean new" to remove remote data contents. * Run "make all" for test build html * Save updated XML source with "svn ci". == Creating Translation PO for a new language == * Get latest source with "svn up". * Run "make xml" to be sure. * Run "cd po4a; cp debian-reference.pot debian-reference.??.po". * Save updated XML source with "svn ci". To test run for language ??, "make xml LANGPO=??". Once translation becomes in good shape, update LANGPO value in Makefile to include ?? . == Updating Translation PO == * Get latest source with "svn up". * If conflicts happen for PO file, seek to merge remote and local one with msguniq and msgcat as described below. (Line numbers are not important to gettext. So do not worry. It is only for translators convienience.) * Run "make xml" to be sure. * Edit "po4a/debian-reference.??.po" to update translation if needed. * Run "make all" for test build html, --> If error, back to Edit "po4a/debian-reference.??.po" files. * Save updated XML source with "svn ci". If your language is not official one yet, you should use: $ make xml "LANGPO=fr" to update french for example. == Updating for new release (squeeze --> squeeze+1) == * Update common.ent target in Makefile to new release names. * Change English source to debian-reference.raw.xml or other pre-XML format. == URL used for translation and its initialization == URL related PO entries are at the top of PO files. They are auto-generated from raw.xml files. If there are more than 1 entity with the same content, they are listed together with comments containing accumulated line numbers and the first entity name only. English document should try to use URLs, whenever possible, which select language automatically as follows. * http://www.debian.org/ * http://www.debian.org/doc/manuals/debian-reference/ * http://www.debian.org/doc/manuals/project-history/ch-leaders These should be kept for translation. (msgid == msgstr) msgid "http://www.debian.org/" msgstr "http://www.debian.org/" For Wikipedia, you should find matching entry for translation. It does not work as easy as Debian documentation. So, PO files should be used to translate URL as follows. In case of UTF-8 characters, use original ones instead of ones with "%" for readability. For Japanese: msgid "http://en.wikipedia.org/wiki/Application_binary_interface" msgstr "http://ja.wikipedia.org/wiki/Application_Programming_Interface" For Japanese: msgid "http://en.wikipedia.org/wiki/Wine_(software)" msgstr "http://ja.wikipedia.org/wiki/Wine" For French: msgid "http://en.wikipedia.org/wiki/Desktop_environment" msgstr "http://fr.wikipedia.org/wiki/Environnement_de_bureau" The initialization of these URL is non-trivial if you do this manually. We initialize URL using English document. Let's take French as example. Easiest way to update debian-reference.fr.po is as follows. $ msgen --no-wrap -o debian-reference.en.po debian-reference.pot $ mv debian-reference.fr.po debian-reference.fr.po.old $ sed -e '/msgid "en"/$,d' Debian", do not change &debian; to translated URL. But look for the following PO file line for URL such as "http://www.debian.org/" and translate it in msgstr "". For this case, automatic locale choice is active on this URL thus it is best not to change as previously stated. If you find no corresponding URL for your language, do not forget to point it to one of "http://en.wikipedia.org/wiki/...". == Utilities for spell check == * spell check efficiently using "wordlist" $ zcat debian-reference.en.txt.gz | wordlist > word.txt $ gedit word.txt ... (autospell check) ... find bad word ... use grep to find text using it ... fix by editor on source == Utilities of PO (split PO etc.) == * Check sanity of PO msgcat some.po >/dev/null I suggest to look for unquoted " and accidental folding of lines. * Reformat PO to one-line msgcat --no-wrap some.po >some.po.new This enables consistent formating. Use wdiff to find changes. * Clean up po for duplicate mv some.po some.old.po msguniq --no-wrap some.old.po >some.po * Merge 2 POs PO files do not merge well if you simply do "svn up". In such case, you need to merge remote one and your local one using gettext tools. If you have no order of preference. $ msgcat --no-wrap some1.po some2.po >some.po Look for #-#-#-#-# where manual resolution is needed. If you line some1.po better than some2.po. $ msgcat --no-wrap --use-first some1.po some2.po >some.po * "po4a-split" to split PO file into several chunks $ po4a-split file.po == VCS locations and history == This DR source is maintained in alioth ddp SVN repository now. VCS usage of Debian Reference (DR) have been changed with time. This can be summarized approximately as follows: 1. DR ver 1 (ALL SGML source) * Souceforge CVS as primary * ddp CVS having copy of Souceforge CVS 2. DR ver 2 (lenny) (XML=English source, SGML=other sources) * Souceforge CVS stalled * ddp CVS having copy of old Souceforge CVS --> This was moved to ddp SVN * git.debian.org having latest source 3. DR ver 2 (squeeze) (XML=English Japanese sources, SGML=other sources) * Souceforge CVS stalled * ddp SVN latest content * git.debian.org stalled Latest SVN repository contents can be inspected/cloned as: $ svn co svn://svn.debian.org/svn/ddp/manuals/trunk manuals # SVN (ro) $ svn co svn+ssh://svn.debian.org/svn/ddp/manuals/trunk manuals # SVN+SSH (r/w) http://svn.debian.org/wsvn/ddp/manuals/trunk/quick-reference/ http://www-master.debian.org/build-logs/ddp/ DDP build results Latest source source package can be downloaded by $ apt-get source debian-reference Older lenny GIT repository contents can be inspected/cloned as: $ sensible-browser http://git.debian.org/?p=users/osamu/debian-reference.git # web $ git clone ssh://git.debian.org/~osamu/public_git/debian-reference.git debian-reference # r/w $ git clone ssh://git.debian.org/git/users/osamu/debian-reference.git debian-reference # r/w $ git clone git://git.debian.org/~osamu/debian-reference.git debian-reference # ro $ git clone git://git.debian.org/git/users/osamu/debian-reference.git debian-reference # ro $ git clone http://git.debian.org/git/users/osamu/debian-reference.git debian-reference # ro Osamu # vim: set tw=78 sts=2 ai expandtab: