Manuel A. Fernandez Montecelo :: Personal Debian page - planet-debianhttps://people.debian.org/~mafm/2019-12-11T02:23:00+00:00Debian GNU/Linux riscv64 port -- Sponsors and Build machines2019-12-11T02:23:00+00:002019-12-11T02:23:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2019-12-11:/~mafm/posts/2019/20191211_debian-gnulinux-riscv64-port-sponsors-and-build-machines/<p>In previous posts about the <code>riscv64</code> port there were mentions about history,
progress and other details, but in this one I want to address the topic of
sponsors and build machines, which even if there are mentions from time to time
(e.g. in talks and slides posted here), it has not been covered in a
comprehensive manner.</p>
<p>And it's only fair that we acknowledge people …</p><p>In previous posts about the <code>riscv64</code> port there were mentions about history,
progress and other details, but in this one I want to address the topic of
sponsors and build machines, which even if there are mentions from time to time
(e.g. in talks and slides posted here), it has not been covered in a
comprehensive manner.</p>
<p>And it's only fair that we acknowledge people and orgs sponsoring and
contributing resources... and about time too. They will appear roughly in
chronological order.</p>
<h2>Bytemark</h2>
<p>From the early days in 2015, when the plans for the port were still very
preliminary, we got support from <a href="https://www.bytemark.co.uk/">Bytemark</a>, which
<a href="https://www.bytemark.co.uk/company/sponsorships/">already supports Debian and other FOSS
projects</a>, and important parts
of the Debian infrastructure are hosted there.</p>
<p>In particular, they sponsored a virtual machine with a good amount of CPU, RAM
and disk space (and good connectivity and great bandwidth with Debian repos
<strong>:-)</strong>) in which many of the early attempts to bootstrap happened, after things
started to get beyond personal laptop territory (see <a href="https://people.debian.org/~mafm/posts/2017/20170422_debian-gnulinux-port-for-risc-v-64-bit-riscv64/">Debian GNU/Linux port for
RISC-V 64-bit
(riscv64)</a>
for more details).</p>
<p>This has been acknowledged here: <a href="https://wiki.debian.org/RISC-V#Credits">Debian RISC-V's wiki page, “Credits”
section</a>:</p>
<blockquote> “Bytemark provides hardware to help to kick-start this
port. Bytemark is a long-time partner of Debian” </blockquote>
<p>and in the post <a href="https://people.debian.org/~mafm/posts/2017/20170422_debian-gnulinux-port-for-risc-v-64-bit-riscv64/">Debian GNU/Linux port for RISC-V 64-bit
(riscv64)</a>
mentioned before, and in slides of talks.</p>
<h2>Kurt Keville, MIT's CSAIL</h2>
<p><strong>Kurt Keville</strong> from <a href="https://web.mit.edu/">MIT</a>'s
<a href="https://www.csail.mit.edu/">CSAIL</a> was helping since early on, in 2016.</p>
<p>He has very kind to help me to travel and give <a href="https://people.debian.org/~mafm/posts/2018/20180608_talk-about-the-debian-gnulinux-riscv64-port-at-risc-v-workshop/">talks about the Debian GNU/Linux
riscv64
port</a>
in RISC-V so-called “workshops” in MIT/CSAIL in 2016 and UPC/Barcelona
Supercomputing Centre in 2018.</p>
<p>Also, and apart from other great help that can never be paid off like
encouragement to go ahead in important moments, in 2018 he set up and couple of
big-ish servers, the network connectivity is excellent and there's also a local
Debian mirror, which is great.</p>
<p>Here we host the <strong>rv-mit-01</strong> to <strong>-18</strong> VMs (see <a href="https://buildd.debian.org/status/architecture.php?a=riscv64">buildd status
pages</a>), building
packages “natively” (inside Qemu, but emulating a <code>riscv64</code> machine). These
were the main builders for most of 2018 of all suites (basically, <code>unstable</code>
─which is the most important and through which all packages enter the archive
that it will eventually be settled as <code>stable</code>─, and <code>experimental</code>).</p>
<p>In the last few months most of <code>unstable</code> is built by hardware machines, but
these ones are still busy with <code>experimental</code> suite, and kept as back-up if
other machines fail, or in cases of extended peak times, we can easily tell them
to start processing packages from <code>unstable</code> again.</p>
<p>This support was really crucial in the early days, without these important
resources the port would probably not be able to lift off. It has been
acknowledged in the past, but only in talks.</p>
<h2>SiFive</h2>
<p><a href="https://www.sifive.com/">SiFive</a> is company created by many of the minds from
UC Berkeley who conceived and created the RISC-V architecture.</p>
<p>At the moment, they are the only to offer hardware that can be purchased
publicly which can be used to build packages ─and as a general purpose GNU/Linux
riscv64 system, for that matter─, the <a href="https://www.sifive.com/boards/hifive-unleashed">HiFive
Unleashed</a>.</p>
<p>They sponsored one board soon after it was premiered at FOSDEM in 2018, which is
named <strong>rv-hfu-01</strong> and hosted by me (though not attached to the buildd network,
but sometimes appears mentioned in build logs or bug reports), in which happen
many of the tries and special builds of packages with difficulties (e.g. builds
for <code>unreleased</code> suite special of debian-ports, with some functionality disabled
or to break dependency cycles, etc.).</p>
<p>They also sponsored other HiFive Unleashed boards, like one intended to become a
<em>porterbox</em> (still in the works) and another one named <strong>rv-rr44-01</strong> and hosted
by <strong>Aurélien Jarno</strong> (there are plans to move them elsewhere, but it's
convenient to have them close to one's hand, in case of problems, like frequent
lock-ups).</p>
<p><strong>rv-rr44-01</strong> is part of the buildd network, and appears listed in the <a href="https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-rr44-01">buildd
status page for
rv-rr44-01</a>,
so you can see the stats and what it's currently building.</p>
<p>Support from SiFive has been acknowledged in the past, but only in talks.</p>
<h2>Aurélien Jarno</h2>
<p>Aside from lots of personal time and skill contributing to this port, and
debian-ports and Debian in general, there are two VMs that have been hosted and
managed by him, that have been building packages for <code>unstable</code> and
<code>experimental</code> since the early days too, along with the <strong>rv-mit-</strong> ones.</p>
<p>They are now hosted elsewhere (but kept their names), more on this below.</p>
<h2>GCC Compile Farm Project / Oregon State University Open Source Lab (OSUOSL)</h2>
<p>In mid-2018 I was approached by <strong>Laurent Guerby</strong>, from the <a href="https://gcc.gnu.org/wiki/CompileFarm">GCC Compile Farm
Project</a>, to offer another two machines
that they would host and we could use.</p>
<p>The GCC Compile Farm Project is a facilitator for free software developers to
get access to resources, mosty to <em>compile</em>/build software projects, as the name
implies. The management system for this project is hosted by
<a href="https://tetaneutral.net/">Tetaneutral</a>, and the machines themselves are hosted
by various organizations around the world. These two machines will actually be
hosted by the <a href="https://osuosl.org/">Oregon State University Open Source Lab
(OSUOSL)</a>. From <strong>Lance Albertson</strong>, OSL's director:</p>
<blockquote> “These machines are owned and hosted by the Oregon State University
Open Source Lab (OSUOSL). The OSUOSL is a non-profit organization and provides
infrastructure hosting for over 160 open source projects including those of
worldwide leaders like the Apache Software Foundation, the Linux Foundation and
Drupal.” </blockquote>
<p>It turns out that the machines are very similar to those at MIT that we were
using, so it was basically doubling resources available.</p>
<p>By the time that we could set the machines up it was the start of “freeze”
period for <strong>buster</strong> release, so basically we moved Aurélien's
<strong>rv-aurel32-01</strong> and <strong>-02</strong> there instead of adding new VMs. Since then and
at the moment they are helping to build <code>unstable</code>, because the hardware
machines cannot cope in peak times.</p>
<p>Links to their buildd status pages:</p>
<ul>
<li><a href="https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-aurel32-01">https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-aurel32-01</a></li>
<li><a href="https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-aurel32-02">https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-aurel32-02</a></li>
</ul>
<p>We also use these machines for some work done by <a href="https://wiki.debian.org/HelmutGrohne/rebootstrap">Helmut Grohne's rebootstrap
project</a>, because it's been
very helpful to bootstrap RISC-V and we want ports in the future to be even more
easy to bootstrap and adopt in Debian <strong>:-)</strong></p>
<h2>Mullvad</h2>
<p><a href="https://mullvad.net/en/">Mullvad</a> is a VPN service from Sweden. They are
providing us with HiFive Unleashed boards, rack space, network, ancillary
hardware, and staff time on site to allow us to manage the boards remotely.</p>
<p>At the moment <strong>rv-mullvad-01</strong> and <strong>-02</strong>, which are part of the buildd
network, are hosted by Mullvad (the hostnames are a bit of a give-away, aren't
they?). We were trying to get more boards there in the recent past, and while
it has not been possible so far, we expect to have more boards there in the
future.</p>
<p>Thanks to this, <strong>it has been possible since about mid-2019 to switch and have
most packages built by hardware machines</strong> (all of them HiFive Unleashed
boards), instead of mostly VMs until then.</p>
<p>This is a very important step in the progress of the port to hopefully become
first-class at some point in the future, and this would not have been possible
without support from Mullvad.</p>
<p>Their support had not been acknowledged publicly, so it was about time and one
of the reasons of this post.</p>
<p>Links to their buildd status pages:</p>
<ul>
<li><a href="https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-mullvad-01">https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-mullvad-01</a></li>
<li><a href="https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-mullvad-02">https://buildd.debian.org/status/architecture.php?a=riscv64&buildd=buildd_riscv64-rv-mullvad-02</a></li>
</ul>
<h2>And finally...</h2>
<p>... if you want to contribute in any way, please get it touch.</p>
<p>Work keeps happening and there are news in other areas that would be worth
mentioning, but I really wanted to keep focus in this post on sponsors and build
machines, so that's all for now.</p>
<p>Happy Building!</p>Debian GNU/Linux riscv64 port in mid 20192019-06-17T23:52:00+00:002019-06-17T23:52:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2019-06-17:/~mafm/posts/2019/20190617_debian-gnulinux-riscv64-port-in-mid-2019/<p>It's been a while since last post (<a href="https://people.debian.org/~mafm/posts/2018/20180608_talk-about-the-debian-gnulinux-riscv64-port-at-risc-v-workshop/">Talk about the Debian GNU/Linux riscv64 port
at RISC-V
workshop</a>),
and sometimes things look very quiet from outside even if the people on the
backstage never stop working. So this is an update on the status of this port
before the release of <a href="https://www.debian.org/releases/buster/">buster</a>, which
should happen in a few weeks and which it will open the way …</p><p>It's been a while since last post (<a href="https://people.debian.org/~mafm/posts/2018/20180608_talk-about-the-debian-gnulinux-riscv64-port-at-risc-v-workshop/">Talk about the Debian GNU/Linux riscv64 port
at RISC-V
workshop</a>),
and sometimes things look very quiet from outside even if the people on the
backstage never stop working. So this is an update on the status of this port
before the release of <a href="https://www.debian.org/releases/buster/">buster</a>, which
should happen in a few weeks and which it will open the way for more changes
that will benefit the port.</p>
<h2>The Big Picture</h2>
<p>First, the big picture(s):</p>
<p style="width: 95%; margin: auto; text-align: center">
<em>Debian-Ports All-time Graph, 2019-06-17</em>
</p>
<p><img alt="Debian-Ports All-time Graph, 2019-06-17" src="https://people.debian.org/~mafm/static/images/graph-ports-big-20190617.png" style="width: 90%" title="Debian-Ports All-time Graph, 2019-06-17"></p>
<p>As it can be seen in the first graph, perhaps with some difficulty, is that the
percent of arch-dependent packages built for <code>riscv64</code> (grey line) has been
around or higher than 80% since mid 2018, just a few months after the port was
added to the infrastructure.</p>
<p>Given than the arch-dependent packages are about half of the Debian['s main,
unstable] archive and that (in simple terms) arch-independent packages can be
used by all ports (provided that the software that they rely on is present,
e.g. a programming language interpreter), this means that around 90% of packages
of the whole archive has been available for this architecture from early on.</p>
<p style="width: 95%; margin: auto; text-align: center">
<em>Debian-Ports Quarter Graph, 2019-06-17</em>
</p>
<p><img alt="Debian-Ports Quarter Graph, 2019-06-17" src="https://people.debian.org/~mafm/static/images/graph-ports-quarter-big-20190617.png" style="width: 90%" title="Debian-Ports Quarter Graph, 2019-06-17"></p>
<p>The second graph shows that the percentages are quite stable (for all
architectures, really: the peaks and dips in the graph only represent <5% of the
total). This is in part due to the freeze for <code>buster</code>, but it usually happens
at other times as well (except in the initial bring-up or in the face of severe
problems), and it really shows that even the <em>second-class</em> ports are in quite
good health in broad terms.</p>
<p><br/>
<strong>Note</strong>: These graphs are for architectures in the <code>debian-ports</code>
infrastructure (which hosts architectures not as well supported as the main
ones, the ones present in stable releases). The graphs are taken from the
<a href="https://buildd.debian.org/stats/">buildd stats page</a>, which also includes the
main supported architectures.</p>
<h3>A little big Thank You</h3>
<p>Together, both graphs are also testament that there are people working on ports
at all times, keeping things working behind the scenes, and that's why from a
high level view it seems that things “just work”.</p>
<p>More in general, aside from the work of porters themselves, there are also
people working on bootstrapping issues that make bringing up ports easier than
in the past, or coping better when toolchain support or other issues deal an
important blow to some ports. And, of course, all other contributors of Debian
help by keeping good tools and building rules that work across architectures,
patching the upstream software for needs of several architectures at the same
time (endianness, width of basic types), many upstream projects are generic
enough that they don't need specific porting, etc.</p>
<p><strong>Thanks to all of you!</strong></p>
<h2>Next Steps</h2>
<h3>Installation on hardware, VMs, etc.</h3>
<p>Due to several reasons, among them the limited availability of hardware able to
run this Debian port and the limited options to use bootloaders during all this
time, the instructions to get Debian running on RISC-V are not the best,
easiest, more elegant or very up to date. This is an area to improve in the
next months.</p>
<p>Meanwhile, there's a <a href="https://wiki.debian.org/RISC-V#OS_.2F_filesystem_images">Debian RISC-V's wiki
page</a> with instructions
to get a <code>chroot</code> working in a <strong>HiFive Unleashed</strong> board as shipped, without
destroying the initial factory set-up.</p>
<p>Specially Vagrant Cascadian and Karsten Merker have been working on the area of
booting the system, and there are instructions to <a href="https://wiki.debian.org/RISC-V#Setting_up_a_riscv64_virtual_machine">set-up a riscv64 Qemu
VM</a> and
boot it with <a href="https://wiki.debian.org/RISC-V#Setting_up_a_riscv64_virtual_machine_with_u-boot_and_opensbi">u-boot and
opensbi</a>.
Karsten is also working to get support in <code>debian-installer</code>, the main/canonical
way to install Debian systems (perhaps less canonical nowadays with the use of
OS images, but still hugely important).</p>
<p>Additionally, it would be nice to have images publicly available and ready to
use, for both <strong>Qemu</strong> and hardware available like the <strong>HiFive Unleashed</strong> (or
others that might show up in time), but although there's been some progress on
that, it's still not ready and available for end users.</p>
<h3>The last 10%+ of the archive</h3>
<p>So, what's the remaining work left to have almost all of the archive built for
this architecture? What's left to port, as such?</p>
<p>The main blockers to get closer to 100% of packages built are basically LLVM and
Rust (which, in turn, depends on LLVM).</p>
<p>Currently there are more than 500 packages from the Rust ecosystem in the
archive (which is about 4%), and they cannot be built and used until Rust has
support for the architecture. And Rust needs LLVM, there's no Rust compiler
based on GCC or other toolchains (as it's the case of Go, for example, in which
there's a <code>gcc-go</code> compiler in addition to their own <code>golang-go</code>), so this is
the only alternative.</p>
<p>Firefox is the main high-level package that depends on Rust, but many packages
also depend on <code>librsvg2</code> to render SVG images, and this library has been
converted to Rust. We're still using the C version for that, but it cannot be
sustained in the long term.</p>
<p>Aside from Rust, other packages directly depend or use LLVM to some extent, and
this is not fully working for riscv64 at the moment, but it is expected that
during 2019 the support of LLVM for riscv64 will be completed.</p>
<p>There are other programming language ecosystems that need attention, but they
represent a really low percentage (only dozens of packages, of more than 12
thousand; and with no dependencies outside that set). And then, of course,
there is long tail of packages that cannot be built due to a missing dependency,
lack of support for the architecture or random failures -- together they make a
substantial number of the total, but they need to be looked at and solved almost
on a case-by-case basis.</p>
<p>Finally, when the gates of the <code>unstable</code> suite open again after the freeze for
the stable release of <code>buster</code>, we will see tools with better support and
patches can be accepted again to support riscv64, so we can hope that things
will improve at a faster rate soon <strong>:-)</strong></p>Talk about the Debian GNU/Linux riscv64 port at RISC-V workshop2018-06-08T21:20:00+00:002018-06-08T21:20:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2018-06-08:/~mafm/posts/2018/20180608_talk-about-the-debian-gnulinux-riscv64-port-at-risc-v-workshop/<p>About a month ago I attended the RISC-V workshop (conference, congress)
co-organised by the Barcelona Supercomputing Center (BSC) and Universitat
Politècnica de Catalunya (UPC).</p>
<p>There I presented a talk with the (unimaginative) name of <strong><em>“Debian GNU/Linux
Port for RISC-V 64-bit”</em></strong>, talking about the same topic as many other posts of
this blog.</p>
<p>There are 2-3 such RISC-V Workshop events per year, one somewhere in Silicon …</p><p>About a month ago I attended the RISC-V workshop (conference, congress)
co-organised by the Barcelona Supercomputing Center (BSC) and Universitat
Politècnica de Catalunya (UPC).</p>
<p>There I presented a talk with the (unimaginative) name of <strong><em>“Debian GNU/Linux
Port for RISC-V 64-bit”</em></strong>, talking about the same topic as many other posts of
this blog.</p>
<p>There are 2-3 such RISC-V Workshop events per year, one somewhere in Silicon
Valley (initially at UC Berkeley, its birthplace) and the others spread around
the world.</p>
<p>The demographics of this gathering are quite different to those of
planet-debian; the people attending usually know a lot about hardware and often
Linux, GNU toolchains and other FOSS, but sometimes very little about the inner
workings of FOSS organisations such as Debian. My talk had these demographics
as target, so a lot of its content will not teach anything new for most readers
of planet-debian.</p>
<p>Still, I know that some readers are interested in parts of this, now that the
slides and videos are published, so here it is:</p>
<ul>
<li><strong><em>“Debian GNU/Linux Port for RISC-V 64-bit”</em></strong><ul>
<li><a href="https://content.riscv.org/wp-content/uploads/2018/05/15.25-15.55-08.05.2018-debian-GNU-Manual-A-Fernandez-Montecelo.pdf">slides</a></li>
<li><a href="https://www.youtube.com/watch?v=hI5EAlLh--Q">video</a></li>
</ul>
</li>
<li>Rest of the event schedule, slides and videos (incl. f.ex. a talk about a
port for Fedora):<ul>
<li><a href="https://riscv.org/2018/05/risc-v-workshop-in-barcelona-proceedings/">https://riscv.org/2018/05/risc-v-workshop-in-barcelona-proceedings/</a></li>
</ul>
</li>
</ul>
<p>Also very relevant is that they were using Debian (our very own riscv64 port,
recently imported into debian-ports infra) in two of the most important hardware
demos in the corridors. The rest were mostly embedded distros to showcase FPS
games like Quake2, Doom or similar.</p>
<p><br/>
All the feedback that I received from many of the attendees about the
availability of the port was very positive and they were very enthusiastic,
basically saying that they and their teams were really delighted to be able to
use Debian to test their different prototypes and designs, and to drive
development.</p>
<p>Also, many used Debian daily in their work and research for other purposes, for
example a couple of people were proudly showing to me Debian installed on their
laptops.</p>
<p>For me, this feedback is a testament of how much of what we do everyday matters
to the world out there.</p>
<p><br/>
For the historical curiosity, I also presented a similar talk in a previous
workshop (2 years back) at CSAIL / MIT.</p>
<p>At that time the port was in a much more incipient state, mostly a proof of
concept (for example the toolchain had not even started to be upstreamed).
Links:</p>
<ul>
<li><strong><em>“Working Towards a Debian RISC-V Port”</em></strong><ul>
<li><a href="https://content.riscv.org/wp-content/uploads/2016/07/Wed1115_Working_Towards_a_Debian_RISC-V_Port.pdf">slides</a></li>
<li><a href="https://youtu.be/NBYbHqPNHGs">video</a></li>
</ul>
</li>
<li>Talks / schedule, also with slides and videos:<ul>
<li><a href="https://riscv.org/2016/07/4th-risc-v-workshop-proceedings/">https://riscv.org/2016/07/4th-risc-v-workshop-proceedings/</a></li>
</ul>
</li>
</ul>Debian GNU/Linux port for RISC-V 64-bit (riscv64) in Debian infrastructure (debian-ports)2018-04-02T01:20:00+00:002018-04-02T01:20:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2018-04-02:/~mafm/posts/2018/20180402_debian-gnulinux-port-for-risc-v-64-bit-riscv64-in-debian-infrastructure-debian-ports/<p><em>tl;dr</em>: We have a new port for <a href="https://wiki.debian.org/RISC-V">RISC-V</a>,
flavour <code>riscv64</code> (64-bits little-endian) in <a href="https://www.ports.debian.org/">Debian
Ports</a>.</p>
<p>And it's doing well.</p>
<p style="width: 95%; margin: auto; text-align: center">
<em>Debian-Ports 2-week Graph, 2018-03-31</em>
</p>
<p><img alt="Debian-Ports 2-week Graph, 2018-03-31" src="https://people.debian.org/~mafm/static/images/graph-ports-week-big-20180331.png" style="width: 90%" title="Debian-Ports 2-week Graph, 2018-03-31"></p>
<p>(PS: Despite the date still in some timezones... no, this is not an
April-Fools-day type of post :-) ).</p>
<h2>Ancient History</h2>
<p>Almost a year ago I wrote a post announcing the availability of a <a href="https://people.debian.org/~mafm/posts/2017/20170422_debian-gnulinux-port-for-risc-v-64-bit-riscv64/">Debian
GNU/Linux port for RISC-V 64-bit
(riscv64)</a>.</p>
<p>At the …</p><p><em>tl;dr</em>: We have a new port for <a href="https://wiki.debian.org/RISC-V">RISC-V</a>,
flavour <code>riscv64</code> (64-bits little-endian) in <a href="https://www.ports.debian.org/">Debian
Ports</a>.</p>
<p>And it's doing well.</p>
<p style="width: 95%; margin: auto; text-align: center">
<em>Debian-Ports 2-week Graph, 2018-03-31</em>
</p>
<p><img alt="Debian-Ports 2-week Graph, 2018-03-31" src="https://people.debian.org/~mafm/static/images/graph-ports-week-big-20180331.png" style="width: 90%" title="Debian-Ports 2-week Graph, 2018-03-31"></p>
<p>(PS: Despite the date still in some timezones... no, this is not an
April-Fools-day type of post :-) ).</p>
<h2>Ancient History</h2>
<p>Almost a year ago I wrote a post announcing the availability of a <a href="https://people.debian.org/~mafm/posts/2017/20170422_debian-gnulinux-port-for-risc-v-64-bit-riscv64/">Debian
GNU/Linux port for RISC-V 64-bit
(riscv64)</a>.</p>
<p>At the time it was an incomplete Debian system, with several of the most
important pieces missing (toolchain: <code>gcc</code>, <code>glibc</code>, <code>binutils</code>), and with
everything kept outside the Debian infrastructure.</p>
<p>The main reason for that was because support for the toolchain had not been
submitted upstream, and the ABI was not very stable. It was changing often and
in important and disruptive ways, without even as much of a public announcement
before the breakage was noticed.</p>
<h2>Last pieces of the toolchain upstreamed, at long last</h2>
<p>Fortunately, over the last year, support has been merged in GNU binutils first
(2.28), then GCC (7), then Linux 4.15 (pre-requisite for glibc), and finally in
glibc 2.27, released in the first days of February.</p>
<p>Support is still not complete, for example <code>riscv32</code> targets have not been
merged in Linux mainline (but this is not important for us, unless someone else
wants to start such a port), basic drivers are still missing from mainline, the
support in GDB and Qemu is becoming available only now, there are many bugs that
need shaking, etc.</p>
<h2>Bootstrapping again, 2018 edition</h2>
<p>Despite the very recent support, the ecosystem and software support is mature
enough that we could, with the only help of <code>qemu-system-riscv64</code>, progress
through the following steps:</p>
<h4>1. Cross-compile a base set of packages</h4>
<p><em>Dates: prior work for a few years (almost since the public announcement of
RISC-V, in 2014); current bootstrapping since end of Feb until ~5th of March.</em></p>
<p>Apart from previous bootstrapping attempts, during the last months prior to
proper starting this one I worked on clearing the path, for example by NMUing
(Debian lingo for fixing without “ack” from maintainer, no reply in
weeks/months, or “ack” agreeing that I went ahead) packages with patches
proposed by Helmut and porters of other architectures (e.g. many from Daniel
Schepler) and without any reply for a long time, or sometimes proposing new
patches or new ways to solve the problems, etc.</p>
<p>After that, at the end of February and along with Karsten and Aurélien, we went
through a round of cross-compiling, mostly with the help of <a href="https://wiki.debian.org/HelmutGrohne/rebootstrap">Helmut Grohne's
rebootstrap</a>, but also solving
additional problems like using gcc-8 rather than gcc-7 because it failed to work
properly for a few days, also pulling parts of the toolchain from <code>experimental</code>
instead of <code>unstable</code> and having different versions around causing conflicts, an
<code>experimental</code> package of glibc 2.27 still not present in <code>unstable</code> at the
time, which rendered packages such as GNU make unable to compile without
patching, etc.</p>
<p>We also cross-compiled a significantly higher amount of packages than those
considered by <code>rebootstrap</code> at the moment; some of them with crude hacks, like
removing build-dependencies or avoiding to build man pages when they needed to
run <code>riscv64</code> native code for that. Sometimes is just faster to cross-compile
and have many packages ready for when they are needed needed, even if sometimes
cross-built packages don't work 100% correctly or not at all.</p>
<p>Many of those package are not essential to <em>run</em> a Debian system, but are needed
later in the initial steps of “native” <em>building</em>. For example, the
compiler/toolchain itself, which the <code>rebootstrap</code> script doesn't attempt to
provide; or GNU <code>make</code> or <code>cmake</code>, which are often not needed to <em>run</em> a Linux
system, but are required to <em>build</em> a great deal of software.</p>
<p>As a result of this phase, we have submitted bugs (or in some cases, uploads
after a period of silence), with many still pending to send or apply, to try to
sort these problems out in a clean way for other ports in the future (and better
rebootstrapping and cross-compilation support in Debian overall), with the use
of different tools like <a href="https://wiki.debian.org/BuildProfileSpec">build
profiles</a> to avoid unwanted
dependencies or to break dependency cycles, or moving the generation of
documentation to an arch:all package, etc (so it's done once for all
architectures, and <code>riscv64</code> or any other arch-builders don't have to bother
about it).</p>
<h4>2. Use this set to launch “native” systems with <code>qemu-system-riscv64</code></h4>
<p><em>Dates: ~5th to ~13th of March.</em></p>
<p>Having built a “native” system from cross-built packages, we could compile
packages natively that were difficult to cross-build, and also to run tests for
packages providing them (so one can have some certainty that the package behaves
sanely), and trying to have as much a clean native system as possible.</p>
<p>For example, one of the first packages needed and that we built natively is Perl
(which is pretty basic for any Debian systems, and that we couldn't get to
cross-compile easily).</p>
<p>We recompiled all (or almost all) packages that we got cross-built “natively”,
and by running tests when possible, we uncovered some latent bugs in the
toolchain and <code>qemu</code>, and in some packages.</p>
<p>The result was a set of packages (that we named “native-1”) enough to run
systems equivalent to those installed in machines acting as automatic builders
for other architectures, capable of building and running native "sbuild",
"schroot", etc. We had to cut some unimportant dependency or feature here and
there, but those changes did not affect the packages of the next set, or only
marginally.</p>
<h4>3. Build “native-2” to import to <code>debian-ports</code></h4>
<p><em>Dates: ~13th to ~23rd of March.</em></p>
<p>With systems installed from “native-1”, we have built again a restricted set of
packages for a very basic Debian system, to act as “seed”, with as few
differences as possible with the current state of Debian unstable, with the aim
to import them in the Debian-ports infrastructure as cleanly as possible.</p>
<p>Intentionally, we built as few packages as possible, so most of the packages
would go through the standard mechanisms.</p>
<p>A few packages have been built with the help of build-profiles support, for
example to avoid dependencies on Java's <code>default-jdk</code> (<code>openjdk-9-jdk</code> at the
moment), because we do not have that one yet.</p>
<p>Another typical case is disabling tests, after attemping to pass them, because a
few cases failed among hundreds of them passed, often due to timeouts (system
emulation being quite slow, this happens very often).</p>
<p>But most packages have been built completely unmodified; and those who were,
will be rebuilt later when their dependencies are satisfied.</p>
<h4>4. Import the base set “native-2” as “seed” and set-up automatic systems</h4>
<p><em>Dates: ~23rd to ~25th of March.</em></p>
<p>Set everything up to build the whole archive in as much of a standard and
unattended way as possible.</p>
<h4>5. Currently, solve problems like breaking dependency cycles</h4>
<p><em>Dates: ~25th of March, still on it.</em></p>
<p>In the current phase, the process as a whole is not “unattended”, because we
have to build packages with a reduced set of features or build-dependencies, to
break cycles (e.g. <code>cmake</code> depending on <code>qt</code> for its GUI tool, <code>qt</code> ecosystem
depending on <code>cmake</code>).</p>
<p>To get close to 100% of the archive built, there are many important dependency
cycles yet to break, some packages need patches specific to the architecture,
etc.</p>
<p>Nevertheless, thousands of packages have been built by now, most of them
completely unattended and in the same way as the other architectures available
in Debian.</p>
<h2>Current Status</h2>
<p>In last few days and weeks, status was sometimes changing by the hour. That,
along with -ENOTIME, are the main reasons why this post has been delayed until
now, and the news have not been publicised widely.</p>
<p>We just focused on getting things done :-)</p>
<p>But to give a general idea of where we are now:</p>
<ul>
<li>Over 4100 packages (> 30% of the source-packages of the whole archive
that are not arch-independent) have been built by now.<ul>
<li>Since arch-independent packages can be installed in this architecture
(e.g. many modules of languages like Python, documentation, etc.), about
65 to 70% of the Debian archive is available to use.</li>
<li>As explained before, most of the packages have been built completely
unattended and in the same way as the other architectures available in
Debian.</li>
<li>Progress of automatic builds as it happens:
<a href="https://buildd.debian.org/status/architecture.php?a=riscv64&suite=sid">https://buildd.debian.org/status/architecture.php?a=riscv64&suite=sid</a></li>
</ul>
</li>
<li>All of this work is not tested on real hardware yet.<ul>
<li>Outside of FPGA implementations, the first hardware publicly/easily
available will only surface in the next few weeks, only in small
quantities and expensive). We hope, and in particular I have a great deal
of confidence, that the software will basically work, but... yeah.</li>
</ul>
</li>
<li>We find still fairly often tests that fail to pass or get locked-up in <code>qemu</code>,
etc.<ul>
<li>It's difficult to know if these are bugs in the software,
compiler/toolchain, emulation, or a combination of them.</li>
<li>Still, since we got over 30% of the arch-dependent packages of a huge
collection like Debian built, passing tests for most of them when
available, things do not look too bad.</li>
</ul>
</li>
<li>We still are breaking dependency cycles that prevent many packages to be
built.<ul>
<li>For example we do not have almost any software depending on complex GUI
(think of KDE, GNOME, browsers), there is still no support for languages
such as Java, etc.</li>
</ul>
</li>
</ul>
<h2>Who</h2>
<p>Listing people working on different areas and at different times:</p>
<ul>
<li>Karsten Merker and I for a long time, participating in RISC-V mailing lists
and some events</li>
<li>Karsten Merker and I cross-building and using rebootstrap, Karsten is still
focusing on that area</li>
<li>Aurélien Jarno helping more actively since the preparation of pre-releases of
glibc 2.27, and other parts of the tool-chain during cross-compilation</li>
<li>Aurélien Jarno and I working closely together in the last phase of
cross-compilation and fine-tuning of the toolchain; then to build the native
sets of packages to bring riscv64 to debian-ports, setting-up buildds, etc.</li>
<li>Additional support by Kurt Keville of the MIT RV128 team for a couple of years
now</li>
<li><a href="https://www.bytemark.co.uk/">Bytemark</a>, one of the big sponsors of Debian
(and other FOSS projects), sponsored a machine where I did most of the work
during the previous years and up to the packages built to import into Debian
ports</li>
</ul>
<p>Apart from these people/resources, we have on our side:</p>
<ul>
<li>all the people working on cross-compilation and multi-arch over the years
(<em>Standing on the Shoulders of Giants</em>, as if it were)</li>
<li>Helmut Grohne's focus and tenacity on rebootstrap over the years, fixing
problems all along (already mentioned previously)</li>
<li>many maintainers adding support eagerly to their packages (sometimes not
trivial at all)</li>
<li>people starting to submit patches and requests to have packages with RISC-V
support ready to install, e.g. <code>qemu-system-riscv64</code></li>
<li>and various people offering help and giving support along the way, too many to
list here and for my poor brain to remember.</li>
</ul>
<h2>How to Use</h2>
<p>Download packages or create your own image from:
<a href="https://wiki.debian.org/RISC-V#Package_repositories">https://wiki.debian.org/RISC-V#Package_repositories</a></p>
<p>We will try to provide pre-built images or easier ways to test in the near
future.</p>
<h2>How to Help</h2>
<p>Please refer to:</p>
<ul>
<li><a href="https://wiki.debian.org/RISC-V">wiki page for RISC-V</a></li>
<li>the <a href="https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-riscv@lists.debian.org">list of open
bugs</a></li>
<li>the <a href="https://buildd.debian.org/status/architecture.php?a=riscv64&suite=sid">list of packages that fail to
build</a>
for one reason or another, analyse, submit fixes or patches</li>
</ul>
<h2>Future Work</h2>
<p>Usual stuff for other ports, like:</p>
<ol>
<li>Be able to build most of the archive</li>
<li>Create images to install or ready to use in <code>qemu</code>, etc.</li>
<li>Make it a first-class, full-supported architecture in future Debian Stable releases</li>
</ol>Debian GNU/Linux port for RISC-V 64-bit (riscv64)2017-04-22T02:20:00+00:002017-04-22T02:20:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2017-04-22:/~mafm/posts/2017/20170422_debian-gnulinux-port-for-risc-v-64-bit-riscv64/<p>This is a post describing my involvement with the <a href="https://wiki.debian.org/RISC-V">Debian GNU/Linux port for
RISC-V</a> (unofficial and not endorsed by Debian
at the moment) and announcing the availability of the repository (still very
much WIP) with packages built for this architecture.</p>
<p>If not interested in the story but you want to check the repository, just jump to the bottom.</p>
<h2>Roots</h2>
<p>A while ago, mostly during 2014 …</p><p>This is a post describing my involvement with the <a href="https://wiki.debian.org/RISC-V">Debian GNU/Linux port for
RISC-V</a> (unofficial and not endorsed by Debian
at the moment) and announcing the availability of the repository (still very
much WIP) with packages built for this architecture.</p>
<p>If not interested in the story but you want to check the repository, just jump to the bottom.</p>
<h2>Roots</h2>
<p>A while ago, mostly during 2014, I was involved in the <a href="https://people.debian.org/~mafm/posts/2015/20150421_about-the-debian-gnulinux-port-for-openrisc-or1k/">Debian port for OpenRISC
(or1k)</a>
─ about which I posted (by coincidence) exactly 2 years ago.</p>
<p>The two of us working on the port stopped in August or September of that year,
after knowing that the copyright of the code to add support for this
architecture in GCC would not be assigned to the <a href="https://www.fsf.org/">FSF</a>, so
it would never be added to GCC upstream ─ unless the original authors changed
their mind (which they didn't) or there was a clean-room reimplementation (which
didn't happen so far).</p>
<p>But a few other key things contributed to the decision to stop working on that
port, which bear direct relationship to this story.</p>
<p>One thing that particularly influenced me to stop working on it was a sense of
lack of purpose, all things considered, for the OpenRISC port that we were
working on.</p>
<p>For example, these chips are sometimes used as part of bigger devices by Samsung
to control or wake up other chips; but it was not clear whether there would ever
be devices with OpenRISC as the main chip, specially in devices powerful enough
to run Linux or similar kernels, and Debian on top. One can use FPGAs to
synthesise OpenRISC or1k, but these are slow, and expensive when using lots of
memory.</p>
<p>Without prospects of having hardware easily available to users, there's not much
point in having a whole Debian port ready to run on hardware that never comes to
be.</p>
<p>Yeah, sure, it's fun to create such a port, but it's tons of work to maintain
and keep it up to-date forever, and with close to zero users it's very
unrewarding.</p>
<p>Another thing that contributed to decide to stop is that, at least in my
opinion, 32-bit was not future-proof enough for general purpose computing,
specially for new devices and ports <em>starting to take off on that time and age</em>.
There was some incipient work to create another OpenRISC design for 64-bits, but
it was still in an early phase.</p>
<p>My secret hope and ultimate goal was to be able to run as much a free-ish
computer as possible <em>as my main system</em>. Still today many people are buying
and using 32-bit devices, like small boards; but very few use it as their main
computer or servers for demanding workloads or complex services. So for me,
even if feasible if one is very austere and dedicated, OpenRISC or1k failed that
test.</p>
<p>And lastly, another thing happened at the time...</p>
<h2>Enter RISC-V</h2>
<p>In August 2014, at the point when we were fully acknowledging the problem of
upstreaming (or rather, lack thereof) the support for OpenRISC in GCC,
<a href="https://riscv.org/">RISC-V</a> was announced to the world, bringing along papers
with suggestive titles such as <em>“<a href="https://aspire.eecs.berkeley.edu/publication/instruction-sets-should-be-free-the-case-for-risc-v/">Instruction Sets Should Be Free: The Case For
RISC-V</a>”</em>
(<a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-146.pdf">pdf</a>) and
articles like <em>“<a href="http://www.eetimes.com/author.asp?section_id=36&doc_id=1323406">RISC-V: An Open Standard for SoCs - The case for an open
ISA</a>”</em> in EE
Times.</p>
<p>RISC-V (as the previous RISC-n <em>marks</em>) had been designed (or rather, <em>was being
designed</em>, because it <em>was</em> and <em>is</em> an as yet unfinished standard) by people
from UC Berkeley, including <a href="https://en.wikipedia.org/wiki/David_Patterson_(computer_scientist)">David
Patterson</a>,
the pioneer of RISC computer designs and co-author of the seminal book
<em>“Computer Architecture: A Quantitative Approach”</em>. Other very capable people
are also also leading the project, doing the design and legwork to make it
happen ─ see the list of <a href="https://riscv.org/contributors/">contributors</a>.</p>
<p>But, apart from throwing names, the project has many other merits.</p>
<p>Similarly to OpenRISC, RISC-V is an open instruction set architecture (ISA), but
with the advantage of being designed in more recent times (thus avoiding some
mistakes and optimising for problems discovered more recently, as technology
evolves); with more resources; with support for instruction widths of 32, 64 and
even 128 bits; with a clean set of standard but optional extensions (atomics,
float, double, quad, vector, ...); and with reserved space to add new extensions
in ordered and compatible ways.</p>
<p>In the view of some people in the OpenRISC community, this unexpected
development in a way made irrelevant the ongoing update of OpenRISC for 64-bits,
and from what I know and for better or worse, all efforts on that front stopped.</p>
<p>Also interestingly (if nothing else, for my purposes of running as much a free
system as possible), was that the people behind RISC-V had strong intentions to
make it useful to create modern hardware, and were pushing heavily towards it
from the beginning.</p>
<p>And together with this announcement or soonly after, it came the promise of
free-ish hardware in the form of the <a href="http://www.lowrisc.org/">lowRISC</a> project.
Although still today it seems to be a long way from actually shipping hardware,
at least there was some prospect of getting it some day.</p>
<p>On top of all that, about the freedom aspect, both the Berkeley and lowRISC
teams engaged since very early on with the free hardware community, including
attending and presenting at OpenRISC events; and lowRISC intended to have as
much free hardware as possible in their planned SoC.</p>
<h2>Starting to work on the Debian port, this time for RISC-V</h2>
<p>So in late 2014 I <em>slowly</em> started to prepare the Debian port, switching on and
off.</p>
<p>The Userland spec was frozen in 2014 just before the announcement, the Privilege
one is still not frozen today, so there was no need to rush.</p>
<p>There were plans to upstream the support in the toolchain for RISC-V (GNU
<code>binutils</code>, <code>glibc</code> and <code>GCC</code>; <code>Linux</code> and other useful software like <code>Qemu</code>) in
2015, but sadly these did not materialise until late 2016 and 2017. One of the
main reasons for the delay was due to the slowness to sort out the copyright
assignment of the code to the FSF (again). Still today, only <code>binutils</code> and
<code>GCC</code> are upstreamed, and <code>Linux</code> and <code>glibc</code> depend on the Privilege spec being
finished, so it will take a while.</p>
<p>After the experience with OpenRISC and the support in GCC, I didn't want to
invest too much time, lest it all became another dead-end due to lack of
upstreaming ─ so I was just cross-compiling here and there, testing <code>Qemu</code>
(which still today is very limited for this architecture, e.g. no network
support and very limited character and block devices) and trying to find and
report bugs in the implementations, and send patches (although I did not
contribute much in the end).</p>
<h2>Incompatible changes in the toolchain</h2>
<p>In terms of compiling packages and building-up a repository, things were
complicate, and less mature and stable than the OpenRISC ones were even back in
2014.</p>
<p>In theory, with the Userland spec being frozen, regular programs (below the
Operating System level) compiled at any time could have run today; but in
practice what happened is that there were several rounds of profound ─or, at
least, disrupting─ changes in the toolchain before and while being upstreamed,
which made the binary packages that I had built to not work at all (changes in
dynamic loader, registers where arguments are stored when jumping functions,
etc.).</p>
<p>These major breakages happened several times already, and kind of unexpectedly ─
at least for the people not heavily involved in the implementation.</p>
<p>When the different pieces are upstreamed it is expected that these breakages
won't happen; but still there's at least the fundamental bit of <code>glibc</code>, which
will probably change things once again in incompatible ways before or while
being upstreamed.</p>
<p>Outside Debian but within the FOSS / Linux world, the main project that I know
of is that some people from Fedora also started a port in mid 2016 and did great
advances, but ─from what I know─ they put the project in the freezer in late
2016 until all such problems are resolved ─ they don't want to spend time
rebootstrapping again and again.</p>
<h2>What happened recently on the Debian front</h2>
<p>In early 2016 I created the page for <a href="https://wiki.debian.org/RISC-V">RISC-V</a> in
the Debian wiki, expecting that things were at last fully stable and the
important bits of the toolchain upstreamed during that year ─ I was too
optimistic.</p>
<p>Some other people (including Debian folks) have been contributing for a while,
in the wiki, mailing lists and IRC channels, and in the RISC-V project mailing
lists ─ you will see their names everywhere.</p>
<p>However, due to the combination of lack of hardware, software not upstreamed and
shortcomings of emulators (chiefly <code>Qemu</code>) make contributions hard and very
tedious, nothing much happened recently visible to the outside world in terms of
software.</p>
<h2>The private repository-in-the-making</h2>
<p>In late 2015 and beginning of 2016, having some free time in my hands and
expecting that all things would coalesce quickly, I started to build a
repository of binary packages in a more systematic way, with most of the basic
software that one can expect in a basic Debian system (including things common
to all Linux systems, and also specific Debian software like <code>dpkg</code> or <code>apt</code>,
and even <code>aptitude</code>!).</p>
<p>After that I also built many others outside the basic system (more than 1000
source packages and 2000 or 3000 arch-dependent binary packages in total),
specially popular libraries (e.g. <code>boost</code>, <code>gtk+</code> version 2 and 3), interpreters
(several versions of <code>lua</code>, <code>perl</code> and <code>python</code>, also version 2 and 3) and in
general packages that are needed to build many other packages (like <code>doxygen</code>).
Unfortunately, some of these most interesting packages do not compile cleanly
(more because of obscure or silly errors than proper porting), so they are not
included at the moment.</p>
<p>I intentionally avoided trying to compile thousands of packages in the archive
which would be of nobody's use at this point; but many more could be compiled
without much effort.</p>
<p>About the <em>how</em>, initially I started cross-compiling and using
<a href="https://wiki.debian.org/HelmutGrohne/rebootstrap">rebootstrap</a>, which was of
great help in the beginning. Some of the packages that I cross-compiled had
bugs that I did not know how to debug without a “live” and “native” (within
emulators) system, so I tried to switch to “natively” built packages very early
on. For that I needed many packages built natively (like <code>doxygen</code> or <code>cmake</code>)
which would be unnecessary if I remained cross-compiling ─ the host tools would
be used in that case.</p>
<p>But this also forced me to <em>eat my own dog food</em>, which even if much slower and
tedious, it was on the whole a more instructive experience; and above all, it
helped to test and make sure that the the tools and the whole stack was working
well enough to build hundreds of packages.</p>
<h2>Why the repository-in-the-making was private</h2>
<p>Until now I did not attempt to make the repository available on-line, for
several reasons.</p>
<p>First because it would be kind of useless to publish files that were not working
or would soon not work, due to the incompatible changes in the toolchain,
rendering many or most of the packages built useless. And because, for many
months now, I expected that things would stabilise and to have something stable
<em>“really soon now”</em> ─ but this didn't happen yet.</p>
<p>Second because of lack of resources and time since mid 2016, and because I got
some packages only compiled thanks to (mostly small and unimportant, but
undocumented and unsaved) hacks, often working around temporary bugs and thus
not worth sending upstream; but I couldn't share the binaries without sharing
the full source and fulfill licenses like the <a href="https://www.gnu.org/licenses/gpl.html">GNU
GPL</a>. I did a new round of clean
rebuilds in the last few weeks, just finished, the result is close to 1000
arch-dependent packages.</p>
<p>And third, because of lack of demand. This changed in the last few weeks, when
other people started to ask me to share the results even if incomplete or not
working properly (I had one request in the past, but couldn't oblige <em>in time</em>
at the time).</p>
<h2>Finally, the work so far: repository now on-line</h2>
<p>So finally, with the great help from <strong>Kurt Keville</strong> from MIT, and <strong>Bytemark</strong>
sponsoring a machine where most of the packages were built, here we have the
repository:</p>
<ul>
<li><a href="http://riscv.mit.edu/debian">http://riscv.mit.edu/debian</a></li>
</ul>
<p>The lines for <code>/etc/apt/sources.list</code> are:</p>
<div class="highlight"><pre><span></span> deb [ arch=riscv64 signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
deb-src [ signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
</pre></div>
<p>The repository is signed with my key as Debian Developer, contained in the file
<code>/usr/share/keyrings/debian-keyring.gpg</code>, which is part of the package
<code>debian-keyring</code> (available from Debian and derivatives).</p>
<h2>WARNING!!</h2>
<p>This repository, though, is very much <strong>WIP</strong>, <strong>incomplete</strong> (some package
dependencies cannot be fulfilled, and it's only a small percentage of the Debian
archive, not trying to be comprehensive at the moment) and <strong>probably does not
work at all in your system</strong> at this point, for the following reasons:</p>
<ul>
<li>I did not create Debian packages for fundamental pieces like <code>glibc</code>, <code>gcc</code>,
<code>linux</code>, etc.; although I hope that this happens soon after the next stable
release (Stretch) is out of the door and the remaining pieces are upstreamed
(help welcome).<br/> </li>
<li>There's no base system image yet, containing the software above plus a boot
loaders and sequence to set up the system correctly. At the moment you have
to provide your own or use one that you can find around the net. But
hopefully we will provide an image soon. (Again, help welcome).<br/> </li>
<li>Due to ongoing changes in the toolchain, if the version of the toolchain in
your image is very new or very old, chances are that the binaries won't work
at all. The one that I used was the master branch on 20161117 of their fork
of the toolchain:
<a href="https://github.com/riscv/riscv-gnu-toolchain">https://github.com/riscv/riscv-gnu-toolchain</a></li>
</ul>
<p>The combination of all these shortcomings is specially unfortunate, because
without <code>glibc</code> provided it will be difficult that you can get the binaries to
run at all; but there are some packages that are arch-dependent but not too tied
to libc or the dynamic loader will not be affected.</p>
<p>At least you can try one the few static packages present in Debian, like the one
in the package <strong><code>bash-static</code></strong>. When one removes moving parts like the
dynamic loader and libc, since the basic machine instructions are stable for
several years now, it should work, but I wouldn't discard some dark magic that
prevents even static binaries from working.</p>
<p>Still, I hope that the respository in its current state is useful to some
people, at least for those who requested it. If one has the environment set-up,
it's easy to unpack the contents of the <code>.deb</code> files and try out the software
(which often is not trivial or very slow to compile, or needs lots of
dependencies to be built first first).</p>
<p>... and finally, even if not useful at all for most people at the moment, by
doing this I also hope that efforts like this spark your interest to contribute
to free software, free hardware, or both! :-)</p>FOSDEM 2017: People, RISC-V and ChaosKey2017-02-08T23:52:00+00:002017-02-08T23:52:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2017-02-08:/~mafm/posts/2017/20170208_fosdem-2017-people-risc-v-and-chaoskey/<p>This year, for the first time, I attended <a href="https://fosdem.org/2017/">FOSDEM</a>.</p>
<p>There I met...</p>
<h2>People</h2>
<p>... including:</p>
<ul>
<li>friends that I don't see very often;</li>
<li>old friends that I didn't expect to see there, some of whom decided to travel
from far away in the last minute;</li>
<li>met people in person for the first time, which previously I had known only
though the internet -- one of whom is a protagonist …</li></ul><p>This year, for the first time, I attended <a href="https://fosdem.org/2017/">FOSDEM</a>.</p>
<p>There I met...</p>
<h2>People</h2>
<p>... including:</p>
<ul>
<li>friends that I don't see very often;</li>
<li>old friends that I didn't expect to see there, some of whom decided to travel
from far away in the last minute;</li>
<li>met people in person for the first time, which previously I had known only
though the internet -- one of whom is a protagonist in a previous blog entry,
about the <a href="https://people.debian.org/~mafm/posts/2015/20150421_about-the-debian-gnulinux-port-for-openrisc-or1k/">Debian port for
OpenRISC</a>;</li>
</ul>
<p>I met new people in:</p>
<ul>
<li>bars/pubs,</li>
<li>restaurants,</li>
<li>breakfast tables at lodgings,</li>
<li>and public transport.</li>
</ul>
<p>... from the first hour to the last hour of my stay in Brussels.</p>
<p>In summary, lots of people around.</p>
<p>I also hoped to meet or spend some (more) time with a few people, but in the end
I didn't catch them, our could not spend as much time with them as I would wish.</p>
<p>For somebody like me who enjoys quiet time by itsef, it was a bit too intensive
in terms of interacting with people. But overall it was a nice winter break,
definitely worth to attend, and even a much better experience than what I had
expected.</p>
<h2>Talks / Events</h2>
<p>Of course, I also attended a few talks, some of which were very interesting;
although the event is so (sometimes uncomfortably) crowded that the rooms were
full more often than not, in which case it was not possible to enter (the doors
were closed) or there were very long queues for waiting.</p>
<p>And with so many talks crammed into a weekend, I had so many schedule clashes
with the talks that I had pre-selected as interesting, that I ended up missing
most of them.</p>
<p>In terms of technical stuff, I have specially enjoyed the talk by Arun Thomas
<a href="https://fosdem.org/2017/schedule/event/riscv/">RISC-V -- Open Hardware for Your Open Source
Software</a>, and some conversations
related with toolchain stuff and other upstream stuff, as well as on the <a href="https://wiki.debian.org/RISC-V">Debian
port for RISC-V</a>.</p>
<p>The talk <a href="https://fosdem.org/2017/schedule/event/dinosaurs/">Resurrecting dinosaurs, what can possibly go wrong? -- How
Containerised Applications could eat our
users</a>, by Richard Brown, was
also very good.</p>
<h2>ChaosKey</h2>
<p>Apart from that, I have witnessed a <em>shady</em> cash transaction in a bus from the
city centre to FOSDEM in exchange for hardware, not very unlike <a href="http://joeyh.name/blog/entry/badUSB/">what I had read
about only days before</a>.</p>
<p>So I could not help but to get involved in a subsequent transaction myself, to
get my hands laid upon a <a href="https://altusmetrum.org/ChaosKey/">ChaosKey</a>.</p>More work on aptitude2016-06-18T16:54:00+00:002016-06-18T16:54:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2016-06-18:/~mafm/posts/2016/20160618_more-work-on-aptitude/<p>The last few months have been a bit of a crazy period of ups and downs, with a
tempest of events beneath the apparent and deceivingly calm surface waters of
being <strong>unemployed</strong> (still at it).</p>
<h2>The daily grind</h2>
<p>Chief activities are, of course, those related to the daily grind of
job-hunting, sending applications, and preparing and attending interviews.</p>
<p>It is demoralising when one searches for many …</p><p>The last few months have been a bit of a crazy period of ups and downs, with a
tempest of events beneath the apparent and deceivingly calm surface waters of
being <strong>unemployed</strong> (still at it).</p>
<h2>The daily grind</h2>
<p>Chief activities are, of course, those related to the daily grind of
job-hunting, sending applications, and preparing and attending interviews.</p>
<p>It is demoralising when one searches for many days or weeks without seeing
anything suitable for one's skills or interests, or other more general life
expectations. And it takes a lot of time and effort to put one's best in the
applications for positions that one is really, <em>really</em>, interested in. And
even for the ones which are <em>meh</em>, for a variety of reasons (e.g. one is not
very suitable for what the offer demands).</p>
<p>After that, not being invited to interviews (or doing very badly at them) is
bad, of course, but quick and not very painful. A swift, merciful end to the
process.</p>
<p>But it's all the more draining when waiting for many weeks ─when not a few
months─ with the uncertainty of not knowing if one is going to be lucky enough
to be summoned for an interview; harbouring some hope ─one has to appear
<em>enthusiastic</em> in the interviews, after all─, while trying to keep it contained
─lest it grows too much─; then in the interview hearing good words and some
praises, and feeling the impression that one will fit in, that one did nicely
and that chances are good ─letting the hope grow again─; start to think about
life changes that the job will require ─to make a quick decision should the
offer finally arrives─; perhaps make some choices and compromises based on the
uncertain result; then wait for a week or two after the interview to know the
result...</p>
<p>... only to end up being unsuccessful.</p>
<p>All the effort and hopes finally get squashed with a cold, short email or
automatic response, or more often than not, complete radio silence from
prospective employers, as an end to a multi-month-long process. An emotional
roller coaster <a name="ref-note-1"></a><sup>[<a href="#note-1">1</a>]</sup>, which
happened to me several times in the last few months.</p>
<h2>All in a day's work</h2>
<p>The months of preparing and waiting for a new job often imply an <em>impasse</em> that
puts many other things that one cares about on hold, and one makes plans that
will never come to pass.</p>
<p>All in a day's (half-year's?) work of an unemployed poor soul.</p>
<p>But not all is bad.</p>
<p>This period was also a busy time doing some plans about life, mid- and
long-term; the usual ─and some really unusual!─ family events; visits to and
from friends, old and new; attending nice little local Debian gatherings or the
bigger gathering of <a href="https://wiki.debian.org/DebianEvents/Europe/2016/DSC">Debian
SunCamp2016</a>, and other
work for side projects or for other events that will happen soon...</p>
<p>And amidst all that, I managed to get some work done on <code>aptitude</code>.</p>
<h2>Two pictures worth (less than) a thousand bugs</h2>
<p>To be precise, worth 709 bugs ─ 488 bugs in the first graph, plus 221 in the
second.</p>
<p>In 2015-11-15 (link to the post <a href="https://people.debian.org/~mafm/posts/2015/20151115_work-on-aptitude/">Work on aptitude</a>):</p>
<p><center>
<img alt="aptitude BTS Graph, 2015-11-15" src="https://people.debian.org/~mafm/static/images/aptitude-bts-graph-20151115.png" title="aptitude BTS Graph, 2015-11-15">
</center></p>
<p>In 2016-06-18:</p>
<p><center>
<img alt="aptitude BTS Graph, 2016-06-18" src="https://people.debian.org/~mafm/static/images/aptitude-bts-graph-20160618.png" title="aptitude BTS Graph, 2016-06-18">
</center></p>
<h2>Numbers</h2>
<p>The <a href="https://www.debian.org/Bugs/">BTS</a> numbers for <code>aptitude</code> right now are:</p>
<ul>
<li>221 (259 if counting all merged bugs independently)</li>
<li>1 Release Critical (but it is an artificial bug to keep it from migrating to
<code>testing</code>)</li>
<li>43 (55 unmerged) with severity Important or Normal</li>
<li>160 (182 unmerged) with severity Minor or Wishlist</li>
<li>17 (21 unmerged) marked as Forwarded or Pending</li>
</ul>
<h2>Highlights</h2>
<p>Beyond graphs and stats, I am specially happy about two achievements in the last
year:</p>
<ol>
<li>
<p><strong>To have <code>aptitude</code> working today</strong>, first and foremost</p>
<p>Apart from the abandon that suffered in previous years, I mean
specifically the critical step of getting it through the troubles of the
last summer, with the GCC-5/C++11 transition in parallel with a
transition of the <code>Boost</code> library (explained in more detail in <a href="https://people.debian.org/~mafm/posts/2015/20151115_work-on-aptitude/">Work on
aptitude</a>).</p>
<p>Without that, possibly <code>aptitude</code> would not have survived until today.</p>
</li>
<li>
<p><strong>Improvements to the suggestions of the resolver</strong></p>
<p>In the version <code>0.8</code>, there were a lot of changes related with improving
the order of the suggestions from the resolver, when it finds conflicts
or other problems with the planned actions.</p>
<p>Historically, but specially in the last few years, there have been many
complaints about the nonsensical or dangerous suggestions from the
resolver. The first solution offered by the resolver was very often
regarded as highly undesirable (for example, removal of many packages),
and preferable solutions like upgrades of one or only a handful of
packages being offered only after many removals; and “keeps” only
offered as last resort.</p>
</li>
</ol>
<p>Perhaps these changes don't get a lot of attention, given that in the first case
it's <em>just to keep working</em> (with few people realising that it could have
collapsed on the spot, if left unattended), and the second can probably go
unnoticed because “<em>it just works</em>” or “<em>it started to work more smoothly</em>”
doesn't get as much immediate attention as “<em>it suddenly broke!</em>”.</p>
<p>Still, I wanted to mention them, because I am quite proud of those.</p>
<h2>Thanks</h2>
<p>Even if I put a lot of work on <code>aptitude</code> in the last year, the results of the
graph and numbers have not been solely achieved by me.</p>
<p>Special thanks go to <a href="https://wiki.debian.org/AxelBeckert">Axel Beckert</a> (abe /
XTaran) and the <a href="https://wiki.debian.org/Teams/Apt"><code>apt</code> team</a>, David
Kalnischkies and Julian Andres Klode ─ who, despite the claim in that page, does
not <em>mostly</em> work <code>python-apt</code> <em>anymore</em>... but also in the main tools.</p>
<p>They help with fixing some of the issues directly, or changing things in <code>apt</code>
that benefit <code>aptitude</code>, testing changes, triaging bugs or commenting on them,
patiently explaining to me why something in <code>libapt</code> doesn't do what I think it
does, and good company in general.</p>
<p>Not the least, for holding <em>impromptu</em> BTS group therapy / support meetings, for
those cases when prolonged exposure to BTS activity starts to induce very bad
feelings.</p>
<p>Thanks also to people who sent their translation updates, notified about
corrections, sent or tested patches, submitted bugs, or tried to help in other
ways. Change logs for details.</p>
<p><a name="notes"></a></p>
<h4>Notes</h4>
<p><a name="note-1"></a> [1] <sup><a href="#ref-note-1">^</a></sup> It's even an example in
the Cambridge Dictionaries Online website, for the entry of <a href="http://dictionary.cambridge.org/dictionary/english/roller-coaster">roller
coaster</a>:</p>
<blockquote>
<p>“<em>He was on an emotional roller coaster for a while when he lost his job.</em>”</p>
</blockquote>Work on aptitude2015-11-15T23:44:00+00:002015-11-15T23:44:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2015-11-15:/~mafm/posts/2015/20151115_work-on-aptitude/<p><a href="https://en.wikipedia.org/wiki/Midsummer">Midsummer</a> for me is also known as
<strong>“Noite do Lume Novo”</strong> (literally “New Fire Night”), one of the big calendar
events of the year, marking the end of the school year and the beginning of
summer.</p>
<p>On this day, there are
<a href="https://en.wikipedia.org/wiki/Bonfires_of_Saint_John#Traditions">celebrations</a>
not very unlike the bonfires in the <a href="https://en.wikipedia.org/wiki/Guy_Fawkes_Night">Guy Fawkes
Night</a> in England or Britain <a
name="ref-note-1"></a><sup>[<a href="#note-1">1</a>]</sup>. It is a bit different in that
it is …</p><p><a href="https://en.wikipedia.org/wiki/Midsummer">Midsummer</a> for me is also known as
<strong>“Noite do Lume Novo”</strong> (literally “New Fire Night”), one of the big calendar
events of the year, marking the end of the school year and the beginning of
summer.</p>
<p>On this day, there are
<a href="https://en.wikipedia.org/wiki/Bonfires_of_Saint_John#Traditions">celebrations</a>
not very unlike the bonfires in the <a href="https://en.wikipedia.org/wiki/Guy_Fawkes_Night">Guy Fawkes
Night</a> in England or Britain <a
name="ref-note-1"></a><sup>[<a href="#note-1">1</a>]</sup>. It is a bit different in that
it is not a single event for the masses, more of a friends and neighbours thing,
and that it lasts for a big chunk of the night (sometimes until morning).
Perhaps for some people, or outside bigger towns or cities, Guy Fawkes Night is
also celebrated in that way ─ and that's why during the first days of November
there are fireworks rocketing and cracking in the neighbourhoods all around.</p>
<p>Like many other <a href="https://en.wikipedia.org/wiki/Bonfire#Regional_traditions">celebrations around the world involving
bonfires</a>, many of
them also happening around the summer solstice, it is supposed to be a time of
renewal of cycles, purification and keeping the evil spirits away; with rituals
to that effect like jumping over the fire ─ when the flames are not high and it
is safe enough.</p>
<p>So it was fitting that, in the middle of June (almost Midsummer in the northern
hemisphere), I learnt that I was about to leave my now-previous job, which is a
pretty big signal and precursor for <strong>renewal</strong> (and it might have something to
do with <strong>purifying</strong> and <strong>keeping the evil away</strong> as well <span
class="smiley">;-)</span> ).</p>
<h2>Whatever... But what does all of this have to do with <code>aptitude</code> or Debian, anyway?</h2>
<p>For one, it was a question of <strong>timing</strong>.</p>
<p>While looking for a new job (and I am still at it), I had more spare time than
usual. <a href="http://debconf15.debconf.org/">DebConf 15 @ Heidelberg</a> was within
sight, and for the first time circumstances allowed me to attend this event.</p>
<p>It also coincided with the time when I re-gained access to commit to
<a href="https://tracker.debian.org/pkg/aptitude"><code>aptitude</code></a> on the 19th of June.
Which means <strong>Renewal</strong>.</p>
<p>End of June was also the time of the announcement of the colossal <a href="https://release.debian.org/transitions/html/libstdc++6.html">GCC-5/C++11
ABI transition</a> in
Debian, that was scheduled to start on the 1st of August, just before the
DebConf. Between 2 and 3 thousand source packages in Debian were affected by
this transition, which a few months later is not yet finished (although the most
important parts were completed by mid-end September).</p>
<p><code>aptitude</code> itself is written in C++, and depends on several libraries written in
C++, like <code>Boost</code>, <code>Xapian</code> and <code>SigC++</code>. All of them had to be compiled with
the new C++11 ABI of GCC-5, in unison and in a particular order, for <code>aptitude</code>
to continue to work (and for minimal breakage). <code>aptitude</code> and some
dependencies did not even compile straight away, so this transition meant that
<code>aptitude</code> needed attention just to keep working.</p>
<p>Having recently being awarded again with the Aptitude Hat, attending DebConf for
the first time and sailing towards the Transition Maelstrom, it was a clear sign
that Something Had to Be Done (to avoid the sideways looks and consequent shame
at DebConf, if nothing else).</p>
<p>Happily (or a bit <em>unhappily</em> for me, but let's pretend...), with the unexpected
free time in my hands, I changed the plans that I had before re-gaining the
Aptitude Hat (some of them involving Debian, but in other ways ─ maybe I will
post about that soon).</p>
<p>In July I worked to fix the problems before the transition started, so
<code>aptitude</code> would be (mostly) ready, or in the worst case broken only for a few
days, while the chain of dependencies was rebuilt. But apart from the changes
needed for the new <code>GCC-5</code>, it was decided at the last minute that <code>Boost</code> 1.55
would not be rebuilt with the new ABI, and that the only version with the new
ABI would be 1.58 (which caused further breakage in <code>aptitude</code>, was added to
<code>experimental</code> only a few days before, and was moved to <code>unstable</code> after the
transition had started). Later, in the first days of the transition, <code>aptitude</code>
was affected for a few days by breakage in the dependencies, due to not being
compiled in sequence according to the transition levels (so with a mix of old
and new ABI).</p>
<p>With the critical intervention of <a href="https://wiki.debian.org/AxelBeckert">Axel
Beckert</a> (abe / XTaran), things were not so
bad as they could have been. He was busy testing and uploading in the critical
days when I was enjoying a small holiday on my way to DebConf, with minimal
internet access and communicating almost exclusively with him; and he promptly
tended the complaints arriving in the Bug Tracking System and asked for rebuilds
of the dependencies with the new ABI. He also brought the packaging up to
shape, which had decayed a bit in the last few years.</p>
<h2>Gruesome Challenges</h2>
<p>But not all was solved yet, more storms were brewing and started to appear in
the horizon, in the form of clouds of fire coming from <a href="https://tracker.debian.org/pkg/apt">nearby
realms</a>.</p>
<p>The APT Deities, which had long ago spilled out their <a href="https://lists.alioth.debian.org/pipermail/aptitude-devel/2012-January/001823.html">secret, inner
challenge</a>
(just the initial paragraphs), were relentless. Moreover, they were present at
Heidelberg in full force, in ─or close to─ their home grounds, and they were
Marching Decidedly towards Victory:</p>
<p><center>
<img alt="apt BTS Graph, 2015-11-15" src="https://people.debian.org/~mafm/static/images/apt-bts-graph-20151115.png" title="apt BTS Graph, 2015-11-15">
</center></p>
<p>In the talk @ DebConf “<a href="https://summit.debconf.org/debconf15/meeting/216/this-apt-has-super-cow-powers/">This APT has Super Cow
Powers</a>”
(video available), by David Kalnischkies, they told us about the niceties of
<code>apt</code> 1.1 (still in <code>experimental</code> but hopefully coming to <code>unstable</code> soon), and
they boasted about getting the lead in our <em>arms race</em> (should I say <em>bugs
race</em>?) by a few open bug reports.</p>
<p>This act of provocation further escalated the tensions. The fierce competition
which had been going on for some time gained new heights. So much so that APT
Deities and our team had to sit together in the outdoor areas of the venue and
have many a <em>weissbier</em> together, while discussing and fixing bugs.</p>
<p>But beneath the calm on the surface, and while pretending to keep good
diplomatic relations, I knew that Something Had to Be Done, again. So I could
only do one thing ─ jump over the bonfire and <strong>Keep the Evil away</strong>, be that
Keep Evil bugs Away or Keep Evil APT Deities Away from winning the challenge, or
both.</p>
<p>After returning from DebConf I continued to dedicate time to the project, more
than a full time job in some weeks, and this is what happened in the last few
months, summarised in another graph, showing the evolution of the BTS for
<code>aptitude</code>:</p>
<p><center>
<img alt="aptitude BTS Graph, 2015-11-15" src="https://people.debian.org/~mafm/static/images/aptitude-bts-graph-20151115.png" title="aptitude BTS Graph, 2015-11-15">
</center></p>
<p>The numbers for <code>apt</code> right now (15th November 2015) are:</p>
<ul>
<li>629 open (731 if counting all merged bugs independently)</li>
<li>0 Release Critical</li>
<li>275 (318 unmerged) with severity Important or Normal</li>
<li>354 (413 unmerged) with severity Minor or Wishlist</li>
<li>0 marked as Forwarded or Pending</li>
</ul>
<p>The numbers for <code>aptitude</code> right now are:</p>
<ul>
<li>488 (573 if counting all merged bugs independently)</li>
<li>1 Release Critical (but it is an artificial bug to keep it from migrating to
<code>testing</code>)</li>
<li>197 (239 unmerged) with severity Important or Normal</li>
<li>271 (313 unmerged) with severity Minor or Wishlist</li>
<li>19 (20 unmerged) marked as Forwarded or Pending</li>
</ul>
<h2>The Aftermath</h2>
<p>As we can see, for the time being I could keep the Evil at bay, both in terms of
bugs themselves and re-gaining the lead in the <em>bugs race</em> ─ the Evil APT
Deities were thwarted again in their efforts.</p>
<p>... More seriously, as most of you suspected, the graph above is not the whole
truth, so I don't want to boast <em>too much</em>. A big part of the reduction in the
number of bugs is because of merging duplicates, closing obsolete bugs, applying
translations coming from multiple contributors, or simple fixes like typos and
useful suggestions needing minor changes. Many of remaining problems are
comparatively more difficult or time consuming that the ones addressed so far
(except perhaps avoiding the immediate breakage of the transition, that took
weeks to solve), and there are many important problems still there, chief among
those is <code>aptitude</code> offering <em><strong>very poor</strong></em> solutions to resolve conflicts.</p>
<p>Still, even the simplest of the changes takes effort, <strong>and triaging hundreds of
bugs is not fun at all</strong> and mostly a thankless effort ─ althought there is the
occasionally kind soul that thanks you for handling a decade-old bug.</p>
<p>If being subjected to the rigours of the BTS and reading and solving hundreds of
bug reports is not <strong>Purification</strong>, I don't know what it is.</p>
<p>Apart from the triaging, there were <strong>118 bugs closed (or pending) due to
changes made in the upstream part or the packaging</strong> in the last few months, and
there are many changes that are not reflected in bugs closed (like most of the
changes needed due to the C++11 ABI transition, bugs and problems fixed that had
no report, and general rejuvenation or improvement of some parts of the code).</p>
<p>How long this will last, I cannot know. I hope to find a job at some point,
which obviously will reduce the time available to work on this.</p>
<p>But in the meantime, for all <code>aptitude</code> users: Enjoy the fixes and new features!</p>
<p><a name="notes"></a></p>
<h4>Notes</h4>
<p><a name="note-1"></a> [1] <sup><a href="#ref-note-1">^</a></sup> Some visitors of the
recent <a href="https://wiki.debian.org/DebianEvents/gb/2015/MiniDebConfCambridge">mini-DebConf @
Cambridge</a>
perhaps thought that the fireworks and throngs gathered were in honour of our
mighty Universal Operating System, but sadly they were not. They might be, some
day. In any case, the reports say that the visitors enjoyed the fireworks.</p>About the Debian GNU/Linux port for OpenRISC or1k2015-04-21T01:04:00+00:002015-04-21T01:04:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2015-04-21:/~mafm/posts/2015/20150421_about-the-debian-gnulinux-port-for-openrisc-or1k/<p>In my <a href="https://people.debian.org/~mafm/posts/2015/20150130_hallo-planet-debian/">previous post</a> I
mentioned my involvement with the <a href="http://opencores.org/or1k/Main_Page">OpenRISC
or1k</a> port. It was the technical activity
in which I spent most time during 2014 (Debian and otherwise, day job aside).</p>
<p>I thought that it would be nice to talk a bit about the port for people who
don't know about it, and give an update for those who do know and care. So this …</p><p>In my <a href="https://people.debian.org/~mafm/posts/2015/20150130_hallo-planet-debian/">previous post</a> I
mentioned my involvement with the <a href="http://opencores.org/or1k/Main_Page">OpenRISC
or1k</a> port. It was the technical activity
in which I spent most time during 2014 (Debian and otherwise, day job aside).</p>
<p>I thought that it would be nice to talk a bit about the port for people who
don't know about it, and give an update for those who do know and care. So this
post explains a bit how it came to be, details about its development, and
finally the current status. It is going to be written as a rather <em>personal</em>
account, for that matter, since I did not get involved enough in the OpenRISC
community at large to learn much about its internal workings and aspects that I
was not directly involved with.</p>
<p>There is not much information about all of this elsewhere, only bits and pieces
scattered here and there, but specially not much public information at all about
the development of the Debian port. There is an
<a href="https://wiki.debian.org/OpenRISC">OpenRISC</a> entry in the Debian wiki, but it
does not contain much information yet. Hopefully, this piece will help a bit to
preserve history and give an insight for future porters.</p>
<h2>First Things First</h2>
<p>I imagine that most people reading this post will be familiar with the
terminology, but just in case, to create a new Debian port means to get a Debian
system (GNU/Linux variant, in this case) to run in the OpenRISC or1k computer
architecture.</p>
<p>Setting to one side all differences between hardware and software, and as
described in their site:</p>
<p><i></p>
<blockquote>
<p>“The aim of the OpenRISC project is to create free and open source computing
platforms”
</i></p>
</blockquote>
<p>It is therefore a good match for the purposes of Debian and Free Software world
in general.</p>
<p>The processor has not been produced in silicon, or not available for the masses
in any case. People with the necessary know-how can download the hardware
description (Verilog) and synthesise it in a FPGA, or otherwise use simulators.
It is not some piece of hardware that people can purchase yet, and there are no
plans to mass-produce it in the near future either.</p>
<p>The two people (including me) involved in this Debian port did not have the
hardware, so we created the port entirely through cross-compiling from other
architectures, and then compiling inside <code>Qemu</code>. In a sense, we were creating a
Debian port for hardware that "does not [phisically] exist". The software that
we built was tested by people who had hardware available in FPGA, though, so it
was at least <em>usable</em>. I understand that people working in the <code>arm64</code> port had
to work similarly in the initial phases, working in the dark without access to
real hardware to compile or test.</p>
<h2>The Spark</h2>
<p>The first time that I heard about the initiative to create the port was in late
February of 2014, in a post which appeared in <a href="http://lwn.net/Articles/588637/">Linux Weekly
News</a> (sent by <a href="https://wiki.debian.org/PaulWise">Paul
Wise</a>) and
<a href="http://linux.slashdot.org/story/14/02/27/2227238/experimental-port-of-debian-to-openrisc">Slashdot</a>.
The <a href="http://lists.opencores.org/pipermail/openrisc/2014-January/001565.html">original
post</a>
announcing it was actually from late January, from <strong>Christian Svensson</strong>
(<strong>blueCmd</strong>):</p>
<p><i></p>
<blockquote>
<p>“Some people know that I've been working on porting Glibc and doing some
toolchain work. My evil master plan was to make a Debian port, and today I'm a
happy hacker indeed!</p>
<p>Below is a link to a screencast of me installing Debian for OpenRISC,
installing python2.7 via apt-get (which you shouldn't do in or1ksim, it takes
ages! (but it works!)) and running a small Python
script. <a href="http://asciinema.org/a/7362">http://asciinema.org/a/7362</a>”
</i></p>
</blockquote>
<p>So, now, what can a Debian Hacker do when reading this? (Even if one's Hackery
Level is not that high, as it is my case). And well, <em>How Hard Can It Be? I
mean, Really?</em></p>
<p>Well, in my own defence, I knew that the answer to the last two questions would
be a resounding <em>“<strong>Very</strong>”</em>. But for some reason the idea grabbed me and I
couldn't help but think that it would be a Really Exciting Project, and that
somehow I would like to get involved. So I wrote to Christian offering my help
after considering it for a few days, around mid March, and he welcomed me
aboard.</p>
<h2>The Ball Was Already Rolling</h2>
<p>Christian had already been in contact with the people behind
<a href="https://wiki.debian.org/DebianBootstrap">DebianBootstrap</a>, and he had already
created the repository
<a href="http://openrisc.debian.net/">http://openrisc.debian.net/</a> with many packages of
the base system and beyond (read: packages <code>name_version_or1k.deb</code> available to
download and install). Still nowadays the packages are not signed with proper
keys, though, so use your judgement if you want to try them.</p>
<p>After a few weeks, I got up to speed with the status of the project and got my
system working with the necessary tools. This meant basically
<code>sbuild</code>/<code>schroot</code> to compile new packages, with the base system that Christian
already got working, installed in a <code>chroot</code>, probably with the help of
<code>debootstrap</code>, and <code>qemu-system-or1k</code> to simulate the system.</p>
<p>Only a few of the packages were different from the version in Debian, like
<code>gcc</code>, <code>binutils</code> or <code>glibc</code> -- they had not been upstreamed yet. <code>sbuild</code> ran
through <code>qemu-system-or1k</code>, so the compilation of new packages could happen
"natively" (running inside <code>Qemu</code>) rather than cross-compiling the packages,
pulling <code>_or1k.deb</code> packages for dependencies from the repository that he had
prepared, and <code>_all.deb</code> packages from <code>snapshots.debian.org</code>.</p>
<p>I started by trying to get the packages that I [co-]maintain in Debian compiled
for this architecture, creating the corresponding <code>_or1k.deb</code>. For most of
them, though, I needed many dependencies compiled before I could even compile my
packages.</p>
<h2>The GNU autotools / autoreconf Problem</h2>
<p>Since very early, many of the packages failed to build with messages such as:</p>
<blockquote>
<p><code>Invalid configuration 'or1k-linux-gnu': machine 'or1k' not recognized</code><br/>
<code>configure: error: /bin/bash ../config.sub or1k-linux-gnu failed</code></p>
</blockquote>
<p>This means that software packages based on GNU <code>autotools</code> and using <code>configure</code>
scripts need recent versions of the files <code>config.sub</code> and <code>config.guess</code> that
they ship in their root directory, to be able to detect the architecture and
generate the code accordingly.</p>
<p>This is counter-intuitive, having into account that GNU <code>autotools</code> were
<a href="https://www.gnu.org/software/automake/faq/autotools-faq.html#What-are-the-_0060_0060Autotools_0027_0027_003f">designed to help with
portability</a>.
Yet, in the case of creating new Debian ports, it meant that unless upstream had
very recent versions of <code>config.{guess,sub}</code>, it would prevent the package to
compile straight away in the new architectures -- even if invoking <code>gcc</code> without
ado would have worked without problems in most cases for native compilation.</p>
<p>Of course this did not only affect <code>or1k</code>, and there was already the
<a href="https://wiki.debian.org/Autoreconf">autoreconf</a> effort underway as a way to
update these files automatically when building Debian packages, pushed by people
porting Debian to the new architectures added in 2013/2014 (<code>mips64el</code>, <code>arm64</code>,
<code>ppc64el</code>), which encountered the same roadblock. This affected <a href="https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=autoreconf;users=debian-devel@lists.debian.org">around a
thousand source
packages</a>
in unstable. A Royal Pain. Also, all of their reverse dependencies (packages
that depended on those to be built) could not be compiled straight away.</p>
<p>The bugs were not Release Critical, though (none of these architectures were
officially accepted at the time), so for people not concerned with the new ports
there was no big incentive to get them fixed. <strong>This problem, which
conceptually is easily solvable, prevented new ports to even attempt compile
vast portions of the archive straight away</strong> (cleanly, without modifications to
the package or to the host system), <strong>for weeks or months</strong>.</p>
<h2>The GNU autotools / autoreconf Solution</h2>
<p>We tackled this problem mainly in two ways.</p>
<p>First, more useful for Debian in general, was to do as other porters were doing
and submit bug reports and patches to Debian packages <strong>requesting them to use
<code>autoreconf</code>, and NMUing packages</strong> (uploading changes to the archive without
the official maintainers' intervention). A few NMUs were made for packages
which had bug reports with patches available for a while, that were in the
critical path to get many other packages compiled, and that were orphan or had
almost no maintainer activity.</p>
<p>The people working in the other new ports, and mainly Ubuntu people which helped
with some of those ports and wanted to support them, had submitted a large
amount of requests since late 2013, so there was no shortage of NMUs to be made.
Some porters, not being Debian Developers, could not easily get the changes
applied; so I also helped a bit the porters of other architectures, specially
later on before the freeze of Jessie, to get as many packages compiled in those
architectures as possible.</p>
<p>The second way was to <strong>create <code>dpkg-buildpackage</code> hooks that updated
unconditionally <code>config.{guess,sub}</code></strong> before attempting to build the package in
the local build system. This local and temporary solution allowed us in the
<code>or1k</code> port to get many <code>_or1k.deb</code> packages in the experimental repository,
which in turn would allow many more packages to compile. After a few weeks, I
set up many <code>sbuilds</code> in a multi-core machine attempting to build
uninterruptedly packages that were not previously built and which had their
dependencies available. Every now and then (typically several times per day in
peak times) I pushed the resulting <code>_or1k.deb</code> files to the repository, so more
packages would have the necessary dependencies ready to attempt to build.</p>
<p>Christian was doing something similar, and by April at peak times, among the two
of us, we were compiling some days more than a hundred packages -- <strong>a huge
amount of packages did not need any change other than up-to-date
<code>config.{guess,sub}</code> files</strong>. At some point, late April, Christian set up
<code>wanna-build</code> in a few hosts to do this more elegantly and smartly than my
method, and more effectively as well.</p>
<h2>Ugly Hacks, Bugs and Shortcomings in the Toolchain and Qemu</h2>
<p>Some packages are extremely important because many other packages need them to
compile (like <code>cmake</code>, <code>Qt</code> or <code>GTK+</code>), and they are themselves very complex and
have dependency loops. They had deeper problems than the <code>autoreconf</code> issue and
needed some <em>seriously dirty</em> hacking to get them built.</p>
<p>To try to get as many packages compiled as possible, we sometimes compiled these
important packages with some functionality disabled, disabling some binary
packages (e.g. Java bindings) or specially disabling documentation (using
<code>DEB_BUILD_OPTIONS=nodoc</code> when possible, and more aggressively when needed by
removing chunks of <code>debian/rules</code>). I tried to use the more aggressive methods
in as few packages as possible, though, about a dozen in total. We also used
<code>DEB_BUILD_OPTIONS=nocheck</code> for speeding up compilation and avoiding build
failures -- many packages' tests failed due to <code>qemu-system-or1k</code> not supporting
multi-threading, which we could do nothing about at the time, but otherwise the
packages mostly passed tests fine.</p>
<p>Due to bugs and shortcomings in <code>Qemu</code> and the toolchain --like the compiler
lacking <em>atomics</em>, missing functionality in <code>glibc</code>, <code>Qemu</code> entering in endless
loops, or programs segfaulting (especially <code>gettext</code>, used by many packages and
causing the packages failing to build)--, we had to resort to some very
<em>creative</em> ways or time-consuming dull work to edit <code>debian/rules</code>, or to create
wrappers of the real programs avoiding or forcing certain options (like <code>gcc
-O0</code>, since <code>-O2</code> made buggy binaries too often).</p>
<p>To avoid having a mix of cleanly compiled and hacked packages in the same
repository, Christian set up a <strong>two-tiered repository system</strong> -- the <em>clean</em>
one and the <em>dirty</em> one. In the <em>dirty</em> one we dumped all of the packages that
we got built, no matter how. The packages in the <em>clean</em> one could use packages
from the <em>dirty</em> one to build, but they themselves were compiled without any
hackery. Of course this was not a completely airtight solution, since they
could contain code injected at build time from the "dirty repository" (e.g. by
static linking), and perhaps other quirks. We hoped to get rid of these
problems later by rebuilding all packages against <em>clean</em> builds of all their
dependencies.</p>
<p>In addition, Christian also spent significant amounts of time working inside the
OpenRISC community, debugging problems, testing and recompiling special versions
of the toolchain that we could use to advance in our compilation of the whole
archive. There were other people in the OpenRISC community implementing the
necessary bits in the toolchain, but I don't know the details.</p>
<h2>Good Progress</h2>
<p>Olof Kindgren wrote the <a href="http://olofkindgren.blogspot.nl/2014/05/openrisc-health-report-april-2014.html">OpenRISC health report April
2014</a>
(actually posted in May), explaining the status at the time of projects in the
broad OpenRISC community, and talking about the software side, Debian port
included. Sadly, I think that there have been no more "health reports" since
then. There was also a new post in Slashdot entitled <a href="http://hardware.slashdot.org/story/14/05/14/1213239/openrisc-gains-atomic-operations-and-multicore-support">OpenRISC Gains Atomic
Operations and Multicore
Support</a>
shortly thereafter.</p>
<p>In the side of the Debian port, from time to time new versions of packages
entered unstable and we started to use those newer versions. Some of them had
nice fixes, like the <code>autoreconf</code> updates, so they did not require local
modifications. In other cases, the new versions failed to build when old ones
had worked (e.g. because the newer versions added support and dependencies of
new versions of <code>gnutls</code>, <code>systemd</code> or other packages not yet available for
<code>or1k</code>), and we had to repeat or create more nasty hacks to get the packages
built again.</p>
<p>But in general, progress was very good. <strong>There were about 10k arch-dependent
packages in Debian at the time, and we got about half of them compiled by the
beginning of May</strong>, counting <em>clean</em> and <em>dirty</em>. And, if I recall correctly,
there were around the same number of arch=all (which can be installed in any
architecture, after the package is built in one of them). Counting both, it
meant that systems using <code>or1k</code> got about 15k packages available, or 75% of the
whole Debian archive (at least "main", we excluded "contrib" and "non-free").
Not bad.</p>
<p>By the middle to end of May, we had about 6k arch-dependent packages compiled,
and 4k to go. The count of packages eventually reached ~6.6k at its peak (I
think that in June/July). Many had been built with hacks and not rebuilt
cleanly yet, but everything was fine until the amount of packages built
plateaued.</p>
<h2>Plateauing</h2>
<p>There were multiple reasons for that. One of them is that after having fixed
the <code>autoreconf</code> issue locally in some packages, new versions were uploaded to
Debian without fixing that problem (in many cases there was no bug report or
patch yet, so it was understandable; in other cases the requests were ignored).
The <code>wanna-build</code> for the <em>clean</em> repository set up by Christian rightly
considered the package out-of-date and prepared to build the more recent
version, that failed. Then, other packages entering the unstable archive and
build-depending on newer versions of those could not be built
("BD-Uninstallable"), until we built the newer versions of the dependencies in
the <em>dirty</em> repository with local hacks. Consequently, the count of cleanly
built packages went back-and-forth, when not backwards.</p>
<p>More challenging was the fact that when creating a new port, language compilers
which are written in that same language need to be built for that architecture
first. Sometimes it is not the compiler, but compile-time or run-time support
for modules of a language are not ported yet. Obviously, as with other
dependencies, large amounts of packages written in those languages are bound to
remain uncompiled for a long time. As Colin Watson explained in <a href="http://www.chiark.greenend.org.uk/~cjwatson/blog/porting-ghc-a-tale-of-two-architectures.html">porting
Haskell's GHC to <code>arm64</code> and
<code>ppc64el</code></a>,
untangling some of the chicken-and-egg problems of language compilers for new
ports is extremely challenging.</p>
<p>Perl and Python are pretty much a pre-requisite of the base Debian system, and
Christian got them working early on. But for example in May, 247 packages
depended on <code>r-base-dev</code> (GNU R) for building, and 736 on <code>ghc</code>, and we did not
have these dependencies compiled. Just counting those two, 1k source packages
of the remaining 4k to 5k to be compiled for the new architecture would have to
wait for a long time. Then there was Java, Mono, etc...</p>
<p>Even more worrying problems were the <strong>pending issues with the toolchain</strong>, like
<em>atomics</em> in <code>glibc</code>, or <code>make check</code> failing for some packages in the <em>clean</em>
repository built with <code>wanna-build</code>. Christian continued to work on the
toolchain and liasing with the rest of the OpenRISC community, I continued to
request more changes to the Debian archive through a few requests to use
<code>autoreconf</code>, and pushing a few more NMUs. Though many requests were attended,
I soon got negative replies/reactions and backed off a bit. In the meantime,
the porters of other new architectures at the time were mostly submitting
requests to support them and not NMUing much either.</p>
<h2>Upstreaming</h2>
<p>Things continued more or less in the same state until the end of the summer.</p>
<p>The new ports needed as many packages built as possible before the evaluation of
which official ports to accept (in early September, I think, the final decision
around the time of the freeze). Porters of the other new architectures (and
maintainers, and other helpful Debian Developers) were by then more active in
pushing for changes, specially remaining <code>autoreconf</code> issues, many of which
benefited <code>or1k</code>. As I said before, I also kept pushing NMUs now and then,
specially during summer, for packages which were not of immediate benefit for
our port but helped the others (e.g. <code>ppc64el</code> needed updates to <code>ltmain.sh</code> for
<code>libtool</code> which were not necessary for <code>or1k</code>, in addition to
<code>config.{guess,sub}</code>).</p>
<p>In parallel in the <code>or1k</code> camp, <strong>there were patches that needed changes to be
sent upstream</strong>, like for example Python's <code>NumPy</code>, that I submitted in May to
the Debian package and upstream, and was uploaded to Debian in September with a
new upstream release. Similar paths were followed between May and September for
packages such as <code>jemalloc</code>, <code>ocaml</code>, <code>gstreamer0.10</code>, <code>libgc</code>, <code>mesa</code>, X.org's
<code>cf</code> module and <code>cmake</code> (patch created by Christian).</p>
<p>In April, Christian had reached the amazing milestone of tracking and getting
all of the contributors of the port of GNU <code>binutils</code> to assign copyright to the
Free Software Foundation (FSF), all of the work was refreshed and upstreamed.
In July or August, he started to gather information about the contributors of
the GCC port, which had started more than a decade ago.</p>
<p>After that, nothing much happened (from the outside) until the end of the year,
when Christian sent a <a href="http://lists.opencores.org/pipermail/openrisc/2014-December/001833.html">message about the status of upstreaming
GCC</a> to
the OpenRISC community. There was only one remaining person to assign the
copyright to the FSF, but it was a blocker. In addition, there was the need to
find one or more maintainers to liaise with upstream, review the patches, fix
the remaining failures in the test suite and keeping the port in good shape. A
few months after that and from what I could gather, the status remains the same.</p>
<h2>Current Status, and The Future?</h2>
<p>In terms of the Debian port, there have not been huge visible changes since the
end of the summer, and not only because of the
<a href="http://www.debian.org/releases/jessie/">Jessie</a> freeze.</p>
<p>It seems that for this effort to keep going forward and be sustainable,
<strong>sorting out the issues with GCC and Glibc is essential</strong>. That means having
the toolchain completely pushed upstream and in good shape, and particularly
<strong>completing the copyright assignment</strong>. Debian will not accept private forks
of those essential packages without a very good reason even in unofficially
supported ports; and from the point of view of porters, working in the remaining
not-yet-built packages with continuing problems in the toolchain is very
frustrating and time-consuming.</p>
<p>Other than that, there is already a significant amount of software available
that could run in an <code>or1k</code> system, so I think that <strong>overall the project has
achieved a significant amount of success</strong>. Granted, KDE and LibreOffice are
not available yet, neither are the tools depending on Haskell or Java. But a
lot of software is available (including things high in the stack, like XFCE),
and in many aspects it should provide a much more functional system that the one
available in Linux (or other free software) systems in the late 1990s. If the
blocking issues are sorted out in the near future, the effort needed to get a
very functional port, on par with the unofficial Debian ports, should not be
that big.</p>
<p>In my opinion, and looking at the big picture, not bad at all for an
architecture whose hardware implementation is not easy to come by, and in which
the port was created almost solely with simulators. That it has been possible
to get this far with such meagre resources, <strong>it's an amazing feat of Free
Software and Debian in particular</strong>.</p>
<p>As for the future, time will tell, as usual. I will try to keep you posted if
there is any significant change in the future.</p>Hallo, Planet Debian!2015-01-30T21:20:00+00:002015-01-30T21:20:00+00:00Manuel A. Fernandez Montecelotag:people.debian.org,2015-01-30:/~mafm/posts/2015/20150130_hallo-planet-debian/<p>Hi!</p>
<p>I just created this blog and asked to get it aggregated to the Debian Planet, so
first things first -- in the initial post I wanted to say <em>Hallooooo!</em> to the
people in the community and to talk about my work and interests in Debian.</p>
<p>Probably most of you never interacted with or even heard of me before. I am
contributor/Maintainer since ~2010 and Developer …</p><p>Hi!</p>
<p>I just created this blog and asked to get it aggregated to the Debian Planet, so
first things first -- in the initial post I wanted to say <em>Hallooooo!</em> to the
people in the community and to talk about my work and interests in Debian.</p>
<p>Probably most of you never interacted with or even heard of me before. I am
contributor/Maintainer since ~2010 and Developer only for ~2 years, and most of
the things that I worked in or packages that I maintain are "low profile" or
unintrusive, except perhaps for work in <code>aptitude</code> and <code>SDL</code> libraries, if you
happen to be interested in these packages or others that I [co-]maintain.</p>
<p>More recently, during 2014, I was helping to bootstrap and bring to life a new
architecture, <a href="http://opencores.org/or1k/Main_Page">OpenRISC or1k</a>. Perhaps I
should devote a future post to explain a bit more about the history, status and
news --or, lately, lack thereof-- about this port. This is one of the main
reasons why I thought that it would be useful to have a blog -- to register
activities for which it is difficult to get information by other means.</p>
<p>Apart from <strong>or1k</strong>, as it often happens with <a href="http://www.debian.org/ports/">porting efforts in
Debian</a>, the work done also helped indirectly
other architectures added last year (<code>mips64el</code>, <code>arm64</code>, <code>ppc64el</code>).
Additionally, a few of my NMUs were directly targetted to help <code>ppc64el</code> to get
ready in time for the architecture evaluation. None of the porters working in
<code>ppc64el</code> were Debian Maintainers/Developers and some of the patches that they
created did not benefit directly the other ports, so in the packages without
active maintainers, the requests were not getting much attention in the crucial
time before the evaluation. In some cases, these packages were in the critical
path to build many other packages or support important user-case scenarios.</p>
<p>So I was very pleased when I learnt that <a href="https://lists.debian.org/debian-devel-announce/2014/11/msg00005.html"><code>arm64</code> and <code>ppc64el</code> will be in the
next stable release --Jessie-- as officially supported
architectures</a>.
They came without much noise or ceremonious celebrations, but I think that this
is a great success story for these architectures and for Debian, and even for
the computing world in general. Time will tell. In the meantime,
congratulations to all people involved!</p>
<p>In addition to porting packages within Debian, I also sent some patches upstream
to get the packages that I maintain compiling in the new architectures, and sent
upstream patches needed to support <strong>or1k</strong> specifically (jemalloc, nspr, libgc,
cmake, components of X.org...).</p>
<p>Enough for the first post.</p>
<p>Just to finish, let me say that after about a decade without a personal website
or blog (the previous ones not about Debian or even computing), here I am again.
Let's see how it goes and I hope to have enough interesting things to tell you
to keep the blog alive.</p>
<p>Or, in other words... <em>“To infinity… and beyond!”</em></p>