#+TITLE: Beyond Trusting Open Source Software
#+AUTHOR: Vagrant Cascadian <vagrant@reproducible-builds.org>
#+EMAIL: vagrant@reproducible-builds.org
#+DATE: LinuxFest Northwest 2026-04-26
#+LANGUAGE:  en
#+OPTIONS:   H:1 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+OPTIONS: ^:nil
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+startup: beamer
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [bigger]
#+latex_header: \mode<beamer>{\usetheme{Madrid}}
#+LaTeX_CLASS_OPTIONS: [aspectratio=169]
#+BEGIN_comment
Beyond Trusting Open Source Software: Reproducible Builds
CC 203
Sun 3:00 pm - 3:30 pm
Lecture (30 Min + Q&A)

Software released under an open-source license and developed using an
open-source model come with many benefits, allowing the ability to
use, study, change, and share not only the software itself, but
similarly engage with a community around the software in a transparent
manner.

One of the strongest assertions is that open-source software is more
secure, as many parties are able to inspect the code. But most code in
the modern day is distributed as precompiled binary code,
indistinguishable from gibberish to many very savvy humans; this makes
the binary code largely impractical to audit. Blind trust is a bit
frightening for a security model!

Reproducible Builds provides a way to build trust that the binaries
produced are the intended result of the source code, by making it
possible for independent third-party verification of binaries to
produce bit-for-bit identical binaries.

This talk will introduce the concepts of Reproducible Builds,
including best practices for developing and releasing software, the
tools available to help diagnose issues, and touch on progress towards
solving a decades-old deeply pervasive security issue...

Learn how to demonstrate trust, rather than simply hoping for it!
#+END_comment

* Who am I

** image
	:PROPERTIES:
	:BEAMER_col: 0.4
	:END:

[[./images/vagrantupsidedown.png]]


** text
	:PROPERTIES:
	:BEAMER_col: 0.4
	:END:

  |                     | Vagrant |
  |---------------------+---------|
  | debian user         |    2001 |
  | debian developer    |    2010 |
  | reproducible builds |    2015 |

* Free and Open Source Software

Free and Open Source Software

#+ATTR_BEAMER: :overlay <+->
- Use
- Study
- Change
- Share
- Community

* A taste of source

from bash 5.0 assoc.c:

#+BEGIN_SRC C
assoc_insert (hash, key, value)
     HASH_TABLE *hash;
     char *key;
     char *value;
{
  BUCKET_CONTENTS *b;

  b = hash_search (key, hash, HASH_CREATE);
  if (b == 0)
    return -1;
  /* If we are overwriting an existing element's value, we're not going to
     use the key.  Nothing in the array assignment code path frees the key
     string, so we can free it here to avoid a memory leak. */
  if (b->key != key)
    free (key);
  FREE (b->data);
  b->data = value ? savestring (value) : (char *)0;
  return (0);
}
#+END_SRC

* Building the software

#+BEGIN_SRC shell
./configure
make
make install
#+END_SRC

* A resulting binary might look like

#+BEGIN_SRC shell

$ head /bin/bash
ELF&@@8 @@88TTTDDPtdDDQtdRtd0<0</lib/ld-linux-aarch64.so.1GNUy;OġUQGNU 04
                                                                          #!JzdAPDDB D  @AJ!Ih@i"r
NL@@@AB
0Iq(h   @(
          H &RD!D
                    $DP`
 @A4@ABf L0 dPCDDBE % 32BX@TD$
 @A%

!0`0@@bBh
         HBH
Xq@ Y       `1B
BdH(0"BB1@
          2
 s0 "Bi$DF0"B 6)4$
=HdL@0( 0D@kBDQH`$yh@(>5R @!% PH
b
RAbN@P@L.<:B@&
              JFD08 `
                     p0D@`
                           H`P30
                                 BL 9E4( B
#+END_SRC

* Reproducible Builds

** text
    :PROPERTIES:
    :BEAMER_col: 0.7
    :END:

https://reproducible-builds.org/docs/definition/

\vspace{\baselineskip}

A build is reproducible if given the same source code, build
environment and build instructions, any party can recreate bit-by-bit
identical copies of all specified artifacts.

** image
    :PROPERTIES:
    :BEAMER_col: 0.3
    :END:

[[./images/reproducible-builds.png]]

* Reproducible Builds At Scale

Debian

#+ATTR_BEAMER: :overlay <+->
- The Universal Operating System
- ~37000 source packages ... and counting
- 380 million lines of code ... and counting!
- ~96% reproducible

* Chaos and Freinds

https://reproducible-builds.org/docs/env-variations/

#+ATTR_BEAMER: :overlay <+->
- Timestamps
- User Information
- Host system information
- Randomness
- So many more!
- Especially Timestamps!

* Deterministic time?

SOURCE_DATE_EPOCH (seconds since 1970-01-01)

https://reproducible-builds.org/docs/source-date-epoch/

Supported in GCC, Clang, and more!

* So you want Reproducible builds

https://reproducible-builds.org/docs/recording/

Providing sufficient information for independent verification:

#+ATTR_BEAMER: :overlay <+->
- ...
- "toolchain" packages at specific versions
- SOURCE_DATE_EPOCH
- Works best with Free and Open Source Software!
- Automated testing (QA, CI, etc.)

* Reprotest

reprotest

#+ATTR_BEAMER: :overlay <+->
- builds something twice with many variations
- displays the differences between results
- https://salsa.debian.org/reproducible/reprotest
- if unreproducible: "bisect" the variations

* diffocope

https://diffoscope.org

\vspace{\baselineskip}

#+ATTR_BEAMER: :overlay <+->
- Recursive and human-readable "diff"
- locates and highlights reproducibility issues
- Supported on many distributions

* diffoscope example

[[./images/diffoscope.png]]

* diffoscope, supported file types

Android APK files, Android boot images, Ar(1) archives, Berkeley DB
database files, Bzip2 archives, Character/block devices, ColorSync
colour profiles (.icc), Coreboot CBFS filesystem images, Cpio
archives, Dalvik .dex files, Debian .buildinfo files, Debian .changes
files, Debian source packages (.dsc), Device Tree Compiler blob files,
Directories, ELF binaries, Ext2/ext3/ext4/btrfs filesystems,
FreeDesktop Fontconfig cache files, FreePascal files (.ppu), Gettext
message catalogues, GHC Haskell .hi files, GIF image files, Git
repositories, GNU R database files (.rdb), GNU R Rscript files (.rds),
Gnumeric spreadsheets, Gzipped files, ISO 9660 CD images, Java .class
files, JavaScript files, JPEG images, JSON files, LLVM IR bitcode
files, MacOS binaries, Microsoft Windows icon files, Microsoft Word
.docx files, Mono 'Portable Executable' files, Ogg Vorbis audio files,
OpenOffice .odt files, OpenSSH public keys, OpenWRT package archives
(.ipk), PDF documents, PGP signed/encrypted messages, PNG images,
PostScript documents, RPM archives, Rust object files (.deflate),
SQLite databases, SquashFS filesystems, Statically-linked binaries,
Symlinks, Tape archives (.tar), Tcpdump capture files (.pcap), Text
files, TrueType font files, XML binary schemas (.xsb), XML files, XZ
compressed files, etc.

* try diffoscope online

And on the World Wide Web!

https://try.diffoscope.org

[[./images/try.diffoscope.org.png]]

* What you get with Reproducible Builds

** text
    :PROPERTIES:
    :BEAMER_col: 0.7
    :END:

Reproducible Builds provides...

#+ATTR_BEAMER: :overlay <+->
- strong confidence...
- that a binary was produced from a given source...
- ...probably!

** image
    :PROPERTIES:
    :BEAMER_col: 0.3
    :END:
[[./images/reproducible-builds.png]]

* Trust

Different levels of trust:

  #+ATTR_BEAMER: :overlay <+->
- curl http://example.net/hackme | sudo sh
- curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
- download files, verify signatures ... run code
- download source, verify signature, compile from source
- emerge --emptytree @world
- rewrite everything in assembly
- build it up from transitors
- I have a beach, some wood, abundant sunshine, and a lot of time

* Trusting Trust

Ken Thompson

Reflections on Trusting Trust, 1984

https://archive.org/details/reflections-on-trusting-trust

https://research.swtch.com/nih

* Building on a solid foundation of turtles

  https://bootstrappable.org

Compiling your C compiler with a C compiler

And a C compiler to compile the other C compiler

...Ad infinitum

* Rust bootstrap

Rust bootstrapping

  #+ATTR_BEAMER: :overlay <+->
- rust 1.95 needs...
- rust 1.94 which needs...
- rust 1.93 which needs...
- ...
- rust 1.54 can be built with mrustc
- mrustc is written in C++
- breaking news, newer mrustc can bootstrap 1.90!

* Diverse Double Compiling

David A. Wheeler

Fully Countering Trusting Trust through Diverse Double-Compiling, 2009

https://dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html

* A beautiful Mes

GNU Mes is a Scheme interpreter and C compiler for bootstrapping the GNU System.

https://www.gnu.org/software/mes/

* We made the same Mes

Bit-for-bit identical Mes built on three different distributions

https://reproducible-builds.org/news/2019/12/21/reproducible-bootstrap-of-mes-c-compiler/

* Beginning with a Mes

GNU Guix: The Reduced Binary Seed Bootstrap

https://guix.gnu.org/en/manual/devel/en/guix.html#Reduced-Binary-Seed-Bootstrap

#+ATTR_BEAMER: :overlay <+->
- ...
- Reduced to 145MB of bootstrap binaries (from 250MB)
- Using Mes and guile...
- Builds from source GCC, binutils, glibc, etc.
- 145MB of binaries is still not really auditable...

* Before The Mes and Beyond

  GNU Guix: The Full-Source Bootstrap

https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/

  Now available via guix pull!

  #+ATTR_BEAMER: :overlay <+->
- hex0 (357-byte binary)
- hex1
- hex2
- M0
- cc_x86
- M2-Planet
- mescc-tools
- Mes
- TinyCC (patched)
- old versions of GCC, binutils, glibc, gzip, tar ...
- modern GCC and almost everything

* Make it live

  https://github.com/fosslinux/live-bootstrap

#+ATTR_BEAMER: :overlay <+->
- A live environment
- From kernel and a bit of source code
- To a reproducibly bootstrapped toolchain
- no pregenerated "source" code shortcuts

* Under that Turtle

How about...

...Without an operating system?

#+ATTR_BEAMER: :overlay <+->
- ...
- UEFI https://git.stikonas.eu/andrius/stage0-uefi
- Bare Metal https://git.savannah.nongnu.org/cgit/stage0.git/tree/

* Forget Trust, Verify

No need to Trust, all we need is:

  #+ATTR_BEAMER: :overlay <+->
- Free/Libre and Open Source Software
- Reproducible Builds
- Bootstrapping
- Diverse compilation
- ... and lots of compile cycles

* Thanks

Help make it happen!

https://reproducible-builds.org/contribute/

https://reproducible-builds.org/donate/

https://reproducible-builds.org/who/sponsors/

* Copyright and attributions
\addtocounter{framenumber}{-1}
\tiny

  Copyright 2019-2023 Vagrant Cascadian <vagrant@reproducible-builds.org>
  Portions by contributors to the reproducible-builds.org website.

  Copyright 2019 Holger Levsen <holger@layer-acht.org>

  This work is licensed under the Creative Commons
  Attribution-ShareAlike 4.0 International License.

  To view a copy of this license, visit
  https://creativecommons.org/licenses/by-sa/4.0/

snippet from bash assoc.c:

  Copyright (C) 2008,2009,2011 Free Software Foundation, Inc.

  Bash is free software: you can redistribute it and/or modify it
  under the terms of the GNU General Public License as published by
  the Free Software Foundation, either version 3 of the License, or
  (at your option) any later version.

  http://www.gnu.org/licenses/
