gagath’s debian devlog

lintian-ng parallel benchmark: tar.gz bottleneck

The current bottleneck of lintian-ng is decompressing the gzipped tar file in memory. Given the structure of gzip-compressed data and tar files, the reading can only be done by one thread. By delegating the checks to other threads, we try to make sure that the reading thread goes as fast as it can.

Function	Duration
parse_orig_seq	4.254 s
parse_orig_rayon	3.069 s
parse_orig_threadpool	2.815 s
parse_orig_read	2.339 s

The orig.tar.gz file used below is the content of an Ubuntu kernel package.

Check files in sequence

The naive implementation. Maybe this will be kept for systems with lower specs, maybe not.

fn parse_orig_seq(path: &Path) -> Result<Vec<Hint>, Box<dyn std::error::Error>> {
    let tar_gz = File::open(path)?;
    let tar = GzDecoder::new(tar_gz);
    let mut archive = Archive::new(tar);

    println!("checking files in sequence");

    let mut count = 0;
    let mut hints = Vec::new();

    for entry in archive.entries()? {
        let item = Item::try_from(&mut entry?)?;
        let mut res = AllChecks::visit_patched_files(&item);
        hints.append(&mut res);
        count = count + 1;
    }

    println!("checked {} items", count);

    Ok(hints)
}

Duration:

./target/release/lintian-ng orig.tar.gz  4.20s user 0.06s system 99% cpu 4.254 total

Check files in parallel, read everything in RAM first

This was me trying to use rayon, but it takes up big amounts of RAM and will be likely to fail on systems with not that much memory.

fn parse_orig_rayon(path: &Path) -> Result<Vec<Hint>, Box<dyn std::error::Error>> {
    let tar_gz = File::open(path)?;
    let tar = GzDecoder::new(tar_gz);
    let mut archive = Archive::new(tar);

    println!("reading everything into RAM");

    let mut items = Vec::new();

    for entry in archive.entries()? {
        let item = Item::try_from(&mut entry?)?;
        items.push(item);
    }

    println!("checking files in parallel RAM");

    let hints = items
        .par_iter()
        .map(AllChecks::visit_patched_files)
        .flatten()
        .collect();

    println!("checked {} items", items.len());

    Ok(hints)
}

Duration:

./target/release/lintian-ng orig.tar.gz --parallel  5.65s user 0.55s system 202% cpu 3.069 total

Check files in parallel, threadpool

Instead of trying to understand rayon producers and consumers, we can use a thread pool and send closures to it to check for every file we encounter.

fn parse_orig_threadpool(path: &Path) -> Result<Vec<Hint>, Box<dyn std::error::Error>> {
    use threadpool::ThreadPool;
    use std::sync::mpsc::channel;

    let tar_gz = File::open(path)?;
    let tar = GzDecoder::new(tar_gz);
    let mut archive = Archive::new(tar);

    println!("checking files using threadpool");

    let pool = ThreadPool::new(num_cpus::get());

    let (tx, rx) = channel();
    for entry in archive.entries()? {
        let tx = tx.clone();

        // Read item fields and content into RAM
        let item = Item::try_from(&mut entry?)?;

        // Send into thread pool
        pool.execute(move || {
            let hints = AllChecks::visit_patched_files(&item);
            tx.send(hints).expect("cannot send to main thread");
        })
    }

    // Drop the last sender to start collecting results
    drop(tx);

    let mut hints = Vec::new();

    while let Ok(h) = rx.recv() {
        let mut h = h;
        hints.append(&mut h);
    }

    Ok(hints)
}

Duration:

./target/release/lintian-ng orig.tar.gz --parallel  6.60s user 0.39s system 248% cpu 2.815 total

Just read the tar file

For reference.

fn parse_orig_read(path: &Path) -> Result<Vec<Hint>, Box<dyn std::error::Error>> {
    let tar_gz = File::open(path)?;
    let tar = GzDecoder::new(tar_gz);
    let mut archive = Archive::new(tar);

    println!("checking files in sequence");

    let mut count = 0;
    let hints = Vec::new();

    for entry in archive.entries()? {
        let _item = Item::try_from(&mut entry?)?;
        count = count + 1;
    }

    println!("checked {} items", count);

    Ok(hints)
}

Duration:

./target/release/lintian-ng orig.tar.gz  2.28s user 0.06s system 99% cpu 2.339 total

social