This "Fun to Program" is a work in progress.

If you find any issue, contact <osamu at debian.org> with a plain ascii (non-html) mail.

All contents are licensed as

(Pick any one of the licenses as you need if you take any parts of this. This is intended to be used for any purposes with least conflicts.)

Please note that the quoted part from the ioputs.c source remains in just LGPL (GNU Lesser General Public License).

What this is

Please consider this as a note of a self-study student.

This is not mean to be a tutorial document to explain how to program nor authoritative document by an expert.

I recorded how I tried to learn programing basics of the modern large software packages written in mixed languages using various complicated libraries. I provide the followings:

  • Learning records of my programing and building binaries.

  • URL links to the essential technical information for learning.

  • Minimal explanation and many simple examples.

  • Some summary list of things to remember.

  • Priority on POSIX shell, Python, and C. (on Debian system)

I hope this information is useful to others.

If the tailing parts of terminal example lines are dropped to fit within 80 character/line, they are indicated by ... at the end of the line. (Or, I forced them to fit using fold command.)

My local user name on this machine is osamu. You may need to replace it with your user name.

Please be careful about the publication date of reference documents since the software infrastructure is a moving target and such documentations may be obsoleted when you read this document.

Tip
If you find any questionable contents in this document, doubt them. If you find errors, let me know.

Background

To be a Debian developer, I encounter situations to deal with large software packages written in a combination of languages such as C, C++, Vala, Python, Perl, Shell, LaTex, DocBook, … with specialty cross platform build script generators such as autotools and cmake. Their build tools invoke commands such as ./configure, make, gcc, and g++. These software packages tend to be linked to many libraries which make them even more complex.

For example:

  • GNOME: gcc + Glib + GTK+ + D-bus + Autoconf + Automake + Autopoint + …

  • KDE: MOC (Metaobject compiler) + g++ + Qt (QtCore, QtGui, …) + D-bus + Cmake + …

Even with all these complexities, using these software package just for backporting is not usually out-of-reach for me as a novice programmer with the help of well designed packaging tools such as debhelper. The package build script can be deceptively simple as the following.

%:
        dh $@

But when I need to tweak something, I get overwhelmed with the complexity. I felt I have to practice program building and packaging with simpler case to understand its fundamentals and to become a better Debian developer.

Also, being a maintainer, it is essential to know how debugger works to use it effectively.

There are new major technical challenges such as multitarch and fortify of binary packages which I did not deal in "Debian New Maintainers' Guide". These needed to be addressed for newer packages, too.

Prerequisites

I expect readers of this to read the followings first.

I have browsed through tutorial contents of these documents … so I expect you to do the same.

Also, I recently found on-line textbooks from Green Tea Press. They are very informative.

Here are extra sources of information …

Tip
Install all dependency packages of devscripts, vim, lua, python, python3, perl, vala, swig, git, manpages-dev, devhelp, …

"Hello World!"

Here are example code snippets to print "Hello World!" and somethings we should be aware of for each language.

Shell

The most basic programing language: Shell.

  • Interpreter (slow)

  • Easy to write a small program.

  • Not easy to write a big program.

  • Test code snippet under the normal console (or by "sh -i").

Source code for the hello shell script
#!/bin/sh
# my first shell program
echo "Hello, world!"
Execution of the hello shell script
$ ls -l ./hello
-rwxr-xr-x 1 osamu osamu 56 Mar 11  2013 ./hello
$ file ./hello
./hello: POSIX shell script, ASCII text executable
$ ./hello
Hello, world!
Tip
Here, the interactive shell to be Bash and the non-interactive shell to be Dash unless explicitly mentioned. They are POSIX shell.

Python

  • Interpreter (with fast JIT compiler)

  • One of the most versatile language.

  • Guide you to write readable program.

  • Good for dynamic data.

  • Test code snippet under the console provided by "python3".

Source code for the hello Python script (simple)
#!/usr/bin/env python3
# My first Python program (bare bones)
print("Hello, world!")

Python programs are usually written as follows:

Source code for the hello Python script (with __name__)
#!/usr/bin/env python3
# My first Python program (nicer style)
def main():
    print("Hello, world!")

if __name__ == '__main__':
    main()

If you use wish to ensure to use the Python interpreter offered by the Debian system, the starting line should use as follows.

Source code for the hello Python script (force the Python interpreter offered by the Debian system)
#!/usr/bin/python3
def main():
    print("Hello, world!")

if __name__ == '__main__':
    main()
Execution of the hello Python script
$ ls -l ./hello
-rwxr-xr-x 1 osamu osamu 99 Mar 11  2013 ./hello
$ file ./hello
./hello: Python script, ASCII text executable
$ ./hello
Hello, world!

Lua

  • Interpreter

  • Small but very versatile.

  • Good for embedding into C/C++.

  • Good for dynamic data.

  • Test code snippet under the console provided by "lua".

Source code for the hello Lua script
#!/usr/bin/lua
-- my first lua program --
print("Hello, world!")
Execution of the hello Lua script
$ ls -l ./hello
-rwxr-xr-x 1 osamu osamu 65 Mar 11  2013 ./hello
$ file ./hello
./hello: Lua script, ASCII text executable
$ ./hello
Hello, world!

Perl

  • Interpreter (with fast JIT compiler)

  • One of the most versatile language with history.

  • Without discipline, you can write mess.

  • Good for dynamic data.

  • Test code snippet under the console provided by "perl -d -e 1".

Source code for the hello Perl script
#!/usr/bin/perl
# my first python program
print("Hello, world!\n");
Execution of the hello Perl script
$ ls -l ./hello
-rwxr-xr-x 1 osamu osamu 68 Mar 11  2013 ./hello
$ file ./hello
./hello: Perl script, ASCII text executable
$ ./hello
Hello, world!

C

  • Super fast binary (Compiler)

  • Very versatile.

  • Not so easy for dynamic data without special libraries.

  • You need to compile code to test it.

  • ABI compatibility issue is less complex.

Source code in C language
#include <stdio.h>
#include <stdlib.h>
/* my first C program */
int main()
{
    printf("Hello, world!\n");
    return EXIT_SUCCESS;
}

Here, I need to include header files:

  • "#include <stdio.h>" for printf.

  • "#include <stdlib.h>" for EXIT_SUCCESS. (EXIT_SUCCESS = 0)

Let’s compile hello.c to create the ELF object hello and run it.

$ ls -l ./hello
-rwxrwxr-x 1 osamu osamu 6696 Aug 21 16:53 ./hello
$ file ./hello
./hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked
, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=2
1d6168f6909d2c2ea3ca54ae75a1cbc0dd5f8fd, not stripped
$ ./hello
Hello, world!

Let’s list linked libraries to the ELF object hello.

    linux-vdso.so.1 (0x00007ffeaeb9a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0dffdae000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0e00157000)

Here, notable libraries are:

  • linux-vdso.so.1 : Linux Virtual Dynamic Shared Object

  • libc.so.6 : The GNU C Library (glibc)

  • /lib64/ld-linux-x86-64.so.2 : dynamic linker/loader

Let’s list symbols defined in the ELF object hello.

                 w __gmon_start__
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses
                 U __libc_start_main@@GLIBC_2.2.5
                 U puts@@GLIBC_2.2.5
00000000004003a8 T _init
0000000000400410 T _start
0000000000400440 t deregister_tm_clones
0000000000400480 t register_tm_clones
00000000004004c0 t __do_global_dtors_aux
00000000004004e0 t frame_dummy
0000000000400506 T main
0000000000400520 T __libc_csu_init
0000000000400590 T __libc_csu_fini
0000000000400594 T _fini
00000000004005a0 R _IO_stdin_used
00000000004006d8 r __FRAME_END__
00000000006006e0 t __frame_dummy_init_array_entry
00000000006006e0 t __init_array_start
00000000006006e8 t __do_global_dtors_aux_fini_array_entry
00000000006006e8 t __init_array_end
00000000006006f0 d __JCR_END__
00000000006006f0 d __JCR_LIST__
00000000006006f8 d _DYNAMIC
00000000006008d0 d _GLOBAL_OFFSET_TABLE_
0000000000600900 D __data_start
0000000000600900 W data_start
0000000000600908 D __dso_handle
0000000000600910 B __bss_start
0000000000600910 b completed.6661
0000000000600910 D _edata
0000000000600910 D __TMC_END__
0000000000600918 B _end

Here, symbol types are:

  • A : value is absolute

  • b : uninitialized data (local)

  • D : initialized data (global)

  • d : initialized data (local)

  • R : read only data (global)

  • r : read only data (local)

  • T : text (code) section (global)

  • t : text (code) section (local)

  • U : undefined

  • w : weak symbol

  • W : weak symbol

Let’s look into how hello.c is compiled by stopping before the assembler by creating assembler code as hello.s with the -S option.

$ cat hello.s
    .file    "hello.c"
    .section    .rodata
.LC0:
    .string    "Hello, world!"
    .text
    .globl    main
    .type    main, @function
main:
.LFB2:
    .cfi_startproc
    pushq    %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    movl    $.LC0, %edi
    call    puts
    movl    $0, %eax
    popq    %rbp
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE2:
    .size    main, .-main
    .ident    "GCC: (Debian 4.9.2-10) 4.9.2"
    .section    .note.GNU-stack,"",@progbits

Please note this is written in the AT&T assembler style. We will get back to its details later.

Tip
Several lines with .cfi_... in the assembler code are the DWARF CFI directives which help debugger to do backtrace on the modern ABI system without frame pointers (FP). See CFI support for GNU assembler (GAS).

C++

  • Super fast binary (Compiler)

  • Very versatile.

  • Good for dynamic data.

  • You need to compile code to test it.

  • ABI compatibility issue is more complex.

Source code in C++ language
#include <iostream>
#include <stdlib.h>
//  my first C++ program
int main ()
{
    using namespace std;
    cout << "Hello World!" << endl;
    return EXIT_SUCCESS;
}

Let’s compile hello.cxx to create the ELF object hello and run it.

$ ls -l ./hello
-rwxrwxr-x 1 osamu osamu 8376 Aug 21 16:53 ./hello
$ file ./hello
./hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked,...
$ ./hello
Hello World!

Let’s list linked libraries to the ELF object hello.

    linux-vdso.so.1 (0x00007ffc141c7000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9638a4500...
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9638744000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f963852e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9638185000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9638d50000)
  • linux-vdso.so.1 : Linux Virtual Dynamic Shared Object

  • libstdc++.so.6 : The GNU Standard C++ Library

  • libm.so.6 : The GNU C Library (glibc, support math functions)

  • libgcc_s.so.1 : The runtime library of GCC

  • libc.so.6 : The GNU C Library (glibc)

  • /lib64/ld-linux-x86-64.so.2 : dynamic linker/loader

Let’s list symbols defined in the ELF object hello.

                 U __cxa_atexit@@GLIBC_2.2.5
                 w __gmon_start__
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses
                 U __libc_start_main@@GLIBC_2.2.5
                 U _ZNSolsEPFRSoS_E@@GLIBCXX_3.4
                 U _ZNSt8ios_base4InitC1Ev@@GLIBCXX_3.4
                 U _ZNSt8ios_base4InitD1Ev@@GLIBCXX_3.4
                 U _ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_@@GL...
                 U _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc@@GLIBC...
0000000000400698 T _init
0000000000400750 T _start
0000000000400780 t deregister_tm_clones
00000000004007c0 t register_tm_clones
0000000000400800 t __do_global_dtors_aux
0000000000400820 t frame_dummy
0000000000400846 T main
000000000040086d t _Z41__static_initialization_and_destruction_0ii
00000000004008aa t _GLOBAL__sub_I_main
00000000004008c0 T __libc_csu_init
0000000000400930 T __libc_csu_fini
0000000000400934 T _fini
0000000000400940 R _IO_stdin_used
0000000000400ac8 r __FRAME_END__
0000000000600ad0 t __frame_dummy_init_array_entry
0000000000600ad0 t __init_array_start
0000000000600ae0 t __do_global_dtors_aux_fini_array_entry
0000000000600ae0 t __init_array_end
0000000000600ae8 d __JCR_END__
0000000000600ae8 d __JCR_LIST__
0000000000600af0 d _DYNAMIC
0000000000600cf8 d _GLOBAL_OFFSET_TABLE_
0000000000600d50 D __data_start
0000000000600d50 W data_start
0000000000600d58 D __dso_handle
0000000000600d60 B __bss_start
0000000000600d60 D _edata
0000000000600d60 D __TMC_END__
0000000000600d80 B _ZSt4cout@@GLIBCXX_3.4
0000000000600e90 b completed.6661
0000000000600e91 b _ZStL8__ioinit
0000000000600e98 B _end

There are many symbols starting with _Z. These are mangled symbols. We need to demangle these symbol with c++filt to make this readable.

                 U __cxa_atexit@@GLIBC_2.2.5
                 w __gmon_start__
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses
                 U __libc_start_main@@GLIBC_2.2.5
                 U std::basic_ostream<char, std::char_traits<char> >::operator<<
(std::basic_ostream<char, std::char_traits<char> >& (*)(std::basic_ostream<char,
 std::char_traits<char> >&))@@GLIBCXX_3.4
                 U std::ios_base::Init::Init()@@GLIBCXX_3.4
                 U std::ios_base::Init::~Init()@@GLIBCXX_3.4
                 U std::basic_ostream<char, std::char_traits<char> >& std::endl<
char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >
&)@@GLIBCXX_3.4
                 U std::basic_ostream<char, std::char_traits<char> >& std::opera
tor<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char>
>&, char const*)@@GLIBCXX_3.4
0000000000400698 T _init
0000000000400750 T _start
0000000000400780 t deregister_tm_clones
00000000004007c0 t register_tm_clones
0000000000400800 t __do_global_dtors_aux
0000000000400820 t frame_dummy
0000000000400846 T main
000000000040086d t __static_initialization_and_destruction_0(int, int)
00000000004008aa t _GLOBAL__sub_I_main
00000000004008c0 T __libc_csu_init
0000000000400930 T __libc_csu_fini
0000000000400934 T _fini
0000000000400940 R _IO_stdin_used
0000000000400ac8 r __FRAME_END__
0000000000600ad0 t __frame_dummy_init_array_entry
0000000000600ad0 t __init_array_start
0000000000600ae0 t __do_global_dtors_aux_fini_array_entry
0000000000600ae0 t __init_array_end
0000000000600ae8 d __JCR_END__
0000000000600ae8 d __JCR_LIST__
0000000000600af0 d _DYNAMIC
0000000000600cf8 d _GLOBAL_OFFSET_TABLE_
0000000000600d50 D __data_start
0000000000600d50 W data_start
0000000000600d58 D __dso_handle
0000000000600d60 B __bss_start
0000000000600d60 D _edata
0000000000600d60 D __TMC_END__
0000000000600d80 B std::cout@@GLIBCXX_3.4
0000000000600e90 b completed.6661
0000000000600e91 b std::__ioinit
0000000000600e98 B _end

Although the source code for both C and C++ are as simple, C++ creates more complicated code linked to more libraries with mangled symbols.

Let’s look into how hello.cxx is compiled by stopping before the assembler by creating assembler code with the -S option and demangling it with c++filt -n.

$ c++filt -n <hello.s |fold
    .file    "hello.cxx"
    .local    std::__ioinit
    .comm    std::__ioinit,1,1
    .section    .rodata
.LC0:
    .string    "Hello World!"
    .text
    .globl    main
    .type    main, @function
main:
.LFB1020:
    .cfi_startproc
    pushq    %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    movl    $.LC0, %esi
    movl    std::cout, %edi
    call    std::basic_ostream<char, std::char_traits<char> >& std::operator
<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&,
 char const*)
    movl    std::basic_ostream<char, std::char_traits<char> >& std::endl<cha
r, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&),
 %esi
    movq    %rax, %rdi
    call    std::basic_ostream<char, std::char_traits<char> >::operator<<(st
d::basic_ostream<char, std::char_traits<char> >& (*)(std::basic_ostream<char, st
d::char_traits<char> >&))
    movl    $0, %eax
    popq    %rbp
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE1020:
    .size    main, .-main
    .type    __static_initialization_and_destruction_0(int, int), @function
__static_initialization_and_destruction_0(int, int):
.LFB1029:
    .cfi_startproc
    pushq    %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    subq    $16, %rsp
    movl    %edi, -4(%rbp)
    movl    %esi, -8(%rbp)
    cmpl    $1, -4(%rbp)
    jne    .L3
    cmpl    $65535, -8(%rbp)
    jne    .L3
    movl    std::__ioinit, %edi
    call    std::ios_base::Init::Init()
    movl    $__dso_handle, %edx
    movl    std::__ioinit, %esi
    movl    std::ios_base::Init::~Init(), %edi
    call    __cxa_atexit
.L3:
    leave
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE1029:
    .size    __static_initialization_and_destruction_0(int, int), .-__static_
initialization_and_destruction_0(int, int)
    .type    _GLOBAL__sub_I_main, @function
_GLOBAL__sub_I_main:
.LFB1030:
    .cfi_startproc
    pushq    %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    movl    $65535, %esi
    movl    $1, %edi
    call    __static_initialization_and_destruction_0(int, int)
    popq    %rbp
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE1030:
    .size    _GLOBAL__sub_I_main, .-_GLOBAL__sub_I_main
    .section    .init_array,"aw"
    .align 8
    .quad    _GLOBAL__sub_I_main
    .hidden    __dso_handle
    .ident    "GCC: (Debian 4.9.2-10) 4.9.2"
    .section    .note.GNU-stack,"",@progbits

For more on C++ ABI issues, see:

Vala

  • C#, Java like source code

  • Super fast binary (C compiler as its backend, no byte code interpreter)

  • Easy support for GLib and GObject.

  • Good for dynamic data.

  • Automatic memory management.

  • You need to compile code to test it.

Simple non-OOP style (no class)

Source code hello-1.vala in Vala language
int main(string[] args) {
    stdout.printf("Hello, world!\n");
    return 0;
}

Let’s compile hello-1.vala to create the ELF object hello-1 and run it.

Loaded package `/usr/share/vala-0.26/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.26/vapi/gobject-2.0.vapi'
cc -o '/path/to/vala/hello-1' '/path/to/vala/hello-1.vala.c' -I/usr/include/glib-...
$ ./hello-1
Hello, world!

You can get the C source as:

$ wc -l hello-1.vala ; wc -l hello-1.c
4 hello-1.vala
35 hello-1.c
$ cat hello-1.c
/* hello-1.c generated by valac 0.26.1, the Vala compiler
 * generated from hello-1.vala, do not modify */


#include <glib.h>
#include <glib-object.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>




gint _vala_main (gchar** args, int args_length1);


gint _vala_main (gchar** args, int args_length1) {
    gint result = 0;
    FILE* _tmp0_ = NULL;
    _tmp0_ = stdout;
    fprintf (_tmp0_, "Hello, world!\n");
    result = 0;
    return result;
}


int main (int argc, char ** argv) {
#if !GLIB_CHECK_VERSION (2,35,0)
    g_type_init ();
#endif
    return _vala_main (argv, argc);
}


Since no OOP techniques are used, the resulting C code is logically as simple as the Vala code except for some #include directives to pertinent libraries automatically included for the Vala code.

OOP style (main outside of class)

Source code hello-2.vala in the Vala language
using GLib;
public class Demo.HelloWorld : GLib.Object {
    public int hello() {
        stdout.printf("Hello, world!\n");
        return 0;
    }
}

int main(string[] args) {
    var obj = new Demo.HelloWorld();
    obj.hello();
    return 0;
}
Tip
Since the use of GLib for the parent class is the default for Vala, the line of using GLib; and the prefix of GLib. in the above example may be dropped as an exception.

Let’s compile hello-2.vala to create the ELF object hello-2 and run it.

Loaded package `/usr/share/vala-0.26/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.26/vapi/gobject-2.0.vapi'
cc -o '/path/to/vala/hello-2' '/path/to/vala/hello-2.vala.c' -I/usr/include/glib-...
$ ./hello-2
Hello, world!

You can get the C source as:

$ wc -l hello-2.vala ; wc -l hello-2.c
13 hello-2.vala
111 hello-2.c
$ cat hello-2.c|sed -e 's/      /    /g'|fold # tab => 4 spaces
/* hello-2.c generated by valac 0.26.1, the Vala compiler
 * generated from hello-2.vala, do not modify */


#include <glib.h>
#include <glib-object.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>


#define DEMO_TYPE_HELLO_WORLD (demo_hello_world_get_type ())
#define DEMO_HELLO_WORLD(obj) (G_TYPE_CHECK_INSTANCE_CAST ((obj), DEMO_TYPE_HELL
O_WORLD, DemoHelloWorld))
#define DEMO_HELLO_WORLD_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST ((klass), DEMO_TY
PE_HELLO_WORLD, DemoHelloWorldClass))
#define DEMO_IS_HELLO_WORLD(obj) (G_TYPE_CHECK_INSTANCE_TYPE ((obj), DEMO_TYPE_H
ELLO_WORLD))
#define DEMO_IS_HELLO_WORLD_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass), DEMO
_TYPE_HELLO_WORLD))
#define DEMO_HELLO_WORLD_GET_CLASS(obj) (G_TYPE_INSTANCE_GET_CLASS ((obj), DEMO_
TYPE_HELLO_WORLD, DemoHelloWorldClass))

typedef struct _DemoHelloWorld DemoHelloWorld;
typedef struct _DemoHelloWorldClass DemoHelloWorldClass;
typedef struct _DemoHelloWorldPrivate DemoHelloWorldPrivate;
#define _g_object_unref0(var) ((var == NULL) ? NULL : (var = (g_object_unref (va
r), NULL)))

struct _DemoHelloWorld {
    GObject parent_instance;
    DemoHelloWorldPrivate * priv;
};

struct _DemoHelloWorldClass {
    GObjectClass parent_class;
};


static gpointer demo_hello_world_parent_class = NULL;

GType demo_hello_world_get_type (void) G_GNUC_CONST;
enum  {
    DEMO_HELLO_WORLD_DUMMY_PROPERTY
};
gint demo_hello_world_hello (DemoHelloWorld* self);
DemoHelloWorld* demo_hello_world_new (void);
DemoHelloWorld* demo_hello_world_construct (GType object_type);
gint _vala_main (gchar** args, int args_length1);


gint demo_hello_world_hello (DemoHelloWorld* self) {
    gint result = 0;
    FILE* _tmp0_ = NULL;
    g_return_val_if_fail (self != NULL, 0);
    _tmp0_ = stdout;
    fprintf (_tmp0_, "Hello, world!\n");
    result = 0;
    return result;
}


DemoHelloWorld* demo_hello_world_construct (GType object_type) {
    DemoHelloWorld * self = NULL;
    self = (DemoHelloWorld*) g_object_new (object_type, NULL);
    return self;
}


DemoHelloWorld* demo_hello_world_new (void) {
    return demo_hello_world_construct (DEMO_TYPE_HELLO_WORLD);
}


static void demo_hello_world_class_init (DemoHelloWorldClass * klass) {
    demo_hello_world_parent_class = g_type_class_peek_parent (klass);
}


static void demo_hello_world_instance_init (DemoHelloWorld * self) {
}


GType demo_hello_world_get_type (void) {
    static volatile gsize demo_hello_world_type_id__volatile = 0;
    if (g_once_init_enter (&demo_hello_world_type_id__volatile)) {
        static const GTypeInfo g_define_type_info = { sizeof (DemoHelloWorldClas
s), (GBaseInitFunc) NULL, (GBaseFinalizeFunc) NULL, (GClassInitFunc) demo_hello_
world_class_init, (GClassFinalizeFunc) NULL, NULL, sizeof (DemoHelloWorld), 0, (
GInstanceInitFunc) demo_hello_world_instance_init, NULL };
        GType demo_hello_world_type_id;
        demo_hello_world_type_id = g_type_register_static (G_TYPE_OBJECT, "DemoH
elloWorld", &g_define_type_info, 0);
        g_once_init_leave (&demo_hello_world_type_id__volatile, demo_hello_world
_type_id);
    }
    return demo_hello_world_type_id__volatile;
}


gint _vala_main (gchar** args, int args_length1) {
    gint result = 0;
    DemoHelloWorld* obj = NULL;
    DemoHelloWorld* _tmp0_ = NULL;
    _tmp0_ = demo_hello_world_new ();
    obj = _tmp0_;
    demo_hello_world_hello (obj);
    result = 0;
    _g_object_unref0 (obj);
    return result;
}


int main (int argc, char ** argv) {
#if !GLIB_CHECK_VERSION (2,35,0)
    g_type_init ();
#endif
    return _vala_main (argv, argc);
}


It is obvious Vala code is much shorter than C.

The Vala class definition of Demo.HelloWorld subclassed from GLib.Object are converted into the C code defining GObject with many macros. These C code macros are cliches to use GLib and GObject in C. The Vala compiler takes care these chores for you.

  • demo, DEMO, Demo: prefix taken from library or application name

  • hello_world, HELLO_WORLD, HelloWorld: object type (= class) name

  • demo_hello_world_get_type(): returning GType of object HELLO_WORLD defined in DEMO

The instantiation of the obj object by the Vala new expression is converted into the C demo_hello_world_new() function which in turn calls the C demo_hello_world_construct() function. The dynamic memory management by the reference counting in the GLib is used to assign memory to the obj object.

The Vala also adds a wrapper macro _g_object_unref0(obj) to the C code which is required to free memory properly for the obj object.

In general, these macro cliches of GObject are defined as follows:

#define FOO_TYPE_BAR            (foo_bar_get_type ())
#define FOO_BAR(obj)            (G_TYPE_CHECK_INSTANCE_CAST ((obj), \
                                FOO_TYPE_BAR, FooBar))
#define FOO_BAR_CLASS(klass)    (G_TYPE_CHECK_CLASS_CAST ((klass),  \
                                FOO_TYPE_BAR, FooBarClass))
#define FOO_IS_BAR(obj)         (G_TYPE_CHECK_INSTANCE_TYPE ((obj), \
                                FOO_TYPE_BAR))
#define FOO_IS_BAR_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass),  \
                                FOO_TYPE_BAR))
#define FOO_BAR_GET_CLASS(obj)  (G_TYPE_INSTANCE_GET_CLASS ((obj),  \
                                FOO_TYPE_BAR, FooBarClass))

Here:

  • foo, FOO, Foo: prefix taken from library or application name (Optional)

  • bar, BAR, Bar: object type (= class) name

  • foo_bar_get_type(): returning GType of object BAR defined in FOO

OOP style (main inside of class)

Source code hello-3.vala in the Vala language
using GLib;
public class Demo.HelloWorld : GLib.Object {
    private int hello() {
        stdout.printf("Hello, world!\n");
        return 0;
    }
    public static int main(string[] args) {
        var obj = new Demo.HelloWorld();
        obj.hello();
        return 0;
    }
}
Tip
Since the use of GLib for the parent class is the default for Vala, the line of using GLib; and the prefix of GLib. in the above example may be dropped as an exception.

Let’s compile hello-3.vala to create the ELF object hello-3 and run it.

Loaded package `/usr/share/vala-0.26/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.26/vapi/gobject-2.0.vapi'
cc -o '/path/to/vala/hello-3' '/path/to/vala/hello-3.vala.c' -I/usr/include/glib-...
$ ./hello-3
Hello, world!

You can get the C source as:

$ wc -l hello-3.vala ; wc -l hello-3.c
13 hello-3.vala
111 hello-3.c
$ cat hello-3.c|sed -e 's/      /    /g'|fold # tab => 4 spaces
/* hello-3.c generated by valac 0.26.1, the Vala compiler
 * generated from hello-3.vala, do not modify */


#include <glib.h>
#include <glib-object.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>


#define DEMO_TYPE_HELLO_WORLD (demo_hello_world_get_type ())
#define DEMO_HELLO_WORLD(obj) (G_TYPE_CHECK_INSTANCE_CAST ((obj), DEMO_TYPE_HELL
O_WORLD, DemoHelloWorld))
#define DEMO_HELLO_WORLD_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST ((klass), DEMO_TY
PE_HELLO_WORLD, DemoHelloWorldClass))
#define DEMO_IS_HELLO_WORLD(obj) (G_TYPE_CHECK_INSTANCE_TYPE ((obj), DEMO_TYPE_H
ELLO_WORLD))
#define DEMO_IS_HELLO_WORLD_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass), DEMO
_TYPE_HELLO_WORLD))
#define DEMO_HELLO_WORLD_GET_CLASS(obj) (G_TYPE_INSTANCE_GET_CLASS ((obj), DEMO_
TYPE_HELLO_WORLD, DemoHelloWorldClass))

typedef struct _DemoHelloWorld DemoHelloWorld;
typedef struct _DemoHelloWorldClass DemoHelloWorldClass;
typedef struct _DemoHelloWorldPrivate DemoHelloWorldPrivate;
#define _g_object_unref0(var) ((var == NULL) ? NULL : (var = (g_object_unref (va
r), NULL)))

struct _DemoHelloWorld {
    GObject parent_instance;
    DemoHelloWorldPrivate * priv;
};

struct _DemoHelloWorldClass {
    GObjectClass parent_class;
};


static gpointer demo_hello_world_parent_class = NULL;

GType demo_hello_world_get_type (void) G_GNUC_CONST;
enum  {
    DEMO_HELLO_WORLD_DUMMY_PROPERTY
};
static gint demo_hello_world_hello (DemoHelloWorld* self);
gint demo_hello_world_main (gchar** args, int args_length1);
DemoHelloWorld* demo_hello_world_new (void);
DemoHelloWorld* demo_hello_world_construct (GType object_type);


static gint demo_hello_world_hello (DemoHelloWorld* self) {
    gint result = 0;
    FILE* _tmp0_ = NULL;
    g_return_val_if_fail (self != NULL, 0);
    _tmp0_ = stdout;
    fprintf (_tmp0_, "Hello, world!\n");
    result = 0;
    return result;
}


gint demo_hello_world_main (gchar** args, int args_length1) {
    gint result = 0;
    DemoHelloWorld* obj = NULL;
    DemoHelloWorld* _tmp0_ = NULL;
    _tmp0_ = demo_hello_world_new ();
    obj = _tmp0_;
    demo_hello_world_hello (obj);
    result = 0;
    _g_object_unref0 (obj);
    return result;
}


int main (int argc, char ** argv) {
#if !GLIB_CHECK_VERSION (2,35,0)
    g_type_init ();
#endif
    return demo_hello_world_main (argv, argc);
}


DemoHelloWorld* demo_hello_world_construct (GType object_type) {
    DemoHelloWorld * self = NULL;
    self = (DemoHelloWorld*) g_object_new (object_type, NULL);
    return self;
}


DemoHelloWorld* demo_hello_world_new (void) {
    return demo_hello_world_construct (DEMO_TYPE_HELLO_WORLD);
}


static void demo_hello_world_class_init (DemoHelloWorldClass * klass) {
    demo_hello_world_parent_class = g_type_class_peek_parent (klass);
}


static void demo_hello_world_instance_init (DemoHelloWorld * self) {
}


GType demo_hello_world_get_type (void) {
    static volatile gsize demo_hello_world_type_id__volatile = 0;
    if (g_once_init_enter (&demo_hello_world_type_id__volatile)) {
        static const GTypeInfo g_define_type_info = { sizeof (DemoHelloWorldClas
s), (GBaseInitFunc) NULL, (GBaseFinalizeFunc) NULL, (GClassInitFunc) demo_hello_
world_class_init, (GClassFinalizeFunc) NULL, NULL, sizeof (DemoHelloWorld), 0, (
GInstanceInitFunc) demo_hello_world_instance_init, NULL };
        GType demo_hello_world_type_id;
        demo_hello_world_type_id = g_type_register_static (G_TYPE_OBJECT, "DemoH
elloWorld", &g_define_type_info, 0);
        g_once_init_leave (&demo_hello_world_type_id__volatile, demo_hello_world
_type_id);
    }
    return demo_hello_world_type_id__volatile;
}


It is obvious Vala code is much shorter than C. Many CPP macros are defined here, too. These are cliches to use GLib and GObject in C. The Vala compiler takes care these chores for you.

  • demo, DEMO, Demo: prefix taken from library or application name

  • hello_world, HELLO_WORLD, HelloWorld: object type (= class) name

  • demo_hello_world_get_type(): returning GType of object HELLO_WORLD defined in DEMO

CLI programs

The command line interface (CLI) program requires to parse its command line arguments. Here are simple example code snippets.

Please note I chose the coding style which is the most legible for each language. I know there are other ways …

Shell

Shell script (short option only)

Shell program with short command line options.

Source code for the shell script: cli_s
#!/bin/sh
verbose=0
while getopts "af:v" opt ; do
    case $opt in
    a)  echo "Option a"
        ;;
    f)  cat "$OPTARG"
        ;;
    v)  verbose=1
        ;;
    -)  break
        ;;
    ?)  echo "Usage: $0: [-a] [-v] [-f file] args ..."
        exit 1
        ;;
    esac
done
shift $(($OPTIND - 1))
printf "Remaining arguments are: %s\n" "$*"

Please note shell variables are used.

  • OPTARG : option argument

  • OPTIND : index of the next element to be processed

Tip
No long option like "--file ...".

The POSIX getopts is available as a shell buildin command of BASH/DASH. (Please note s at the end. The older Bell Labs derived external command is called getopt.)

Shell script (long and short option)

Shell program with long and short command line options.

Source code for the shell script: cli_l
#!/bin/sh
verbose=0
while [ "$#" -gt "0" ]; do
    case $1 in
    -a)    echo "Option a"
        ;;
    -f|--file)
        shift
        cat $1
        ;;
    -v)    verbose=1
        ;;
    --long)    echo "Option long"
        ;;
    --longextra)
        shift
        echo "Option longextra with $1"
        ;;
    --)    shift
        break
        ;;
    -*)    echo "Unknown option: $1"
        echo "Usage: $0 [-a] [-f|--file file] [-v] args ..."
        exit 1
        ;;
    *)    break
        ;;
    esac
    shift
done
printf "Remaining arguments are: %s\n" "$*"
Tip
You must type as "-a -f ..." and not as "-af ..."

Here I avoided the use of getopts/getopt for long option support to gain maximum portability.

Python

Python script with argparse

Simple example program with long and short command line options.

Source code for Python with argparse
#!/usr/bin/python3
# vim:se tw=0 sts=4 ts=4 et cindent:
def main():
    import argparse, sys
    parser = argparse.ArgumentParser(description='This is skelton program.')
    parser.add_argument('-a', action="store_true", dest="opt_a",
                    default=False, help='operation mode')
    parser.add_argument('-f', '--file', action="store", dest="filename",
                    help='input filename')
    parser.add_argument('-v', action="store_true", dest="verbose",
                    default=False, help='vebosity')
    parser.add_argument('args', nargs='*', help='arguments')
    parser.add_argument('--long', action="store_true", dest="opt_long",
                    help='long mode')
    parser.add_argument('--longextra', action="store", dest="long",
                    help='long extra mode')
    args = parser.parse_args()
    if args.opt_a:
        print("option a")
    if args.filename:
        print("filename = " + args.filename)
        with open(args.filename) as f:
            for line in f:
                print (line[:-1])
    if args.verbose:
        print("option verbose")
    if args.opt_long:
        print("option long")
    if args.long:
        print("longextra = " + args.long)
    if args.args:
        print("args = " , args.args)

if __name__ == '__main__':
    main()
Execution examples with auto generated help
usage: cli [-h] [-a] [-f FILENAME] [-v] [--long] [--longextra LONG]
           [args [args ...]]

This is skelton program.

positional arguments:
  args                  arguments

optional arguments:
  -h, --help            show this help message and exit
  -a                    operation mode
  -f FILENAME, --file FILENAME
                        input filename
  -v                    vebosity
  --long                long mode
  --longextra LONG      long extra mode
$ ./cli -avf BOGUS.txt --long --longextra=FOO cli BOGUS.txt
option a
filename = BOGUS.txt
This is BOGUS.txt here.
EOF here :-)
option verbose
option long
longextra = FOO
args =  ['cli', 'BOGUS.txt']
$ ./cli -b
usage: cli [-h] [-a] [-f FILENAME] [-v] [--long] [--longextra LONG]
           [args [args ...]]
cli: error: unrecognized arguments: -b
Tip
Older programs may use getopt module which is not as object oriented as argparse.

See:

Actual module file on the system can be obtained by running the Python interpreter interactively as follows.

$ /usr/bin/python3 -q
>>> import argparse
>>> print (argparse.__file__)
/usr/lib/python3.2/argparse.py

Python script with cliapp

cliapp is a Python framework for Unix-like command line programs and provides a unified supports for the command line and the configuration file.

This seems to be interesting and is used by the backup program obnam(1).

Source code for Python with cliapp
#!/usr/bin/env python
# vim:se tw=0 sts=4 ts=4 et cindent:
'''
Example using cliapp framework. (Python2 only)
'''
import cliapp
import logging

class ExampleApp(cliapp.Application):

    def add_settings(self):
        self.settings.boolean(['append', 'a'], 'append mode')
        self.settings.string(['file', 'f'], 'configuration filename',
                metavar='FILENAME')
        self.settings.boolean(['verbose', 'v'], 'verbose mode')
        self.settings.boolean(['long'], 'long mode')
        self.settings.string(['longextra'], 'longextra mode',
                metavar='LONG')
    def process_inputs(self, args):
        self.counts = 0
        cliapp.Application.process_inputs(self, args)
        self.output.write('There were %s lines in %s.\n'
            % (self.counts, args))
        print('append=%s' % self.settings['append'])
        print('verbose=%s' % self.settings['verbose'])
        print('long=%s' % self.settings['long'])
        print('longextra=%s' % self.settings['longextra'])
        if self.settings['file']:
           print("filename = %s" % self.settings['file'])
           with open(self.settings['file']) as f:
               for line in f:
                   print (line[:-1])

    def process_input_line(self, name, line):
        self.counts += 1

if __name__ == '__main__':
    app = ExampleApp('Cli', version='0.1',
        description="Cli as a command line program example",
        epilog="Copyright (C) 2012 Osamu Aoki")
    app.settings.config_files = ['example.conf']
    app.run()
Execution examples with auto generated help
Traceback (most recent call last):
  File "./cli", line 6, in <module>
    import cliapp
ImportError: No module named cliapp
$ ./cli -avf BOGUS.txt --long --longextra=FOO cli BOGUS.txt
Traceback (most recent call last):
  File "./cli", line 6, in <module>
    import cliapp
ImportError: No module named cliapp
$ ./cli -b
Traceback (most recent call last):
  File "./cli", line 6, in <module>
    import cliapp
ImportError: No module named cliapp

Lua

Lua script (long and short option)

Simple example program with long and short command line options.

Source code for the Lua script: cli
#!/usr/bin/lua
verbose = 0
i = 1
while (i <= #arg) do
    if (arg[i] == '-a') then
        print("Option a")
    elseif (arg[i] == '-f' or arg[i] == '--file') then
        i = i +1
        for line in io.lines(arg[i]) do
            print(line)
        end
    elseif (arg[i] == '-v') then
        verbose = 1
    elseif (arg[i] == '--long') then
        print("Option long")
    elseif (string.sub(arg[i],1,11) == '--longextra') then
        print("Option longextra with " .. string.sub(arg[i], 13))
    elseif (arg[i] == '--') then
        i = i +1
        break
    elseif (string.sub(arg[i], 1, 1) == '-') then
        print("Unknown option: " .. arg[i])
        print("Usage: " .. arg[0] ..
            " [-a] [-f|--file file] [-v] [--long] [--longextra=value] [args...]")
        return(1)
    else
        break
    end
    i = i + 1
end
i0 = i - 1
for i = 1, #arg - i0, 1 do
    print("ARG[" .. i .. "] = " .. arg[i + i0])
end
Execution examples with auto generated help
Unknown option: --help
Usage: ./cli [-a] [-f|--file file] [-v] [--long] [--longextra=value] [args...]
$ ./cli -a -v -f BOGUS.txt --long --longextra=FOO cli BOGUS.txt
Option a
This is BOGUS.txt here.
EOF here :-)
Option long
Option longextra with FOO
ARG[1] = cli
ARG[2] = BOGUS.txt
Tip
You must type as "-a -f ..." and not as "-af ..."

Perl

Perl script with Getopt::Long

Simple example program with long and short command line options.

Source code for Perl with Getopt::Long
#!/usr/bin/perl -w
use strict;
use Getopt::Long;
# Global variables
use vars qw(%opt $verbose $helpmsg);
%opt = ();
$verbose = 0;
$helpmsg = "Usage: $0 [-a] [-v] [-f|--file name] [--long] [--longextra extra]...\n";

sub main()
{
    &GetOptions(
        "a"=>\$opt{'a'},
        "f|file=s"=>\$opt{'f'},
        "h|help"=>\$opt{'h'},
        "v"=>\$opt{'v'},
        "long"=>\$opt{'long'},
        "longextra=s"=>\$opt{'longextra'}) or die $helpmsg;
    if ($opt{'a'}) {
        print "Option a\n";
    }
    if ($opt{'h'}) {
        print $helpmsg;
    }
    if ($opt{'f'}) {
        print("Option f with $opt{'f'}\n");
        open(INPUT_FILE, $opt{'f'})
            or die "Couldn't open $opt{'f'}!";
        while (<INPUT_FILE>) {
            print "$_";
        }
        close(INPUT_FILE)
    }
    if ($opt{'v'}) {
        $verbose = 1;
    }
    if ($opt{'long'}) {
        print "Option long\n";
    }
    if ($opt{'longextra'}) {
        print "Option f with $opt{'longextra'}\n";
    }
    if ($ARGV[0]) {
        print "Other things found on the command line:\n";
    }
    foreach (@ARGV) {
        print "ARG=$_\n";
    }
}

main();
Tip
Older programs with short command line options may use functions such as getopt("af:v") or getopts("af:v") with use Getopt::Std; which set global variables $opt_a, $opt_f, $opt_v.
Execution examples with auto generated help
Usage: ./cli [-a] [-v] [-f|--file name] [--long] [--longextra extra]...
$ ./cli -a -v -f BOGUS.txt --long --longextra=FOO cli BOGUS.txt
Option a
Option f with BOGUS.txt
This is BOGUS.txt here.
EOF here :-)
Option long
Option f with FOO
Other things found on the command line:
ARG=cli
ARG=BOGUS.txt
$ ./cli -b
Unknown option: b
Usage: ./cli [-a] [-v] [-f|--file name] [--long] [--longextra extra]...

C

Writing a code with the iteration over unknown length lines is not easy with C since it requires dynamic memory allocation for the line buffer. We will come back to such coding later.

C with getopt()

Simple example program with short command line options.

Source code for C with getopt()
/* a short command line option example */

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h> /* getopt */
int main(int argc, char **argv)
{
    int c;
    int verbose = 0;
    while ((c = getopt(argc, argv, "af:v")) != -1) {
        switch (c) {
        case 'a':
            printf("option a\n");
            break;
        case 'f':
            printf("option f with %s.\n", optarg);
            break;
        case 'v':
            verbose = 1;
            break;
        default:
            printf("unknown option = %c = ascii(0%x)\n",
                    optopt, optopt);
            printf("return value of getopt() = %c\n", c);
        }
    }
    printf("verbose mode = %i\n", verbose);
    if (optind < argc) {
        int j = 1;
        while (optind < argc) {
            printf("argument[%i]=%s\n", j, argv[optind++]);
            j++;
        }
    }
    return EXIT_SUCCESS;
}

Please note few global external variables used.

  • char *optarg : pointer to the option argument string

  • int optind : index for the next argument to be processed

  • int opterr : set to zero for quieting error message

  • int optopt :

C with getopt_long()

Simple example program with long and short command line options.

Source code for C with getopt_long()
/* a long command line option example */

#include <stdlib.h>
#include <stdio.h>
#include <getopt.h>        /* getopt_long */

int main(int argc, char **argv)
{
    int c;
    int i = 0;
    int verbose = 0;
    static struct option longopts[] = {
        {"append", no_argument, 0, 'a'},
        {"verbose", no_argument, 0, 'v'},
        {"file", required_argument, 0, 'f'},
        {"long", no_argument, 0, 0},
        {"longextra", required_argument, 0, 0},
        {NULL, 0, NULL, 0}
    };
    while ((c = getopt_long(argc, argv, "af:v", longopts, &i)) != -1) {
        switch (c) {
        case 0: /* for longopts[all][3] == 0 */
            printf ("option %s", longopts[i].name);
            if (optarg)
                printf (" with arg %s", optarg);
            printf ("\n");
            break;
        case 'a':
            printf("option a\n");
            break;
        case 'f':
            printf("option f with %s.\n", optarg);
            break;
        case 'v':
            verbose = 1;
            break;
        default:
            printf("unknown option = %c = ascii(0%x)\n", optopt, optopt);
            printf("return value of getopt_long() = %c\n", c);
        }
    }
    printf("verbose mode = %i\n", verbose);
    if (optind < argc) {
        int j = 1;
        while (optind < argc) {
            printf("argument[%i]=%s\n", j, argv[optind++]);
            j++;
        }
    }
    return EXIT_SUCCESS;
}

Please note few global external variables used just like getopt().

In order to avoid GNU vs. BSD incompatibility, the following may be a good idea:

  • Do not define -W option.

  • Third component of longopts[] must be 0. (as in the above example)

  • Always put all of arguments after all the options when calling the compiled program with getopt_long().

C++

The golden rule of C++ programing: "If you can not find something in the C++ Standard Library, look for it in Boost libraries."

Simple example program with long and short command line options.

Source code for C++ with boost::program_options
#include <boost/program_options.hpp>
#include <iostream>
#include <iterator>
#include <stdlib.h>
using namespace std;

// A helper function to simplify the main part.
template<class T>
ostream& operator<<(ostream& os, const vector<T>& v)
{
    copy(v.begin(), v.end(), ostream_iterator<T>(cout, " "));
    return os;
}


int main(int argc, char **argv)
{
    string file;
    string extra;
    try {
        namespace po = boost::program_options;
        po::options_description visible("SYNTAX: cli [OPTION ...] files ...");
        visible.add_options()
            ("append,a", "append option")
            ("help,h", "help message")
            ("verbose,v", "verbose option")
            ("file,f", po::value<string>(&file), "file option")
            ("long", "long option")
            ("longextra", po::value<string>(&extra), "longextra option")
        ;
        po::options_description hidden;
        hidden.add_options()
            ("input-file", po::value< vector<string> >(), "input file")
        ;
        po::positional_options_description p;
        p.add("input-file", -1);

        po::options_description desc;
        desc.add(visible).add(hidden);

        po::variables_map vm;
        po::store(po::command_line_parser(argc, argv)
            .options(desc).positional(p).run(), vm);
        po::notify(vm);

        if (vm.count("help")) {
            cout << visible << "\n";
            return EXIT_FAILURE;
        }
        if (vm.count("append")) {
            cout << "append mode\n";
        }
        if (vm.count("verbose")) {
            cout << "verbose mode\n";
        }
        if (vm.count("file")) {
            cout << "Filename: "
                 << file << "\n";
        } else {
            cout  << "Filename: (unset)\n";
        }
        if (vm.count("long")) {
            cout << "long mode\n";
        }
        if (vm.count("longextra")) {
            cout << "Longextra: "
                 << extra << "\n";
        } else {
            cout  << "Longextra: (unset)\n";
        }
        if (vm.count("input-file")) {
            cout << "Input files are: "
                 << vm["input-file"].as< vector<string> >() << "\n";
        }
    }
    catch(exception& e) {
        cerr << "E: " << e.what() << "\n";
        return EXIT_FAILURE;
    }
    catch(...) {
        cerr << "E: unknown type!\n";
    }

    return EXIT_SUCCESS;
}

Let’s compile cli.cxx to create the ELF object cli and run it.

/usr/bin/ld: cannot find -lboost_program_options
collect2: error: ld returned 1 exit status
$ ./cli -avf BOGUS.txt --long --longextra=FOO X1 X2
/path/to/cxx/../../bin/p: 1: eval: ./cli: not found
$ ./cli -b
/path/to/cxx/../../bin/p: 1: eval: ./cli: not found
$ ./cli --help
/path/to/cxx/../../bin/p: 1: eval: ./cli: not found

Vala

Simple Vala example programs with long and short command line options using GLib.

Simple non-OOP style (no class)

Source code cli-1.vala in Vala language
static bool append = false;
static bool verbose = false;
static bool op_long = false;
static string filename = null;
static string longextra = null;

// sources is a string array containing all non-option arguments
// https://mail.gnome.org/archives/vala-list/2009-March/msg00090.html
[CCode (array_length = false, array_null_terminated = true)]
static string[] sources;

const OptionEntry[] options = {
    { "append",  'a', 0, OptionArg.NONE,     ref append,
        "Set append mode",     null },
    { "verbose", 'v', 0, OptionArg.NONE,     ref verbose,
        "Set verbose mode",    null },
    { "file",    'f', 0, OptionArg.FILENAME, ref filename,
        "Use F file",         "F"},
    { "long",      0, 0, OptionArg.NONE,     ref op_long,
        "Set long mode",      null },
    { "longextra", 0, 0, OptionArg.STRING,   ref longextra,
        "Set longextra to M", "M" },
    { "",    0, 0, OptionArg.FILENAME_ARRAY, ref sources,
        null,                "FILE..." },
    { null }
};


int main (string[] args) {
    // initialize locale
    Intl.setlocale (LocaleCategory.ALL, "");
    //
    stdout.printf ("command  = %s\n", args[0]);
    stdout.printf ("basename = %s\n", Path.get_basename(args[0]));
    try {
        // parse command line options with GLib
        var opt_context = new OptionContext ("- Cli");
        opt_context.set_help_enabled (true);
        opt_context.add_main_entries (options, null);
        opt_context.parse (ref args);
    } catch (OptionError e) {
        stdout.printf ("%s\n", e.message);
        stdout.printf (
"Run '%s --help' to see a full list of available command line options.\n",
            args[0]);
        return 1;
    }

    // print variables
    if (verbose) {
        stdout.printf ("Cli: verbose ON\n");
    }
    if (append) {
        stdout.printf ("Cli: appemd ON\n");
    }
    if (op_long) {
        stdout.printf ("Cli: long ON\n");
    }
    if (filename != null) {
        stdout.printf ("Cli: file = %s\n", filename);
    }
    if (longextra != null) {
        stdout.printf ("Cli: longextra = %s\n", longextra);
    }
    if (sources != null) {
        int i = 0;
        foreach (string s in sources) {
            stdout.printf("sources[%i]=%s\n", i, s);
            i++;
        }
    }
    return 0;
}

Tip
The above example skips specifying GLib for the parent class since it is the default for Vala.

Let’s compile cli-1.vala to create the ELF object cli-1 and run it.

Loaded package `/usr/share/vala-0.26/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.26/vapi/gobject-2.0.vapi'
cc -o '/path/to/vala/cli-1' '/path/to/vala/cli-1.vala.c' -I/usr/include/glib-2.0 ...
$ ./cli-1 -avf BOGUS.txt --long --longextra=FOO X1 X2
command  = ./cli-1
basename = cli-1
Cli: verbose ON
Cli: appemd ON
Cli: long ON
Cli: file = BOGUS.txt
Cli: longextra = FOO
sources[0]=X1
sources[1]=X2
$ ./cli-1 -b
command  = ./cli-1
basename = cli-1
Unknown option -b
Run './cli-1 --help' to see a full list of available command line options.
$ ./cli-1 --help
command  = ./cli-1
basename = cli-1
Usage:
  cli-1 [OPTION...] FILE... - Cli

Help Options:
  -h, --help        Show help options

Application Options:
  -a, --append      Set append mode
  -v, --verbose     Set verbose mode
  -f, --file=F      Use F file
  --long            Set long mode
  --longextra=M     Set longextra to M

You can get the C source as:

$ wc -l cli-1.vala ; wc -l cli-1.c
75 cli-1.vala
223 cli-1.c
$ cat cli-1.c |sed -e 's/       /    /g'|fold  # tab => 4 spaces
/* cli-1.c generated by valac 0.26.1, the Vala compiler
 * generated from cli-1.vala, do not modify */


#include <glib.h>
#include <glib-object.h>
#include <stdlib.h>
#include <string.h>
#include <locale.h>
#include <stdio.h>

#define _g_free0(var) (var = (g_free (var), NULL))
#define _g_option_context_free0(var) ((var == NULL) ? NULL : (var = (g_option_co
ntext_free (var), NULL)))
#define _g_error_free0(var) ((var == NULL) ? NULL : (var = (g_error_free (var),
NULL)))


extern gboolean append;
gboolean append = FALSE;
extern gboolean verbose;
gboolean verbose = FALSE;
extern gboolean op_long;
gboolean op_long = FALSE;
extern gchar* filename;
gchar* filename = NULL;
extern gchar* longextra;
gchar* longextra = NULL;
extern gchar** sources;
gchar** sources = NULL;

gint _vala_main (gchar** args, int args_length1);
static gint _vala_array_length (gpointer array);

const GOptionEntry options[7] = {{"append", 'a', 0, G_OPTION_ARG_NONE, &append,
"Set append mode", NULL}, {"verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose, "Set
verbose mode", NULL}, {"file", 'f', 0, G_OPTION_ARG_FILENAME, &filename, "Use F
file", "F"}, {"long", (gchar) 0, 0, G_OPTION_ARG_NONE, &op_long, "Set long mode"
, NULL}, {"longextra", (gchar) 0, 0, G_OPTION_ARG_STRING, &longextra, "Set longe
xtra to M", "M"}, {"", (gchar) 0, 0, G_OPTION_ARG_FILENAME_ARRAY, &sources, NULL
, "FILE..."}, {NULL}};

gint _vala_main (gchar** args, int args_length1) {
    gint result = 0;
    FILE* _tmp0_ = NULL;
    gchar** _tmp1_ = NULL;
    gint _tmp1__length1 = 0;
    const gchar* _tmp2_ = NULL;
    FILE* _tmp3_ = NULL;
    gchar** _tmp4_ = NULL;
    gint _tmp4__length1 = 0;
    const gchar* _tmp5_ = NULL;
    gchar* _tmp6_ = NULL;
    gchar* _tmp7_ = NULL;
    gboolean _tmp18_ = FALSE;
    gboolean _tmp20_ = FALSE;
    gboolean _tmp22_ = FALSE;
    const gchar* _tmp24_ = NULL;
    const gchar* _tmp27_ = NULL;
    gchar** _tmp30_ = NULL;
    gint _tmp30__length1 = 0;
    GError * _inner_error_ = NULL;
    setlocale (LC_ALL, "");
    _tmp0_ = stdout;
    _tmp1_ = args;
    _tmp1__length1 = args_length1;
    _tmp2_ = _tmp1_[0];
    fprintf (_tmp0_, "command  = %s\n", _tmp2_);
    _tmp3_ = stdout;
    _tmp4_ = args;
    _tmp4__length1 = args_length1;
    _tmp5_ = _tmp4_[0];
    _tmp6_ = g_path_get_basename (_tmp5_);
    _tmp7_ = _tmp6_;
    fprintf (_tmp3_, "basename = %s\n", _tmp7_);
    _g_free0 (_tmp7_);
    {
        GOptionContext* opt_context = NULL;
        GOptionContext* _tmp8_ = NULL;
        GOptionContext* _tmp9_ = NULL;
        GOptionContext* _tmp10_ = NULL;
        GOptionContext* _tmp11_ = NULL;
        _tmp8_ = g_option_context_new ("- Cli");
        opt_context = _tmp8_;
        _tmp9_ = opt_context;
        g_option_context_set_help_enabled (_tmp9_, TRUE);
        _tmp10_ = opt_context;
        g_option_context_add_main_entries (_tmp10_, options, NULL);
        _tmp11_ = opt_context;
        g_option_context_parse (_tmp11_, &args_length1, &args, &_inner_error_);
        if (G_UNLIKELY (_inner_error_ != NULL)) {
            _g_option_context_free0 (opt_context);
            if (_inner_error_->domain == G_OPTION_ERROR) {
                goto __catch0_g_option_error;
            }
            _g_option_context_free0 (opt_context);
            g_critical ("file %s: line %d: unexpected error: %s (%s, %d)", __FIL
E__, __LINE__, _inner_error_->message, g_quark_to_string (_inner_error_->domain)
, _inner_error_->code);
            g_clear_error (&_inner_error_);
            return 0;
        }
        _g_option_context_free0 (opt_context);
    }
    goto __finally0;
    __catch0_g_option_error:
    {
        GError* e = NULL;
        FILE* _tmp12_ = NULL;
        GError* _tmp13_ = NULL;
        const gchar* _tmp14_ = NULL;
        FILE* _tmp15_ = NULL;
        gchar** _tmp16_ = NULL;
        gint _tmp16__length1 = 0;
        const gchar* _tmp17_ = NULL;
        e = _inner_error_;
        _inner_error_ = NULL;
        _tmp12_ = stdout;
        _tmp13_ = e;
        _tmp14_ = _tmp13_->message;
        fprintf (_tmp12_, "%s\n", _tmp14_);
        _tmp15_ = stdout;
        _tmp16_ = args;
        _tmp16__length1 = args_length1;
        _tmp17_ = _tmp16_[0];
        fprintf (_tmp15_, "Run '%s --help' to see a full list of available comma
nd line options.\n", _tmp17_);
        result = 1;
        _g_error_free0 (e);
        return result;
    }
    __finally0:
    if (G_UNLIKELY (_inner_error_ != NULL)) {
        g_critical ("file %s: line %d: uncaught error: %s (%s, %d)", __FILE__, _
_LINE__, _inner_error_->message, g_quark_to_string (_inner_error_->domain), _inn
er_error_->code);
        g_clear_error (&_inner_error_);
        return 0;
    }
    _tmp18_ = verbose;
    if (_tmp18_) {
        FILE* _tmp19_ = NULL;
        _tmp19_ = stdout;
        fprintf (_tmp19_, "Cli: verbose ON\n");
    }
    _tmp20_ = append;
    if (_tmp20_) {
        FILE* _tmp21_ = NULL;
        _tmp21_ = stdout;
        fprintf (_tmp21_, "Cli: appemd ON\n");
    }
    _tmp22_ = op_long;
    if (_tmp22_) {
        FILE* _tmp23_ = NULL;
        _tmp23_ = stdout;
        fprintf (_tmp23_, "Cli: long ON\n");
    }
    _tmp24_ = filename;
    if (_tmp24_ != NULL) {
        FILE* _tmp25_ = NULL;
        const gchar* _tmp26_ = NULL;
        _tmp25_ = stdout;
        _tmp26_ = filename;
        fprintf (_tmp25_, "Cli: file = %s\n", _tmp26_);
    }
    _tmp27_ = longextra;
    if (_tmp27_ != NULL) {
        FILE* _tmp28_ = NULL;
        const gchar* _tmp29_ = NULL;
        _tmp28_ = stdout;
        _tmp29_ = longextra;
        fprintf (_tmp28_, "Cli: longextra = %s\n", _tmp29_);
    }
    _tmp30_ = sources;
    _tmp30__length1 = _vala_array_length (sources);
    if (_tmp30_ != NULL) {
        gint i = 0;
        gchar** _tmp31_ = NULL;
        gint _tmp31__length1 = 0;
        i = 0;
        _tmp31_ = sources;
        _tmp31__length1 = _vala_array_length (sources);
        {
            gchar** s_collection = NULL;
            gint s_collection_length1 = 0;
            gint _s_collection_size_ = 0;
            gint s_it = 0;
            s_collection = _tmp31_;
            s_collection_length1 = _tmp31__length1;
            for (s_it = 0; s_it < _tmp31__length1; s_it = s_it + 1) {
                gchar* _tmp32_ = NULL;
                gchar* s = NULL;
                _tmp32_ = g_strdup (s_collection[s_it]);
                s = _tmp32_;
                {
                    FILE* _tmp33_ = NULL;
                    gint _tmp34_ = 0;
                    const gchar* _tmp35_ = NULL;
                    gint _tmp36_ = 0;
                    _tmp33_ = stdout;
                    _tmp34_ = i;
                    _tmp35_ = s;
                    fprintf (_tmp33_, "sources[%i]=%s\n", _tmp34_, _tmp35_);
                    _tmp36_ = i;
                    i = _tmp36_ + 1;
                    _g_free0 (s);
                }
            }
        }
    }
    result = 0;
    return result;
}


int main (int argc, char ** argv) {
#if !GLIB_CHECK_VERSION (2,35,0)
    g_type_init ();
#endif
    return _vala_main (argv, argc);
}


static gint _vala_array_length (gpointer array) {
    int length;
    length = 0;
    if (array) {
        while (((gpointer*) array)[length]) {
            length++;
        }
    }
    return length;
}


Since no OOP techniques are used, the resulting C code does not have GObject related macros.

But some libraries from GLib are used, the resulting C code has automatically generated memory management codes such as _g_option_context_free0, _g_error_free0, and _g_free0.

OOP style (main outside of class)

Source code cli-2.vala in Vala language
class Cli : Object {
    private static bool append = false;
    private static bool verbose = false;
    private static bool op_long = false;
    private static string filename = null;
    private static string longextra = null;

    // sources is a string array containing all non-option arguments
    // https://mail.gnome.org/archives/vala-list/2009-March/msg00090.html
    [CCode (array_length = false, array_null_terminated = true)]
    private static string[] sources;

    private const OptionEntry[] options = {
        { "append",  'a', 0, OptionArg.NONE,     ref append,
            "Set append mode",     null },
        { "verbose", 'v', 0, OptionArg.NONE,     ref verbose,
            "Set verbose mode",    null },
        { "file",    'f', 0, OptionArg.FILENAME, ref filename,
            "Use F file",         "F"},
        { "long",      0, 0, OptionArg.NONE,     ref op_long,
            "Set long mode",      null },
        { "longextra", 0, 0, OptionArg.STRING,   ref longextra,
            "Set longextra to M", "M" },
        { "",    0, 0, OptionArg.FILENAME_ARRAY, ref sources,
            null,                "FILE..." },
        { null }
    };

    public int parse (string[] args) {
        // parse command line
        try {
            // parse command line options with GLib
            var opt_context = new OptionContext ("- Cli");
            opt_context.set_help_enabled (true);
            opt_context.add_main_entries (options, null);
            opt_context.parse (ref args);
        } catch (OptionError e) {
            stdout.printf ("%s\n", e.message);
            stdout.printf (
    "Run '%s --help' to see a full list of available command line options.\n",
                args[0]);
            return 1;
        }
        return 0;
    }

    public int run () {
        // print variables
        if (verbose) {
            stdout.printf ("Cli: verbose ON\n");
        }
        if (append) {
            stdout.printf ("Cli: appemd ON\n");
        }
        if (op_long) {
            stdout.printf ("Cli: long ON\n");
        }
        if (filename != null) {
            stdout.printf ("Cli: file = %s\n", filename);
        }
        if (longextra != null) {
            stdout.printf ("Cli: longextra = %s\n", longextra);
        }
        if (sources != null) {
            int i = 0;
            foreach (string s in sources) {
                stdout.printf("sources[%i]=%s\n", i, s);
                i++;
            }
        }
        return 0;
    }

}
int main (string[] args) {
    // initialize locale
    Intl.setlocale (LocaleCategory.ALL, "");
    //
    stdout.printf ("command  = %s\n", args[0]);
    stdout.printf ("basename = %s\n", Path.get_basename(args[0]));

    var cli = new Cli ();
    cli.parse (args);
    return cli.run ();
}
Tip
The above example skips specifying GLib for the parent class since it is the default for Vala.

Let’s compile cli-2.vala to create the ELF object cli-2 and run it.

Loaded package `/usr/share/vala-0.26/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.26/vapi/gobject-2.0.vapi'
cc -o '/path/to/vala/cli-2' '/path/to/vala/cli-2.vala.c' -I/usr/include/glib-2.0 ...
$ ./cli-2 -avf BOGUS.txt --long --longextra=FOO X1 X2
command  = ./cli-2
basename = cli-2
Cli: verbose ON
Cli: appemd ON
Cli: long ON
Cli: file = BOGUS.txt
Cli: longextra = FOO
sources[0]=X1
sources[1]=X2
$ ./cli-2 -b
command  = ./cli-2
basename = cli-2
Unknown option -b
Run './cli-2 --help' to see a full list of available command line options.
$ ./cli-2 --help
command  = ./cli-2
basename = cli-2
Usage:
  cli-2 [OPTION...] FILE... - Cli

Help Options:
  -h, --help        Show help options

Application Options:
  -a, --append      Set append mode
  -v, --verbose     Set verbose mode
  -f, --file=F      Use F file
  --long            Set long mode
  --longextra=M     Set longextra to M

You can get the C source as:

$ wc -l cli-2.vala ; wc -l cli-2.c
86 cli-2.vala
323 cli-2.c
$ cat cli-2.c |sed -e 's/       /    /g'|fold  # tab => 4 spaces
/* cli-2.c generated by valac 0.26.1, the Vala compiler
 * generated from cli-2.vala, do not modify */


#include <glib.h>
#include <glib-object.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <locale.h>


#define TYPE_CLI (cli_get_type ())
#define CLI(obj) (G_TYPE_CHECK_INSTANCE_CAST ((obj), TYPE_CLI, Cli))
#define CLI_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST ((klass), TYPE_CLI, CliClass))
#define IS_CLI(obj) (G_TYPE_CHECK_INSTANCE_TYPE ((obj), TYPE_CLI))
#define IS_CLI_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass), TYPE_CLI))
#define CLI_GET_CLASS(obj) (G_TYPE_INSTANCE_GET_CLASS ((obj), TYPE_CLI, CliClass
))

typedef struct _Cli Cli;
typedef struct _CliClass CliClass;
typedef struct _CliPrivate CliPrivate;
#define _g_option_context_free0(var) ((var == NULL) ? NULL : (var = (g_option_co
ntext_free (var), NULL)))
#define _g_error_free0(var) ((var == NULL) ? NULL : (var = (g_error_free (var),
NULL)))
#define _g_free0(var) (var = (g_free (var), NULL))
#define _g_object_unref0(var) ((var == NULL) ? NULL : (var = (g_object_unref (va
r), NULL)))

struct _Cli {
    GObject parent_instance;
    CliPrivate * priv;
};

struct _CliClass {
    GObjectClass parent_class;
};


static gpointer cli_parent_class = NULL;
static gboolean cli_append;
static gboolean cli_append = FALSE;
static gboolean cli_verbose;
static gboolean cli_verbose = FALSE;
static gboolean cli_op_long;
static gboolean cli_op_long = FALSE;
static gchar* cli_filename;
static gchar* cli_filename = NULL;
static gchar* cli_longextra;
static gchar* cli_longextra = NULL;
static gchar** cli_sources;
static gchar** cli_sources = NULL;

GType cli_get_type (void) G_GNUC_CONST;
enum  {
    CLI_DUMMY_PROPERTY
};
gint cli_parse (Cli* self, gchar** args, int args_length1);
gint cli_run (Cli* self);
Cli* cli_new (void);
Cli* cli_construct (GType object_type);
static void cli_finalize (GObject* obj);
gint _vala_main (gchar** args, int args_length1);
static gint _vala_array_length (gpointer array);

static const GOptionEntry CLI_options[7] = {{"append", 'a', 0, G_OPTION_ARG_NONE
, &cli_append, "Set append mode", NULL}, {"verbose", 'v', 0, G_OPTION_ARG_NONE,
&cli_verbose, "Set verbose mode", NULL}, {"file", 'f', 0, G_OPTION_ARG_FILENAME,
 &cli_filename, "Use F file", "F"}, {"long", (gchar) 0, 0, G_OPTION_ARG_NONE, &c
li_op_long, "Set long mode", NULL}, {"longextra", (gchar) 0, 0, G_OPTION_ARG_STR
ING, &cli_longextra, "Set longextra to M", "M"}, {"", (gchar) 0, 0, G_OPTION_ARG
_FILENAME_ARRAY, &cli_sources, NULL, "FILE..."}, {NULL}};

gint cli_parse (Cli* self, gchar** args, int args_length1) {
    gint result = 0;
    GError * _inner_error_ = NULL;
    g_return_val_if_fail (self != NULL, 0);
    {
        GOptionContext* opt_context = NULL;
        GOptionContext* _tmp0_ = NULL;
        GOptionContext* _tmp1_ = NULL;
        GOptionContext* _tmp2_ = NULL;
        GOptionContext* _tmp3_ = NULL;
        _tmp0_ = g_option_context_new ("- Cli");
        opt_context = _tmp0_;
        _tmp1_ = opt_context;
        g_option_context_set_help_enabled (_tmp1_, TRUE);
        _tmp2_ = opt_context;
        g_option_context_add_main_entries (_tmp2_, CLI_options, NULL);
        _tmp3_ = opt_context;
        g_option_context_parse (_tmp3_, &args_length1, &args, &_inner_error_);
        if (G_UNLIKELY (_inner_error_ != NULL)) {
            _g_option_context_free0 (opt_context);
            if (_inner_error_->domain == G_OPTION_ERROR) {
                goto __catch0_g_option_error;
            }
            _g_option_context_free0 (opt_context);
            g_critical ("file %s: line %d: unexpected error: %s (%s, %d)", __FIL
E__, __LINE__, _inner_error_->message, g_quark_to_string (_inner_error_->domain)
, _inner_error_->code);
            g_clear_error (&_inner_error_);
            return 0;
        }
        _g_option_context_free0 (opt_context);
    }
    goto __finally0;
    __catch0_g_option_error:
    {
        GError* e = NULL;
        FILE* _tmp4_ = NULL;
        GError* _tmp5_ = NULL;
        const gchar* _tmp6_ = NULL;
        FILE* _tmp7_ = NULL;
        gchar** _tmp8_ = NULL;
        gint _tmp8__length1 = 0;
        const gchar* _tmp9_ = NULL;
        e = _inner_error_;
        _inner_error_ = NULL;
        _tmp4_ = stdout;
        _tmp5_ = e;
        _tmp6_ = _tmp5_->message;
        fprintf (_tmp4_, "%s\n", _tmp6_);
        _tmp7_ = stdout;
        _tmp8_ = args;
        _tmp8__length1 = args_length1;
        _tmp9_ = _tmp8_[0];
        fprintf (_tmp7_, "Run '%s --help' to see a full list of available comman
d line options.\n", _tmp9_);
        result = 1;
        _g_error_free0 (e);
        return result;
    }
    __finally0:
    if (G_UNLIKELY (_inner_error_ != NULL)) {
        g_critical ("file %s: line %d: uncaught error: %s (%s, %d)", __FILE__, _
_LINE__, _inner_error_->message, g_quark_to_string (_inner_error_->domain), _inn
er_error_->code);
        g_clear_error (&_inner_error_);
        return 0;
    }
    result = 0;
    return result;
}


gint cli_run (Cli* self) {
    gint result = 0;
    gboolean _tmp0_ = FALSE;
    gboolean _tmp2_ = FALSE;
    gboolean _tmp4_ = FALSE;
    const gchar* _tmp6_ = NULL;
    const gchar* _tmp9_ = NULL;
    gchar** _tmp12_ = NULL;
    gint _tmp12__length1 = 0;
    g_return_val_if_fail (self != NULL, 0);
    _tmp0_ = cli_verbose;
    if (_tmp0_) {
        FILE* _tmp1_ = NULL;
        _tmp1_ = stdout;
        fprintf (_tmp1_, "Cli: verbose ON\n");
    }
    _tmp2_ = cli_append;
    if (_tmp2_) {
        FILE* _tmp3_ = NULL;
        _tmp3_ = stdout;
        fprintf (_tmp3_, "Cli: appemd ON\n");
    }
    _tmp4_ = cli_op_long;
    if (_tmp4_) {
        FILE* _tmp5_ = NULL;
        _tmp5_ = stdout;
        fprintf (_tmp5_, "Cli: long ON\n");
    }
    _tmp6_ = cli_filename;
    if (_tmp6_ != NULL) {
        FILE* _tmp7_ = NULL;
        const gchar* _tmp8_ = NULL;
        _tmp7_ = stdout;
        _tmp8_ = cli_filename;
        fprintf (_tmp7_, "Cli: file = %s\n", _tmp8_);
    }
    _tmp9_ = cli_longextra;
    if (_tmp9_ != NULL) {
        FILE* _tmp10_ = NULL;
        const gchar* _tmp11_ = NULL;
        _tmp10_ = stdout;
        _tmp11_ = cli_longextra;
        fprintf (_tmp10_, "Cli: longextra = %s\n", _tmp11_);
    }
    _tmp12_ = cli_sources;
    _tmp12__length1 = _vala_array_length (cli_sources);
    if (_tmp12_ != NULL) {
        gint i = 0;
        gchar** _tmp13_ = NULL;
        gint _tmp13__length1 = 0;
        i = 0;
        _tmp13_ = cli_sources;
        _tmp13__length1 = _vala_array_length (cli_sources);
        {
            gchar** s_collection = NULL;
            gint s_collection_length1 = 0;
            gint _s_collection_size_ = 0;
            gint s_it = 0;
            s_collection = _tmp13_;
            s_collection_length1 = _tmp13__length1;
            for (s_it = 0; s_it < _tmp13__length1; s_it = s_it + 1) {
                gchar* _tmp14_ = NULL;
                gchar* s = NULL;
                _tmp14_ = g_strdup (s_collection[s_it]);
                s = _tmp14_;
                {
                    FILE* _tmp15_ = NULL;
                    gint _tmp16_ = 0;
                    const gchar* _tmp17_ = NULL;
                    gint _tmp18_ = 0;
                    _tmp15_ = stdout;
                    _tmp16_ = i;
                    _tmp17_ = s;
                    fprintf (_tmp15_, "sources[%i]=%s\n", _tmp16_, _tmp17_);
                    _tmp18_ = i;
                    i = _tmp18_ + 1;
                    _g_free0 (s);
                }
            }
        }
    }
    result = 0;
    return result;
}


Cli* cli_construct (GType object_type) {
    Cli * self = NULL;
    self = (Cli*) g_object_new (object_type, NULL);
    return self;
}


Cli* cli_new (void) {
    return cli_construct (TYPE_CLI);
}


static void cli_class_init (CliClass * klass) {
    cli_parent_class = g_type_class_peek_parent (klass);
    G_OBJECT_CLASS (klass)->finalize = cli_finalize;
}


static void cli_instance_init (Cli * self) {
}


static void cli_finalize (GObject* obj) {
    Cli * self;
    self = G_TYPE_CHECK_INSTANCE_CAST (obj, TYPE_CLI, Cli);
    G_OBJECT_CLASS (cli_parent_class)->finalize (obj);
}


GType cli_get_type (void) {
    static volatile gsize cli_type_id__volatile = 0;
    if (g_once_init_enter (&cli_type_id__volatile)) {
        static const GTypeInfo g_define_type_info = { sizeof (CliClass), (GBaseI
nitFunc) NULL, (GBaseFinalizeFunc) NULL, (GClassInitFunc) cli_class_init, (GClas
sFinalizeFunc) NULL, NULL, sizeof (Cli), 0, (GInstanceInitFunc) cli_instance_ini
t, NULL };
        GType cli_type_id;
        cli_type_id = g_type_register_static (G_TYPE_OBJECT, "Cli", &g_define_ty
pe_info, 0);
        g_once_init_leave (&cli_type_id__volatile, cli_type_id);
    }
    return cli_type_id__volatile;
}


gint _vala_main (gchar** args, int args_length1) {
    gint result = 0;
    FILE* _tmp0_ = NULL;
    gchar** _tmp1_ = NULL;
    gint _tmp1__length1 = 0;
    const gchar* _tmp2_ = NULL;
    FILE* _tmp3_ = NULL;
    gchar** _tmp4_ = NULL;
    gint _tmp4__length1 = 0;
    const gchar* _tmp5_ = NULL;
    gchar* _tmp6_ = NULL;
    gchar* _tmp7_ = NULL;
    Cli* cli = NULL;
    Cli* _tmp8_ = NULL;
    gchar** _tmp9_ = NULL;
    gint _tmp9__length1 = 0;
    gint _tmp10_ = 0;
    setlocale (LC_ALL, "");
    _tmp0_ = stdout;
    _tmp1_ = args;
    _tmp1__length1 = args_length1;
    _tmp2_ = _tmp1_[0];
    fprintf (_tmp0_, "command  = %s\n", _tmp2_);
    _tmp3_ = stdout;
    _tmp4_ = args;
    _tmp4__length1 = args_length1;
    _tmp5_ = _tmp4_[0];
    _tmp6_ = g_path_get_basename (_tmp5_);
    _tmp7_ = _tmp6_;
    fprintf (_tmp3_, "basename = %s\n", _tmp7_);
    _g_free0 (_tmp7_);
    _tmp8_ = cli_new ();
    cli = _tmp8_;
    _tmp9_ = args;
    _tmp9__length1 = args_length1;
    cli_parse (cli, _tmp9_, _tmp9__length1);
    _tmp10_ = cli_run (cli);
    result = _tmp10_;
    _g_object_unref0 (cli);
    return result;
}


int main (int argc, char ** argv) {
#if !GLIB_CHECK_VERSION (2,35,0)
    g_type_init ();
#endif
    return _vala_main (argv, argc);
}


static gint _vala_array_length (gpointer array) {
    int length;
    length = 0;
    if (array) {
        while (((gpointer*) array)[length]) {
            length++;
        }
    }
    return length;
}


It is obvious that the Vala code is much shorter than the generated C code. The Vala compiler took care chores of generating CPP macros for GObject with.

  • No prefix.

  • cli, CLI, Cli: object type (= class) name

  • cli_get_type(): returning GType of object CLI

OOP style (main inside of class)

Source code cli-3.vala in Vala language
class Cli : Object {
    private static bool append = false;
    private static bool verbose = false;
    private static bool op_long = false;
    private static string filename = null;
    private static string longextra = null;

    // sources is a string array containing all non-option arguments
    // https://mail.gnome.org/archives/vala-list/2009-March/msg00090.html
    [CCode (array_length = false, array_null_terminated = true)]
    private static string[] sources;

    private const OptionEntry[] options = {
        { "append",  'a', 0, OptionArg.NONE,     ref append,
            "Set append mode",     null },
        { "verbose", 'v', 0, OptionArg.NONE,     ref verbose,
            "Set verbose mode",    null },
        { "file",    'f', 0, OptionArg.FILENAME, ref filename,
            "Use F file",         "F"},
        { "long",      0, 0, OptionArg.NONE,     ref op_long,
            "Set long mode",      null },
        { "longextra", 0, 0, OptionArg.STRING,   ref longextra,
            "Set longextra to M", "M" },
        { "",    0, 0, OptionArg.FILENAME_ARRAY, ref sources,
            null,                "FILE..." },
        { null }
    };

    private int parse (string[] args) {
        // parse command line
        try {
            // parse command line options with GLib
            var opt_context = new OptionContext ("- Cli");
            opt_context.set_help_enabled (true);
            opt_context.add_main_entries (options, null);
            opt_context.parse (ref args);
        } catch (OptionError e) {
            stdout.printf ("%s\n", e.message);
            stdout.printf (
    "Run '%s --help' to see a full list of available command line options.\n",
                args[0]);
            return 1;
        }
        return 0;
    }

    private int run () {
        // print variables
        if (verbose) {
            stdout.printf ("Cli: verbose ON\n");
        }
        if (append) {
            stdout.printf ("Cli: appemd ON\n");
        }
        if (op_long) {
            stdout.printf ("Cli: long ON\n");
        }
        if (filename != null) {
            stdout.printf ("Cli: file = %s\n", filename);
        }
        if (longextra != null) {
            stdout.printf ("Cli: longextra = %s\n", longextra);
        }
        if (sources != null) {
            int i = 0;
            foreach (string s in sources) {
                stdout.printf("sources[%i]=%s\n", i, s);
                i++;
            }
        }
        return 0;
    }

    public static int main (string[] args) {
        // initialize locale
        Intl.setlocale (LocaleCategory.ALL, "");
        //
        stdout.printf ("command  = %s\n", args[0]);
        stdout.printf ("basename = %s\n", Path.get_basename(args[0]));

        var cli = new Cli ();
        cli.parse (args);
        return cli.run ();
    }

}
Tip
The above example skips specifying GLib for the parent class since it is the default for Vala.

Let’s compile cli-3.vala to create the ELF object cli-3 and run it.

Loaded package `/usr/share/vala-0.26/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.26/vapi/gobject-2.0.vapi'
cc -o '/path/to/vala/cli-3' '/path/to/vala/cli-3.vala.c' -I/usr/include/glib-2.0 ...
$ ./cli-3 -avf BOGUS.txt --long --longextra=FOO X1 X2
command  = ./cli-3
basename = cli-3
Cli: verbose ON
Cli: appemd ON
Cli: long ON
Cli: file = BOGUS.txt
Cli: longextra = FOO
sources[0]=X1
sources[1]=X2
$ ./cli-3 -b
command  = ./cli-3
basename = cli-3
Unknown option -b
Run './cli-3 --help' to see a full list of available command line options.
$ ./cli-3 --help
command  = ./cli-3
basename = cli-3
Usage:
  cli-3 [OPTION...] FILE... - Cli

Help Options:
  -h, --help        Show help options

Application Options:
  -a, --append      Set append mode
  -v, --verbose     Set verbose mode
  -f, --file=F      Use F file
  --long            Set long mode
  --longextra=M     Set longextra to M

You can get the C source as:

$ wc -l cli-3.vala ; wc -l cli-3.c
87 cli-3.vala
323 cli-3.c
$ cat cli-3.c |sed -e 's/       /    /g'|fold  # tab => 4 spaces
/* cli-3.c generated by valac 0.26.1, the Vala compiler
 * generated from cli-3.vala, do not modify */


#include <glib.h>
#include <glib-object.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <locale.h>


#define TYPE_CLI (cli_get_type ())
#define CLI(obj) (G_TYPE_CHECK_INSTANCE_CAST ((obj), TYPE_CLI, Cli))
#define CLI_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST ((klass), TYPE_CLI, CliClass))
#define IS_CLI(obj) (G_TYPE_CHECK_INSTANCE_TYPE ((obj), TYPE_CLI))
#define IS_CLI_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass), TYPE_CLI))
#define CLI_GET_CLASS(obj) (G_TYPE_INSTANCE_GET_CLASS ((obj), TYPE_CLI, CliClass
))

typedef struct _Cli Cli;
typedef struct _CliClass CliClass;
typedef struct _CliPrivate CliPrivate;
#define _g_option_context_free0(var) ((var == NULL) ? NULL : (var = (g_option_co
ntext_free (var), NULL)))
#define _g_error_free0(var) ((var == NULL) ? NULL : (var = (g_error_free (var),
NULL)))
#define _g_free0(var) (var = (g_free (var), NULL))
#define _g_object_unref0(var) ((var == NULL) ? NULL : (var = (g_object_unref (va
r), NULL)))

struct _Cli {
    GObject parent_instance;
    CliPrivate * priv;
};

struct _CliClass {
    GObjectClass parent_class;
};


static gpointer cli_parent_class = NULL;
static gboolean cli_append;
static gboolean cli_append = FALSE;
static gboolean cli_verbose;
static gboolean cli_verbose = FALSE;
static gboolean cli_op_long;
static gboolean cli_op_long = FALSE;
static gchar* cli_filename;
static gchar* cli_filename = NULL;
static gchar* cli_longextra;
static gchar* cli_longextra = NULL;
static gchar** cli_sources;
static gchar** cli_sources = NULL;

GType cli_get_type (void) G_GNUC_CONST;
enum  {
    CLI_DUMMY_PROPERTY
};
static gint cli_parse (Cli* self, gchar** args, int args_length1);
static gint cli_run (Cli* self);
gint cli_main (gchar** args, int args_length1);
Cli* cli_new (void);
Cli* cli_construct (GType object_type);
static void cli_finalize (GObject* obj);
static gint _vala_array_length (gpointer array);

static const GOptionEntry CLI_options[7] = {{"append", 'a', 0, G_OPTION_ARG_NONE
, &cli_append, "Set append mode", NULL}, {"verbose", 'v', 0, G_OPTION_ARG_NONE,
&cli_verbose, "Set verbose mode", NULL}, {"file", 'f', 0, G_OPTION_ARG_FILENAME,
 &cli_filename, "Use F file", "F"}, {"long", (gchar) 0, 0, G_OPTION_ARG_NONE, &c
li_op_long, "Set long mode", NULL}, {"longextra", (gchar) 0, 0, G_OPTION_ARG_STR
ING, &cli_longextra, "Set longextra to M", "M"}, {"", (gchar) 0, 0, G_OPTION_ARG
_FILENAME_ARRAY, &cli_sources, NULL, "FILE..."}, {NULL}};

static gint cli_parse (Cli* self, gchar** args, int args_length1) {
    gint result = 0;
    GError * _inner_error_ = NULL;
    g_return_val_if_fail (self != NULL, 0);
    {
        GOptionContext* opt_context = NULL;
        GOptionContext* _tmp0_ = NULL;
        GOptionContext* _tmp1_ = NULL;
        GOptionContext* _tmp2_ = NULL;
        GOptionContext* _tmp3_ = NULL;
        _tmp0_ = g_option_context_new ("- Cli");
        opt_context = _tmp0_;
        _tmp1_ = opt_context;
        g_option_context_set_help_enabled (_tmp1_, TRUE);
        _tmp2_ = opt_context;
        g_option_context_add_main_entries (_tmp2_, CLI_options, NULL);
        _tmp3_ = opt_context;
        g_option_context_parse (_tmp3_, &args_length1, &args, &_inner_error_);
        if (G_UNLIKELY (_inner_error_ != NULL)) {
            _g_option_context_free0 (opt_context);
            if (_inner_error_->domain == G_OPTION_ERROR) {
                goto __catch0_g_option_error;
            }
            _g_option_context_free0 (opt_context);
            g_critical ("file %s: line %d: unexpected error: %s (%s, %d)", __FIL
E__, __LINE__, _inner_error_->message, g_quark_to_string (_inner_error_->domain)
, _inner_error_->code);
            g_clear_error (&_inner_error_);
            return 0;
        }
        _g_option_context_free0 (opt_context);
    }
    goto __finally0;
    __catch0_g_option_error:
    {
        GError* e = NULL;
        FILE* _tmp4_ = NULL;
        GError* _tmp5_ = NULL;
        const gchar* _tmp6_ = NULL;
        FILE* _tmp7_ = NULL;
        gchar** _tmp8_ = NULL;
        gint _tmp8__length1 = 0;
        const gchar* _tmp9_ = NULL;
        e = _inner_error_;
        _inner_error_ = NULL;
        _tmp4_ = stdout;
        _tmp5_ = e;
        _tmp6_ = _tmp5_->message;
        fprintf (_tmp4_, "%s\n", _tmp6_);
        _tmp7_ = stdout;
        _tmp8_ = args;
        _tmp8__length1 = args_length1;
        _tmp9_ = _tmp8_[0];
        fprintf (_tmp7_, "Run '%s --help' to see a full list of available comman
d line options.\n", _tmp9_);
        result = 1;
        _g_error_free0 (e);
        return result;
    }
    __finally0:
    if (G_UNLIKELY (_inner_error_ != NULL)) {
        g_critical ("file %s: line %d: uncaught error: %s (%s, %d)", __FILE__, _
_LINE__, _inner_error_->message, g_quark_to_string (_inner_error_->domain), _inn
er_error_->code);
        g_clear_error (&_inner_error_);
        return 0;
    }
    result = 0;
    return result;
}


static gint cli_run (Cli* self) {
    gint result = 0;
    gboolean _tmp0_ = FALSE;
    gboolean _tmp2_ = FALSE;
    gboolean _tmp4_ = FALSE;
    const gchar* _tmp6_ = NULL;
    const gchar* _tmp9_ = NULL;
    gchar** _tmp12_ = NULL;
    gint _tmp12__length1 = 0;
    g_return_val_if_fail (self != NULL, 0);
    _tmp0_ = cli_verbose;
    if (_tmp0_) {
        FILE* _tmp1_ = NULL;
        _tmp1_ = stdout;
        fprintf (_tmp1_, "Cli: verbose ON\n");
    }
    _tmp2_ = cli_append;
    if (_tmp2_) {
        FILE* _tmp3_ = NULL;
        _tmp3_ = stdout;
        fprintf (_tmp3_, "Cli: appemd ON\n");
    }
    _tmp4_ = cli_op_long;
    if (_tmp4_) {
        FILE* _tmp5_ = NULL;
        _tmp5_ = stdout;
        fprintf (_tmp5_, "Cli: long ON\n");
    }
    _tmp6_ = cli_filename;
    if (_tmp6_ != NULL) {
        FILE* _tmp7_ = NULL;
        const gchar* _tmp8_ = NULL;
        _tmp7_ = stdout;
        _tmp8_ = cli_filename;
        fprintf (_tmp7_, "Cli: file = %s\n", _tmp8_);
    }
    _tmp9_ = cli_longextra;
    if (_tmp9_ != NULL) {
        FILE* _tmp10_ = NULL;
        const gchar* _tmp11_ = NULL;
        _tmp10_ = stdout;
        _tmp11_ = cli_longextra;
        fprintf (_tmp10_, "Cli: longextra = %s\n", _tmp11_);
    }
    _tmp12_ = cli_sources;
    _tmp12__length1 = _vala_array_length (cli_sources);
    if (_tmp12_ != NULL) {
        gint i = 0;
        gchar** _tmp13_ = NULL;
        gint _tmp13__length1 = 0;
        i = 0;
        _tmp13_ = cli_sources;
        _tmp13__length1 = _vala_array_length (cli_sources);
        {
            gchar** s_collection = NULL;
            gint s_collection_length1 = 0;
            gint _s_collection_size_ = 0;
            gint s_it = 0;
            s_collection = _tmp13_;
            s_collection_length1 = _tmp13__length1;
            for (s_it = 0; s_it < _tmp13__length1; s_it = s_it + 1) {
                gchar* _tmp14_ = NULL;
                gchar* s = NULL;
                _tmp14_ = g_strdup (s_collection[s_it]);
                s = _tmp14_;
                {
                    FILE* _tmp15_ = NULL;
                    gint _tmp16_ = 0;
                    const gchar* _tmp17_ = NULL;
                    gint _tmp18_ = 0;
                    _tmp15_ = stdout;
                    _tmp16_ = i;
                    _tmp17_ = s;
                    fprintf (_tmp15_, "sources[%i]=%s\n", _tmp16_, _tmp17_);
                    _tmp18_ = i;
                    i = _tmp18_ + 1;
                    _g_free0 (s);
                }
            }
        }
    }
    result = 0;
    return result;
}


gint cli_main (gchar** args, int args_length1) {
    gint result = 0;
    FILE* _tmp0_ = NULL;
    gchar** _tmp1_ = NULL;
    gint _tmp1__length1 = 0;
    const gchar* _tmp2_ = NULL;
    FILE* _tmp3_ = NULL;
    gchar** _tmp4_ = NULL;
    gint _tmp4__length1 = 0;
    const gchar* _tmp5_ = NULL;
    gchar* _tmp6_ = NULL;
    gchar* _tmp7_ = NULL;
    Cli* cli = NULL;
    Cli* _tmp8_ = NULL;
    gchar** _tmp9_ = NULL;
    gint _tmp9__length1 = 0;
    gint _tmp10_ = 0;
    setlocale (LC_ALL, "");
    _tmp0_ = stdout;
    _tmp1_ = args;
    _tmp1__length1 = args_length1;
    _tmp2_ = _tmp1_[0];
    fprintf (_tmp0_, "command  = %s\n", _tmp2_);
    _tmp3_ = stdout;
    _tmp4_ = args;
    _tmp4__length1 = args_length1;
    _tmp5_ = _tmp4_[0];
    _tmp6_ = g_path_get_basename (_tmp5_);
    _tmp7_ = _tmp6_;
    fprintf (_tmp3_, "basename = %s\n", _tmp7_);
    _g_free0 (_tmp7_);
    _tmp8_ = cli_new ();
    cli = _tmp8_;
    _tmp9_ = args;
    _tmp9__length1 = args_length1;
    cli_parse (cli, _tmp9_, _tmp9__length1);
    _tmp10_ = cli_run (cli);
    result = _tmp10_;
    _g_object_unref0 (cli);
    return result;
}


int main (int argc, char ** argv) {
#if !GLIB_CHECK_VERSION (2,35,0)
    g_type_init ();
#endif
    return cli_main (argv, argc);
}


Cli* cli_construct (GType object_type) {
    Cli * self = NULL;
    self = (Cli*) g_object_new (object_type, NULL);
    return self;
}


Cli* cli_new (void) {
    return cli_construct (TYPE_CLI);
}


static void cli_class_init (CliClass * klass) {
    cli_parent_class = g_type_class_peek_parent (klass);
    G_OBJECT_CLASS (klass)->finalize = cli_finalize;
}


static void cli_instance_init (Cli * self) {
}


static void cli_finalize (GObject* obj) {
    Cli * self;
    self = G_TYPE_CHECK_INSTANCE_CAST (obj, TYPE_CLI, Cli);
    G_OBJECT_CLASS (cli_parent_class)->finalize (obj);
}


GType cli_get_type (void) {
    static volatile gsize cli_type_id__volatile = 0;
    if (g_once_init_enter (&cli_type_id__volatile)) {
        static const GTypeInfo g_define_type_info = { sizeof (CliClass), (GBaseI
nitFunc) NULL, (GBaseFinalizeFunc) NULL, (GClassInitFunc) cli_class_init, (GClas
sFinalizeFunc) NULL, NULL, sizeof (Cli), 0, (GInstanceInitFunc) cli_instance_ini
t, NULL };
        GType cli_type_id;
        cli_type_id = g_type_register_static (G_TYPE_OBJECT, "Cli", &g_define_ty
pe_info, 0);
        g_once_init_leave (&cli_type_id__volatile, cli_type_id);
    }
    return cli_type_id__volatile;
}


static gint _vala_array_length (gpointer array) {
    int length;
    length = 0;
    if (array) {
        while (((gpointer*) array)[length]) {
            length++;
        }
    }
    return length;
}


It is obvious that the Vala code is much shorter than the generated C code. The Vala compiler took care chores of generating CPP macros for GObject with.

  • No prefix.

  • cli, CLI, Cli: object type (= class) name

  • cli_get_type(): returning GType of object CLI

Fibonacci numbers

The Fibonacci numbers are a sequence of integers, starting with 0, 1 and continuing 1, 2, 3, 5, 8, 13, …, each new number being the sum of the previous two.

Let’s check simple code snippets to obtain Fibonacci numbers implemented in different languages to study the following:

  • basic while-loop syntax

  • integer overflow behavior

Shell

Before we start, let’s check the integer overflow behavior of the shell on the 64-bit GNU/Linux platform.

Dash behavior for the integer overflow etc.
4611686018427387904    4611686018427387904
$ echo "$(echo 2^63-1 | bc)  $((9223372036854775807))"
9223372036854775807  9223372036854775807
$ echo "$(echo 2^63 | bc)    $((9223372036854775808))"
9223372036854775808    9223372036854775807
$ echo "$(echo 2^64-1 | bc) $((18446744073709551615))"
18446744073709551615 9223372036854775807
$ echo "$(echo 2^64 | bc)   $((18446744073709551616))"
18446744073709551616   9223372036854775807
$ echo "$((2**62))"
/path/to/sh/../../bin/p: 1: eval: arithmetic expression: expecting primary: "2**
62"
Bash behavior for the integer overflow etc.
4611686018427387904    4611686018427387904
$ echo "$(echo 2^63-1 | bc)  $((9223372036854775807))"
9223372036854775807  9223372036854775807
$ echo "$(echo 2^63 | bc)    $((9223372036854775808))"
9223372036854775808    -9223372036854775808
$ echo "$(echo 2^64-1 | bc) $((18446744073709551615))"
18446744073709551615 -1
$ echo "$(echo 2^64 | bc)   $((18446744073709551616))"
18446744073709551616   0
$ echo "$((2**62))"
4611686018427387904

The arithmetic expression always returns the largest positive integer when overflows for Dash.

The "**" operator for arithmetic expression in "$((...))" is not POSIX but an usable extension for Bash.

Tip
For arbitrary precision calculation, always use "bc".

Here is a shell program for the Fibonacci numbers.

Source code for the shell script: fibo
#!/bin/sh
N="$1"
A=0
B=1
while [ "$B" -lt "$N" ]; do
    echo "$B"
    C=$B
    B=$(($A+$B))
    A=$C
    if [ "$B" -lt "0" ]; then
        echo "E: OVER FLOW at $B, stopped."
        exit 1
    fi
done

Even with some overflow protection is coded in this to avoid infinite loop, if you provide an argument to "./fibo" which is larger than "$(echo '2^63' | bc)" then this program will fail in a cryptic way.

Error with the integer overflow for the shell script
160500643816367088
259695496911122585
420196140727489673
679891637638612258
1100087778366101931
1779979416004714189
2880067194370816120
4660046610375530309
7540113804746346429
E: OVER FLOW at -6246583658587674878, stopped.
$ ./fibo "$(echo 2^63 | bc)"|tail
./fibo: 5: [: Illegal number: 9223372036854775808
Tip
Do not abuse shell code when integer overflow may occur. Its response to the overflow is cryptic.

Python

Here is a Python program for the Fibonacci numbers.

Source code for the Python script: fibo
#!/usr/bin/python3
import sys
n = int(sys.argv[1])
a, b = 0, 1
while b < n:
    print(b)
    a, b = b, a + b
Tip
Python3 integer supports the arbitrary precision number which does not overflow :-)

Lua

Here is a Lua program for the Fibonacci numbers.

Source code for the Lua script: fibo
#!/usr/bin/lua
    n = arg[1] +0;
    a = 0;
    b = 1;
    while ( b < n ) do
        print(b);
        a, b = b, a + b;
    end

Lua automatically converts large integer numbers to the double type.

Behavior with the large integer for the Lua script
27777890035288
44945570212853
72723460248141
1.1766903046099e+14
1.9039249070914e+14
3.0806152117013e+14
4.9845401187926e+14
8.0651553304939e+14
1.3049695449287e+15
2.111485077978e+15
Tip
Be careful with the behavior of Lua for large integer numbers :-)

Perl

Here is a Perl program for the Fibonacci numbers.

Source code for the Perl script: fibo
#!/usr/bin/perl -w
    $n = "$ARGV[0]" + 0;
    $a = 0;
    $b = 1;
    while ( $b < $n ) {
        print "$b\n";
        $c = $b;
        $b = $a + $b;
        $a = $c;
    }

Perl automatically converts large integer numbers to the double type.

Behavior with the large integer for the Perl script
420196140727489673
679891637638612258
1100087778366101931
1779979416004714189
2880067194370816120
4660046610375530309
7540113804746346429
12200160415121876738
1.97402742198682e+19
3.19404346349901e+19
Tip
Be careful with the behavior of Perl for large integer numbers :-)

C

Before we start, let’s check the integer overflow behavior of "strtol" on the 64-bit GNU/Linux platform using simple test code. (This is more like how Dash behaves as of 2013.)

C long behavior for the integer overflow
#include <stdlib.h>
#include <stdio.h>
int main(int argc, char **argv)
{
    long n = strtol(argv[1], NULL, 10);
    printf("Parsed as %ld\n", n);
    return EXIT_SUCCESS;
}

Let’s compile and test this C code.

Testing the C strtol with the integer overflow
$ ./strtol-test 9223372036854775807
Parsed as 9223372036854775807
$ ./strtol-test 9223372036854775808
Parsed as 9223372036854775807
$ ./strtol-test 18446744073709551615
Parsed as 9223372036854775807
$ ./strtol-test 18446744073709551616
Parsed as 9223372036854775807

The C strtol always returns the largest positive integer when overflows,

Here is a C program for the Fibonacci numbers.

Source code for the C: fibo.c
#include <stdlib.h>
#include <stdio.h>
int main(int argc, char **argv)
{
    long n = strtol(argv[1], NULL, 10);
    long a = 0;
    long b = 1;
    long c;
    while (b < n) {
        printf("%ld\n", b);
        c = b;
        b = a + b;
        a = c;
        if (b < 0) {
            printf("E: overflow %ld\n", b);
            exit(EXIT_FAILURE);
        }
    }
    return EXIT_SUCCESS;
}

The C long sufferes overflow and needs to be dealt carefully.

Error with the integer overflow for C
160500643816367088
259695496911122585
420196140727489673
679891637638612258
1100087778366101931
1779979416004714189
2880067194370816120
4660046610375530309
7540113804746346429
E: overflow -6246583658587674878
Tip
Be careful with overflow.

The size of C variables are platform dependent. Let’s check it with simple program.

Source code for the C: checksize.c
#include <stdio.h>
#include <stdlib.h>

#define checksize(type, var)    \
    type var ; \
    printf("The size of '%s' is %li Byte(s).\n", #type, sizeof(var))

int main(int argc, char** argv)
{
    checksize(char, varchar);
    checksize(short, varshort);
    checksize(int, varint);
    checksize(long, varlong);
    checksize(long long, varlonglong);
    checksize(float, varfloat);
    checksize(double, vardouble);
    checksize(long double, varlongdouble);
    return EXIT_SUCCESS;
}
Size of variables on C
Linux 3.16.0-4-amd64 x86_64 GNU/Linux
$ gcc -Wall -o checksize checksize.c
$ ./checksize
The size of 'char' is 1 Byte(s).
The size of 'short' is 2 Byte(s).
The size of 'int' is 4 Byte(s).
The size of 'long' is 8 Byte(s).
The size of 'long long' is 8 Byte(s).
The size of 'float' is 4 Byte(s).
The size of 'double' is 8 Byte(s).
The size of 'long double' is 16 Byte(s).

Vala

Here is a vala program for the Fibonacci numbers.

Source code for the Vala: fibo.vala
int main (string[] args) {
    long n = long.parse(args[1]);
    long a = 0;
    long b = 1;
    long c;
    while (b < n) {
        stdout.printf ("%ld\n", b);
        c = b;
        b = a + b;
        a = c;
        if (b < 0) {
            stdout.printf ("E: overflow %ld\n", b);
            return 0;
        }
    }
    return 0;
}

The vala long (=C long) sufferes overflow and needs to be dealt carefully.

Error with the integer overflow for Vala
$ ./fibo "$(echo 2^63-1 | bc)"|tail
160500643816367088
259695496911122585
420196140727489673
679891637638612258
1100087778366101931
1779979416004714189
2880067194370816120
4660046610375530309
7540113804746346429
E: overflow -6246583658587674878
Tip
Be careful with overflow.

Prime numbers

A prime number (or a prime) is a natural number greater than 1 that has no positive divisors other than 1 and itself.

Let’s check simple code snippets to obtain prime numbers via the same trial division algorithm implemented in different languages to study the following:

  • basic for-loop syntax

  • relative execution speed

Please note this algorhism to obtain prime numbers is not the best one.

Shell

Here is a shell program to search the prime numbers.

Source code for the Shell script: prime
#!/bin/sh
N_MAX=$1
P="" # list containing all primes found
for N in $(seq 2 $N_MAX); do
    FLAG_PRIME=1 # True
    # search for all pimes found
    for I in $P; do
        N_DIV_I=$(($N / $I))
        N_MOD_I=$(($N % $I))
        if [ $N_MOD_I = 0 ]; then
            FLAG_PRIME=0 # False
            break # found not to be prime
        fi
        if [ $N_DIV_I -lt $I ]; then
            break # no use doing more i-loop if N < I*I
        fi
    done
    if [ $FLAG_PRIME = 1 ] ; then
        P="$P $N"
    fi
done
# space separated to line break separated
echo "$P" |xargs -n1 echo

Let’s check time to compute prime numbers up to some numbers.

Behavior of the shell script
real 15.59
user 13.16
sys 0.24

Python

Here is a Python program for the prime numbers.

Source code for the Python script: prime
#!/usr/bin/python3
import sys
n_max = int(sys.argv[1])
p = [] # list containing all primes found
for n in range(2, n_max):
    flag_prime = True
    # search for all pimes found
    for i in p:
        (n_div_i, n_mod_i) = divmod(n,i)
        if n_mod_i == 0:
            flag_prime = False
            break # found not to be prime
        if n_div_i < i:
            break # no loop over i if n < i*i
    if flag_prime:
        p.append(n)
for i in p:
    print(i)
Behavior of the Perl script
real 0.09
user 0.09
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
real 2.77
user 2.77
sys 0.00

Lua

Here is a Lua program for the prime numbers with BASIC style "for".

Source code for the Lua script: prime
#!/usr/bin/lua
n_max = tonumber(arg[1])
True = 1
False = 0
p = {}; -- list containing all primes found
for n = 2, n_max do
    flag_prime = True
    -- search for all pimes found
    for ii = 1, (#p) do
        i = p[ii]
        n_div_i = math.floor(n / i)
        n_mod_i = n % i
        if n_mod_i == 0 then
            flag_prime = False
            break -- found not to be prime
        end
        if n_div_i < i then
            break -- no use doing more i-loop if n < i*i
        end
    end
    if flag_prime == True then
        table.insert(p,n)
    end
end
for ii = 1, (#p) do
    print(p[ii])
end

Here is an alternative Lua program with "`for … in `" and a iterator function.

Source code for the Lua script with "`for … in `": prime0
#!/usr/bin/lua
n_max = tonumber(arg[1])
True = 1
False = 0
function iter(a) -- ordered iterator
  local i = 0
  return function()
    i = i + 1
    return a[i]
  end
end
p = {}; -- list containing all primes found
for n = 2, n_max do
    flag_prime = True
    -- search for all pimes found
    for i in iter(p) do
        n_div_i = math.floor(n / i)
        n_mod_i = n % i
        if n_mod_i == 0 then
            flag_prime = False
            break -- found not to be prime
        end
        if n_div_i < i then
            break -- no use doing more i-loop if n < i*i
        end
    end
    if flag_prime == True then
        table.insert(p,n)
    end
end
for i in iter(p) do
    print(i)
end
Behavior of the Lua script
real 0.10
user 0.09
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
real 3.27
user 3.26
sys 0.00
$ /usr/bin/time -p ./prime0 "$(echo 2^20 | bc)">/dev/null
real 2.78
user 2.78
sys 0.00

Perl

Here is a Perl program for the prime numbers.

Source code for the Perl script: prime
#!/usr/bin/perl -w
use constant false => 0;
use constant true  => 1;
$n_max = "$ARGV[0]" + 0;
@p = (); # list containing all primes found
for ($n = 2; $n < $n_max; $n++) {
    $flag_prime = true;
    # search for all pimes found
    foreach $i (@p) {
        $n_div_i = int($n / $i);
        $n_mod_i = $n % $i;
        if ($n_mod_i == 0) {
            # found to be non-prime
            $flag_prime = false;
            last; # found not to be prime
        }
        if ($n_div_i < $i) {
            last; # no use doing more i-loop if n < i*i
        }
    }
    if ($flag_prime == true) {
        push (@p, $n);
    }
}
foreach $i (@p) {
    print "$i\n";
}
Behavior of the Perl script
real 0.09
user 0.09
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
real 2.77
user 2.77
sys 0.00

C

C with the array

Here is a C program with the array for the prime numbers.

Source code for the C program with the array: prime.c
#include <stdlib.h>
#include <stdio.h>

#define ARRAY_SIZE 100000000
#define TRUE 1
#define FALSE 0

int main (int argc, char ** argv) {
    long n_max = atol (argv[1]);
    long j = 1;
    long p[ARRAY_SIZE];
    long n, k, i, n_div_i, n_mod_i;
    int flag_prime;
    for (n = 2; n < n_max; n++) {
        flag_prime = TRUE;
        /* search for all pimes found */
        for (k = 1; k < j; k++) {
            i = p[k];
            n_div_i = n / i;
            n_mod_i = n % i;
            if (n_mod_i == 0) {
                flag_prime = FALSE;
                break; /* found not to be prime */
            }
            if (n_div_i < i) {
                break; /* no use doing more i-loop if n < i*i */
            }
        }
        if (flag_prime == TRUE) {
            p[j]=n;
            j++;
            if (j>ARRAY_SIZE) {
                printf ("E: Overflow array after %ld\n", p[j-1]);
                return EXIT_FAILURE;
            }
        }
    }
    for (k=1; k < j; k++) {
        printf ("%ld\n", p[k]);
    }
    return EXIT_SUCCESS;
}

Let’s compile this into a binary executable.

Compilation of the C program with the array
Behavior of the C program with the array
Command terminated by signal 11
real 0.00
user 0.00
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
Command terminated by signal 11
real 0.00
user 0.00
sys 0.00

C with the list

Here is a C program with the list for the prime numbers.

Source code for the C program with the list: prime.c
#include <stdlib.h>
#include <stdio.h>
#define TRUE 1
#define FALSE 0

struct _primelist {
    long prime;
    struct _primelist *next;
    };
typedef struct _primelist primelist;

primelist *head=NULL, *tail=NULL;

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    for (p = head; p != NULL ; p = p->next) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
    }
    return flag;
}

int main(int argc, char **argv) {
    primelist *p=NULL, *q=NULL;
    long n, n_max;
    n_max = atol(argv[1]);
    for (n = 2; n <= n_max; n++) {
        if (checkprime(n)) {
            q= calloc(1, sizeof(primelist));
            q->prime = n;
            if (head == NULL) {
                head = q;
            } else {
                tail->next=q;
            }
            tail = q;
        }
    }
    for (p = head; p != NULL; p = p->next) {
        printf ("%ld\n", p->prime);
    }
    for (p = head; p != NULL; q = p->next, free(p), p = q) {}
    return EXIT_SUCCESS;
}
Tip
It is a good practice to free memory properly to avoid common errors in C dynamic memory allocation even when OS may take care.

Let’s compile this into a binary executable.

Compilation of the C program with the list
Behavior of the C program with the list
real 0.00
user 0.00
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
real 0.13
user 0.13
sys 0.00

C with the list (variants)

Since this is very good example to learn C with the dynamic memory allocation, I made a combination code of several styles.

Source code for the C program with the list: prime-all.c
#include <stdlib.h>
#include <stdio.h>
#define TRUE 1
#define FALSE 0

#if STYLE==1
typedef struct _primelist {
    long prime;
    struct _primelist *next;
    } primelist;
#elif STYLE==2
typedef struct _primelist primelist;
struct _primelist {
    long prime;
    primelist *next;
    };
#else /* default */
struct _primelist {
    long prime;
    struct _primelist *next;
    };
typedef struct _primelist primelist;
#endif

primelist *head=NULL, *tail=NULL;

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
#if STYLE == 4
    p = head;
    while(p) {
#else /* default */
    for (p = head; p != NULL ; p = p->next) {
#endif
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
#if STYLE == 4
        p = p->next;
#endif
    }
    return flag;
}

int main(int argc, char **argv) {
    primelist *p=NULL, *q=NULL;
    long n, n_max;
    n_max = atol(argv[1]);
#if STYLE == 4
    n = 1;
    while(n <= n_max) {
        n++;
#else /* default */
    for (n = 2; n <= n_max; n++) {
#endif
        if (checkprime(n)) {
#if STYLE == 3
            q = malloc(sizeof(primelist));
            q->next = NULL;
#else /* default */
            q= calloc(1, sizeof(primelist));
#endif
            q->prime = n;
            if (head == NULL) {
                head = q;
            } else {
                tail->next=q;
            }
            tail = q;
        }
    }
#if STYLE == 4
    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
        free(p);
        p = q;
    }
#else /* default */
    for (p = head; p != NULL; p = p->next) {
        printf ("%ld\n", p->prime);
    }
    for (p = head; p != NULL; q = p->next, free(p), p = q) {}
#endif
    return EXIT_SUCCESS;
}

This code uses CPP macro to change its style.

  • "#define STYLE 0" enables the basic style of the struct and typedef combination.

  • "#define STYLE 1" enables the shorthand style of the struct and typedef combination.

  • "#define STYLE 2" enables the alternative style of the struct and typedef combination using the typedef feature allowing the incomplete type. (Vala generated C code uses this style.)

  • "#define STYLE 3" enables the alternative style of memory allocation using malloc instead of calloc.

  • "#define STYLE 4" enables the while-loop style instead of for-loop.

Here, I used the _primelist tag for struct to distinguish from the primelist identifier for typedef. The _primelist tags can be replaced with primelist without problem in C since struct tags and typedefs are in separate namespaces in C (But C++ is a different story …).

The first 3 memory allocation styles generate exactly the same code. The difference between calloc and malloc codes is because only calloc sets the allocated memory to zero.

The "-D NAME=DEFINITION" compiler option is processed effectively as "#define NAME DEFINITION" at the top of the source. (See more in the info page for CPP.)

Let’s compile this into 5 binaries using this "-D NAME=DEFINITION" compiler option.

Compilation of the C program with the list in several styles
$ gcc -O9 -Wall -D'STYLE=1' -o prime1 prime-all.c
$ gcc -O9 -Wall -D'STYLE=2' -o prime2 prime-all.c
$ gcc -O9 -Wall -D'STYLE=3' -o prime3 prime-all.c
$ gcc -O9 -Wall -D'STYLE=4' -o prime4 prime-all.c

All these generated binary executable programs work the same (i.e., the same execution speed).

The prime-all.c code can be simplified to prime.c by removing preprocessor conditionals with unifdef(1).

Removal of preprocessor conditionals from prime-all.c
$ unifdef -D'STYLE=1' -o prime1.c prime-all.c
$ unifdef -D'STYLE=2' -o prime2.c prime-all.c
$ unifdef -D'STYLE=3' -o prime3.c prime-all.c
$ unifdef -D'STYLE=4' -o prime4.c prime-all.c

Vala (Glib)

Here is a Vala program for the prime numbers using only Glib.

(This is a bad program example. I know this is problematic only after making this program and analyzing the problem I faced. See Prime: Benchmark.)

Source code for the Vala program: prime.vala
int main (string[] args) {
    long n_max = long.parse(args[1]);
    var p = new List<long> ();
    for (long n = 2; n < n_max; n++) {
        bool flag_prime = true;
        // search for all pimes found
        foreach (long i in p) {
            long n_div_i = n / i;
            long n_mod_i = n % i;
            if (n_mod_i == 0) {
                flag_prime = false;
                break; // found not to be prime
            }
            if (n_div_i < i) {
                break; // no use doing more i-loop if n < i*i
            }
        }
        if (flag_prime == true) {
            p.append(n);
        }
    }
    foreach (long i in p) {
        stdout.printf ("%lld\n", i);
    }
    return 0;
}
Behavior of the Vala program
real 0.05
user 0.05
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
real 7.20
user 7.20
sys 0.00

Vala (Gee)

Here is a Vala program for the prime numbers using Gee.

(This is an improved program example over previous Vala (GLib) one. See Prime: Benchmark.)

ArrayList<G> provided by Gee is used instead of List<G> provided by Glib. This type of list structure is very fast for adding data at the end of the list.

Source code for the Vala program: prime.vala
using Gee;
int main (string[] args) {
    long n_max = long.parse(args[1]);
    var p = new ArrayList<long> ();
    for (long n = 2; n < n_max; n++) {
        bool flag_prime = true;
        // search for all pimes found
        foreach (long i in p) {
            long n_div_i = n / i;
            long n_mod_i = n % i;
            if (n_mod_i == 0) {
                flag_prime = false;
                break; // found not to be prime
            }
            if (n_div_i < i) {
                break; // no use doing more i-loop if n < i*i
            }
        }
        if (flag_prime == true) {
            p.add(n);
        }
    }
    foreach (long i in p) {
        stdout.printf ("%lld\n", i);
    }
    return 0;
}
Behavior of the Vala program
real 0.01
user 0.01
sys 0.00
$ /usr/bin/time -p ./prime "$(echo 2^20 | bc)">/dev/null
real 0.47
user 0.46
sys 0.00

Benchmark

Here are benchmark results of various program languages.

Table 1. Speed benchmark of various program languages
Program real(2^16) user(2^16) sys(2^16) real(2^20) user(2^20) sys(2^20)

Shell

15.59

13.16

0.24

 — 

 — 

 — 

Python

0.12

0.12

0.00

3.76

3.76

0.00

Lua

0.10

0.09

0.00

3.27

3.26

0.00

Lua (for … in)

 — 

 — 

 — 

2.78

2.78

0.00

Perl

0.09

0.09

0.00

2.77

2.77

0.00

C (array)

0.00

0.00

0.00

0.00

0.00

0.00

C (list)

0.00

0.00

0.00

0.13

0.13

0.00

Vala (glib)

0.05

0.05

0.00

7.20

7.20

0.00

Vala (gee)

0.01

0.01

0.00

0.47

0.46

0.00

Here, the time reported by the /usr/bin/time -p command is in POSIX standard 1003.2 style:

  • real: Elapsed real (wall clock) time used by the process, in seconds.

  • user: Total number of CPU-seconds that the process used directly (in user mode), in seconds.

  • sys: Total number of CPU-seconds used by the system on behalf of the process (in kernel mode), in seconds.

Please disregard minor differences such as 20% since this benchmark is not meant to be very accurate.

The speed differences are several orders of magnitudes.

A code written in any compiler language is faster than one written in any interpreter in normal situation.

But looking at real (2^20) case, the Vala program with GLib result is very odd. For now, let me just say "The root cause is the way GLib is used. There are many operations to add an item at the end of the long list. This is slow o(n) operation."

This will be analyzed in Profiling prime.

Tip
The use of the compiler does not guarantee the faster code. Algorithm is important.

Process

Here are some practice results to play with process and inter process communication (IPC) (signal and network socket) on Debian wheezy.

Signal

Signal is explained in signal(7).

Here are default actions for notable signals.

  • Default action is to terminate the process.

    • SIGHUP = 1 : Death of controlling process

    • SIGINT = 2 : Interrupt from keyboard (Ctrl-C)

    • SIGKILL = 9 : Kill signal (non-trappable)

    • SIGALRM = 14 : Timer signal from alarm(2)

    • SIGTERM = 15 : Termination signal (default for kill)

    • SIGUSR1 = 10 : User-defined signal 1

    • SIGUSR2 = 12 : User-defined signal 2

  • Default action is to terminate the process and dump core.

    • SIGQUIT = 3 : Quit from keyboard (Ctrl-\)

    • SIGTRAP = 5 : Trace/breakpoint trap

    • SIGSEGV = 11 : Invalid memory reference

  • Default action is to continue the process

    • SIGCONT = 18 : Continue if stopped

  • Default action is to stop the process.

    • SIGSTOP = 19 : Stop process (non-trappable)

    • SIGTSTP = 20 : Stop typed at tty (Ctrl-Z)

  • Default action is to ignore the signal.

    • SIGWINCH = 28 : Gracefully shutdown the worker processes (httpd, non-POSIX)

I can check signal assignment situation quickly with the Bash builtin "kill -l" command. (Not with other shells and command!)

List of signal names and signal numbers with the Bash builtin "kill -l"
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX
Tip
Please note that SIGABRT and SIGIOT share the same signal value 6; and SIGIO and SIGLOST share the same signal value 29.

Shell: kill

The kill commands behave differently depending on which one used. Let’s check their syntax to send signals.

The example of kill commands
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill
-l [sigspec]
$ dash -c kill
dash: 1: kill: Usage: kill [-s sigspec | -signum | -sigspec] [pid | job]... or
kill -l [exitstatus]
$ /bin/kill

Usage:
 kill [options] <pid> [...]

Options:
 <pid> [...]            send signal to every <pid> listed
 -<signal>, -s, --signal <signal>
                        specify the <signal> to be sent
 -l, --list=[<signal>]  list all signal names, or convert one to a name
 -L, --table            list all signal names in a nice table

 -h, --help     display this help and exit
 -V, --version  output version information and exit

For more details see kill(1).
$ bash -c "kill -l 15"
TERM
$ dash -c "kill -l 15"
TERM
$ /bin/kill -l15
TERM
  • Only the Bash builtin kill can use jobspec such as %1 as its argument.

  • If signal number argument is given to "/bin/kill -l", there should be no space before it.

Shell: trap

The trap command can trap signals.

Example shell code with trap and infinite loop.
#!/bin/sh
trap_hup() {
    echo "SIGHUP detected."
}
trap trap_hup HUP
trap "echo 'SIGTSTP detected.'" TSTP
trap "" TERM
while true; do sleep 0.01;done

Let’s play with this on the Bash shell.

Example of kill used with ./sigtrap
$ ./sigtrap &
[1] 12692
$ kill -s HUP %1
SIGHUP detected.
$ kill -s TSTP %1
SIGTSTP detected.
$ kill -n 1 %1
SIGHUP detected.
$ kill -1 %1
SIGHUP detected.
$ ps
  PID TTY          TIME CMD
12589 pts/9    00:00:00 bash
12692 pts/9    00:00:31 sigtrap
12697 pts/9    00:00:00 ps
$ jobs
[1]+  Running                 ./sigtrap &
$ kill -s INT %1
$ jobs
[1]+  Interrupt               ./sigtrap
$ jobs
$ dash
$ ./sigtrap &
$ ps
  PID TTY          TIME CMD
12589 pts/9    00:00:00 bash
12705 pts/9    00:00:00 dash
12710 pts/9    00:00:10 sigtrap
12712 pts/9    00:00:00 ps
$ kill -s INT 12710
$ ps
  PID TTY          TIME CMD
12589 pts/9    00:00:00 bash
12705 pts/9    00:00:00 dash
12733 pts/9    00:00:00 ps
[1] + Interrupt                  ./sigtrap
$ ps
  PID TTY          TIME CMD
12589 pts/9    00:00:00 bash
12705 pts/9    00:00:00 dash
12734 pts/9    00:00:00 ps

If you are not on the Bash shell, you have to use PID "10951", instead of jobspec "%1".

When writing a shell script which is usually the Dash shell, PID changes every invocation. The use of kill is non-trivial for such case. So we use the killall(1) command instead.

Example of killall used with ./sigtrap
$ ps
  PID TTY          TIME CMD
10485 pts/2    00:00:00 sh
10486 pts/2    00:00:00 sh
10487 pts/2    00:00:00 sigtrap
10488 pts/2    00:00:00 ps
10489 pts/2    00:00:00 sleep
$ jobs
[1] + Running
$ killall -s HUP sigtrap
SIGHUP detected.
$ killall -s TSTP sigtrap
SIGTSTP detected.
$ killall -s 1 sigtrap
SIGHUP detected.
$ killall -HUP sigtrap
$ ps
  PID TTY          TIME CMD
10485 pts/2    00:00:00 sh
10486 pts/2    00:00:00 sh
10487 pts/2    00:00:00 sigtrap
10545 pts/2    00:00:00 sleep
10547 pts/2    00:00:00 ps
$ jobs
[1] + Running
$ killall -s TERM sigtrap
$ ps
SIGHUP detected.
  PID TTY          TIME CMD
10485 pts/2    00:00:00 sh
10486 pts/2    00:00:00 sh
10487 pts/2    00:00:00 sigtrap
10549 pts/2    00:00:00 ps
$ jobs
[1] + Running
$ killall -s KILL sigtrap
$ ps
  PID TTY          TIME CMD
10485 pts/2    00:00:00 sh
10486 pts/2    00:00:00 sh
10550 pts/2    00:00:00 sleep
10552 pts/2    00:00:00 ps
$ jobs
[1] + Killed

The most common use of the trap command is to set up a hook script for the program exit. Here a special signal EXIT is used in addition to other normal signals with the trap command. This EXIT signal is raised when shell script itself exits without external signals.

Example shell code with trap and remove a file.
#!/bin/sh
set -x
trap 'echo "Exit! Removing foo.tmp."; rm foo.tmp' EXIT HUP INT TERM
touch foo.tmp
# actual code to use foo.tmp
ls -l foo.tmp
sleep 1 # timing to ensure ls output

Let’s play with this.

Example of ./traprm
ls: cannot access foo.tmp: No such file or directory
$ ./traprm
+ trap echo "Exit! Removing foo.tmp."; rm foo.tmp EXIT HUP INT TERM
+ touch foo.tmp
+ ls -l foo.tmp
-rw-rw-r-- 1 osamu osamu 0 Aug 21 16:54 foo.tmp
+ sleep 1
+ echo Exit! Removing foo.tmp.
Exit! Removing foo.tmp.
+ rm foo.tmp
$ ls -l foo.tmp
ls: cannot access foo.tmp: No such file or directory

Shell: new process

Shell can start a new process easily by:

  • starting another shell program as a command.

  • starting a block of shell code list enclosed in ( ... ) as subshell.

Tip
Use of & after starting the new process enables the current shell to continue processing the subsequent part of the current shell and the new process in parrallel. It moves the new process to the background.
Tip
Use the alternative syntax of a block of shell code list enclosed in { ... ; } to execute them in the current shell environment.

Python: new process

Python can start a new process easily by:

  • starting a command "foo" using the os.system("foo") function after "import os".

  • starting a command "foo" using the os.popen("foo") function after "import os".

Tip
The new subprocess module can be used to replace many process related modules and functions to make them more secure easily. See subprocess — Subprocess management

Lua: new process

Lua can start a new process easily by:

  • starting a command "foo" using the os.execute("foo").

  • starting a command "foo" using the io.popen("foo").

C: new process

Let’s execute a command as a new process from the C program in 3 ways.

  • system(3) : The system() function starts a command by calling "/bin/sh -c" command, and returns after its completion. During execution of the command, SIGCHLD will be blocked, and SIGINT and SIGQUIT will be ignored.

  • popen(3) : The popen() function opens a process by creating a pipe(read-only or write-only), forking, and invoking the shell.

  • fork(2) + exec(3) : The fork() function creates a new process by duplicating the calling process. The exec() family of functions replaces the current process image with a new process image.

C: system

Let’s start a command from the C program with system(3) while checking environment variables.

Command execution: system.c
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit getenv system */
#include <stdio.h>      /* [f]printf perror */

int
main(int argc, char* argv[])
{
    printf("main    HOME    = %s\n", getenv("HOME"));
    printf("main    PATH    = %s\n", getenv("PATH"));
    printf("main    TERM    = %s\n", getenv("TERM"));
    printf("main    DISPLAY = %s\n", getenv("DISPLAY"));
    printf("--- Let's fake shell command prompt with system\n");
    printf("$ ls -li system.c\n");
    system("ls -li system.c");
    return EXIT_SUCCESS;

}
Run ./system
$ ./system
main    HOME    = /home/osamu
main    PATH    = /path/to/c/../../bin:(CURDIR)/bin:(CURDIR)/bin:/home/osamu/bin:...
main    TERM    = dumb
main    DISPLAY = :0
--- Let's fake shell command prompt with system
$ ls -li system.c
4252301 -rw-rw-r-- 1 osamu osamu 545 Mar 12  2013 system.c

C: popen

Let’s start a child process with popen(3) and pass messages via pipe from the child process to the parent process.

Command execution: popen.c
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit getenv */
#include <stdio.h>      /* [f]printf perror */
#include <unistd.h>     /* getpid getppid fork exec sleep */
#include <sys/types.h>  /* getpid getppid */

int
main(int argc, char* argv[])
{
    FILE* f;
    char * p;
    size_t size;
    p = malloc(1024);
    size = (size_t) 1024;
    fprintf(stderr, "main    PID     = %d\n", getpid());
    fprintf(stderr, "main    PPID    = %d\n", getppid());
    fprintf(stderr, "--- Let's fake shell command prompt with popen.\n");
    fprintf(stderr, "$ ls -li *.c; echo PPID=$PPID; echo PID=$$\n");
    f = popen("ls -li *.c; echo PPID=$PPID; echo PID=$$", "r");
    if (f < 0) {
        perror("E: popen");
        exit(EXIT_FAILURE);
    }
    fprintf(stderr, "--- Let's print result of popen.\n");
    while (getline(&p, &size, f) >= 0) {
        fprintf(stderr, "%s", p);
    }
    pclose(f);
    return EXIT_SUCCESS;

}
Run ./popen
$ echo PID=$$
PID=10230
$ echo PPID=$PPID
PPID=10229
$ ./popen
main    PID     = 10241
main    PPID    = 10230
--- Let's fake shell command prompt with popen.
$ ls -li *.c; echo PPID=$PPID; echo PID=$$
--- Let's print result of popen.
4252303 -rw-rw-r-- 1 osamu osamu 2056 Mar 12  2013 forkexec.c
4252298 -rw-rw-r-- 1 osamu osamu  942 Mar 12  2013 popen.c
4252305 -rw-rw-r-- 1 osamu osamu 1292 Mar 17  2013 sigaction.c
4252302 -rw-rw-r-- 1 osamu osamu  723 Mar 16  2013 signal.c
4252301 -rw-rw-r-- 1 osamu osamu  545 Mar 12  2013 system.c
PPID=10241
PID=10242
$ ./popen 2>/dev/null
$ ./popen >/dev/null
main    PID     = 10247
main    PPID    = 10230
--- Let's fake shell command prompt with popen.
$ ls -li *.c; echo PPID=$PPID; echo PID=$$
--- Let's print result of popen.
4252303 -rw-rw-r-- 1 osamu osamu 2056 Mar 12  2013 forkexec.c
4252298 -rw-rw-r-- 1 osamu osamu  942 Mar 12  2013 popen.c
4252305 -rw-rw-r-- 1 osamu osamu 1292 Mar 17  2013 sigaction.c
4252302 -rw-rw-r-- 1 osamu osamu  723 Mar 16  2013 signal.c
4252301 -rw-rw-r-- 1 osamu osamu  545 Mar 12  2013 system.c
PPID=10247
PID=10248

C: fork + exec

Let’s start a child process with fork(2) and pass messages from the child process to the parent process via pipe.

Fork and exec example: forkexec.c
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit getenv */
#include <stdio.h>      /* [f]printf perror */
#include <string.h>     /* strlen */
#include <unistd.h>     /* getpid getppid fork exec sleep */
#include <sys/types.h>  /* getpid getppid wait */
#include <sys/wait.h>   /* wait */

int
main(int argc, char* argv[])
{
    pid_t p;
    int pipefd[2], e;
    char buf;
    char * b = "MESSAGE: passed from child to parent via pipe\n";
    fprintf(stderr, "parent  PID     = %d\n", getpid());
    fprintf(stderr, "parent  PPID    = %d\n", getppid());
    if (pipe(pipefd) == -1) {   /* create pipe */
        perror("E: pipe");
        exit(EXIT_FAILURE);
    }
    fprintf(stderr, "--- Let's fork ...\n");
    p = fork();
    if (p == -1) {
        perror("E: Fork error\n");
        exit(EXIT_FAILURE);
    } else if (p == 0) {    /* child process */
        close(pipefd[0]);   /* Close unused read end fd */
        printf("child   PID     = %d\n", getpid());
        printf("child   PPID    = %d\n", getppid());
        printf("child   write to pipe\n");
        write(pipefd[1], b, strlen(b));
        dup2(pipefd[1], STDOUT_FILENO);
        printf("child>  ls -l forkexec.c\n");
        e = execlp("/bin/ls", "ls", "-l", "forkexec.c", NULL);   /* execute ls -l */
        close(pipefd[1]);   /* Close used write end fd */
        _exit(e);           /* idiom to end child process "immediately" */
    } else {                /* parent process */
        close(pipefd[1]);   /* Close unused write end fd */
        fprintf(stderr, "parent  PID     = %d\n", getpid());
        fprintf(stderr, "parent  PPID    = %d\n", getppid());
        fprintf(stderr, "parent  forked  = %d\n", p);
        fprintf(stderr, "parent  read from pipe\n");
        while (read(pipefd[0], &buf, 1) > 0) {
            write(STDERR_FILENO, &buf, 1);
        }
        close(pipefd[0]);   /* Close used read end fd */
        wait(NULL);         /* Wait for child */
        fprintf(stderr, "parent  child exited.\n");
    }
    return EXIT_SUCCESS;
}
Run ./forkexec
$ ./forkexec
parent  PID     = 10364
parent  PPID    = 10353
--- Let's fork ...
parent  PID     = 10364
parent  PPID    = 10353
parent  forked  = 10365
parent  read from pipe
child   PID     = 10365
child   PPID    = 10364
child   write to pipe
MESSAGE: passed from child to parent via pipe
child>  ls -l forkexec.c
-rw-rw-r-- 1 osamu osamu 2056 Mar 12  2013 forkexec.c
parent  child exited.
$
$ ./forkexec 2>/dev/null
child   PID     = 10367
child   PPID    = 10366
child   write to pipe
$
$ ./forkexec >/dev/null
parent  PID     = 10368
parent  PPID    = 10353
--- Let's fork ...
parent  PID     = 10368
parent  PPID    = 10353
parent  forked  = 10369
parent  read from pipe
MESSAGE: passed from child to parent via pipe
-rw-rw-r-- 1 osamu osamu 2056 Mar 12  2013 forkexec.c
parent  child exited.

This last method uses the exec(3) family of functions with various argument usages.

  • l-suffix: execl, execlp, execle

    • The argument list available to the new program is directly used as the later part of the arguments.

  • v-suffix: execv, execvp, execvpe

    • An array of pointers to null-terminated strings are used to represent the argument list available to the new program.

  • p-suffix: execlp execvp, execvpe

    • This indicates that the function duplicates the shell function for searching an executable file indicated by the $PATH environment variable.

  • e-suffix : execle execvpe

    • This indicates that the function specifys the environment of the executed program via the argument envp.

C: signal macro

Since the signal number assignment is system dependent, we should use macro to specify signal in the C program.

Signal macros defined in /usr/include/asm/signal.h.
#define SIGHUP           1
#define SIGINT           2
#define SIGQUIT          3
#define SIGILL           4
#define SIGTRAP          5
#define SIGABRT          6
#define SIGIOT           6
#define SIGBUS           7
#define SIGFPE           8
#define SIGKILL          9
#define SIGUSR1         10
#define SIGSEGV         11
#define SIGUSR2         12
#define SIGPIPE         13
#define SIGALRM         14
#define SIGTERM         15
#define SIGSTKFLT       16
#define SIGCHLD         17
#define SIGCONT         18
#define SIGSTOP         19
#define SIGTSTP         20
#define SIGTTIN         21
#define SIGTTOU         22
#define SIGURG          23
#define SIGXCPU         24
#define SIGXFSZ         25
#define SIGVTALRM       26
#define SIGPROF         27
#define SIGWINCH        28
#define SIGIO           29
#define SIGLOST         29
#define SIGPWR          30
#define SIGSYS          31
#define SIGRTMIN        32

C: signal(2)

For the simplicity, I start with the traditional signal(2).

Tip
The POSIX.1 sigaction(2) is more portable than the signal(2). The effects of signal(2) in a multithreaded process are unspecified. See signal(2) for details.
Signal example: signal.c
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit */
#include <stdio.h>      /* [f]printf perror */
#include <signal.h>     /* signal */
#include <unistd.h>     /* getpid pause */

void
handler(int signum)
{
    printf("\n=== SIGNAL %d handler ===\n", signum);
    printf("First round ... ignored.  Next round activated.\n");
    (void) signal(SIGINT, SIG_DFL); /* set signal behavior to default */
}
int
main(int argc, char* argv[])
{
    printf("PID=%d", getpid());
    (void) signal(SIGINT, handler); /* set signal behavior to handler */
    (void) signal(SIGHUP, SIG_IGN); /* set signal behavior to ignore */
    while(1) {
        pause();    /* waiting for signal */
    }
    return EXIT_SUCCESS;
}
Run ./signal, etc.
$ ./signal &
$  jobs
[1] + Running
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10261 10260  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10262 10261  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10273 10262  0  80   0 -  1018 -      pts/2    00:00:00 signal
0 R  1000 10280 10278  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s INT signal
PID=10273
=== SIGNAL 2 handler ===
First round ... ignored.  Next round activated.
Killed signal(10273) with signal 2
$  jobs
[1] + Running
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10261 10260  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10262 10261  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10273 10262  0  80   0 -  1018 -      pts/2    00:00:00 signal
0 R  1000 10293 10291  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s INT signal
Killed signal(10273) with signal 2
$  jobs
[1] + Interrupt
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10261 10260  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10262 10261  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 R  1000 10306 10304  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ ./signal &
$  jobs
[1] + Running
$ ./signal &
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10261 10260  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10262 10261  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10307 10262  0  80   0 -  1018 -      pts/2    00:00:00 signal
0 R  1000 10314 10312  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s HUP signal
Killed signal(10307) with signal 1
$  jobs
[1] + Running
$ ./signal &
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10261 10260  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10262 10261  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10307 10262  0  80   0 -  1018 -      pts/2    00:00:00 signal
0 R  1000 10327 10325  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s KILL signal
Killed signal(10307) with signal 9
$  jobs
[1] + Killed
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10261 10260  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10262 10261  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 R  1000 10340 10338  0  80   0 -  2654 -      pts/2    00:00:00 ps

C: sigaction(2)

Let’s try basically the same with the new sigaction(2) while preventing the reentrant signal situation.

Signal example: sigaction.c
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit */
#include <stdio.h>      /* [f]printf perror */
#include <signal.h>     /* signaction */
#include <unistd.h>     /* getpid pause */

static struct sigaction saint, sahup;

void
handler(int signum)
{
    printf("\n=== SIGNAL %d handler ===\n", signum);
    printf("First round ... ignored.  Next round activated.\n");
    saint.sa_handler = SIG_DFL;     /* signal behavior to default */
    sigaction(SIGINT, &saint, 0);   /* set signal */

}
int
main(int argc, char* argv[])
{
    printf("PID=%d", getpid());
    saint.sa_flags = 0;             /* default */
    sigemptyset(&saint.sa_mask);    /* create a open mask */
    /* sigaddset(&saint.sa_mask, SIGINT); no need since no SA_NODEFER */
    saint.sa_handler = handler;     /* signal behavior to handler */
    sigaction(SIGINT, &saint, 0);   /* set signal */

    sahup.sa_flags = SA_NODEFER;    /* set SA_NODEFER */
    sigemptyset(&sahup.sa_mask);    /* create a open mask */
    sigaddset(&sahup.sa_mask, SIGHUP); /* mask SIGHUP since SA_NODEFER */
    sahup.sa_handler = SIG_IGN;     /* signal behavior to ignore */
    sigaction(SIGHUP, &sahup, 0);   /* set signal */

    while(1) {
        pause();    /* waiting for signal */
    }
    return EXIT_SUCCESS;
}
Run ./sigaction, etc.
$ ./sigaction &
$  jobs
[1] + Running
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10093 10092  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10094 10093  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10105 10094  0  80   0 -  1018 -      pts/2    00:00:00 sigaction
0 R  1000 10112 10110  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s INT sigaction
PID=10105
=== SIGNAL 2 handler ===
First round ... ignored.  Next round activated.
Killed sigaction(10105) with signal 2
$  jobs
[1] + Running
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10093 10092  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10094 10093  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10105 10094  0  80   0 -  1018 -      pts/2    00:00:00 sigaction
0 R  1000 10125 10123  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s INT sigaction
Killed sigaction(10105) with signal 2
$  jobs
[1] + Interrupt
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10093 10092  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10094 10093  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 R  1000 10138 10136  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ ./sigaction &
$  jobs
[1] + Running
$ ./sigaction &
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10093 10092  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10094 10093  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10139 10094  0  80   0 -  1018 -      pts/2    00:00:00 sigaction
0 R  1000 10146 10144  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s HUP sigaction
Killed sigaction(10139) with signal 1
$  jobs
[1] + Running
$ ./sigaction &
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10093 10092  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10094 10093  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10139 10094  0  80   0 -  1018 -      pts/2    00:00:00 sigaction
0 R  1000 10159 10157  0  80   0 -  2654 -      pts/2    00:00:00 ps
$ killall -v -s KILL sigaction
Killed sigaction(10139) with signal 9
$  jobs
[1] + Killed
$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000 10093 10092  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 S  1000 10094 10093  0  80   0 -  1082 -      pts/2    00:00:00 sh
0 R  1000 10172 10170  0  80   0 -  2654 -      pts/2    00:00:00 ps

In above example, SA_NODEFER was set for the SIGHUP side of code just to show how to prevent reentrant signal situation explicitly by the sigaddset(3) command.

Tip
You can set up fine grained signal mask with the sigprocmask(3).

httpd in C

Let’s try coding a simple httpd server program, httpd6. This can provide good insight into how the httpd server works with the bottom up approach. Here is the design guide line I used.

  • IPv6 protocol.

  • the port number as the first argument (required, range=1024-60000)

  • the document root path as the optional second argument.

  • the default document root path as "~/public_html"

  • no CGI script support

  • no ".." in the URL string for security.

  • write simple for the easier to understand code.

Tip
See nweb: a tiny, safe Web server (static pages only) for IPv4 protocol example code. It comes with detailed explanation on how things work. (My example is basically a rewrite for IPv6 in my coding style.)
Common header : common.h
/* vi:set ts=4 sts=4 expandtab: */
#include <stdio.h>          /* [fs]printf */
#include <stdlib.h>         /* exit getpid getenv */
#include <string.h>         /* strncmp */
#include <unistd.h>         /* lseek read write fork sleep */
#include <sys/types.h>      /* lseek getpid BSD_portability */
#include <errno.h>          /* errno */
#include <fcntl.h>          /* open close */
#include <signal.h>         /* sigaction */
#include <sys/socket.h>     /* socket bind listen accept */
#include <arpa/inet.h>      /* htonl htons */

#define WEB_ROOT    "public_html/"
#define LOG_FILE    "httpd6.log"
#define MAXFD       32
#define MAXBACKLOG  64
#define PIPE_BUF    (8*1024)

/* provide "FILE *f;" by the caller routine */
/* ## is GNU extension to allow zero __VA_ARGS__ */
#define log_printf(fmt, ...)  \
    f = fopen(LOG_FILE, "a");\
    (void) fprintf(f, fmt, ##__VA_ARGS__);\
    (void) fflush(f);\
    fclose(f);\
    sync();

void httpd6(int fdsock, int hit);
int main (int argc, char **argv);

This is a common header file. Most notable is a macro log_printf which enables to write to the log file with printf syntax.

Routine to daemonize httpd6: main.c
/* vi:set ts=4 sts=4 expandtab: */
#include "common.h"

int main (int argc, char **argv)
{
    FILE *f;              /* for log_printf */
    int port, fdbl, hit, fdsock, pid, i;
    socklen_t addrlen;
    static struct sockaddr_in6 client_address, server_address;
    static struct sigaction sacld, sahup;
    if (argc < 1 || argc > 3) {
        (void) fprintf(stderr, "Mini-httpd6:\nUsage: httpd6 [port [PATH]]\n");
        exit(EXIT_FAILURE);
    } else if (argc == 1) {
        port = 8080;
    } else {
        if ((int) strlen(argv[1]) > 6) {
            (void) fprintf(stderr, "E: port is more than 5 digits.\n");
            exit(EXIT_FAILURE);
        }
        port = atoi(argv[1]);
    }
    if (port < 1024 || port > 60000) {
        (void) fprintf(stderr, "E: Invalid port number as argument. "
                "Try 1024-60000.\n");
        exit(EXIT_FAILURE);
    }
    if (argc == 3) {
        if ((int) strlen(argv[2]) > 81) {
            (void) fprintf(stderr, "E: PATH is more than 80 chars.\n");
            exit(EXIT_FAILURE);
        }
        if (chdir(argv[2])) {
            (void) fprintf(stderr, "E: Can not change to directory %s.\n", argv[2]);
            exit(EXIT_FAILURE);
        }
    } else { /* default argc = 1 or 2 */
        if (chdir(getenv("HOME"))) {
            (void) fprintf(stderr, "E: Can not change to directory %s.\n",
                getenv("HOME"));
            exit(EXIT_FAILURE);
        }
        if (chdir(WEB_ROOT)) {
            (void) fprintf(stderr, "E: Can not change to directory %s.\n",
                WEB_ROOT);
            exit(EXIT_FAILURE);
        }
    }
    sacld.sa_flags = 0;             /* default */
    sigemptyset(&sacld.sa_mask);    /* create a open mask */
    sacld.sa_handler = SIG_IGN;     /* ignore child death */
    sigaction(SIGINT, &sacld, 0);   /* set signal */
    sahup.sa_flags = 0;             /* default */
    sigemptyset(&sahup.sa_mask);    /* create a open mask */
    sahup.sa_handler = SIG_IGN;     /* ignore terminal hungup */
    sigaction(SIGINT, &sahup, 0);   /* set signal */
    for (i = 0; i <= MAXFD; i++) {
        (void) close(i);            /* close STDIN, STDOUT, ... */
    }
    pid = fork();
    if (pid == -1) {
        log_printf("ERROR: system call: fork, %s, exiting pid=%d.\n",
                strerror(errno), getpid());
        exit(EXIT_FAILURE);
    } else if (pid != 0) {
        (void) fprintf(stderr, "I: Sucessfully daemonize httpd6.\n");
        return EXIT_SUCCESS;        /* parent exit and deamonize */
    }
    if (setpgrp()) {                /* set PGID of child to PGID of parent */
        log_printf("ERROR: system call: setgrp, %s, exiting pid=%d.\n",
                strerror(errno), getpid());
        exit(EXIT_FAILURE);
    }
    log_printf("LOG: httpd starting with parent pid=%d.\n", getpid());
    fdbl = socket(AF_INET6, SOCK_STREAM, 0);
    if (fdbl == -1) {
        log_printf("ERROR: system call: socket, %s, exiting pid=%d.\n",
                strerror(errno), getpid());
        exit(EXIT_FAILURE);
    }
    server_address.sin6_family = AF_INET6;              /* IPV6 , see ip(7)*/
    server_address.sin6_addr = in6addr_any;
    server_address.sin6_port = htons(port);             /* host to network */
    if (bind(fdbl, (struct sockaddr *) &server_address,
            sizeof(server_address)) == -1) {
        log_printf("ERROR: system call: bind, %s, exiting pid=%d.\n",
                    strerror(errno), getpid());
        exit(EXIT_FAILURE);
    } else {
        log_printf("LOG: bind: with IPv6 accept, port=%d.\n", port);
    }
    if (listen(fdbl, MAXBACKLOG) == -1) {
        log_printf("ERROR: system call: listen, %s, exiting pid=%d.\n",
                strerror(errno), getpid());
        exit(EXIT_FAILURE);
    }
    log_printf("LOG: httpd successfuly bind and listen at port %i.\n", port);
    for (hit = 1; ; hit++) {
        addrlen = sizeof(client_address);
        fdsock = accept(fdbl, (struct sockaddr *) &client_address, &addrlen);
        if (fdsock == -1) {
            log_printf("ERROR: system call: accept, %s, exiting pid=%d.\n",
                    strerror(errno), getpid());
            exit(EXIT_FAILURE);
        }
        log_printf("LOG: httpd hit=%i.\n", hit);
        pid = fork();
        if (pid == -1) {
            log_printf("ERROR: system call: fork, %s, exiting pid=%d.\n",
                    strerror(errno), getpid());
            exit(EXIT_FAILURE);
        } else if (pid != 0) {
            (void) close(fdsock);                   /* parent */
        } else {
            (void) close(fdbl);                     /* child */
            log_printf("LOG: httpd forked child pid=%i.\n", getpid());
            httpd6(fdsock, hit);                     /* no return */
        }
    }
}

This essentially starts a daemonized httpd6 process. The daemonized httpd6 process is the infinite loop routine. It creates a network socket and listens to the TCP connection port specified on the command line and assume IPv6 connection. Every time` httpd6` is hit, it starts a child process to handle the HTTP request.

Main routine providing httpd6: httpd6.c
/* vi:set ts=4 sts=4 expandtab: */
#include "common.h"

void httpd6(int fdsock, int hit)
{
    char buf[PIPE_BUF +1];
    FILE *f;              /* for log_printf */
    ssize_t r, i, j;
    size_t l, m;
    int fd, k;
    struct pair {
        char *ext;
        char *filetype;
    };
    struct pair extlist[] = {
            {"html", "text/html" },
            {"htm",  "text/html" },
            {"css",  "text/css" },
            {"js",   "text/javascript" },
            {"jpeg", "image/jpeg"},
            {"jpg",  "image/jpg" },
            {"png",  "image/png" },
            {"gif",  "image/gif" },
            {"ico",  "image/ico" },
            {"gz",   "image/gz"  },
            {"xz",   "image/xz"  },
            {"tar",  "image/tar" },
            {"zip",  "image/zip" },
            {NULL, NULL}};
    /* messages */
    char *mes403 =
"HTTP/1.1 403 Forbidden\nContent-Length: 185\n"
"Connection: close\n"
"Content-Type: text/html\n\n"
"<html><head>\n"
"<title>403 Forbidden</title>\n"
"</head><body>\n"
"<h1>Forbidden</h1>\n"
"The requested URL, file type or operation is forbidden on this webserver.\n"
"</body></html>\n";

    char *mes404 =
"HTTP/1.1 404 Not Found\nContent-Length: 136\n"
"Connection: close\n"
"Content-Type: text/html\n\n"
"<html><head>\n"
"<title>404 Not Found</title>\n"
"</head><body>\n"
"<h1>Not Found</h1>\n"
"The requested URL was not found on this server.\n"
"</body></html>\n";

    char *http_header =
"HTTP/1.1 200 OK\nServer: httpd\nContent-Length: %ld\n"
"Connection: close\n"
"Content-Type: %s\n"
"charset=utf-8\n\n";

    /* Start of the routine to handle the HTTP request*/
    r = read(fdsock, buf, PIPE_BUF);
    if (r == -1) {
        log_printf("ERROR: system call: read, "
                "%s, hit=%d exiting pid=%d.\n", strerror(errno), hit, getpid());
        exit(EXIT_FAILURE);
    }
    if (r == 0) {
        (void) write(fdsock, mes403, sizeof(mes403));
        log_printf("ERROR: 403 Forbidden: zero length request, "
                "hit=%d exiting pid=%d.\n", hit, getpid());
        exit(EXIT_FAILURE);
    }

    buf[r] = '\0';  /* null terminated */
    r++;            /* size of buf used */
    for (i = 0; i < r; i++) {
        if (buf[i] == '\r' || buf[i] == '\n') {
            buf[i] = '_';
        }
    }
    log_printf("READ(%d): %s\n", hit, buf);
    if (strncmp(buf, "GET ", 4) && strncmp(buf, "get ", 4)) {
        (void) write(fdsock, mes403, sizeof(mes403));
        log_printf("ERROR: 403 Forbidden: Only simple GET supported, "
                "hit=%d exiting pid=%d.\n", hit, getpid());
        exit(EXIT_FAILURE);
    }
    /* string is "GET /FILE_PATH " + other extras */
    for (i = 4; i < r; i++) {
        if (buf[i] == ' ') {
            if (buf[i - 1] == '/') {
                (void) strcpy(&buf[i], "index.html");
            } else {
                buf[i] = '\0';
            }
            break;
        }
    }
    for (j = 4; j < i - 1; j++) {
        if (buf[j] == '.' && buf[j + 1] == '.') {
            (void) write(fdsock, mes403, sizeof(mes403));
            log_printf("ERROR: 403 Forbidden: URL with \"..\" is illegal, "
                    "hit=%d exiting pid=%d.\n", hit, getpid());
            exit(EXIT_FAILURE);
        }
    }
    l = strlen(buf);
    for (k =0; extlist[k].ext != NULL; k++) {
        m = strlen(extlist[k].ext);
        if (!strncmp(&buf[l - m], extlist[k].ext, m)) {
            break;
        }
    }
    if (extlist[k].ext == NULL) {
        (void) write(fdsock, mes403, sizeof(mes403));
        log_printf("ERROR: 403 Forbidden: file extension type is illegal, "
                "hit=%d exiting pid=%d.\n", hit, getpid());
        exit(EXIT_FAILURE);
    }
    log_printf("SEND(%d): %s as %s\n", hit, &buf[5], extlist[k].filetype);
    fd = open(&buf[5], O_RDONLY); /* open the file for reading */
    if (fd == -1) {
        (void) write(fdsock, mes404, sizeof(mes404));
        log_printf("ERROR: 404 Not Found: failed to open file, "
                "hit=%d exiting pid=%d.\n", hit, getpid());
        exit(EXIT_FAILURE);
    }
    l = lseek(fd, (off_t)0, SEEK_END);      /* file length */
    (void) lseek(fd, (off_t)0, SEEK_SET);
    (void) sprintf(buf, http_header, l, extlist[k].filetype);
    (void) write(fdsock, buf, strlen(buf)); /* send header */
    while ((r = read(fd, buf, PIPE_BUF)) > 0) {
        (void) write(fdsock, buf, r);       /* send file in block */
    }
    sleep(1);               /* allow socket to drain before signal */
    close(fdsock);
    log_printf("CLOSE(%d)\n", hit);
    exit(EXIT_FAILURE);     /* terminate child */
}

This is the routine to handle the HTTP request in the child process of the daemon. Since I put HTML protocol related text data in the codes, this is very top heavy. But this is relatively simple :-)

You can see this httpd6 can serve as IPv6 web server.

$ ./httpd6 8080 ../01_static
$ wget localhost:8080
--2013-03-25 21:57:02--  http://localhost:8080/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2175 (2.1K) [text/html]
Saving to: ‘index.html’

     0K ..                                                    100% 86.8M=0s

2013-03-25 21:57:02 (86.8 MB/s) - ‘index.html’ saved [2175/2175]

httpd in Python

Coding in the Python environment provides us with bigger building blocks to work with than that in the C environment.

The http.server module in the Python3 library contains basic HTTP server classes. Let’s use this.

Just like the argparse module in CLI programs: Python, I read the library source comments to get good idea.

$ /usr/bin/python3 -q
>>> print(sys.path)
['', '/usr/lib/python3.2', '/usr/lib/python3.2/plat-linux2', '/usr/lib/python3.2
/lib-dynload', '/usr/local/lib/python3.2/dist-packages', '/usr/lib/python3/dist-
packages']
>>> import http.server
>>> print (http.server.__file__)
/usr/lib/python3.2/http/server.py
>>> exit()
$ head /usr/lib/python3.2/http/server.py
"""HTTP server classes.

Note: BaseHTTPRequestHandler doesn't implement any HTTP request; see
SimpleHTTPRequestHandler for simple implementations of GET, HEAD and POST,
and CGIHTTPRequestHandler for CGI scripts.

It does, however, optionally implement HTTP/1.1 persistent connections,
as of version 0.3.

Notes on CGIHTTPRequestHandler

The test function towards the end of the /usr/lib/python3.2/http/server.py file seems to be good template to start with if its HTTP handler is changed to SimpleHTTPRequestHandler.

Simple Python HTTP server: httpd
#!/usr/bin/python3

import http.server
import sys

def run(HandlerClass = http.server.BaseHTTPRequestHandler,
         ServerClass = http.server.HTTPServer, protocol="HTTP/1.0"):
    """Test the HTTP request handler class.

    This runs an HTTP server on port 8080 (or the first command line
    argument).

    """

    if sys.argv[1:]:
        port = int(sys.argv[1])
    else:
        port = 8080
    server_address = ('', port)

    HandlerClass.protocol_version = protocol
    httpd = ServerClass(server_address, HandlerClass)

    sa = httpd.socket.getsockname()
    print("Serving HTTP on", sa[0], "port", sa[1], "...")
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        print("\nKeyboard interrupt received, exiting.")
        httpd.server_close()
        sys.exit(0)

if __name__ == '__main__':
    run(HandlerClass=http.server.SimpleHTTPRequestHandler)

You can see this httpd can serve as IPv4 web server.

$ ./httpd 8080 &
Serving HTTP on 0.0.0.0 port 8080 ...
$ wget -O index2.html localhost:8080
--2013-03-25 22:03:26--  http://localhost:8080/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8080... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
127.0.0.1 - - [25/Mar/2013 22:03:27] "GET / HTTP/1.1" 200 -
HTTP request sent, awaiting response... 200 OK
Length: 2175 (2.1K) [text/html]
Saving to: ‘index2.html’

     0K ..                                                    100% 5.23M=0s

2013-03-25 22:03:26 (5.23 MB/s) - ‘index2.html’ saved [2175/2175]

See:

GCC

The gccintro package provides a good tutorial "Introduction to GCC by Brian J. Gough" for the GCC basics to compile C programs.

GCC version

Check gcc version and defaults:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-b...
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10)

Basic options

Basic GCC syntax from the top few lines of its manpage:

gcc [-c|-S|-E] [-std=standard]
    [-g] [-pg] [-Olevel]
    [-Wwarn...] [-pedantic]
    [-Idir...] [-Ldir...]
    [-Dmacro[=defn]...] [-Umacro]
    [-foption...] [-mmachine-option...]
    [-o outfile] [@file] infile...

The manpage for gcc is too long. Here are the part I should remember.

  • -c : preprocess=Yes compile=Yes assemble=Yes link=No

  • -S : preprocess=Yes compile=Yes assemble=No link=No

  • -E : preprocess=Yes compile=No assemble=No link=No

  • -std=standard : specify standard conformance

    • C : -ansi is -std=c89 , default is -std=gnu89

    • C++ : -ansi is -std=c++98 , default is -std=gnu++98

  • -Wall : enables all the warnings.

  • -pedantic : warnings by strict ISO C and ISO C++ conformance

  • -g : produce debug information for gdb(1)

  • -pg : produce extra code for gprof(1)

  • -O0 : no optimization

  • -O1 : some optimization

  • -O2 : lots of optimization

  • -O3 : yet more optimization

  • -I<dir> : search the <dir> directory for header files

  • -L<dir> : search the <dir> directory for library files

  • -l<library> : search the <library> library when linking

  • -D<macro>[=<defn>] : predefine the <macro> macro (<defn> or 1)

  • -U<macro> : undefine the <macro> macro

  • -f<option> : set the <option> machine-independent flag

  • -m<machine-option> : set the <machine-option> machine-dependent flag

  • -o<outfile> : output in the <outfile> file

  • @<file> : read command-line options from <file>.

  • -v verbose output. (list defined symbols etc.)

  • -Q compiler print out each function name etc.

  • -Wp,<option> : pass option as an <option> directly to the preprocessor.

  • -Wa,<option> : pass option as an <option> directly to the assembler.

  • -Wl,<option> : pass option as an <option> directly to the linker.

  • -fpic : generate position-independent code (smaller code)

  • -fPIC : generate position-independent code (larger code)

  • -fpie : generated code for position independent executable (smaller code)

  • -fPIE : generated code for position independent executable (larger code)

  • -pie : generated position independent executable (smaller code, linker option)

  • -PIE : generated position independent executable (larger code, linker option)

Please note that gcc uses no space after the command switch and a single leading - even for long option.

The current C defualt is -std=gnu90 which is GNU dialect of ISO C90 including some C99 features.

The current C++ defualt is -std=gnu++98 which is GNU dialect of 1998 ISO C++ standard plus amendments including some C++11 features.

Tip
The meaning of inline in C is different between the default -std=gnu90 and the rest of the world (-std=gnu99|-std=c99|...). See An Inline Function is As Fast As a Macro.

Assembler code

The GCC with the -S option produces the assembler code output written in the AT&T assembler style as shown in the "Hello World!" example.

It is not so difficult to grock roghly what the GCC generated assembler code does. (Writing some code in the assembler from scratch requires serious knowledge.)

Some basic register names, command mnemonic names, and command mnemonic suffix conventions need to be noted.

  • Command mnemonic names and quasi-C equivalents:

    • "mov op1, op2" : "op2 = op1"

    • "mov (op1), op2" : "op2 = *op1"

    • "lea (op1), op2" : "op2 = op1" (load effective address)

    • "add op1, op2" : "op2 += op1"

    • "sub op1, op2" : "op2 -= op1"

    • "test op1, op2" : set flags based on "op2 & op1"

    • "cmp op1, op2" : set flags based on "op2 - op1"

    • "call op1" : call function at op1

      • Push the next instruction address %rip + 2 to the stack and jump to the op1 address

    • "ret" : return to the callee procedure

      • Pop the next instruction address %rip from the stack.

    • "jmp op" : jump unconditional to op

    • "je op" : jump equal to to op

    • "jne op" : jump not-equal to op

    • "jg op" : jump greater to op

    • "jge op" : jump greater-or-equal to op

    • "jl op" : jump less to op

    • "jle op" : jump less-or-equal to op

    • "jz op" : jump zero to op

    • "jnz op" : jump non-zero to op

  • Command mnemonic suffix indicating data width:

    • "b" : 8 bit (byte)

    • "w" : 16 bit (word)

    • "l" : 32 bit (long)

    • "q" : 64 bit (quadruple word)

  • Command mnemonic arguments:

    • "$" : immediate value following

    • "Offs(Base,Index,Scale)" : the value stored at the address Base + Index * Scale + Offs (where, Scale = 1, 2, 4, 8).

  • Register names and their data width:

    • 64 bit: %rax, %rbx, %rcx, %rdx, %rdi, %rsi, %rbp, %rsp, …

    • 32 bit: %eax, %ebx, %ecx, %edx, %edi, %esi, %ebp, %esp, …

Tip
"mov op1, op2" moves data "op1op2" in the AT&T assembler style (GCC default); while "mov op1, op2" moves data "op1op2" in the Intel assembler style (NASM default). These are in the opposite order.
Table 2. Examples of assembly codes
AT&T Intel quasi-C

movq $0x12345678, %rax

mov rax, 12345678h

rax = 0x12345678

movq $0xff, %rax

mov rax, 0ffh

rax = 0xff

movq -8(%rbp), %rax

mov rax, [rbp-8]

rax = *(rbp - 8)

movq -0x10(%rbp, %rdx, 8), %rax

mov rax, [rbp+rdx*8-10h]

rax = *(rbp + rdx * 8 - 0x10)

movq (%rcx), %rax

mov rax, [rcx]

rax = *(rcx)

movq %rcx, %rax

mov rax, rcx

rax = rcx

leaq 8(,%rcx,8), %rax

lea rax, [rcx*8+8]

rax = rcx * 8 + 8

leaq (%rbx,%rcx,4), %rax

lea rax, [rbx+rcx*4]

rax = rbx + rcx * 4

Some basic 64-bit (= 8 bytes) integer ABI conventions under the x86-64 (amd64) Linux need to be noted.

  • registers for function call return values

    • "%rax" : the 1st function return integer value (a.k.a. accumulator register)

    • "%rdx" : the 2nd function return integer value (a.k.a. data register)

    • "%xmm0" : the 1st function return double precision floating point value (128-bit SSE2 register)

    • "%xmm1" : the 2nd function return double precision floating point value (128-bit SSE2 register)

  • registers for managing the stack

    • "%rsp" : the stack pointer to the top of the stack. (%rsp a.k.a. stack pointer register)

    • "%rbp" : the frame pointing to the base of the stack frame. (%rbp a.k.a. stack base pointer register)

  • temporary registers

    • "%r10", "%r11", "%xmm8" - "%xmm15"

  • callee-saved registers

    • "%rbx", "%r12" - "%r15"

  • registers to pass function arguments

    • "%rdi" : the 1st function argument integer passed (a.k.a. destination index register)

    • "%rsi" : the 2nd function argument integer passed (a.k.a. source index register)

    • "%rdx" : the 3rd function argument integer passed (a.k.a. data register)

    • "%rcx" : the 4th function argument integer passed (a.k.a. counter register)

    • "%r8" : the 5th function argument integer passed

    • "%r9" : the 6th function argument integer passed

    • "%xmm2" - "%xmm7" : the function argument floating point passed

  • stack data usages (the stack grows from the high address to the low address)

    • stack "*(%rbp + 8*(n-5))" : the last (n-th) function argument passed

    • stack "*(%rbp + 16)" : the 7th function argument passed

    • stack "*(%rbp + 8)" : the function return address

    • stack "*(%rbp)" : the old %rbp value (%rbp: frame pointer)

    • stack "*(%rbp - 8)" : the 1st local variable

    • stack "*(%rbp - 16)" : the 2nd local variable

    • stack "*(%rbp - 24)" : the 3rd local variable

    • stack "*(%rsp)" : the top local variable (%rsp: stack pointer)

There are some memory alignment requirements of x86-64 under GCC/Linux.

  • 8-byte aligned: long, double, pointer

  • 16-byte aligned: SSE2 instructions, long double

Tip
These register usages and function call conventions are architecture and OS specific. For example, i386 passes all function arguments in the stack by pushing them in the right-to-left order.
Tip
There are some strange situation on fdivp and fdivrp: Debian Bug #372528: as/i386: reverses fdivp and fdivrp

String in C function

This tricky problem of string in C function becomes simple when you inspect the code under the assembler.

Here is a C code string-array.c which manipulates a string.

string-array.c with "char[]"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char** argv) {
    char foo[] = "abcdefgh";
    printf("Before foo[] = '%s'\n\n", foo);
    foo[3] = '@';
    printf("After  foo[] = '%s'\n\n", foo);
    return EXIT_SUCCESS;
}

This string-array.c compiles fine and runs without problem.

Compile and run of string-array.c
$ ./string-array
Before foo[] = 'abcdefgh'

After  foo[] = 'abc@efgh'

Here is a similar looking buggy C code string-pointer.c.

string-pointer.c with "char *"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char** argv) {
    char* bar = "abcdefgh";
    printf("Before bar* = '%s'\n\n", bar);
    bar[3] = '@';
    printf("After  bar* = '%s'\n\n", bar);
    return EXIT_SUCCESS;
}

This string-pointer.c compiles fine but fails to run.

Compile and run of string-pointer.c
$ ./string-pointer
Segmentation fault

This reason can be elucidated by looking into their assembler codes by compiling with the -S option.

Assembler code from string-array.c
$ cat string-array.s
    .file    "string-array.c"
    .section    .rodata
.LC0:
    .string    "Before foo[] = '%s'\n\n"
.LC1:
    .string    "After  foo[] = '%s'\n\n"
    .text
    .globl    main
    .type    main, @function
main:
.LFB2:
    .cfi_startproc
    pushq    %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    subq    $32, %rsp
    movl    %edi, -20(%rbp)
    movq    %rsi, -32(%rbp)
    movabsq    $7523094288207667809, %rax
    movq    %rax, -16(%rbp)
    movb    $0, -8(%rbp)
    leaq    -16(%rbp), %rax
    movq    %rax, %rsi
    movl    $.LC0, %edi
    movl    $0, %eax
    call    printf
    movb    $64, -13(%rbp)
    leaq    -16(%rbp), %rax
    movq    %rax, %rsi
    movl    $.LC1, %edi
    movl    $0, %eax
    call    printf
    movl    $0, %eax
    leave
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE2:
    .size    main, .-main
    .ident    "GCC: (Debian 4.9.2-10) 4.9.2"
    .section    .note.GNU-stack,"",@progbits

Here, upon execution of main function, the stack space for storing data[] is dynamically secured and the value of "abcdefgh" is stored into the stack space by the somewhat obfuscated assignment operation as below:

        movl    $1684234849, -16(%rbp)
        movl    $1751606885, -12(%rbp)
  • local data: low address = address pointed by %rbp with offset -16

    • 1684234849 = 0x64636261

    • ,, = 'd' * 0x1000000 + 'c' * 0x10000 + 'b' * 0x100 + 'a'

  • local data: high address = address pointed by %rbp with offset -12

    • 1751606885 = 0x68676665

    • ,, = 'h' * 0x1000000 + 'g' * 0x10000 + 'f' * 0x100 + 'e'

Please note x86-64 (=amd64) is little endian architecture (LSB first memory mapping) thus 'a' = 0x61 comes first in the stack.

Assembler code from string-pointer.c
$ cat string-pointer.s
    .file    "string-pointer.c"
    .section    .rodata
.LC0:
    .string    "abcdefgh"
.LC1:
    .string    "Before bar* = '%s'\n\n"
.LC2:
    .string    "After  bar* = '%s'\n\n"
    .text
    .globl    main
    .type    main, @function
main:
.LFB2:
    .cfi_startproc
    pushq    %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    subq    $32, %rsp
    movl    %edi, -20(%rbp)
    movq    %rsi, -32(%rbp)
    movq    $.LC0, -8(%rbp)
    movq    -8(%rbp), %rax
    movq    %rax, %rsi
    movl    $.LC1, %edi
    movl    $0, %eax
    call    printf
    movq    -8(%rbp), %rax
    addq    $3, %rax
    movb    $64, (%rax)
    movq    -8(%rbp), %rax
    movq    %rax, %rsi
    movl    $.LC2, %edi
    movl    $0, %eax
    call    printf
    movl    $0, %eax
    leave
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE2:
    .size    main, .-main
    .ident    "GCC: (Debian 4.9.2-10) 4.9.2"
    .section    .note.GNU-stack,"",@progbits

Here, the value of "abcdefgh" is stored in the section marked as .rodata, i.e., read-only. So the ./string-pointer command tries to overwrite this read-only data and causes segmentation error.

This execution time error can be moved to compilation time error by adding "const" to the line defining the string.

Compile error for string-const-pointer.c with "const char *"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char** argv) {
    const char* bar = "abcdefgh";
    printf("Before bar* = '%s'\n\n", bar);
    bar[3] = '@';
    printf("After  bar* = '%s'\n\n", bar);
    return EXIT_SUCCESS;
}

This string-const-pointer.c fails to compile.

Compile error of string-const-pointer.c
string-const-pointer.c: In function ‘main’:
string-const-pointer.c:8:12: error: assignment of read-only location ‘*(bar + 3u)...
     bar[3] = '@';
            ^

Buffer overflow protection

Enabling macro _FORTIFY_SOURCE with -D option substitutes high risk functions in the GNU libc library to protect against the buffer overflow risk. This requires gcc to be run with -O1 or higher optimization. This works on all CPU architectures as long as the source code is linked to the GNU libc library.

  • -D_FORTIFY_SOURCE=2 -O2

GCC’s Stack Smashing Protector (SSP) to protect against the buffer overflow risk of unknown cause was developed by IBM and originally called ProPolice. This only works on some CPU architectures. SSP can be enabled by the GCC flag:

  • SSP on for all: -fstack-protector-all

  • SSP on: -fstack-protector

  • SSP off: -fno-stack-protector

Let’s try these compiler options using an example bof.c code having the buffer overflow risk.

bof.c with the buffer overflow risk:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define DESTLEN 8
int main(int argc, char** argv)
{
    char dest[DESTLEN];
    if (argc == 2) {
        printf(">>> Before the possible buffer over flow >>>\n");
        strcpy(dest, argv[1]);
        printf("<<< After the possible buffer over flow <<<\n");
    } else {
        fprintf(stderr,"Usage: %s ARG\n", argv[0]);
        fprintf(stderr,"  Length(ARG) < %i bytes\n", DESTLEN);
        exit(EXIT_FAILURE);
    }
    return EXIT_SUCCESS;
}
Buffer overflow protection: None
$ ./bof-unsafe "0123456789"
>>> Before the possible buffer over flow >>>
<<< After the possible buffer over flow <<<
Buffer overflow protection: -D_FORTIFY_SOURCE=2
$ ./bof-fortify "0123456789"
*** buffer overflow detected ***: ./bof-fortify terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731ff)[0x7f6a20e481ff]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6a20ecb4c7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf46e0)[0x7f6a20ec96e0]
./bof-fortify[0x400578]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f6a20df6b45]
./bof-fortify[0x4005f5]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 10646427                           /path/to...
00600000-00601000 rw-p 00000000 08:01 10646427                           /path/to...
017bd000-017de000 rw-p 00000000 00:00 0                                  [heap]
7f6a20bbf000-7f6a20bd5000 r-xp 00000000 08:01 5373972                    /lib/x86...
7f6a20bd5000-7f6a20dd4000 ---p 00016000 08:01 5373972                    /lib/x86...
7f6a20dd4000-7f6a20dd5000 rw-p 00015000 08:01 5373972                    /lib/x86...
7f6a20dd5000-7f6a20f74000 r-xp 00000000 08:01 5374250                    /lib/x86...
7f6a20f74000-7f6a21174000 ---p 0019f000 08:01 5374250                    /lib/x86...
7f6a21174000-7f6a21178000 r--p 0019f000 08:01 5374250                    /lib/x86...
7f6a21178000-7f6a2117a000 rw-p 001a3000 08:01 5374250                    /lib/x86...
7f6a2117a000-7f6a2117e000 rw-p 00000000 00:00 0
7f6a2117e000-7f6a2119e000 r-xp 00000000 08:01 5374060                    /lib/x86...
7f6a21370000-7f6a21373000 rw-p 00000000 00:00 0
7f6a2139a000-7f6a2139e000 rw-p 00000000 00:00 0
7f6a2139e000-7f6a2139f000 r--p 00020000 08:01 5374060                    /lib/x86...
7f6a2139f000-7f6a213a0000 rw-p 00021000 08:01 5374060                    /lib/x86...
7f6a213a0000-7f6a213a1000 rw-p 00000000 00:00 0
7ffe8eb7e000-7ffe8eb9f000 rw-p 00000000 00:00 0                          [stack]
7ffe8ebe7000-7ffe8ebe9000 r-xp 00000000 00:00 0                          [vdso]
7ffe8ebe9000-7ffe8ebeb000 r--p 00000000 00:00 0                          [vvar]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscal...
>>> Before the possible buffer over flow >>>
Aborted
Buffer overflow protection: -fstack-protector --param=ssp-buffer-size=4
$ ./bof-safe "0123456789"
*** stack smashing detected ***: ./bof-safe terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731ff)[0x7f41df2281ff]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f41df2ab4c7]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x0)[0x7f41df2ab490]
./bof-safe[0x40073b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f41df1d6b45]
./bof-safe[0x4005b9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 10646476                           /path/to...
00600000-00601000 rw-p 00000000 08:01 10646476                           /path/to...
00ef0000-00f11000 rw-p 00000000 00:00 0                                  [heap]
7f41def9f000-7f41defb5000 r-xp 00000000 08:01 5373972                    /lib/x86...
7f41defb5000-7f41df1b4000 ---p 00016000 08:01 5373972                    /lib/x86...
7f41df1b4000-7f41df1b5000 rw-p 00015000 08:01 5373972                    /lib/x86...
7f41df1b5000-7f41df354000 r-xp 00000000 08:01 5374250                    /lib/x86...
7f41df354000-7f41df554000 ---p 0019f000 08:01 5374250                    /lib/x86...
7f41df554000-7f41df558000 r--p 0019f000 08:01 5374250                    /lib/x86...
7f41df558000-7f41df55a000 rw-p 001a3000 08:01 5374250                    /lib/x86...
7f41df55a000-7f41df55e000 rw-p 00000000 00:00 0
7f41df55e000-7f41df57e000 r-xp 00000000 08:01 5374060                    /lib/x86...
7f41df750000-7f41df753000 rw-p 00000000 00:00 0
7f41df77a000-7f41df77e000 rw-p 00000000 00:00 0
7f41df77e000-7f41df77f000 r--p 00020000 08:01 5374060                    /lib/x86...
7f41df77f000-7f41df780000 rw-p 00021000 08:01 5374060                    /lib/x86...
7f41df780000-7f41df781000 rw-p 00000000 00:00 0
7fff0ab4d000-7fff0ab6e000 rw-p 00000000 00:00 0                          [stack]
7fff0aba8000-7fff0abaa000 r-xp 00000000 00:00 0                          [vdso]
7fff0abaa000-7fff0abac000 r--p 00000000 00:00 0                          [vvar]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscal...
>>> Before the possible buffer over flow >>>
<<< After the possible buffer over flow <<<
Aborted
Buffer overflow protection: -fstack-protector-all
$ ./bof-safest "0123456789"
*** stack smashing detected ***: ./bof-safest terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731ff)[0x7f3e6ff5e1ff]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f3e6ffe14c7]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x0)[0x7f3e6ffe1490]
./bof-safest[0x40073b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f3e6ff0cb45]
./bof-safest[0x4005b9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 10646478                           /path/to...
00600000-00601000 rw-p 00000000 08:01 10646478                           /path/to...
00979000-0099a000 rw-p 00000000 00:00 0                                  [heap]
7f3e6fcd5000-7f3e6fceb000 r-xp 00000000 08:01 5373972                    /lib/x86...
7f3e6fceb000-7f3e6feea000 ---p 00016000 08:01 5373972                    /lib/x86...
7f3e6feea000-7f3e6feeb000 rw-p 00015000 08:01 5373972                    /lib/x86...
7f3e6feeb000-7f3e7008a000 r-xp 00000000 08:01 5374250                    /lib/x86...
7f3e7008a000-7f3e7028a000 ---p 0019f000 08:01 5374250                    /lib/x86...
7f3e7028a000-7f3e7028e000 r--p 0019f000 08:01 5374250                    /lib/x86...
7f3e7028e000-7f3e70290000 rw-p 001a3000 08:01 5374250                    /lib/x86...
7f3e70290000-7f3e70294000 rw-p 00000000 00:00 0
7f3e70294000-7f3e702b4000 r-xp 00000000 08:01 5374060                    /lib/x86...
7f3e70486000-7f3e70489000 rw-p 00000000 00:00 0
7f3e704b0000-7f3e704b4000 rw-p 00000000 00:00 0
7f3e704b4000-7f3e704b5000 r--p 00020000 08:01 5374060                    /lib/x86...
7f3e704b5000-7f3e704b6000 rw-p 00021000 08:01 5374060                    /lib/x86...
7f3e704b6000-7f3e704b7000 rw-p 00000000 00:00 0
7ffd72fdd000-7ffd72ffe000 rw-p 00000000 00:00 0                          [stack]
7ffd73167000-7ffd73169000 r-xp 00000000 00:00 0                          [vdso]
7ffd73169000-7ffd7316b000 r--p 00000000 00:00 0                          [vvar]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscal...
>>> Before the possible buffer over flow >>>
<<< After the possible buffer over flow <<<
Aborted

Library

Static and dynamic libraries

Compiling source while stopping at object file can be done with the -c option. You can bunch such object files into a single archive/object. This is called library.

  • static library: libfoo.a

    • simple archive of object files (*.o) as "ar rcs libfoo.a *.o"

    • *.a may be used just like bunch of *.o files while linking.

  • dynamic library: libfoo.so

    • all object files (*.o) compiled with gcc option -fPIC.

    • shared object file created by "gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0 *.o".

    • associated symbolic links created by "ldconfig".

Tip
In order to load a library file with the GCC -l option, its name must start with lib.

I do not go in details here but the gccintro package provides a good tutorial "Introduction to GCC by Brian J. Gough" with examples.

See also:

GNU C Library

The Debian libc6:amd64 package offeres embedded GNU C library which contains the standard libraries that are used by nearly all programs on the system.

Shared libraries offered by the libc6:amd64 package.
/lib/x86_64-linux-gnu/libthread_db-1.0.so
/lib/x86_64-linux-gnu/ld-2.19.so
/lib/x86_64-linux-gnu/libBrokenLocale-2.19.so
/lib/x86_64-linux-gnu/libdl-2.19.so
/lib/x86_64-linux-gnu/libutil-2.19.so
/lib/x86_64-linux-gnu/libnss_files-2.19.so
/lib/x86_64-linux-gnu/libnss_nis-2.19.so
/lib/x86_64-linux-gnu/libresolv-2.19.so
/lib/x86_64-linux-gnu/libSegFault.so
/lib/x86_64-linux-gnu/libcidn-2.19.so
/lib/x86_64-linux-gnu/libnss_dns-2.19.so
/lib/x86_64-linux-gnu/libpthread-2.19.so
/lib/x86_64-linux-gnu/librt-2.19.so
/lib/x86_64-linux-gnu/libcrypt-2.19.so
/lib/x86_64-linux-gnu/libnss_compat-2.19.so
/lib/x86_64-linux-gnu/libmemusage.so
/lib/x86_64-linux-gnu/libnss_nisplus-2.19.so
/lib/x86_64-linux-gnu/libm-2.19.so
/lib/x86_64-linux-gnu/libc-2.19.so
/lib/x86_64-linux-gnu/libpcprofile.so
/lib/x86_64-linux-gnu/libanl-2.19.so
/lib/x86_64-linux-gnu/libnss_hesiod-2.19.so
/lib/x86_64-linux-gnu/libnsl-2.19.so

libc

Most of the standard C library functions are included in the libc library.

You do not mention linking to the libc library explicitly using the -l option to GCC. It is always linked.

Most of the basic functions of the libc library are explained in many C programing tutorials such as "The C programming Language, by B. W. Kerninghan and Dennis M. Ritchie". I will skip most of those mentioned in such tutorials.

The GNU C Library manual is also good source of information.

libc: macro

There are some macros defined in the libc library. They tend to make programs easier to read.

Macro for exit status.
#define    EXIT_FAILURE    1    /* Failing exit status.  */
#define    EXIT_SUCCESS    0    /* Successful exit status.  */
Tip
The exit status value matches with the shell convention. But some programs return -1 as the non-zero value instead of 1 when errors are encountered.
Tip
Defining TRUE and FALSE macros as the Boolean context values for 1 and 0 are popular in the C program. They are not defined in the libc library. So normally, user has to define them.

libc: error.h

Here are some notable items for the error handling of the libc library.

  • The errno integer variable is set to non-zero value when library functions encounter the error.

  • The strerror(errno) function returns a pointer to a string that describes the meaning of the error for errno.

  • The perror("foo") produces a message on the standard error output for errno with "foo: <error message>"

The macros for the error are explained in the manpage errno(3).

The values of the macros for the error are defined in <errno.h> header file which is a bit convoluted. Arch dependent symlinks are marked as (*).:

<errno.h>

defines

"extern int errno;".

includes

<bits/errno.h>.

<bits/errno.h> (*)

includes

<linux/errno.h>.

<linux/errno.h>

includes

<asm/errno.h>.

<asm/errno.h> (*)

includes

<asm-generic/errno.h>.

<asm-generic/errno.h>

defines

many error values.

includes

<asm-generic/errno-base.h>.

<asm-generic/errno-base.h>

defines

important error values from 1 to 34.

Tip
Make sure to include <error.h> in the header if a program needs to deal with the libc library generated errors.

libc: string operations

Unfortunately, some C string functions are known to be troublemakers.

Table 3. Safe coding recommendations by busybox coders

troublemaker functions

overrun concerns

recommendation

strcpy(3)

dest string

safe_strncpy

strncpy(3)

may fail to 0-terminate dst

safe_strncpy

strcat(3)

dest string

strncat(3)

gets(3)

string it gets

fgets(3)

getwd(3)

buf string

getcwd(3)

[v]sprintf(3)

str buffer

[v]snprintf(3)

realpath(3)

path buffer

use with pathconf(3)

[vf]scanf(3)

its arguments

just avoid it

Although [vf]scanf(3) are marked as "just avoid it", it is not the end of the world for the coding of the scanf-like logic.

The combination of getline(3) and sscanf(3) is the most portable solution for the safe scanf alternative. If the incoming data is not delimited by the newline "\n" code, getdelim(3) may alternatively be used in place of getline(3).

The use the "m" flag in the format string, as described in scanf(3) is the other solution for the safe scanf alternative. (You need newer POSIX.1-2008 complient libc.) It uses "[vf]scanf("%ms", &char)" with free instead of "[vf]scanf("%s", char)" alone.

libc: safe_strncpy

The safe_strncpy recommended by busybox coders is essentially the 0-terminate guranteed strncpy with the safe data length. Since it is missing in the libc library, it should be emulated by adding a custom function definition as:

Alternative safe string copy function safe_strncpy: safe_strncpy.c.
#include <string.h>
char* safe_strncpy(char *dst, const char *src, size_t size)
{
    if (size == 0) {
        return dst;
    } else {
        size--;
        dst[size] = '\0';
        return strncpy(dst, src, size);
    }
}
Alternative safe string copy function safe_strncpy: safe_strncpy.h.
char* safe_strncpy(char *dst, const char *src, size_t size);

Let’s test string copy functions: strcpy, strncpy, and safe_strncpy.

Test code for string copy functions: test_strcpy.c.
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "safe_strncpy.h"

char s1[10], s2[10];
int main(int argc, char ** argv)
{
    char *sc1 = "0123456789"; /* 11 bytes */
    char *sc2 = "I gotcha!";  /* 10 bytes */

    printf("%s\n", "Constant strings");
    printf("\tsc1 = %s\n", sc1);
    printf("\tsc2 = %s\n", sc2);
    safe_strncpy(s1, sc1, 10);
    safe_strncpy(s2, sc2, 10);
    printf("%s\n", "Copied strings: safe_strncpy\n"
        "\t\tIt should drop 9 at the end to be safe.");
    printf("\ts1  = %s\n", s1);
    printf("\ts2  = %s\n", s2);
    strncpy(s1, sc1, 10);
    printf("%s\n", "Copied strings: strncpy(s1, sc1, 10)\n"
        "\t\tprintf(..., s1) can not stop at the end.");
    printf("\ts1  = %s\n", s1);
    printf("\ts2  = %s\n", s2);
    strcpy(s1, sc1);
    printf("%s\n", "Copied strings: strcpy(s1, sc1)\n"
        "\t\tstrcpy overwrites onto s2.");
    printf("\ts1  = %s\n", s1);
    printf("\ts2  = %s\n", s2);
    return EXIT_SUCCESS;
}
Test result of string copy functions by test_strcpy.c.
$ ./test_strcpy
Constant strings
    sc1 = 0123456789
    sc2 = I gotcha!
Copied strings: safe_strncpy
        It should drop 9 at the end to be safe.
    s1  = 012345678
    s2  = I gotcha!
Copied strings: strncpy(s1, sc1, 10)
        printf(..., s1) can not stop at the end.
    s1  = 0123456789
    s2  = I gotcha!
Copied strings: strcpy(s1, sc1)
        strcpy overwrites onto s2.
    s1  = 0123456789
    s2  = I gotcha!

Only safe_strncpy works safely as seen above.

libc: file operations

File operation in C can be done with different levels.

  • low level file descriptor based operations:

    • open(2): open and possibly create the file and return a new file descriptor

    • close(2): close the file descriptor

    • lseak(2): reposition read/write file offset associated with the file descriptor

    • read(2): read from the file descriptor

    • write(2): write to the file descriptor

    • mmap(2): map file associated with the file descriptor into memory

    • fcntl(2): manipulate file descriptor

Predefined file descriptor macros
#define    STDIN_FILENO    0    /* Standard input.  */
#define    STDOUT_FILENO    1    /* Standard output.  */
#define    STDERR_FILENO    2    /* Standard error output.  */
  • high level stream IO based operations:

    • fopen(3): open and possibly create the file and associate the stream with the file

    • fclose(3): close the stream

    • feof(3): test the end-of-file indicator for the stream

    • ferror(3): test the error indicator for the stream

    • getc(3): read a character from the binary stream

    • putc(3): write a character to the binary stream

    • fread(3): read blocks of data from the binary stream

    • fwrite(3): write blocks of data to the binary stream

    • fprintf(3): formatted text stream output conversion

    • fscanf(3): formatted text stream input conversion

Predefined file stream macros
#define stdin stdin
#define stdout stdout
#define stderr stderr

Let’s learn fundamentals of file operation by creating simple codes such as size of a file or copy a file. These example codes are not meant to be the fastest nor the shortest code.

libc: size of a file

We can think of 4 different methods to obtain the size of a file.

  • char : read and count characters

  • block : read blocks and count characters

  • lseek : move file offset and count characters

  • stat : obtain file size from the directory it belongs

Since stat method works only for real files but not for symlinks, lseek method seems to be the most popular one used.

Here are benchmark results of these methods using perf (See Debug: level 4: perf).

Table 4. Speed benchmark of various methods to obtain the file size.
Performance counter stats char block lseek stat

seconds time elapsed

0.060168937

0.004092420

0.002638296

0.002588425

task-clock

59.126037

3.051782

1.616662

1.634113

context-switches

5

0

0

0

cpu-migrations

1

1

0

1

page-faults

193

193

191

191

cycles

154,305,859

2,428,799

<not counted>

1,297,107

stalled-cycles-frontend

32,413,607

1,237,525

787,158

805,663

stalled-cycles-backend

3,089,296

828,200

612,162

627,711

branches

87,138,178

382,322

209,395

208,836

branch-misses

14,795

<not counted>

9,023

<not counted>

If you wish to do more than just counting characters, other methods may give good starting point for such programs. I will list all the source of size.c as below.

Read size of a file (char)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdio.h>      /* printf, perror */
#include <errno.h>      /* perror */
#include <stdlib.h>     /* exit */
#include <fcntl.h>      /* open */
#include <sys/stat.h>   /* open */
#include <sys/types.h>  /* open lseek */
#include <unistd.h>     /* lseek */
#include <locale.h>     /* setlocale */

int
main(int argc, char* argv[])
{
    FILE *f;
    off_t size = 0;
    if (argc != 2) {
        printf("E: Need a filename as an argument.\n");
        return EXIT_FAILURE;
    }
    f = fopen(argv[1], "r");
    if (f == NULL) {
        perror("E: Can not open input file");
        exit(EXIT_FAILURE);
    }

    for (;;) {
        fgetc(f);
        if (ferror(f)) {
            perror("E: Error reading input file");
            exit(EXIT_FAILURE);
        }
        if (feof(f)) {
            break;
        } else {
            size += 1;
        }
    }
    if (fclose(f)) {
        perror("E: Can not close input file");
        exit(EXIT_FAILURE);
    }

    setlocale(LC_ALL,"");
    return (printf("\nFile size: %'zi\n", size));
}
Read size of a file (block)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h> /* exit, malloc */
#include <stdio.h>  /* printf, perror, freed */
#include <errno.h>  /* perror */
#include <locale.h> /* setlocale */
#define BUFFSIZE (1024*4)
int
main(int argc, char* argv[])
{
    FILE *f;
    char *buf;
    size_t n = BUFFSIZE, i, size = 0;
    if (argc != 2) {
        printf("E: Need a filename as an argument.\n");
        return EXIT_FAILURE;
    }
    if ((buf = (char *) malloc(n)) == NULL) {
        perror("E: Can not make a buffer");
        exit(EXIT_FAILURE);
    }
    if ((f = fopen(argv[1], "r")) == NULL) {
        perror("E: Can not open input file");
        exit(EXIT_FAILURE);
    }
    for (;;) {
        i = fread(buf, 1, n, f);
        if (ferror(f)) {
            perror("E: Error reading input file");
            exit(EXIT_FAILURE);
        } else if (i == 0) {
            break;
        } else {
            size += i;
        }
    }
    if (fclose(f)) {
        perror("E: Can not close input file");
        exit(EXIT_FAILURE);
    }
    setlocale(LC_ALL,"");
    return (printf("\nFile size: %'zi\n", size));
}
Read size of a file (lseek)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdio.h>      /* printf, perror */
#include <errno.h>      /* perror */
#include <stdlib.h>     /* exit */
#include <fcntl.h>      /* open */
#include <sys/stat.h>   /* open */
#include <sys/types.h>  /* open, lseek */
#include <unistd.h>     /* lseek */
#include <locale.h>     /* setlocale */

int
main(int argc, char* argv[])
{
    int fd;
    off_t size;
    if (argc != 2) {
        printf("E: Need a filename as an argument.\n");
        return EXIT_FAILURE;
    }
    if ((fd = open(argv[1], O_RDONLY)) == -1) {
        perror("E: Can not open input file");
        exit(EXIT_FAILURE);
    }
    size = lseek(fd, 0, SEEK_END);
    setlocale(LC_ALL,"");
    printf("\nFile size: %'zi\n", size);
    return EXIT_SUCCESS;
}
Read size of a file (stat)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdio.h>      /* printf, perror */
#include <errno.h>      /* perror */
#include <stdlib.h>     /* exit */
#include <sys/types.h>  /* stat */
#include <sys/stat.h>   /* stat */
#include <unistd.h>     /* stat */
#include <locale.h>     /* setlocale */

int
main(int argc, char* argv[])
{
    struct stat st;
    off_t size;
    if (argc != 2) {
        printf("E: Need a filename as an argument.\n");
        return EXIT_FAILURE;
    }
    if (stat(argv[1], &st) == -1) {
        perror("E: Can not stat input file");
        exit(EXIT_FAILURE);
    }
    size = st.st_size;
    setlocale(LC_ALL,"");
    printf("\nFile size: %'zi\n", size);
    return EXIT_SUCCESS;
}

All the above example can be compiled as follows.

Example of compiling size.c.
$ gcc -Wall -o size size.c

libc: copy a file

We can think of 5 different methods to copy a file.

  • char : copy a character at a time

  • block : copy a block (4 KiB) at a time

  • block big : copy a block (4 MiB) at a time

  • mmap memcpy : use mmap(2) to map input and output files while copying data with memcpy(3).

  • mmap write : use mmap(2) to map input file while writing data with write(2) from the memory.

Here are benchmark results of these methods using perf (See Debug: level 4: perf).

Table 5. Speed benchmark of various methods to copy a large file about 2.4 MiB.
Performance counter stats char block block big mmap memcpy mmap write

seconds time elapsed

0.089096423

0.015654253

0.016963338

0.018813373

0.018040972

task-clock

78.600617

5.916846

7.286212

8.114573

6.747344

context-switches

12

5

5

5

5

cpu-migrations

0

0

0

0

0

page-faults

132

132

715

1,283

694

cycles

213,622,432

8,787,602

9,975,035

9,506,449

8,258,812

stalled-cycles-frontend

46,249,968

5,289,960

4,939,101

4,771,343

4,421,422

stalled-cycles-backend

8,069,075

3,617,957

3,660,122

3,965,894

3,223,960

branches

118,042,429

1,614,855

1,639,492

1,555,953

1,442,130

branch-misses

20,800

12,048

10,295

8,539

<not counted>

Table 6. Speed benchmark of various methods to copy a small file about 2 KiB.
Performance counter stats char block block big mmap memcpy mmap write

seconds time elapsed

0.002350776

0.001894332

0.001848917

0.002023421

0.002016262

task-clock

1.335467

1.008275

0.954096

1.053580

1.013171

context-switches

1

1

1

1

1

cpu-migrations

0

0

0

0

0

page-faults

132

132

132

117

111

cycles

<not counted>

731,955

769,188

606,346

616,053

stalled-cycles-frontend

696,792

625,996

648,758

579,552

552,678

stalled-cycles-backend

536,087

495,138

513,756

455,248

434,162

branches

153,252

125,526

127,090

102,813

99,467

branch-misses

3,292

<not counted>

2,232

2,474

<not counted>

The char method works the slowest as expected.

The block method works slightly faster than all other methods excluding the char method which is significantly slower.

If you wish to do more than just copying a file, other methods may give good starting point for such programs. For example, if many programs access the same file simultaneously, use of mmap(2) should have major advantage over simple block method. I will list all the source of cp.c as below.

Read copy a file (char)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit */
#include <stdio.h>      /* printf, perror */
#include <errno.h>      /* perror */
#include <locale.h>     /* setlocale */

int
main(int argc, char* argv[])
{
    FILE *fi, *fo;
    int i;
    if (argc != 3) {
        printf("E: Need 2 filenames as arguments.\n");
        return EXIT_FAILURE;
    }
    if ((fi = fopen(argv[1], "r")) == NULL) {
        perror("E: Can not open input file");
        return EXIT_FAILURE;
    }
    if ((fo = fopen(argv[2], "w")) == NULL) {
        perror("E: Can not open output file");
        return EXIT_FAILURE;
    }
    for (;;) {
        i = getc(fi);
        if (ferror(fi)) {
            perror("E: Error reading input file");
            exit(EXIT_FAILURE);
        } else if (feof(fi)) {
            break;
        } else {
            putc(i, fo);
        }
    }
    if (fclose(fi)) {
        perror("E: Can not close input file");
        exit(EXIT_FAILURE);
    }
    if (fclose(fo)) {
        perror("E: Can not close output file");
        exit(EXIT_FAILURE);
    }
    return EXIT_SUCCESS;
}
Copy a file (block)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h> /* exit, malloc */
#include <stdio.h>  /* printf, perror, freed */
#include <errno.h>  /* perror */
#include <locale.h> /* setlocale */
#define BUFFSIZE (1024*4)

int
main(int argc, char* argv[])
{
    FILE *fi, *fo;
    char *buf;
    size_t n = BUFFSIZE, i;
    if (argc != 3) {
        printf("E: Need 2 filenames as arguments.\n");
        return EXIT_FAILURE;
    }
    if ((fi = fopen(argv[1], "r")) == NULL) {
        perror("E: Can not open input file");
        return EXIT_FAILURE;
    }
    if ((fo = fopen(argv[2], "w")) == NULL) {
        perror("E: Can not open output file");
        return EXIT_FAILURE;
    }
    if ((buf = (char *) malloc(n)) == NULL) {
        perror("E: Can not make a buffer");
        exit(EXIT_FAILURE);
    }
    for (;;) {
        i = fread(buf, 1, n, fi);
        if (ferror(fi)) {
            perror("E: Error reading input file");
            exit(EXIT_FAILURE);
        } else if (i == 0) {
            break;
        } else {
            if (fwrite(buf, 1, i, fo) != i) {
                perror("E: Error writing output file");
                exit(EXIT_FAILURE);
            }
        }
    }
    if (fclose(fi)) {
        perror("E: Can not close input file");
        exit(EXIT_FAILURE);
    }
    if (fclose(fo)) {
        perror("E: Can not close output file");
        exit(EXIT_FAILURE);
    }
    return EXIT_SUCCESS;
}

Copy a file (big block) replaces #define BUFFSIZE (1024*4) in the above with #define BUFFSIZE (1024*1024*4).

Copy a file (mmap+memcpy)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h> /* exit, malloc */
#include <stdio.h>  /* printf, perror */
#include <errno.h>  /* perror */
#include <fcntl.h>      /* open */
#include <unistd.h>     /* lseek, write */
#include <sys/stat.h>   /* open */
#include <sys/types.h>  /* open, lseek */
#include <sys/mman.h>   /* mmap */
#include <locale.h> /* setlocale */
#include <string.h>     /* memcpy */
int
main(int argc, char* argv[])
{
    int fdi, fdo;
    void *src, *dst;
    size_t size;
    ssize_t offset;
    if (argc != 3) {
        printf("E: Need 2 filenames as arguments.\n");
        return EXIT_FAILURE;
    }
    if ((fdi = open(argv[1], O_RDONLY)) < 0) {
        perror("E: Can not open input file");
        exit(EXIT_FAILURE);
    }
    size = lseek(fdi, 0, SEEK_END);
    src = mmap(NULL, size, PROT_READ, MAP_SHARED, fdi, 0);
    if (src == (void *) -1) {
        perror("E: Can not map input file");
        exit(EXIT_FAILURE);
    }
    if ((fdo = open(argv[2], O_CREAT | O_RDWR | O_TRUNC, 00666)) < 0) {
        perror("E: Can not open output file");
        exit(EXIT_FAILURE);
    }
    offset = lseek(fdo, size -1, SEEK_SET);
    if (offset == (ssize_t) -1) {
        perror("E: lseek() error");
        exit(EXIT_FAILURE);
    }
    offset = write(fdo, "", 1); /* dummy write at the end */
    if (offset == (ssize_t) -1) {
        perror("E: write() error");
        exit(EXIT_FAILURE);
    }
    dst = mmap(NULL, size, PROT_WRITE, MAP_SHARED, fdo, 0);
    if (dst == MAP_FAILED) {
        perror("E: Can not map output file");
        exit(EXIT_FAILURE);
    }
    memcpy(dst, src, size);
    if (munmap(src, size)) {
        perror("E: Can not unmap input file");
        exit(EXIT_FAILURE);
    }
    if (munmap(dst, size)) {
        perror("E: Can not unmap output file");
        exit(EXIT_FAILURE);
    }
    if (close(fdi)) {
        perror("E: Can not close input file");
        exit(EXIT_FAILURE);
    }
    if (close(fdo)) {
        perror("E: Can not close output file");
        exit(EXIT_FAILURE);
    }
    return EXIT_SUCCESS;
}
Tip
The dummy write of 1 byte of 0 to the output file after lseek is the idiom to set the size of the output file with mmap(2). It will be overwritten by the subsequent memcpy(3).
Copy a file (mmap+write)
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>     /* exit, malloc */
#include <stdio.h>      /* printf, perror */
#include <errno.h>      /* perror */
#include <fcntl.h>      /* open */
#include <unistd.h>     /* stat */
#include <sys/types.h>  /* open, lseek */
#include <sys/stat.h>   /* open stat */
#include <sys/mman.h>   /* mmap */
#include <locale.h>     /* setlocale */
#include <string.h>     /* memcpy */
int
main(int argc, char* argv[])
{
    int fdi, fdo;
    struct stat st;
    void *src;
    size_t size;
    ssize_t offset;
    if (argc != 3) {
        printf("E: Need 2 filenames as arguments.\n");
        return EXIT_FAILURE;
    }
    if ((fdi = open(argv[1], O_RDONLY)) < 0) {
        perror("E: Can not open input file");
        exit(EXIT_FAILURE);
    }
    if (stat(argv[1], &st) == -1) {
        perror("E: Can not stat input file");
        exit(EXIT_FAILURE);
    }
    size = st.st_size;
    src = mmap(NULL, size, PROT_READ, MAP_SHARED, fdi, 0);
    if (src == (void *) -1) {
        perror("E: Can not map input file");
        exit(EXIT_FAILURE);
    }
    if ((fdo = open(argv[2], O_CREAT | O_RDWR | O_TRUNC, 00666)) < 0) {
        perror("E: Can not open output file");
        exit(EXIT_FAILURE);
    }
    offset = write(fdo, src, size);
    if (offset == (ssize_t) -1) {
        perror("E: write() error");
        exit(EXIT_FAILURE);
    }
    if (munmap(src, size)) {
        perror("E: Can not unmap input file");
        exit(EXIT_FAILURE);
    }
    if (close(fdi)) {
        perror("E: Can not close input file");
        exit(EXIT_FAILURE);
    }
    if (close(fdo)) {
        perror("E: Can not close output file");
        exit(EXIT_FAILURE);
    }
    return EXIT_SUCCESS;
}
Example of compiling cp.c.
$ gcc -Wall -o cp cp.c

libc: setlocale

For decimal conversion functions provided by the libc library such as printf(3), the 3-digit-grouping behavior depends on the locale. Use setlocale(3) to set the locale.

Localization example of printf as lprintf.c
/* vi:set ts=4 sts=4 expandtab: */
#include <stdlib.h>
#include <stdio.h>
#include <locale.h>

int
main(int argc, char* argv[])
{
    int n = 12345678;
    printf("init        %%i  -> %i\n", n);
    printf("init        %%'i -> %'i\n", n);
    setlocale(LC_ALL,"");
    printf("env         %%i  -> %i\n", n);
    printf("env         %%'i -> %'i\n", n);
    setlocale(LC_ALL,"C");
    printf("C           %%i  -> %i\n", n);
    printf("C           %%'i -> %'i\n", n);
    setlocale(LC_ALL,"en_US.UTF-8");
    printf("en_US.UTF-8 %%i  -> %i\n", n);
    printf("en_US.UTF-8 %%'i -> %'i\n", n);
    setlocale(LC_ALL,"fr_FR.UTF-8");
    printf("fr_FR.UTF-8 %%i  -> %i\n", n);
    printf("fr_FR.UTF-8 %%'i -> %'i\n", n);
    setlocale(LC_ALL,"de_DE.UTF-8");
    printf("de_DE.UTF-8 %%i  -> %i\n", n);
    printf("de_DE.UTF-8 %%'i -> %'i\n", n);
    setlocale(LC_ALL,"ja_JP.UTF-8");
    printf("ja_JP.UTF-8 %%i  -> %i\n", n);
    printf("ja_JP.UTF-8 %%'i -> %'i\n", n);
    return EXIT_SUCCESS;
}
Run lprintf
$ ./lprintf
init        %i  -> 12345678
init        %'i -> 12345678
env         %i  -> 12345678
env         %'i -> 12,345,678
C           %i  -> 12345678
C           %'i -> 12345678
en_US.UTF-8 %i  -> 12345678
en_US.UTF-8 %'i -> 12,345,678
fr_FR.UTF-8 %i  -> 12345678
fr_FR.UTF-8 %'i -> 12 345 678
de_DE.UTF-8 %i  -> 12345678
de_DE.UTF-8 %'i -> 12.345.678
ja_JP.UTF-8 %i  -> 12345678
ja_JP.UTF-8 %'i -> 12,345,678
Tip
The text translation mechanism also uses the locale. See gettext(3) and "info gettext".

libm

Although most of standard C library functions are included in the libc library, some math related library functions are in the separate libm library.

So such program requires to be linked not just to libc but also to libm.

Let’s consider math.c to calculate sin(60 degree).

Source code: math.c
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
double
sindeg(double x)
{
    return sin(x * 3.141592 / 180.0);
}

int
main()
{
    double x, y;
    x = 60.0;
    y = sindeg(x);
    printf("angle = %f degree, sin(angle) = %f\n", x, y);
    exit(0);
}

Let’s compile math.c while linking to the libm library to create an ELF executable object math and run it.

$ ./math
angle = 60.000000 degree, sin(angle) = 0.866025
Tip
The linked library is specified after the -l option to GCC while removing the leading lib from the library name.

Let’s list linked libraries to the ELF executable object hello.

    linux-vdso.so.1 (0x00007ffd089c8000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f679316a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6792dc1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f679346b000)
  • linux-vdso.so.1 : Linux Virtual Dynamic Shared Object

  • libm.so.6 : The GNU C Library (glibc, support math functions)

  • libc.so.6 : The GNU C Library (glibc)

  • /lib64/ld-linux-x86-64.so.2 : dynamic linker/loader

We can split math.c into 3 files and compile each code piece.

Source code: math-main.c containing the main() function only
#include <stdio.h>
#include <stdlib.h>
#include "sindeg.h"
int main()
{
    double x, y;
    x = 60.0;
    y = sindeg(x);
    printf("angle = %f degree, sin(angle) = %f\n", x, y);
    exit(0);
}
Source code: sindeg.c containing the sindeg() function only
#include <math.h>
double sindeg(double x)
{
    return sin(x * 3.141592 / 180.0);
}
Source code: sindeg.h containing the header information of sindeg()
double sindeg(double x);

Let’s compile these into separate object files (*.o files) with the -c option and link them into a executable math-main specified with the -o option.

Building math-main via separate object files and running it.
$ gcc -Wall -c sindeg.c
$ gcc -Wall -o math-main math-main.o sindeg.o -lm
$ ./math-main
angle = 60.000000 degree, sin(angle) = 0.866025

libdl

libdl provides the following generic functions for the dynamic loader.

  • dlopen(3): POSIX

  • dlerror(3): POSIX

  • dlsym(3): POSIX

  • dlclose(3): POSIX

  • dladdr(3): Glibc extensions

  • dlvsym(3): Glibc extensions

Let’s convert math.c to math-dl.c which uses libm via dynamic linking loader provided by libdl. Here, function names prefixed with v are defined as wrappers for the dlopen, dlsym, and dlclose functions providing verbose error reporting. So the core is just main().

Source code: math-dl.c
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

void *
vdlopen(const char *filename, int flag) {
    void *dl_handle;
    dl_handle = dlopen(filename, RTLD_LAZY);
    if (!dl_handle) {
        printf("Load %s library, error: %s\n", filename, dlerror());
        exit(EXIT_FAILURE);
    } else {
        printf("Load %s library, success!\n", filename);
    }
    return dl_handle;
}

void *
vdlsym(void *handle, const char *symbol) {
    double (*func)(double); /* Yep ... this is only for double */
    func = dlsym(handle, symbol);
    if (!func) {
        printf("Load %s symbol, error: %s\n", symbol, dlerror());
        exit(EXIT_FAILURE);
    } else {
        printf("Load %s symbol, success!\n", symbol);
    }
    return func;
}

int
vdlclose(void *handle) {
    int i;
    i = dlclose(handle);
    if (i) {
        printf("Unload error %i: %s\n", i, dlerror());
        exit(EXIT_FAILURE);
    } else {
        printf("Unload, success!\n");
    }
    return i;
}

int
main()
{
    double x, y;
    x = 60.0;
    char *lib = "libm.so";
    char *method = "sin";
    void *dl_handle;
    double (*func)(double);
    dl_handle = vdlopen(lib, RTLD_LAZY);
    func = vdlsym(dl_handle, method);
    y = (*func)(x * 3.141592 / 180.0);
    printf("angle = %f degree, sin(angle) = %f\n", x, y);
    vdlclose(dl_handle);
    exit(0);
}

Let’s compile this math-dl.c by not directly linking to libm but to libdl.

Building math-dl and running it.
$ ./math-dl
Load libm.so library, success!
Load sin symbol, success!
angle = 60.000000 degree, sin(angle) = 0.866025
Unload, success!

The -rdynamic option is used to add all symbols to the dynamic symbol table to permit backtraces with the use of dlopen. The -ldl option is for the libdl.so library. You do not need the -lm option here but need libm.so library installed in the system.

libpthread

The thread can efficiently implement parallelism for shared memory multiprocessor architectures, such as SMPs. The thread creation does not copy ever resources like the fork-exec multiprocessing mechanism of the UNIX-like system. POSIX thread is supported by the modern GNU/Linux (with [Linux kernel 2.6 or newer]) with the libpthread library. Here are some references and tutorials.

Tip
We should focus on reading tutorials which are written after the native POSIX thread library (NPTL) support. This means tutorial should be newer than 2006.

The actual execution speed of a program on the modern CPU can be affected by many issues other than the utilization of CPU cores:

I am no expert to tell how all these fit together. But seeing is believing. Let’s use a touched-up C programs to list prime numbers based on while-loop style in C with the list (variants) to to experiment with the POSIX thread programing. This algorithm has some sequential nature. So the task of partitioning the program into parallel and mostly independent code is non-trivial. The observed execution time figures are significantly different.

  • prime5.c: fast

    • single-threaded program, a uninterrupted tight code.

  • prime6.c: slow

    • multi-threaded program, a thread started for each integer.

  • prime7.c: very slow

    • multi-threaded program, fixed number of threads are started and controlled via semaphore.

  • prime8.c: very fast

    • multi-threaded program, fixed number of threads are started only for the time consuming large number portion while each thread is written as a uninterrupted tight code.

Here is a benchmark results of execution times for these programs listed below.

Table 7. Speed benchmark of various program languages
Program real(2^20) user(2^20) sys(2^20) real(2^24) user(2^24) sys(2^24)

prime5.c

0.13

0.13

0.00

4.94

4.94

0.00

prime6.c

0.21

0.08

0.43

3.41

1.42

7.01

prime7.c

0.09

0.04

0.23

1.53

0.86

3.60

prime8.c

0.04

0.14

0.00

1.63

5.32

0.03

Here, the time reported by the /usr/bin/time -p command is in POSIX standard 1003.2 style:

  • real: Elapsed real (wall clock) time used by the process, in seconds.

  • user: Total number of CPU-seconds that the process used directly (in user mode), in seconds.

  • sys: Total number of CPU-seconds used by the system on behalf of the process (in kernel mode), in seconds.

It seems that the user time and the sys time add up all multi-threaded time figures so they may end up in larger figure than the real time figure for multi-threaded programs. There are a similar sar command offered by the sysstat and atsar packages which has more functionalities. But if you are looking for more insight for the time, you should consider using the perf command. See Debug: level 2; perf.

Tip
Unless properly used, the use of the thread mechanism does not guarantee the faster code.

prime5.c: single-threaded program, a uninterrupted tight code.

Source code for the C program: prime5.c
#include <stdlib.h>
#include <stdio.h>
#define TRUE 1
#define FALSE 0

struct _primelist {
    long prime;
    struct _primelist *next;
    };
typedef struct _primelist primelist;

primelist *head=NULL, *tail=NULL;

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    p = head;
    while(p) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
        p = p->next;
    }
    return flag;
}

int main(int argc, char **argv) {
    primelist *p=NULL, *q=NULL;
    long n, n_max;
    n_max = atol(argv[1]);
    head = calloc(1, sizeof(primelist));
    tail = head;
    tail->prime = 2;
    n = 2;
    while(n <= n_max) {
        n++;
        if (checkprime(n)) {
            q= calloc(1, sizeof(primelist));
            tail->next = q;
            tail = q;
            tail->prime = n;
        }
    }
    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
            free(p);
            p = q;
    }
    return EXIT_SUCCESS;
}
Behavior of the C program: prime5.c
real 0.13
user 0.13
sys 0.00
$ /usr/bin/time -p ./prime5 "$(echo 2^24 | bc)">/dev/null
real 4.94
user 4.94
sys 0.00

prime6.c: multi-threaded program, a thread started for each integer.

Source code for the Vala program: prime6.c
#include <stdlib.h>
#include <stdio.h>
#include <semaphore.h>
#include <pthread.h>
#define TRUE 1
#define FALSE 0
#define TMAX 64L

struct _primelist {
    long prime;
    struct _primelist *next;
    };
typedef struct _primelist primelist;

struct _thdata {
    pthread_t           th;
    int                 flag;
};
typedef struct _thdata thdata;

primelist *head=NULL, *tail=NULL;

thdata       thd[TMAX];

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    p = head;
    while(p) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
        p = p->next;
    }
    return flag;
}

int main(int argc, char **argv) {
    primelist *p=NULL, *q=NULL;
    long n, n_max, m;
    n_max = atol(argv[1]);
    /* thdata = calloc(TMAX, sizeof(thdata)); */
    head = calloc(1, sizeof(primelist));
    tail = head;
    tail->prime = 2;
    n = 2; /* last number checking for prime */
    m = 2; /* last number checked  for prime */
    while(m < n_max) {
        if ((n + 1 - m < TMAX) && (n + 1 <= m * m) && (n + 1 <= n_max)) {
            n = n + 1;
            /* start checkprime(n) */
            if (pthread_create(&thd[n%TMAX].th,
                        NULL,
                        (void *) checkprime,
                        (void *) n) ) {
                printf ("E: error creating thread at %li\n", n);
            }
        }
        if ((n + 1 - m >= TMAX) || (n + 1 > m * m) || (n + 1 > n_max) ) {
            m++;
            /* close checkprime(m) */
            pthread_join(thd[m%TMAX].th, (void *) &thd[m%TMAX].flag);
            if (thd[m%TMAX].flag) {
                /* if prime, update list with m */
                q = calloc(1, sizeof(primelist));
                    tail->next = q;
                    tail = q;
                tail->prime = m;
            }
        }
    }
    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
            /* free(p); */
            p = q;
    }
    return EXIT_SUCCESS;
}
Behavior of the C program: prime6.c
real 0.21
user 0.08
sys 0.43
$ /usr/bin/time -p ./prime6 "$(echo 2^20 | bc)">/dev/null
real 3.41
user 1.42
sys 7.01

prime7.c: multi-threaded program, fixed number of threads are started and controlled via semaphore.

Source code for the Vala program: prime7.c
#include <stdlib.h>
#include <stdio.h>
#include <semaphore.h>
#include <pthread.h>
#define TRUE 1
#define FALSE 0
#define TMAX 64L

struct _primelist {
    long prime;
    struct _primelist *next;
    };
typedef struct _primelist primelist;

struct _thdata {
    pthread_t           th;
    long                n;
    int                 flag;
    sem_t               read_ready;
    sem_t               read_done;
    sem_t               write_ready;
    sem_t               write_done;
};
typedef struct _thdata thdata;

primelist *head=NULL, *tail=NULL;

thdata       thd[TMAX];

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    p = head;
    while(p) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
        p = p->next;
    }
    return flag;
}

void subthread(thdata *thd) {
    while(TRUE) {
        sem_post(&(thd->read_ready));
        sem_wait(&(thd->read_done));
        thd->flag = checkprime(thd->n);
        sem_post(&(thd->write_ready));
        sem_wait(&(thd->write_done));
    }
}

int main(int argc, char **argv) {
    primelist *p=NULL, *q=NULL;
    long n, n_max, m;
    int i;
    n_max = atol(argv[1]);
    head = calloc(1, sizeof(primelist));
    tail = head;
    tail->prime = 2;
    for (i=0;i<TMAX;i++){
        sem_init(&thd[i].read_ready, 0, 0);
        sem_init(&thd[i].read_done, 0, 0);
        sem_init(&thd[i].write_ready, 0, 0);
        sem_init(&thd[i].write_done, 0, 0);
        if (pthread_create(&thd[i].th,
                NULL,
                (void *) subthread,
                (void *) &(thd[i]) ) ) {
            printf ("E: error creating thread at %i\n", i);
        }
    }
    n = 2; /* last number started   checking of prime*/
    m = 2; /* last number completed checking of prime */
    while(m < n_max) {
        if ((n + 1 - m < TMAX) && (n + 1 <= m * m) && (n + 1 <= n_max)) {
            n = n + 1;
            /* start checkprime(n) */
            sem_wait(&(thd[n%TMAX].read_ready));
            thd[n%TMAX].n = n;
            sem_post(&(thd[n%TMAX].read_done));
        }
        if ((n + 1 - m >= TMAX) || (n >= m * m) || (n >= n_max) ) {
            m++;
            /* close checkprime(m) */
            sem_wait(&(thd[m%TMAX].write_ready));
            if (thd[m%TMAX].flag) {
                /* if prime, update list with m */
                q = calloc(1, sizeof(primelist));
                    tail->next = q;
                    tail = q;
                tail->prime = m;
            }
            sem_post(&(thd[m%TMAX].write_done));
        }
    }
    for (i=0;i<TMAX;i++){
        if (pthread_cancel(thd[i].th)) {
            printf ("E: error canseling thread at %i\n", i);
        }

    }
    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
            /* free(p); */
            p = q;
    }
    return EXIT_SUCCESS;
}
Behavior of the C program: prime7.c
real 0.09
user 0.04
sys 0.23
$ /usr/bin/time -p ./prime7 "$(echo 2^20 | bc)">/dev/null
real 1.53
user 0.86
sys 3.60

prime8.c: multi-threaded program, fixed number of threads are started only for the time consuming large number portion while each thread is written as a uninterrupted tight code.

Source code for the Vala program: prime8.c
#include <stdlib.h>
#include <stdio.h>
#include <semaphore.h>
#include <pthread.h>
#define TRUE 1
#define FALSE 0
#define TMAX 64L

struct _primelist {
    long prime;
    struct _primelist *next;
    };
typedef struct _primelist primelist;

primelist *head=NULL, *tail=NULL;

struct _thdata {
    pthread_t           th;
    long                n0;
    long                n1;
    primelist           *head;
    primelist           *tail;
};
typedef struct _thdata thdata;

thdata       thd[TMAX];

int checkprime(long n) {
    primelist *p;
    long i, n_div_i, n_mod_i;
    int flag;
    flag = TRUE;
    p = head;
    while(p) {
        i = p->prime;
        n_div_i = n / i;
        n_mod_i = n % i;
        if (n_mod_i == 0) {
            flag = FALSE;
            break; /* found not to be prime */
        }
        if (n_div_i < i) {
            break; /* no use doing more i-loop if n < i*i */
        }
        p = p->next;
    }
    return flag;
}

void subthread(thdata *thd) {
    long i;
    primelist *p=NULL, *q=NULL;
    thd->head = NULL;
    for (i = thd->n0; i <= thd->n1; i++) {
        if (checkprime(i)) {
            q = calloc(1, sizeof(primelist));
            q->prime = i;
            if (!thd->head) {
                thd->head = q;
                p = q;
            } else {
                p->next = q;
                p = q;
            }
            thd->tail = q;
        }
    }
}

int main(int argc, char **argv) {
    primelist *p=NULL, *q=NULL;
    long n, n_max, i, nd;
    n_max = atol(argv[1]);
    head = calloc(1, sizeof(primelist));
    tail = head;
    tail->prime = 2;
    n = 2;
    while((n - 1) * (n - 1) <= n_max) {
        n++;
        if (checkprime(n)) {
            q= calloc(1, sizeof(primelist));
            tail->next = q;
            tail = q;
            tail->prime = n;
        }
    }
    nd = (n_max - n ) / (long) TMAX + 1L;
    for (i=0; i < TMAX; i++) {
        /* TMAX thread of checkprime loop */
        thd[i].n0 = n;
        thd[i].n1 = n + nd;
        if (thd[i].n1 >= n_max) {
            thd[i].n1 = n_max;
        }
        n = thd[i].n1;
        if (pthread_create(&thd[i].th,
                NULL,
                (void *) subthread,
                (void *) &(thd[i]) ) ) {
            printf ("E: error creating thread at %li\n", i);
        }
    }
    for (i=0; i < TMAX; i++) {
        /* TMAX thread of checkprime loop */
        if (pthread_join(thd[i].th, (void *) NULL) ) {
            printf ("E: error joining thread at %li\n", i);
        }
        tail->next = thd[i].head;
        tail = thd[i].tail;
    }

    p=head;
    while(p) {
        printf ("%ld\n", p->prime);
        p = p->next;
    }
    p=head;
    while(p) {
        q = p->next;
            free(p);
            p = q;
    }
    return EXIT_SUCCESS;
}
Behavior of the C program: prime8.c
real 0.04
user 0.14
sys 0.00
$ /usr/bin/time -p ./prime8 "$(echo 2^24 | bc)">/dev/null
real 1.63
user 5.32
sys 0.03

Actually, this program is buggy for smaller than 1090. We will debug this later.

Buggy behavior of the C program for 1090: prime8.c
Segmentation fault
139
$ ./prime8 "1091">/dev/null; echo $?
0

ELF

In this chapter, I will play with ELF files generated from the same hello.c program file as the one in the Hello World: C with the debug information to learn how it works.

See the following for the he ELF:

An ELF file has two views:

  • The section header lists the set of sections of the binary. (readelf -S)

  • The program header shows the segments used at run-time. (readelf -l)

Tip
Both readelf and objdump offer similar functions. The readelf command exists independently of the BFD library, so if there is a bug in the BFD objdump command then readelf will not be affected. Only the objdump command can do disassemble.

Compile hello-gdb

Let’s compile the hello.c program file into the ELF executable hello-gdb with the additional flags.

  • -g flag produces debugging information in the operating system’s native format (DWARF on GNU/Linux).

  • -v flag produces verbose output of the commands executed to run the stages of compilation.

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-
bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,g
o,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --e
nable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-
threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --en
able-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable
-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-
plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enabl
e-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-
java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-j
ar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64
 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multia
rch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enabl
e-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gn
u --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10)
COLLECT_GCC_OPTIONS='-v' '-Wall' '-g' '-o' 'hello-gdb' '-mtune=generic' '-march=
x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1 -quiet -v -imultiarch x86_64-linux-gnu he
llo.c -quiet -dumpbase hello.c -mtune=generic -march=x86-64 -auxbase hello -g -W
all -version -o /tmp/ccrN97vs.s
GNU C (Debian 4.9.2-10) version 4.9.2 (x86_64-linux-gnu)
    compiled by GNU C version 4.9.2, GMP version 6.0.0, MPFR version 3.1.2-p3, M
PC version 1.0.2
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../x8
6_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/4.9/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
GNU C (Debian 4.9.2-10) version 4.9.2 (x86_64-linux-gnu)
    compiled by GNU C version 4.9.2, GMP version 6.0.0, MPFR version 3.1.2-p3, M
PC version 1.0.2
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: dc5310582b2a0cacbb4cea323963a0c0
COLLECT_GCC_OPTIONS='-v' '-Wall' '-g' '-o' 'hello-gdb' '-mtune=generic' '-march=
x86-64'
 as -v --64 -o /tmp/ccuttySQ.o /tmp/ccrN97vs.s
GNU assembler version 2.25 (x86_64-linux-gnu) using BFD version (GNU Binutils fo
r Debian) 2.25
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/4
.9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/g
cc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/:/usr/lib/gcc/x86_64-linux-gnu/4.
9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib/:
/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:
/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-Wall' '-g' '-o' 'hello-gdb' '-mtune=generic' '-march=
x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/collect2 -plugin /usr/lib/gcc/x86_64-linux-gn
u/4.9/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
 -plugin-opt=-fresolution=/tmp/cch48Jff.res -plugin-opt=-pass-through=-lgcc -plu
gin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-th
rough=-lgcc -plugin-opt=-pass-through=-lgcc_s --sysroot=/ --build-id --eh-frame-
hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 -
o hello-gdb /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crt1.o /
usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x
86_64-linux-gnu/4.9/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gc
c/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu
/4.9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-lin
ux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../.. /tmp/ccutt
ySQ.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --n
o-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.9/crtend.o /usr/lib/gcc/x86_64-linux
-gnu/4.9/../../../x86_64-linux-gnu/crtn.o

Here the gcc command steps through 3 stages of compilation with 3 sub-commands:

  • /usr/lib/gcc/x86_64-linux-gnu/4.7/cc1 which is roughly the gcc -S command (Compiler from the C code to the assembler code).

  • as which is the assembler. (The gcc -c stops after this.)

  • /usr/lib/gcc/x86_64-linux-gnu/4.7/collect2 which is roughly equivalent of the good old ld (linker).

FYI:

/path/to/c/../../bin/p: 1: eval: /usr/lib/gcc/x86_64-linux-gnu/4.7/cc1: not foun
d
$ /usr/lib/gcc/x86_64-linux-gnu/4.7/cc1 --help=C
/path/to/c/../../bin/p: 1: eval: /usr/lib/gcc/x86_64-linux-gnu/4.7/cc1: not foun
d
   ... (snip)
$ /usr/lib/gcc/x86_64-linux-gnu/4.7/collect2 --help
/path/to/c/../../bin/p: 1: eval: /usr/lib/gcc/x86_64-linux-gnu/4.7/collect2: not
 found
   ... (snip)

In order to understand how linker works to create an executable from an object file, let me make the hello.o and hello-gdb.o ELF object files with the gcc -c command.

$ gcc -Wall -g -o hello-gdb.o -c hello.c

So including the hello file generated in Hello World!: C, we have 4 ELF files to play with.

Table 8. Summary of options and generated files from hello.c.
object file executable file

option

file

option

file

-c

hello.o

hello

-c -g

hello-gdb.o

-g

hello-gdb

size

Here is the size of the ELF files.

Size of the ELF file generated from hello.c.
-rwxrwxr-x 1 osamu osamu 6696 Aug 21 16:53 hello
-rwxrwxr-x 1 osamu osamu 7728 Aug 21 16:53 hello-gdb
-rw-rw-r-- 1 osamu osamu 3304 Aug 21 16:53 hello-gdb.o
-rw-rw-r-- 1 osamu osamu 1488 Aug 21 16:53 hello.o
$ size  hello.o hello hello-gdb.o hello-gdb
   text       data        bss        dec        hex    filename
     91          0          0         91         5b    hello.o
   1216        560          8       1784        6f8    hello
     91          0          0         91         5b    hello-gdb.o
   1216        560          8       1784        6f8    hello-gdb

The GNU size command lists the section sizes — and the total size — for each of the object or archive files in its argument list.

  • The text column is the size of the executable instructions of a program in the decimal expression.

  • The data column is the size of the initialized data that contribute to the program’s memory image in the decimal expression.

  • The bss column is the size of the uninitialized data that contributes to the program’s memory image in the decimal expression.

  • The dec column is the total size of the text, data, and bss sections in the decimal expression.

  • The hex column is the total size of the text, data, and bss sections in the hexadecimal expression.

Since the difference between hello and hello-gdb is the debug information, they share the same values with size(1) but their file size is different with ls(1).

The text column of hello-gdb is much larger than that of hello-gdb.o since the linker seems to link few additional object files such as crt1.o, crti.o, etc.

These will be elucidated from the following results.

readelf --file-header

Here are the ELF file header of the hello file.

ELF file Header of the hello file.
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          656 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         13
  Section header string table index: 10
Tip
-h = --file-header

Let’s compare the ELF file headers of the compiled hello.c.

Table 9. ELF file headers of the compiled hello.c
Item hello.o hello-gdb.o hello hello-gdb

Magic

7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00

Class

ELF64

Data

2’s complement, little endian

Version (EI_VERSION)

1 (current)

OS/ABI

UNIX - System V

ABI Version

0

Type

REL (Relocatable file)

EXEC (Executable file)

Machine

Advanced Micro Devices X86-64

Version (e_version)

0x1

0x1

0x1

0x1

Entry point address

0x0

0x0

0x400410

0x400410

Start of program headers (bytes into file)

0

0

64

64

Start of section headers (bytes into file)

656

1960

4776

5488

Flags

0x0

0x0

0x0

0x0

Size of this header (bytes)

64

64

64

64

Size of program headers (bytes)

0

0

56

56

Number of program headers

0

0

8

8

Size of section headers (bytes)

64

64

64

64

Number of section headers

13

21

30

35

Section header string table index

10

18

27

32

readelf --sections

The section header lists the set of sections of the binary.

Here are the sections in these ELF files.

Sections of the hello-gdb file.
There are 35 section headers, starting at offset 0x1570:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .interp           PROGBITS         0000000000400200  00000200
       000000000000001c  0000000000000000   A       0     0     1
  [ 2] .note.ABI-tag     NOTE             000000000040021c  0000021c
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .note.gnu.build-i NOTE             000000000040023c  0000023c
       0000000000000024  0000000000000000   A       0     0     4
  [ 4] .gnu.hash         GNU_HASH         0000000000400260  00000260
       000000000000001c  0000000000000000   A       5     0     8
  [ 5] .dynsym           DYNSYM           0000000000400280  00000280
       0000000000000060  0000000000000018   A       6     1     8
  [ 6] .dynstr           STRTAB           00000000004002e0  000002e0
       000000000000003d  0000000000000000   A       0     0     1
  [ 7] .gnu.version      VERSYM           000000000040031e  0000031e
       0000000000000008  0000000000000002   A       5     0     2
  [ 8] .gnu.version_r    VERNEED          0000000000400328  00000328
       0000000000000020  0000000000000000   A       6     1     8
  [ 9] .rela.dyn         RELA             0000000000400348  00000348
       0000000000000018  0000000000000018   A       5     0     8
  [10] .rela.plt         RELA             0000000000400360  00000360
       0000000000000048  0000000000000018  AI       5    12     8
  [11] .init             PROGBITS         00000000004003a8  000003a8
       000000000000001a  0000000000000000  AX       0     0     4
  [12] .plt              PROGBITS         00000000004003d0  000003d0
       0000000000000040  0000000000000010  AX       0     0     16
  [13] .text             PROGBITS         0000000000400410  00000410
       0000000000000182  0000000000000000  AX       0     0     16
  [14] .fini             PROGBITS         0000000000400594  00000594
       0000000000000009  0000000000000000  AX       0     0     4
  [15] .rodata           PROGBITS         00000000004005a0  000005a0
       0000000000000012  0000000000000000   A       0     0     4
  [16] .eh_frame_hdr     PROGBITS         00000000004005b4  000005b4
       0000000000000034  0000000000000000   A       0     0     4
  [17] .eh_frame         PROGBITS         00000000004005e8  000005e8
       00000000000000f4  0000000000000000   A       0     0     8
  [18] .init_array       INIT_ARRAY       00000000006006e0  000006e0
       0000000000000008  0000000000000000  WA       0     0     8
  [19] .fini_array       FINI_ARRAY       00000000006006e8  000006e8
       0000000000000008  0000000000000000  WA       0     0     8
  [20] .jcr              PROGBITS         00000000006006f0  000006f0
       0000000000000008  0000000000000000  WA       0     0     8
  [21] .dynamic          DYNAMIC          00000000006006f8  000006f8
       00000000000001d0  0000000000000010  WA       6     0     8
  [22] .got              PROGBITS         00000000006008c8  000008c8
       0000000000000008  0000000000000008  WA       0     0     8
  [23] .got.plt          PROGBITS         00000000006008d0  000008d0
       0000000000000030  0000000000000008  WA       0     0     8
  [24] .data             PROGBITS         0000000000600900  00000900
       0000000000000010  0000000000000000  WA       0     0     8
  [25] .bss              NOBITS           0000000000600910  00000910
       0000000000000008  0000000000000000  WA       0     0     1
  [26] .comment          PROGBITS         0000000000000000  00000910
       0000000000000039  0000000000000001  MS       0     0     1
  [27] .debug_aranges    PROGBITS         0000000000000000  00000949
       0000000000000030  0000000000000000           0     0     1
  [28] .debug_info       PROGBITS         0000000000000000  00000979
       000000000000009f  0000000000000000           0     0     1
  [29] .debug_abbrev     PROGBITS         0000000000000000  00000a18
       0000000000000042  0000000000000000           0     0     1
  [30] .debug_line       PROGBITS         0000000000000000  00000a5a
       000000000000003c  0000000000000000           0     0     1
  [31] .debug_str        PROGBITS         0000000000000000  00000a96
       00000000000000c3  0000000000000001  MS       0     0     1
  [32] .shstrtab         STRTAB           0000000000000000  00000b59
       0000000000000148  0000000000000000           0     0     1
  [33] .symtab           SYMTAB           0000000000000000  00000ca8
       0000000000000690  0000000000000018          34    50     8
  [34] .strtab           STRTAB           0000000000000000  00001338
       0000000000000237  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), l (large)
  I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)
Tip
-S = --sections = --section-headers

Let’s compare the ELF file sections of the compiled hello.c.

Table 10. ELF file sections of the compiled hello.c
section type hello.o hello-gdb.o hello hello-gdb size

NULL

0

0

0

0

.interp

PROGBITS

N/A

N/A

0x1c

0x1c

text

.note.ABI-tag

NOTE

N/A

N/A

0x20

0x20

text

.note.gnu.build-i

NOTE

N/A

N/A

0x24

0x24

text

.dynsym

DYNSYM

N/A

N/A

0x60

0x60

text

.dynstr

STRTAB

N/A

N/A

0x3d

0x3d

text

.gnu.hash

GNU_HASH

N/A

N/A

0x1c

0x1c

text

.gnu.version

VERSYM

N/A

N/A

0x8

0x8

text

.gnu.version_r

VERNEED

N/A

N/A

0x20

0x20

text

.rela.dyn

RELA

N/A

N/A

0x18

0x18

text

.rela.plt

RELA

N/A

N/A

0x48

0x48

text

.init

PROGBITS

N/A

N/A

0x1a

0x1a

text

.plt

PROGBITS

N/A

N/A

0x40

0x40

text

.text

PROGBITS

0x15

0x15

0x182

0x182

text

.fini

PROGBITS

N/A

N/A

0x9

0x9

text

.rodata

PROGBITS

0xe

0xe

0x12

0x12

text

.eh_frame

PROGBITS

0x38

0x38

0xf4

0xf4

text

.eh_frame_hdr

PROGBITS

N/A

N/A

0x34

0x34

text

.dynamic

DYNAMIC

N/A

N/A

0x1d0

0x1d0

data

.got

PROGBITS

N/A

N/A

0x8

0x8

data

.got.plt

PROGBITS

N/A

N/A

0x30

0x30

data

.data

PROGBITS

0

0

0x10

0x10

data

.jcr

PROGBITS

N/A

N/A

0x8

0x8

data

.fini_array

FINI_ARRAY

N/A

N/A

0x8

0x8

data

.init_array

INIT_ARRAY

N/A

N/A

0x8

0x8

data

.bss

NOBITS

0

0

0x8

0x8

bss

.comment

PROGBITS

0x1e

0x1e

0x39

0x39

???

.debug_info

PROGBITS

N/A

0x9f

N/A

0x9f

???

.debug_abbrev

PROGBITS

N/A

0x42

N/A

0x42

???

.debug_aranges

PROGBITS

N/A

0x30

N/A

0x30

???

.debug_line

PROGBITS

N/A

0x3c

N/A

0x3c

???

.debug_str

PROGBITS

N/A

0xfc

N/A

0xc3

???

.symtab

SYMTAB

0x108

0x180

0x618

0x690

???

.strtab

STRTAB

0x13

0x13

0x237

0x237

???

.shstrtab

STRTAB

0x61

0xb0

0x108

0x148

???

Here, N/A stands for "not available".

Let’s compaire the .text size reported by size and readelf -S.

Table 11. Comparison of the reported size.
file size readelf

hello.o

text = 91 = 0x5b

.text + .rodata + .eh_frame = 0x5b

hello