Ganeti (a cluster of debian machines on your laptop)

What is Ganeti

Ganeti is a virtual machine cluster manager.

It manages a collection of nodes
It can use various hypervisors
- kvm, xen (hvm or pvm)
- in development: lxc
It manages various types of backend disks
- drbd, lvm, files
It has a simple API to support different OSes
You can use it to schedule virtual machines (instances) on the nodes
It supports redundancy, and live migration, without the need for shared storage

You can also use it to quickly create and dispose development machines.

Ganeti Development and Debian Release status

ganeti 1.2.6 in lenny:
- has a bug with xen hvm support, hopefully can be fixed in a point release
- only supports xen
- ganeti 1.2.8 is the last update for the 1.2 series, which is now deprecated
ganeti2 (2.0.1) in sid:
- uploaded during debconf
- supports kvm, xen-hvm and xen-pvm
- will upload to backports.org as it reaches testing
- 2.0 is in active development for bugfixes and "minor" features
- branch 'master' in git is the current stable, 'next' is the next one
ganeti 'branch-2.1' is the unstable development branch, in git
- new shiny features and bugs
- don't use in production
- doc/design-2.1.rst contains an outline of the previewed changes
- if you want to submit any major change, this is where to start :)

Structure of Ganeti

Daemons

Ganeti-masterd

Runs on the ganeti master node:

Manage the job queue.
Handle concurrent job execution.
Manage the cluster config.
Manage replication of data.

Ganeti-noded

Runs on all nodes:

Simple https rpc daemon: receives rpc calls from masterd and performs the appropriate actions
Does all system-level operations, but is stateless

Ganeti-rapi

Runs on the master:

Simple https rest interface to the cluster
Useful to program your own interface to Ganeti

More to come

Read the Ganeti 2.1 design doc for future daemons and their purpose.

Scripts

gnt-cluster: cluster level operations
gnt-instance: instance commands
gnt-node: node management
gnt-job: job management
gnt-os: os listing/diagnosing
gnt-backup: instance import/export
gnt-debug: well, debug

Tools

lvmstrap: quickly bootstrapping lvm on all unused disks
cfgshell: a less scary 'vi /var/lib/ganeti/config.data'
cfgupgrade: upgrade from Ganeti 1.2 to 2.0
htools (external to ganeti) (and, for now, not in debian)

Ganeti as a development/testing environment

How to create the environment

This is geared at a one-node portable secure installation. Don't use it on a real cluster: follow the install.html Ganeti doc.

Add a bridge to your machine

cat >>/etc/network/interfaces <<EOF
iface dummy0 inet static
address 192.168.99.254
netmask 255.255.255.0

iface br0 inet static
address 192.168.99.254
netmask 255.255.255.0
bridge_ports dummy0
pre-up /sbin/ifup dummy0
up /etc/init.d/dnsmasq restart
down /etc/init.d/dnsmasq stop
EOF

You can also set up dnsmasq to only start when br0 is up, if you want.

Securely bind your daemons

cat >/etc/default/ganeti <<EOF
NODED_ARGS="-b 192.168.99.254"
RAPI_ARGS="-b 192.168.99.254"
EOF

RAPI=/bin/true works as well :)

iptables rules

$IPT -N FWD_VIRTUAL
$IPT -t nat -N PST_VIRTUAL

$IPT -A FORWARD -m state --state INVALID -j DROP
$IPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A FORWARD -j FWD_VIRTUAL

$IPT -t nat -A POSTROUTING -j PST_VIRTUAL

$IPT -F FWD_VIRTUAL
$IPT -A FWD_VIRTUAL -s 192.168.99.0/24 -i br0 -j ACCEPT
$IPT -A FWD_VIRTUAL -s 192.168.99.0/24 -i tap+ -j ACCEPT

$IPT -t nat -F PST_VIRTUAL
$IPT -t nat -A PST_VIRTUAL -s 192.168.99.0/24 -o wlan0 -j MASQUERADE
$IPT -t nat -A PST_VIRTUAL -s 192.168.99.0/24 -o wlan1 -j MASQUERADE
$IPT -t nat -A PST_VIRTUAL -s 192.168.99.0/24 -o eth0 -j MASQUERADE
$IPT -t nat -A PST_VIRTUAL -s 192.168.99.0/24 -o ppp0 -j MASQUERADE

Add a few /etc/hosts entries

cat <<EOF >/etc/hosts
192.168.99.254 myhost.mobile.example.com myhost
192.168.99.253 cluster.mobile.example.com cluster
192.168.99.10 i0.mobile.example.com i0
192.168.99.11 i1.mobile.example.com i1
192.168.99.12 i2.mobile.example.com i2
192.168.99.13 i3.mobile.example.com i3
EOF

Also make sure you have a FQDN in /etc/hostname

Install Ganeti

apt-get install ganeti2 ganeti-instance-debootstrap

ganeti-instance-debootstrap is customized through /etc/default/ganeti-instance-debootstrap. Unfortunately for now different installations are necessary to use different config files on the fly. (This will change in Ganeti 2.1)

Using Ganeti

gnt-cluster init [--no-lvm-storage] -t kvm --enabled-hypervisors=kvm cluster
# install your kvm kernel
gnt-instance add -t file -n myhost -o debootstrap -s 512M i4
# ...
ping i0
ssh i0 # easy with OS hooks
gnt-instance remove i0

hooks (diversion)

Useful to personalize the installed OS
An example 'ethers' ganeti hook is in git
With the example 'interfaces' hook in ganeti it can be used to automate network setup

Nice uses

Whatever you can do with virtual machines, plus a very fast deployment:

Play with different/newer kernels
Play with different networking setups (bridge, routed, multicast tunnels)
Quickly build or test a package on its own machine
Securely test network protocols/daemons

Conclusion

This is a showcase for how I use Ganeti to quickly create experimental development environments on the go, on my laptop.