Set up an LXC Host using Ubuntu and ZFS

Background

I needed a host that manages LXC instances (Linux Kernel Containers) on my local network. I wanted the LXC instances to fetch IP addresses from my already existing, local DHCP server. I also wanted to take advantage of the ZFS snapshot and deduplication features, as I will have lots of LXC instances based on the same root image.

So this manual outlines the steps to install LXC and ZFS on a minimal Ubuntu 14.04 LTS. I pieced the parts relevant for my situation together from various tutorials and sources out there. Enjoy.

Step 1: Install Base OS

Start out with a plain vanilla Ubuntu 14.04 LTS Server installer. The partition layout needs to be done manually, as the installer does not support ZFS partitions. I decided to go with the following, very simple layout. No LUKS, no LVM, just 3 primary partitions. A small OS partition and a decent swap partition. The third partition filling all the available space on the disk, but being marked as "do not use" in the installer, so it will be created, but not mounted. This will become our ZFS pool later. This runs on a hardware RAID, so no need to pool multiple disks in ZFS here.

/dev/sda1     /           10 GB
   /dev/sda2     swap        min(RAM,8GB)
   /dev/sda3     "do not use this partition"

Step 2: Install Packages

Refresh everything and install a few required packages:

# apt-get update
# apt-get dist-upgrade
# apt-get install vim software-properties-common bridge-utils

Step 3: Configure Bridged Network

Since the LXC instances will have IPs in the same subnet as the LXC host itself, the host's primary interface, "eth0", must not be configured with an IP. Instead, a bridge interface with the LXC host's primary IP needs to be created. The following /etc/network/interfaces file will accomplish that. My LXC host's IP will be 192.168.5.50.

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface without IP
iface eth0 inet manual

# The bridge interface for the LXCs and the host itself
auto br0
iface br0 inet static
address 192.168.5.50
netnask 255.255.255.0
gateway 192.168.5.1
dns-nameservers 192.168.5.1
bridge_ports eth0

If you are using the UFW firewall on the LXC host, you need to allow bridged traffic across by adding the following into /etc/ufw/before.rules, before the final COMMIT:

# Allow bridged traffic to be forwarded
-I FORWARD -m physdev --physdev-is-bridged -j ACCEPT

Restart the network (or reboot) to make the new network configuration active.

Step 4: Install and Configure ZFS and zpool

Add the PPA for the ZFS modules, then install the required packages:

# add-apt-repository ppa:zfs-native/stable
# apt-get update
# apt-get install ubuntu-zfs

Create the zpool on the previously created but un-used /dev/sda3 partition:

# zpool create lxc /dev/sda3
# zpool set listsnapshots=on lxc
# zfs create -o dedup=on lxc/containers
# zfs create lxc/tmp
# chmod 1777 /lxc/tmp

Step 5: Install and Configure LXC

Add packages:

# apt-get install lxc
# cp /etc/lxc/default.conf{,.orig}

Optional: if you want to use swap space with cgroups, add this to the GRUB_CMDLINE_LINUX_DEFAULT parameter in /etc/default/grub:

cgroup_enable=memory swapaccount=1

and then run

# update-grub

and reboot.


Tweak the configuration in /etc/lxc/default.conf:

lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

Tweak the configuration in /etc/lxc/lxc.conf to use the ZFS backend:

lxc.lxcpath = /lxc/containers
lxc.bdev.zfs.root = lxc/containers

Step 6: Start Creating Containers

You are now ready to start creating LXC instances. They should come up with a DHCP address from the DHCP server in the same subnet.

# lxc-create -t ubuntu -n my_lxc_1 -B zfs

To limit the amount of RAM and swap space for a container, make sure you enabled swap support above. Then add something like this to the container's config file:

lxc.cgroup.memory.limit_in_bytes = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 1G