Filesystem Resilience with dm-crypt, BTRFS, and OverlayFS - Linux

This is a guide to gaining multiple levels of resilience for a filesystem through the combination of dm-crypt, BTRFS, and OverlayFS.

We will be encrypting a partition, formatting it with BTRFS, snapshotting it, then mounting it into an OverlayFS. Through this setup we will gain privacy with dm-crypt, protection against faulty upgrades with btrfs snapshotting, and protection against human error with OverlayFS. I will be using Arch Linux and Pacman to create this setup but it translates very easily to other Linux distros.

Prerequisites

# Install btrfs-progs
$ pacman -Sy btrfs-progs

Now let’s setup our dm-crypt container using LUKS.

# Create encrypted partition
$ cryptsetup [options] luksFormat /dev/sdXY

# Open the encrypted partition
$ cryptsetup open /dev/sdXY [crypt]

Now, let’s format the newly created device with BTRFS

$ mkfs.btrfs -L MyFilesystem /dev/mapper/[crypt]

Now, let’s mount the device into an OverlayFS

# Create the required directories
$ mkdir overlay
$ mkdir overlay/lower
$ mkdir overlay/upper
$ mkdir overlay/work

# Create the directory we will mount to
$ mkdir mnt

# Mount the encrypted device
$ mount /dev/mapper/[crypt] overlay/lower

# Mount the OverlayFS
$ mount -t overlayfs overlayfs -o lowerdir=overlay/lower,upperdir=overlay/upper,workdir=overlay/work mnt

Snapshotting

You can use any snapshotting scheme you’d like at this point but I would highly suggest using the built in “btrfs subvolume set-default” command to mount a default subvolume/snapshot when mounting so you don’t have to edit the fstab or mounting script each time you want to switch subvolumes. Since the directory “mnt” is an OverlayFS we will need to work with the “overlay/lower” directory. I will demonstrate a subvolume scheme here.

# Currently the root subvolume (5) is mounted by default
# Create a snapshots directory
$ mkdir overlay/lower/.snapshots

# Create an "actively" mounted subvolume
$ btrfs sub snap overlay/lower overlay/lower/.snapshots/ACTIVE

# Create a backup subvolume
$ btrfs sub snap overlay/lower/.snapshots/ACTIVE overlay/lower/.snapshots/$(date +%F)

# List the subvolumes
$ btrfs sub list overlay/lower
ID 261 gen 192 top level 5 path .snapshots/ACTIVE
ID 266 gen 190 top level 5 path .snapshots/2016-03-19

# Now set the default mounted subvolume
$ btrfs sub set 261 overlay/lower

#
# Disclaimer
#
# The 'top level' id number for the above subvolume,
# as well as the id numbers in the subsequent sections,
# will be more accurate on your system;
# reflecting the actual subvolume ID each was snapshotted from
# and not just the root subvolume (5) as is listed above

Now you can do anything you want in the “overlay/lower” directory. Once you remount the OverlayFS you will see the changes.

Persisting Changes

You’re probably asking, “Well, how do I update the system or keep any file changes?”. You can simply use rsync to copy your changes to disc.

# Mount the root subvolume
$ mkdir btrfs-root
$ mount /dev/mapper/[crypt] btrfs-root -o subvol=/

# Create a snapshot of the "active" subvolume for rolling back changes
$ btrfs sub snap btrfs-root/.snapshots/ACTIVE btrfs-root/.snapshots/$(date +%F)

# Make some changes to the OverlayFS
$ touch mnt/file1 mnt/file2

# Persist the changes (don't forget the trailing forward slash)
$ rsync -ravv mnt/ btrfs-root/.snapshots/ACTIVE/

When we remount the overlay filesystem we should see that our changes have peristed.

Roll Back Changes

If something goes wrong after you’ve peristed your changes, you can simply set the default mounting subvolume and remount the partition

# List the subvolumes available
$ btrfs sub list overlay/lower
ID 261 gen 192 top level 5 path .snapshots/ACTIVE
ID 266 gen 190 top level 5 path .snapshots/2016-03-19-root

# Set the default mounting subvolume to the backup
$ btrfs sub set 266 overlay/lower

# Remount the BTRFS partition and the OverlayFS
$ umount mnt
$ umount overlay/lower
$ mount /dev/mapper[crypt] overlay/lower
$ mount -t overlay overlay -o lowerdir=overlay/lower,upperdir=overlay/upper,workdir=overlay/work mnt

If we want to create a new ACTIVE subvolume from our backup moving forward we need to do some shuffling in the ‘btrfs-root’ directory

# List subvolumes available
$ btrfs sub list btrfs-root
ID 261 gen 192 top level 5 path .snapshots/ACTIVE
ID 266 gen 190 top level 5 path .snapshots/2016-03-19

# Shuffle around the ACTIVE subvolume
$ mv btrfs-root/.snapshots/ACTIVE btrfs-root/.snapshots/ACTIVE.old

# Create new subvolume from backup
$ btrfs sub snap btrfs-root/.snapshots/2016-03-19 btrfs-root/.snapshots/ACTIVE

# List subvolumes available
$ btrfs sub list btrfs-root
ID 261 gen 192 top level 5 path .snapshots/ACTIVE.old
ID 266 gen 190 top level 5 path .snapshots/2016-03-19
ID 267 gen 192 top level 5 path .snapshots/ACTIVE

# Set the default mounted partition to the new ACTIVE partition
$ btrfs sub set 267 btrfs-root

# Remount the ACTIVE partition and OverlayFS
$ umount mnt
$ umount overlay/lower
$ mount /dev/mapper/[crypt] overlay/lower
$ mount -t overlay overlay -o lowerdir=overlay/lower,upperdir=overlay/upper,workdir=overlay/work mnt

Boot a BTRFS root partition into an OverlayFS

To use this setup as a root partition, we need to choose a snapshot heirarchy and naming convention and stick to it.

/
|--.snapshots
  |--root
     |--ACTIVE
     |--2016-1-23
     |--2016-1-24<br>  |--lxc
     |--lxc1
     |--lxc1/rootfs
     |--lxc2
     |--lxc2/rootfs

First create a BTRFS partition, create the subvolume hierarchy, install the system, set the default mounting subvolume, then reboot

# Create a BTRFS partition
$ mkfs.btrfs -L ROOT /dev/sdXY<br>
# Mount the partition
$ mount /dev/sdaXY mnt<br>
# Create a .snapshots subvolume
$ btrfs sub create /.snapshots<br>
# Snapshot the root filesystem to start building a system
$ btrfs sub snap / .snapshots/BASE<br>
# Set the default mounting subvolume
$ btrfs sub list mnt
ID 155 gen 192 top level 5 path .snapshots
ID 156 gen 190 top level 155 path .snapshots/BASE<br>
$ btrfs sub set 156 mnt<br>
# Remount the BTRFS partition
$ umount mnt && mount /dev/sdXY mnt<br>
# Install a base Arch system in the partition
$ pacstrap -i base base-devel mnt<br>
# chroot into the partition and install grub
# Read the Arch wiki beginner's guide to set time, locale, and hostname
$ arch-chroot mnt /bin/bash
$ pacman -S grub os-prober
$ grub-install --target=i386-pc /dev/sdX
$ grub-mkconfig -o /boot/grub/grub.cfg<br>
# Exit the chroot
$ exit<br>
# Snapshot the BASE installation
$ btrfs sub snap mnt/.snapshots/BASE mnt/.snapshots/ACTIVE<br>
# Set the default mounting subvolume
$ btrfs sub list mnt
ID 155 gen 192 top level 5 path .snapshots
ID 156 gen 190 top level 155 path .snapshots/BASE
ID 157 gen 190 top level 156 path .snapshots/ACTIVE<br>
$ btrfs sub set 157 mnt<br>
# Set the Grub bootloader where to look for our grub.cfg file
# If you rename the logically named ACTIVE subvolume you will
    # have to perform this step again
$ arch-chroot mnt /bin/bash
$ grub-install --target=i386-pc /dev/sdX
$ grub-mkconfig -o /boot/grub/grub.cfg<br>
# Edit the grub config
$ vi /boot/grub/grub.cfg
# Remove the rootflags on the linux line
# The line will look something like this BEFORE
linux /.snapshots/root/ACTIVE/boot/vmlinuz-linux root=UUID=[uuid] rw quiet rootflags=subvol=/.snapshots/root/ACTIVE
# and AFTER
linux /.snapshots/root/ACTIVE/boot/vmlinuz-linux root=UUID=[uuid] rw quiet<br>
# Remove the subvol option from fstab
$ vi /etc/fstab
# The line will look something like the BEFORE
UUID=[uuid]/btrfs &#xA0; &#xA0; rw,relatime,space_cache,subvol=/.snapshots/root/ACTIVE0 0
# and AFTER
UUID=[uuid]/btrfs &#xA0; &#xA0; rw,relatime,space_cache0 0

At this point you have a working snapshotting scheme so you can snapshot ACTIVE then make updates at your leisure. But in order to implement the overlayfs we need some init scripts to mount root correctly. Let’s incorporate liveroot into our boot.

# You should still be in a chroot
# Install liveroot for mounting root in an overlayFS
# Choose the option that says "n|N : use standard oroot hook"
$ yaourt -S liveroot<br>
# Or do it manually
$ git clone <a href="https://aur.archlinux.org/liveroot.git">https://aur.archlinux.org/liveroot.git</a>
$ cd liveroot
$ makepkg -si<br>
# Now edit grub.cfg to include this kernel script
$ vi /boot/grub/grub.cfg<br>
# Add "oroot=compressed" to the line
linux /.snapshots/root/ACTIVE/boot/vmlinuz-linux root=UUID=[uuid] rw quiet
# So it looks like this
linux /.snapshots/root/ACTIVE/boot/vmlinuz-linux root=UUID=[uuid] rw quiet oroot=compressed<br>
# Now edit mkinitcpio.conf
$ vi /etc/mkinitcpio.conf<br>
# Add the correct modules
MODULES="zram ext2 btrfs overlay"<br>
# Include oroot as a kernel script
HOOKS="base udev oroot autodetect modconf block filesystems keyboard fsck"
# Run mkinitcpio -p linux to remake your initramfs and linux image
$ mkinitcpio -p linux<br>
# Exit the chroot, unmount, and reboot

Snapshotting with Root Mounted in an overlayFS

$ mkdir -p /tmp/mnt
$ mount /dev/sdXY /tmp/mnt -o subvol=/
$ btrfs sub snap /tmp/mnt/.snapshots/root/ACTIVE /tmp/mnt/.snapshots/root/$(date +%Y%m%d-%H%M)<br>
# Cleanup
$ umount /tmp/mnt

Persist Changes to Root Mounted in an OverlayFS

# Create a file /.rsync-exclude to skip syncing some folders
$ vi /.rsync-exclude<br>
# ----begin /.rsync-exclude
/.snapshots
/dev
/mnt
/proc
/run
/sys
tmp
/root
# ----end /.rsync-exclude<br>
# First create snapshot
$ mkdir -p /tmp/mnt
$ mount /dev/sdXY /tmp/mnt -o subvol=/
$ btrfs sub snap /tmp/mnt/.snapshots/root/ACTIVE /tmp/mnt/.snapshots/root/$(date +%Y%m%d-%H%M)<br>
# Persist changes
$ rsync -ravvx --delete --exclude-from=/.rsync-exclude / /tmp/mnt/.snapshots/root/ACTIVE | grep -v uptodate | grep -v hiding<br>
# Cleanup
$ umount /tmp/mnt

And finally, we have a rollback solution and user error resistence. By removing the ‘subvol’ lines from fstab and grub.cfg we can set the default mounting subvolume to rollback to a previous snapshot. If you have the /boot directory in your snapshots, you don’t have to worry about your vmlinuz-linux or initramfs-linux-fallback.img having incorrect modules. To rollback, use the steps provided earlier for a regular BTRFS partition.