Ephemeral LXC Swarm For Local Testing

So, you need to create a swarm of Linux Containers to test multiple clients but don’t want the containers to stick around when you’re finished. We can use ephemeral LXC’s that utilize OverlayFS on harddisk or temporary storage in RAM. We do need a little Bash scripting knowledge. But, when we include the use of Tmux, we gain a large increase in power, flexibility, and productivity. What we need:

  • Base LXC with software installed
  • Base-project LXC with project specific configuration files
  • Bash script to create the environment

Let’s create the Base LXC and install any software our swarm may need

$ lxc-create -t /usr/share/lxc/templates/lxc-archlinux -n base

# Edit base/config to add network connectivity
lxc.net.0.type = none

$ lxc-start -n base; lxc-attach -n base
root@base $ pacman -S <list of software>
root@base $ exit
$ lxc-stop -n base

Now, let’s create our base project LXC with various mounts and configuration files.

$ lxc-create -s -B overlayfs -n base -N base-project

# edit base-project/config to add mounts
lxc.mount.entry = /home/http home/http none bind,ro,create=dir 0 0

# Copy any project files into the container
$ rsync -a <my project files> base-project/delta0/<project files path>

Now we can write a bash script that will spawn our swarm.

# start-swarm.sh

#!/bin/bash

lxc-copy -s -e -n base-project -N one
lxc-copy -s -e -n base-project -N two
lxc-copy -s -e -n base-project -N three
lxc-copy -s -e -n base-project -N four

# Make script executable and run it
$ chmod +x start-swarm.sh
$ ./start-swarm.sh

The ephemeral LXC’s will have already been started. Once you stop each container, they will be automatically deleted. From here let’s create a Tmux session to attach to each client in the swarm.

# start-swarm.sh

#/bin/bash

# Tmux session name
sname="my-session"

# Sleep required after attaching to LXC
attach_delay=0.5

# Use variable to hold lengthy command for sending keystrokes to tmux pane
send_keys="tmux send-keys -t $sname"

# Create the LXC swarm
lxc-copy -s -e -n base-project -N one
lxc-copy -s -e -n base-project -N two
lxc-copy -s -e -n base-project -N three
lxc-copy -s -e -n base-project -N four

# Create Tmux Session
tmux start-server
TMUX= tmux new-session -d -s $sname -n $sname

# Create 3 more splits in current window
tmux split-window -t $sname:0 -v
tmux split-window -t $sname:0 -v
tmux split-window -t $sname:0 -v

# Set Tmux window 0 pane layout
tmux select-layout -t $sname:0 tiled

# First machine
$send_keys:0.0 "lxc-attach -n one" C-m
sleep $attach_delay
$send_keys:0.0 "cd /project/directory" C-m

# Second machine
$send_keys:0.1 "lxc-attach -n two" C-m
sleep $attach_delay
$send_keys:0.1 "cd /project/directory" C-m

# Third machine
$send_keys:0.2 "lxc-attach -n three" C-m
sleep $attach_delay
$send_keys:0.2 "cd /project/directory" C-m

# Fourth machine
$send_keys:0.3 "lxc-attach -n four" C-m
sleep $attach_delay
$send_keys:0.3 "cd /project/directory" C-m

tmux select-window -t $sname
tmux select-pane -t 0

if [ -z "$TMUX" ]; then
    tmux -u attach-session -t $sname
else
    tmux -u switch-client -t $sname
fi

Currently this setup assumes the swarm will use the same IP address as the host machine. Since setting up dynamic IP addresses in ephemeral containers is daunting we can just modify our script to create plain containers in a temporary directory then modify each container’s config file. First we will edit the base project configuration to use a veth network type.

# base-project/config
...
# Combine containers into group for easy shutdown
lxc.group = swarm

# Necessary to give each container a separate IP address
lxc.net.0.type = veth
lxc.net.0.name = veth0
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.ipv4.address = 10.0.0.101
lxc.net.0.ipv4.gateway = auto

Now, edit the start-swarm.sh script to create our session.

# start-swarm.sh

#!/bin/bash

# LXC directory
lxc_dir=/tmp/lxc

# Tmux session name
sname="my-session"

# Sleep required after attaching to LXC
attach_delay=0.5

# Use variable to hold lengthy command for sending keystrokes to tmux pane
send_keys="tmux send-keys -t $sname"

# Create the LXC directory
mkdir -p $lxc_dir

# Create the LXC swarm, notice they are not ephemeral
lxc-copy -P $lxc_dir -s -n base-project -N one
lxc-copy -P $lxc_dir -s -n base-project -N two
lxc-copy -P $lxc_dir -s -n base-project -N three
lxc-copy -P $lxc_dir -s -n base-project -N four

# Create a bridge device for the swarm to connect to
brctl addbr br0 > /dev/null 2>&1
ifconfig br0 10.0.0.100
ifconfig br0 up

# Edit the base-project/config that was created
# in each ephemeral LXC to give each a unique
# IP address
sed -i 's/101/101/' $lxc_dir/one/config
sed -i 's/101/102/' $lxc_dir/two/config
sed -i 's/101/103/' $lxc_dir/three/config
sed -i 's/101/104/' $lxc_dir/four/config

# Start the containers
lxc-start -P $lxc_dir -n one
lxc-start -P $lxc_dir -n two
lxc-start -P $lxc_dir -n three
lxc-start -P $lxc_dir -n four

# Create Tmux Session
tmux start-server
TMUX= tmux new-session -d -s $sname -n $sname

# Create 3 more splits in current window
tmux split-window -t $sname:0 -v
tmux split-window -t $sname:0 -v
tmux split-window -t $sname:0 -v

# Set Tmux window 0 pane layout
tmux select-layout -t $sname:0 tiled

# First container
$send_keys:0.0 "lxc-attach -P $lxc_dir -n one" C-m
sleep $attach_delay
$send_keys:0.0 "cd /project/directory" C-m

# Second container
$send_keys:0.1 "lxc-attach -P $lxc_dir -n two" C-m
sleep $attach_delay
$send_keys:0.1 "cd /project/directory" C-m

# Third container
$send_keys:0.2 "lxc-attach -P $lxc_dir -n three" C-m
sleep $attach_delay
$send_keys:0.2 "cd /project/directory" C-m

# Fourth container
$send_keys:0.3 "lxc-attach -P $lxc_dir -n four" C-m
sleep $attach_delay
$send_keys:0.3 "cd /project/directory" C-m

tmux select-window -t $sname
tmux select-pane -t 0

if [ -z "$TMUX" ]; then
    tmux -u attach-session -t $sname
else
    tmux -u switch-client -t $sname
fi

Now to cleanup our session when we’re finished testing:

$ tmux kill-session -t my-session
$ lxc-autostart -P /tmp/lxc -kAg swarm
$ rm -rf /tmp/lxc

If we want to streamline the shutdown process, we can add this after the variable declarations in our start-swarm.sh script

# start-swarm.sh

...

if [[ "$1" == "down" ]]; then
    tmux kill-session -t $sname
    lxc-autostart -P $lxc_dir -kAg swarm
    rm -rf $lxc_dir
    exit
fi

...