Linux Apache Mysql Php in an Lxc Container

After fumbling around with Docker, and getting nowhere trying to use Apache in a container, I’ve decided to use a Linux Container (LXC) to setup a portable development environment. Why did I decide upon LXC over Docker? I couldn’t get the Apache daemon to continue executing within Docker. I’m also more used to building environments on a command line. At first, I decided to try Docker due to its built in portability; with LXC you have to backup and migrate your containers manually. But with its single config file per container and excellent compressibility, I was able to avoid many headaches.

Another advantage of LXC over Docker is the learning curve. You don’t need to learn esoteric option flags when you are starting, stopping, and attaching containers. All of the LXC commands are symmantically named and easily listable with shell tab completion. Also, no need to learn how to write Dockerfiles (migraine city). And once you have a running container, you simply attach to it and use it just like you would a virtual machine. I will be using Arch Linux but these steps translate to all Linux distributions. Check out the Pacman Rosetta to see the equivalent package manager commands on other distributions.

A Brief Overview

• Install LXC
• Create a container
• Install packages
• Configure Network and Mounts
• Cleanup and backup
• Hooks
• Gotchas

Install LXC

Here’s the easiest of all the steps. We will install LXC and arch-install-scripts. We need the arch-install-scripts to chroot into our LXC root filesytem path (this is optional).

$ pacman -S lxc arch-install-scripts

Create a Container

As of lxc 3.0, templates have been removed in favor of a tool called distrobuilder. You can create a container manually with this tool, use templates from the lxc-templates repo, or use the ‘download’ option.

$ lxc-create -n [container_name] -t download

Now we will create our container. You can think of it like a virtual machine. You can install any package or service and run these when you boot it up. You will be starting the container using the [container_name] but you can change it later with some copying and minor config file edits.

$ lxc-create -n [container_name] -t /usr/share/lxc/templates/lxc-archlinux

# Use the below commands if you are using BTRFS
$ lxc-create -n vanilla -B btrfs -t /usr/share/lxc/templates/lxc-archlinux
$ lxc-copy -s -B btrfs -n vanilla -N [container_name]

# Use the below commands if you are using ZFS
$ lxc-create -n vanilla -B zfs -t /usr/share/lxc/templates/lxc-archlinux
$ lxc-copy -s -B zfs -n vanilla -N [container_name]

# Note: the -t option accepts a template file from /usr/share/lxc/templates/

Recommended Container Scheme

Though you can create containers with a BTRFS or ZFS backing store I highly recommend against it. This will make creating backup snapshots a nightmare when the number of containers grows. It will also make updating these containers tedious at best. I do however recommend creating all of your containers on a BTRFS or ZFS partition or subvolume. This will allow you to create backup snapshots of every container at once. This could make restoring backups of individual containers more complex, but there is another remedy.

Create a master container that you install every piece of software you may need for every container. Keep this container bare bones. This means don’t edit any configuration files within this container. Keep it vanilla. Then use overlayfs as a backing store for each subsequent container where the new container’s rootfs uses the master container as the lower filesystem layer. This allows you to update the master container’s software at your leisure. When you want the updated version on the copied container, simply restart it. Here is an example of the steps you need to take to accomplish this scheme.

# Move to the default container location
$ cd /var/lib/lxc
# Create the master container
$ lxc-create -t /usr/share/templates/lxc-archlinux -n base
# Allow the master container to use the host machine internet
# Edit base/config
lxc.net.0.type = none

# Create a "snapshot" of the master container
$ lxc-stop -n base
$ lxc-copy -s -B overlayfs -n base -N apache

# Test that our new container works
$ lxc-start -n apache

# Attach to the container and start the Apache web server
$ lxc-attach -n apache
$ systemctl start httpd
Failed to start apache.service: Unit apache.service not found.

# Install Apache in the master container, restart the apache container and test again
$ lxc-start -n base
$ lxc-attach -n base
$ pacman -S apache
$ exit
$ lxc-stop -r -n apache
$ lxc-attach -n apache
$ systemctl start httpd

Install packages

Now is the fun part. To start things off, let’s change directory to our container.

$ cd /var/lib/lxc/ && ls

There will be a folder in this directory with a name of [container_name]. At this point, if you are using BTRFS or ZFS with Copy-on-write enabled in this directory, the LXC will be created in a subvolume. The lxc-copy and lxc-snapshot commands use their native BTRFS or ZFS counterparts to save you drive space and time. Only changes to the subsequent snapshots will be written leaving your original snapshot intact. From here, there are a few ways to install packages into the container. You can use arch-chroot that was installed with arch-install-scripts.

$ arch-chroot /var/lib/lxc/[container_name] /bin/bash

Or, you can startup and attach to the container with LXC.

$ lxc-start -n [container_name]
$ lxc-attach -n [container_name]

Once you’re inside your container, you can install any packages you’d like but since this is a tutorial on getting a LAMP stack working…

$ pacman -S apache php php-apache php-gd php-mcrypt \
php-pear php-mongodb phpmyadmin mariadb mongodb

Configuring the LAMP stack is outside the scope of this tutorial. You can set everything up exactly the way you would on a bare metal server. If you need help, consult this tutorial.

Configure Network and Mounts

Easy

To share the physical network device on your host machine with the container, you merely need to set a single config option.

# /var/lib/lxc/[container_name]/config
lxc.net.0.type=none

This is the most transparent way to connect the container to the outside world. All incoming requests to your host machine will also be received by the container. So incoming requests on port 80 are sent to the container since the Apache instance inside the container is the one that opened it. You will be able to set the vhost names in your hosts file to correspond to your local device

# /etc/hosts on host machine
127.0.0.1 vhost-in-container.local

Hard

For running multiple containers we need to configure a bridge connection on our host with which our containers will connect. We will use netctl to create our bridge device. In this config file, (eth0) is the device on your host machine with which you connect to the internet. You can use a wireless device here as well.

# /etc/netctl/lxcbridge

Description="LXC Bridge"
Interface=br0
Connection=bridge
BindsToInterfaces=(eth0)
IP=static
Address=10.0.0.1/24
SkipForwardingDelay=yes

Now let’s start the br0 device and forward ip packets to our container. We will set up the container to talk to this device later.

$ netctl start lxcbridge
$ sysctl net.ipv4.ip_forward=1
$ iptables -t nat -A POSTROUTING -s 10.0.0.100/32 -j MASQUERADE

# Note: You can use a larger subnet in the iptables command.
# Note: Since using multiple containers is common we will use scripts to set this up automatically so when one container is stopped we only remove the iptables rule associated with that container.

So far, we will only be able to talk to our container with its IP address from our host. If we want other machines to talk to our container we need to forward our ports from our host to our container.

$ iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 \
-j DNAT --to-destination 10.0.0.100:80

You won’t be able to ping the container just yet. If you have the container running, please stop it.

$ lxc-stop -n [container_name]

Now, let’s edit the container config file. Here, eth0 is a device inside the container. It isn’t a physical device but is created when you start up the container.

# /var/lib/lxc/[container_name]/config

lxc.net.0.type=veth
lxc.net.0.link=br0
lxc.net.0.ipv4=10.0.0.100
lxc.net.0.ipv4.gateway=10.0.0.1
lxc.net.0.flags=up
lxc.net.0.name=eth0
lxc.net.0.mtu=1500

You can now ping the container at 10.0.0.100. If you are or are not using an iptables rule to forward your ports you can set your hosts file entry like this following.

# /etc/hosts on host machine
10.0.0.100 vhost-in-container.local

! Important !

There is one issue with this “hard” setup in that the container will not be able to reach the outside world using curl, npm, composer, or any other service that uses any port you’ve set an iptables rule to forward from your host machine. This is because the container will make a request to the bridge on port 80 which is then forwarded to your host machine. The host machine finds the rule you set in iptables and forwards it right back to the container. So if you are merely doing local development without the need of the outside world to contact your container, you can remove those rules from iptables that forward ports from the host machine. If you aren’t using multiple containers, then just use the lxc.network.type=none config option as a remedy. If you need more help, please consult this guide.

Mounts

So we can migrate this container easily, we will mount host directories with the config file.

# /var/lib/lxc/[container_name]/config

...
#Mounts
lxc.mount.entry = /[path]/[to]/[host]/[public_html] home/http none bind.rw 0.0
lxc.mount.entry = /[path]/[to]/[host]/[public_html]/db/mysql var/lib/mysql none bind.rw 0.0
lxc.mount.entry = /[path]/[to]/[host]/[public_html]/db/mongodb var/lib/mongodb none bind.rw 0.0

If you mount database directories like I do, you will likely have some serious permission problems. You can use these commands to straighten things out. You will need to be inside the container so that our uids and gids are mapped correctly.

#Enter our container
$ lxc-start -n [container_name]
$ lxc-attach -n [container_name]

#Fix directory and file ownership
$ chown -R you:http /[path]/[to]/[public_html]

#fix directory and file permissions
$ find /home/http -type d -exec chmod 775 {} \;
$ find /home/http -type f -exec chmod 664 {} \;

#fix database ownership
$ chown -R mysql:mysql /home/http/db/mysql
$ chown -R mongodb:mongodb /home/http/db/mongodb

#fix database permissions
$ find /home/http/db/mysql -type d -exec chmod 770 {} \;
$ find /home/http/db/mysql -type f -exec chmod 660 {} \;

$ find /home/http/db/mongodb -type d -exec chmod 770 {} \;
$ find /home/http/db/mongodb -type f -exec chmod 660 {} \;

If you want to keep your uids and gids in sync between your host machine and the container, you can user usermod:

# Find uid and gid of mysql in host
$ lxc-attach -n [container_name]
$ cd /var/lib/mysql<br>$ ls -la .
drwsrwsr-x 10 600 600 &#xA0;16K Feb 22 04:55 .

# Now change uid and gid of mysql to 600
$ usermod -u 600 mysql<br>$ groupmod -g 600 mysql

# Do the same for mongodb
$ cd /var/lib/mongodb
$ ls -la<br>drwx------ 7 601 daemon 4.0K Feb 21 21:47 .

# Change uid of mongodb to 601
$ usermod -u 601 mongodb

Cleanup and backup

We should be rock ’n rollin’ by now! But let’s do some cleanup and make it easy to start and stop our container. First the cleanup.

# Enter the container
$ lxc-start -n [container_name]
$ lxc-attach -n [container_name]

# Enable LAMP services
$ systemctl enable httpd mysqld mongodb

# Remove cached packages<br># Don't use this if you are using the recommended setup<br>#   and inside a container that uses overlayfs
$ pacman -Scc

# copy our bash file
$ exit
# Should now be in host console
$ cp /[path]/[to]/[host]/.bashrc /var/lib/lxc/[container_name]/rootfs/root/

Let’s create a bash script to start and stop our container. Remember if we used lxc.network.type=none then we don’t need any of the brctl, netctl, sysctl, or iptables commands.

# /home/[user]/lamp.sh

#!/bin/bash

if lxc-info -n [container_name] | grep -q STOPPED; then
 echo "Starting Server"
 netctl start lxcbridge
 systemctl restart NetworkManager
 sysctl net.ipv4.ip_forward=1
 iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.100:80
 iptables -t nat -A POSTROUTING -s 10.0.0.100/32 -j MASQUERADE
 lxc-start -n [container_name]
else
 echo "Stopping Server"
 lxc-stop -n [container_name]
 # Stop the lxcbridge if we only have one container
 #netctl stop lxcbridge
 #systemctl restart NetworkManager
 sysctl net.ipv4.ip_forward=0
 iptables -t nat -D PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.100:80
 iptables -t nat -D POSTROUTING -s 10.0.0.100/32 -j MASQUERADE
fi

We can now backup our container as such.

$ tar -caf lxc-lamp-container.xz /var/lib/lxc/[container_name]/

Restore

# Restore a backup
$ cd /var/lib/lxc
$ lxc-create -n [lxc to restore] -t /usr/share/lxc/templates/lxc-archlinux<br>or<br>$ lxc-copy -s -n [lxc to clone from] -N [lxc to restore]<br>then
$ tar -xvsf /[path]/[to]/[backup]/my_lxc_backup.tar

Hooks

The above startup/shutdown script can be created with hooks so your LXC is portable to other systems.

# /var/lib/lxc/[container_name]/config
...
lxc.hook.pre-start=/var/lib/lxc/[container_name]/pre-start
lxc.hook.post-stop=/var/lib/lxc/[container_name]/post-stop

# /var/lib/lxc/[container_name]/pre-start

#!/bin/bash

if ip addr | grep -q br0; then
    printf "Bridge is up"
else
    brctl addbr br0
    ifconfig br0 up
    ifconfig br0 10.0.0.1
fi

sysctl net.ipv4.ip_forward=1

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.100:80
iptables -t nat -A POSTROUTING -s 10.0.0.100/32 -j MASQUERADE

# /var/lib/lxc/[container_name]/post-stop

#!/bin/bash

iptables -t nat -D PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.100:80
iptables -t nat -D POSTROUTING -s 10.0.0.100/32 -j MASQUERADE

Now make those hooks executable.

$ cd /var/lib/lxc/[container_name]
$ chmod +x pre-start post-stop

Gotchas!

If your internet cuts out when using netctl, you will have to restart your host machine DHCP.

#If you use dhcpcd
$ systemctl restart dhcpcd

#If you use NetworkManager
$ systemctl restart NetworkManager

If this still doesn’t fix the issue, try switching to bridge-utils. Here’s what your startup script will look like.

# /home/[user]/lamp.sh

#!/bin/bash

if lxc-info -n [container_name] | grep -q STOPPED; then
 echo "Starting Server"
 #netctl start lxcbridge
 #systemctl restart NetworkManager

 # Yeah Bro!
 brctl addbr br0
 ifconfig br0 up
 ifconfig br0 10.0.0.1

 sysctl net.ipv4.ip_forward=1
 iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.100:80
 iptables -t nat -A POSTROUTING -s 10.0.0.100/32 -j MASQUERADE
 lxc-start -n [container_name]
else
 echo "Stopping Server"
 lxc-stop -n [container_name]
 # Don't disconnect the br0 as other containers might still use it
 #netctl stop lxcbridge
 #systemctl restart NetworkManager
 sysctl net.ipv4.ip_forward=0
 iptables -t nat -D PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.100:80
 iptables -t nat -D POSTROUTING -s 10.0.0.100/32 -j MASQUERADE
fi

You can use name based virtual hosts in Apache but if you use port based virtual hosts you will have to open every port manually.

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.0.0.100:8080
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 8081 -j DNAT --to-destination 10.0.0.100:8081
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 8082 -j DNAT --to-destination 10.0.0.100:8082
#etc....

If you restore a container from a backup…

$ cd /var/lib/lxc/
$ tar -xvsf /[path]/[to]/lxc-lamp-container.xz

…and some of your services don’t work, you will have to reinstall the packages and restart/reenable the services. Don’t worry. Your config files will remain intact.

#Enter container
$ lxc-start -n [container_name]
$ lxc-attach -n [container_name]

$ pacman -S apache mysql mongodb

$ systemctl restart httpd mysqld mongodb
$ systemctl enable httpd mysqld mongodb

In version 2.0.0 ’lxc-clone’ has been removed and replaced with ’lxc-copy’. To rename a container you can:

$ lxc-copy -n [oldname] -R [newname]<br>

If your container creation gets stuck creating GPG keys, open a new terminal on the host machine and…

$ ls -R /

If you are using OpenVPN in a container that uses a bridged connection with a service available to your LAN, you might need to add a route to your LAN.

$ lxc-attach -n [container_name]

$ ip route
# You will see the routes available
# Your LAN is likely not listed

# Add a route to your LAN on the correct interface
# You can add this line to the autodev hook to persist it
$ ip route add 192.168.1.0/24 via 10.0.0.1 dev eth0

# Now check to see if your gateway is reachable
$ ping 192.168.1.1

Learn More…

https://linuxcontainers.org

If you’ve found this information useful, please consider purchasing time on a VPS through Linode using this referral link:

https://www.linode.com/?r=bd946ee78d4313eea22763d1d2f7d447fd795ef7