Network and Filesystem Isolation with LXC and virtenv

For my memory comparison of light Linux desktops I needed a tool that would allow me to install on my computer about 20 window managers/desktop environments. After looking at several common virtualization packages, I ended up using Linux containers and virtenv for the job.

LXC and virtenv

Probably the best way to describe virtenv is as a graphic interface for Linux containers utilities developed and distributed by LXC project. Linux containers is the virtualization technology build into Linux kernel, available in any kernel after 2.6.32. LXC is still under heavy development, with lots of new features emerging in 3.x Linux kernel series.

The virtual machines (VM) are driven without any overhead by the kernel already running on the computer. You don’t need to run a different kernel in the virtual machine, run only the processes you need, without even going trough the regular SysV or Linux init. This all means that memory is used very conservatively.

For example, on a 1GB RAM computer you can run easily 10 SSH/DHCP servers, or 10 different xorg/X11 servers with LXDE window managers on top.

virtenv program is build using Qt4, and it is very simple to use. Download it as source code and compile it, or downlaod a .deb file for Ubuntu and install it as

$ sudo apt-get install xserver-xephyr bridge-utils lxc jwm
$ sudo dpkg -i virtrenv_0.8_1.deb

Once installed, you would start it from command line as virtenv, or from System Tools menu.

Filesystem isolation

virtenv mounts copy-on-write the current filesystem installed on the host computer. By default /home directory is not imported in the container. Any modification to the filesystem in the container will stay in the container, the host filesystem remains untouched.

This feature is useful to isolate potentially malicious processes from the host system, install (apt-get) and try new software packages, chroot system for running servers etc.

Network isolation

In my memory comparison exercise, virtenv provided me with filesystem and xorg/X11 server isolation. All this was based on LXC facilities in the existing Linux kernel. LXC has also support for networking stack isolation. I will describe in the rest of the article a setup with one virtual machine connected by an isolated network (br0) to the host computer.

First thing first, we need to create and configure br0 bridge device:

$ sudo brctl addbr br0
$ sudo ifconfig br0 10.0.0.1/24
$ ifconfig br0
br0       Link encap:Ethernet  HWaddr fa:cc:c0:01:44:a2  
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f8cc:c0ff:fe01:44a2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:238 (238.0 B)

Then, I start virtenv and create a new virtual machine vm1.

Creating a new virtual machine

I configure it for no graphic xserver support, bridge networking and I assign it an IP address of 10.0.0.10/24.

Configuring the virtual machine

Once configured, the virtual machine is started automatically by virtenv. The VM has its own xterm acting as a console. I can now ping from this virtual machine the host interface at 10.0.0.1.

Virtual machine console

This is all that is to it, nothing more! This kind of network isolation is very useful for software development, testing, trying out new software etc. You can even stream youtube videos in a such a virtual machine with xserver support enabled.

Conclusion

I find it funny how cloud providers sell virtual machines using VM RAM size for pricing. For them, more memory means more money. Such a virtual machine has its own kernel, a full Linux support system (cron, logger etc.) and all the necessary init routines. 512MB or 1024MB is something usual for a VM.

LXC is approaching the same problem from a different angle. There is only one kernel running the host and also the virtual machines. You can build really small virtual machines this way. Some people are also calling them application containers:

Virtual machine	RAM Memory
ISC DHCP server, OpenSSH server, rsyslogd	20MB
Apache2 server, OpenSSH server, rsyslogd	22MB
xserver, LXDE window manager	44MB
xserver, KDE Plasma desktop environment	209MB

ezchroot

ezchroot is a small script to chroot into OpenVZ containers. Once inside, you can update or modify the container software. The operation is similar to ezlxc.

#!/bin/bash

if [ $# -gt 0 ]; then
	echo
else
	echo "Usage: ezchroot directory"
	exit 1
fi

cp -L /etc/resolv.conf $1/etc/.
mount -t proc none $1/proc
mount --rbind /dev $1/dev
mount --rbind /sys $1/sys

echo "entering chroot directory"
env NAME=chroot chroot $1 /bin/bash
umount $1/proc
umount $1/dev
umount $1/sys
echo "chroot exited"

Equal Cost Multipath

Equal Cost Multipath (ECMP) is a network load-balancing method that enables the coexistence of multiple network paths form one source node to a destination node. The two or more paths between the nodes have the same routing cost, thus the traffic will be split evenly across, avoiding congestion and increasing the bandwidth.

ECMP is also a network redundancy method. In case one ECMP link fails, the traffic will move on the remaining links with minimal interruption in service.

For ECMP to work, a router will need special support in the forwarding plane and in the routing protocols deployed in the network. For experimentation I will use two Linux virtual machines on my host computer. The Linux kernel has an excellent ECMP implementation, as for the routing protocol I will use OSPF.

The virtual machines are set using Easy LXC. Each machine owns a slice of the Linux kernel running on the host computer, with full network and process separation. In each virtual machine I run one instance of RCP100. RCP100 is a router control plane for Linux platforms, supporting among other things OSPF and ECMP. RCP100 also features a CISCO-like command line interface (CLI) which simplifies router operation for people already skilled in configuring commercial routers.

The network diagram is as follows:

ECMP test network

The two virtual machines are plugged into two Linux bridges, br0 and br1. On router connections I use network addresses in the range 10.20.x.0/24 range. Router IDs are defined in 192.168.10.x range and are implemented as loopback interfaces on the router. The loopback interfaces are redistributed into OSPF and should be advertised by OSPF throughout the network. The relevant configuration for the two routers looks like this:

Router r0:

r0#show running-config | no-more 
hostname r0
[snip]
!
router ospf
  router-id 192.168.10.0
  network 10.20.0.0/16 area 0
  redistribute connected loopback
!
interface ethernet eth0
  ip address 10.20.0.2/24
  ip mtu 1500
  no shutdown
!
interface ethernet eth1
  ip address 10.20.1.2/24
  ip mtu 1500
  no shutdown
!
interface loopback 0
  ip address 192.168.10.0/32
  ip mtu 16436
!
[snip]
r0#

Router r1:

r1#show running-config | no-more 
hostname r1
[snip]
router ospf
  router-id 192.168.10.1
  network 10.20.0.0/16 area 0
  redistribute connected loopback
!
interface ethernet eth0
  ip address 10.20.0.3/24
  ip mtu 1500
  no shutdown
!
interface ethernet eth1
  ip address 10.20.1.3/24
  ip mtu 1500
  no shutdown
!
interface loopback 0
  ip address 192.168.10.1/32
  ip mtu 16436
!
r1#

We login into the routers by telnet 0 in the virtual machine control terminals. The routing tables will show the equal cost multipath routes installed for the neighbor’s router ID. As we configured them earlier as loopbacks, router IDs are real routes, we can ping them for example.

Router r0:

r0#show ip route
Codes: C - connected, S - static, R - RIP, B - blackhole, O - OSPF
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2

C    10.20.0.0/24 is directly connected, eth0
C    10.20.1.0/24 is directly connected, eth1
O E2 192.168.10.1/32[110/20] via 10.20.0.3, eth0
                             via 10.20.1.3, eth1
r0#

Router r1:

r1#show ip route
Codes: C - connected, S - static, R - RIP, B - blackhole, O - OSPF
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2

C    10.20.0.0/24 is directly connected, eth0
C    10.20.1.0/24 is directly connected, eth1
O E2 192.168.10.0/32[110/20] via 10.20.0.2, eth0
                             via 10.20.1.2, eth1
r1#

With the ECMP routes installed, we can try now to see how the traffic is distributed across the two links. We login into router 0 and from there we telnet several times into r1 at 192.168.10.1.

r0#telnet 192.168.10.1
Trying 192.168.10.1...
Connected to 192.168.10.1.
Escape character is '^]'.
User: rcp
Password: 
r1>exit
Connection closed by foreign host.
r0#

Each telnet connection goes trough one path or the other alternatively. Once the telnet session is established, the traffic stays on the specific path until the telnet connection is closed. This is accomplished in the Linux kernel by hashing source/destination address and port numbers, and associating specific hash values with specific connections.

The following picture shows two Wireshark windows side by side, while pinging r1 from r0. The ICMP requests are spread over the two links, r1 will respond on the link it received the request.

ECMP traffic in two WIreshark windows

Easy LXC: Running OpenVZ containers in LXC

ezlxc (Easy LXC) is a small Bash script for running virtual machines (VM) using Linux Containers (LXC). The VM containers are based on the templates provided by OpenVZ project.

In an LXC environment, a single Linux kernel is shared between the host and the virtual machines. Only the essential needed services are run in VMs. The VM is basically a chroot-based environment with the added network/process separation provided by LXC virtualization. Memory requirements for this type of setup is very low – my old dual-core AMD computer with 1GB of RAM memory runs easily 10 VMs.

ezlxc is based on ssh-template script developed and distributed with LXC. I have tested it on a Fedora 17 x86_64 computer and it will probably work without modifications on any recent Linux distribution. Copy the script below in a text editor and save it as ezlxc. Make the file executable (chmod +x ezlxc) and copy it in /usr/local/bin/directory. The copying is performed as root, in fact all the operations below can only be performed as root.

#!/bin/bash

echo "1" > /proc/sys/net/ipv4/ip_forward
brctl addbr br0
ifconfig br0 up
brctl addbr br1
ifconfig br1 up

generate_rclocal() {

cat <<EOF > etc/rc.local
echo "rc.local running..."
mount proc /proc -t proc
mount tmpfs /dev/shm -t tmpfs -o size=16m
mount devpts /dev/pts -t devpts
mount -t sysfs sys sys/

#****************************************
# initialize the rest of the system here
#****************************************

#/bin/echo "*   default gateway..."
#/sbin/route add default gw 10.0.0.1

echo "*   starting  crond..."
/usr/sbin/crond -n &

echo "*   starting  rsyslog..."
/sbin/rsyslogd -c 5

echo "*   starting sshd..."
/usr/sbin/sshd

echo "*   starting control session..."
/bin/bash
echo "control session exited, VM shutting down..."

#****************************************
# shutdown all processes here
#****************************************
pkill crond
pkill rsyslogd
pkill sshd
umount /sys
umount /dev/shm
umount /dev/pts
umount /proc

EOF

}

create_fs() {
	if [ ! -f etc/rc.local ]
	then
		echo "creating a default /etc/rc.local, please modify it!"
		generate_rclocal
		chmod +x etc/rc.local
	fi
	
	if [ ! -f etc/ssh/ssh_host_rsa_key ]
	then
		echo "configuring sshd"
		ssh-keygen -t rsa -f etc/ssh/ssh_host_rsa_key -N ""
		ssh-keygen -t dsa -f etc/ssh/ssh_host_dsa_key -N ""
	fi
	
	
	echo "populating dev directory"
	mknod -m 666 dev/null c 1 3
	mknod -m 666 dev/zero c 1 5
	mknod -m 666 dev/random c 1 8
	mknod -m 666 dev/urandom c 1 9
	mkdir -m 755 dev/pts
	mkdir -m 1777 dev/shm
	mknod -m 666 dev/tty c 5 0
	mknod -m 600 dev/console c 5 1
	mknod -m 666 dev/tty0 c 4 0
	mknod -m 666 dev/full c 1 7
	mknod -m 600 dev/initctl p
	mknod -m 666 dev/ptmx c 5 2
}

do_exec() {
	echo "chrooting to $1"
	chroot $1 /bin/bash -i /etc/rc.local
	sleep 2
	
	echo "VM exited"
}




if [ $# -gt 0 ]; then
	echo
else
	echo "Usage: ezlxc name"
	echo "where"
	echo "	name: virtual machine directory"
	exit 1
fi

case "$1" in
lxc-exec)
	do_exec $2
	;;
	
*)
	# directory setup
	DIRNAME=`dirname $1`
	VMNAME=`basename $1`
	echo "switching to $1 directory"
	cd $1
	create_fs
	cd -
	
	# create/start/destroy virtual machine
	echo "creating virtual machine"
	lxc-create -n $VMNAME
	echo "starting virtual machine"
	lxc-execute -n $VMNAME -f $1/lxc.conf $0 lxc-exec $1
	echo "destroying virtual machine"
	lxc-destroy -n $VMNAME
	mount -t proc proc /proc
	;;
esac

The steps involved in building and running a container are described below.

1. Download fedora-17-x86_64 template from OpenVZ OS template page.

2. Create a new directory for the container and unpack the template.

# cd ~
# mkdir vm1
# cd vm1
# tar -xzvj fedora-17-x86_64.tar.gz

3. In vm1 directory, create a lxc.conf file with the following content:

lxc.utsname = vm1
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = 4a:49:43:49:a0:11
lxc.network.ipv4 = 10.0.0.10/24

This is the configuration file for the container.

• vm1 is the name of the container;
• the container has a network interface with the specified MAC and IP addresses;
• the network interface is connected to a bridge interface br0 on the host.

4. Start the container:

# cd ~
# ezlxc vm1
switching to vm1 directory
creating a default /etc/rc.local, please modify it!
populating dev directory
creating virtual machine
‘vm1' created
starting virtual machine
chrooting to vm1
rc.local running...
*   starting  crond...
*   starting  rsyslog...
*   starting sshd...
*   starting control session...
[root@vm1 /]#

In this moment, the current terminal is logged into the new virtual machine. This is the controlling terminal. As long as this terminal is active, the virtual machine is running. To stop it, just type exit or close the terminal window.

By default, the virtual machine runs three processes: crnod (UNIX job scheduler), rsyslog (system logger) and sshd (OpenSSH server). The processes are started from /etc/rc.local file inside the container (~/vm1/etc/rc.local).

# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  14788   572 pts/0    S    23:23   0:00 /usr/lib64/lxc/lxc-init --
root         2  0.0  0.1 112092  1248 pts/0    S    23:23   0:00 /bin/bash /usr/local/bin/ezlxc
root         7  0.0  0.2 114288  1804 pts/0    S    23:23   0:00 /bin/bash -i /etc/rc.local
root        20  0.0  0.1 118392  1400 pts/0    S    23:23   0:00 /usr/sbin/crond -n
root        22  0.0  0.3 177796  3432 ?        Sl   23:23   0:00 /sbin/rsyslogd -c 5
root        26  0.0  0.1  77604  1244 ?        Ss   23:23   0:00 /usr/sbin/sshd
root        27  0.0  0.2 114292  1960 pts/0    S    23:23   0:00 /bin/bash
root        38  0.0  0.1 115728  1164 pts/0    R+   23:28   0:00 ps aux

You can open and modify /etc/rc.local and start more processes. Also, in this file you can set various other parameters such as the default gateway. A typical rc.local file generated by ezlxc looks like this:

echo “rc.local running..."
mount proc /proc -t proc
mount tmpfs /dev/shm -t tmpfs -o size=16m
mount devpts /dev/pts -t devpts

#****************************************
# initialize the rest of the system here
#****************************************

#/bin/echo “*   default gateway..."
#/sbin/route add default gw 10.0.0.1

echo “*   starting  crond..."
/usr/sbin/crond -n &

echo “*   starting  rsyslog..."
/sbin/rsyslogd -c 5

echo “*   starting sshd..."
/usr/sbin/sshd

echo “*   starting control session..."
/bin/bash
echo “control session exited, VM shutting down...”

#****************************************
# shutdown all processes here
#****************************************
pkill crond
pkill rsyslogd
pkill sshd
umount /dev/shm
umount /dev/pts
umount /proc

More servers and applications can be installed in VM (yum install …) and started in rc.local.

5. The IP address of the VM was set to 10.0.0.10. We need to set an IP address on bridge br0 on the host also. Open a new terminal and configure br0 interface:

# ifconfig br0 10.0.0.1

In this moment we can ping the virtual machine from the host

# ping 10.0.0.10
PING 10.0.0.10 (10.0.0.10) 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_req=1 ttl=64 time=0.196 ms
64 bytes from 10.0.0.10: icmp_req=2 ttl=64 time=0.113 ms
64 bytes from 10.0.0.10: icmp_req=3 ttl=64 time=0.115 ms
^C
--- 10.0.0.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.113/0.141/0.196/0.039 ms

6. Back in the controlling terminal, we set a root password. This allows us to access the VM using SSH:

[root@vm1 /]# passwd root
Changing password for user root.
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.
[root@vm1 /]#

From the previous host terminal we can SSH into the VM:

# ssh root@10.0.0.10
root@10.0.0.10‘s password: 
Last login: Wed Oct 24 23:44:57 2012 from 10.0.0.1
[root@vm1 ~]#

7. To stop the VM type exit in the controlling terminal:

[root@vm1 /]# exit
exit
control session exited, VM shutting down...
VM exited
destroying virtual machine