This file describes how to create a load-balanced virtual network of
XEN guests (virtual machines) on a XEN Debian server. It is assumed
that the XEN server and xen-tools are already installed.

Step 1: Creating the XEN virtual machine using CentOS as guest OS
-------

- Fix rinse installation of CentOS 5. Add the following lines to
/etc/rinse/centos-5.packages:

nss
nspr
python-iniparse

- Optionally, install the packages for CentOS 5 version from a new
location. Modify /etc/rinse/rinse.conf:

[centos-5]
mirror       = http://mirror.bytemark.co.uk/centos/5/os/i386/CentOS/
mirror.amd64 = http://mirror.bytemark.co.uk/centos/5/os/x86_64/CentOS/

- Create the XEN virtual machine with the following command. Modify
parameters to suit needs:

xen-create-image --hostname node01 --dist centos-5 --install-method \
rinse --size 4Gb --swap 1Gb --memory 2Gb --ip 192.168.1.101 \
--netmask 255.255.255.0 --gateway 192.168.1.1 --passwd

Step 2: Bring the new virtual machine online and open a console to it
-------

- Run the following command:

xm create /etc/xen/node01.cfg -c

Step 3: Set up the network settings on the node
-------

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce

- Setup the ipip tunnel to the load balancer:

ifconfig tunl0 <public IP of load-balancer> netmask 255.255.255.255

- Optionally, set up another public IP address on eth0:1 (to access
this node directly, which is good when developing or debugging):

On the node, type:
ifconfig eth0:1 <IP> broadcast <IP> netmask 255.255.255.255

On the gateway (XEN server / Dom0), type:
route add -host <IP> <vifX.Y>

The vifX.Y must be the one used by XEN for the virtual machine on
which the node runs.

Step 4: Install and start the HTTP server
-------

yum install httpd.x86_64
echo "node01" > /var/www/html/index.html
echo "ok" > /var/www/html/status.html
/etc/init.d/httpd start

The status.html file is used by the load balancer to determine if the
node is online or not so that it knows if it can use it.

Step 5: Configure the load balancer to use the new node
-------

On the load balancer node (lb), edit /etc/ha.d/conf/ldirectord.cf:
real=192.168.1.101:80 ipip 10

Test that the new node is load-balanced (you should see node01 after a
few refreshes, in case you have more than 1 node):
http://<public Load-balancer IP address>/

A few remarks:

• You need at least 2 public IP addresses on your server, but I would recommend 3 or more. One will be used by the server itself (to connect to Dom0), one will be used by the load balancer (your web requests will go to that IP and will be distributed among the available nodes) and at least another one should be assigned to one of the nodes to access it directly and not through the load balancer, which is very important for development and debugging of your load-balanced application.

• The load balancer in my setup is another (virtual) node of the system… a XEN guest. This is needed because the load balancer cannot be on the same machine as the router for your virtual network, which will be Dom0 (i.e.: the main server itself, on which XEN and everything else runs). Why not? Well, because if you use the same IP for both your router and your load balancer when a packet is sent from one of your end nodes to a client it will have the source IP set to the public IP of the load balancer. Now, the router will receive a packet from “outside” that needs to be routed with the same IP address as its own IP address and will say “no, no! this isn’t right” and will drop that packet as being “source martian”. Google a bit for “source martian problem” and you’ll get a clearer picture.

My first task was to create this system setup on my development server. Basically, the system has N nodes with a load-balancer in front of them. Since I only have one server, the obvious choice is to use XEN and create a virtual machine for every node of the system (including the load balancer). All nodes of the system (i.e.: XEN virtual machines) will sit on a private virtual network which is, again, created on just one real computer (my development server).

So far, so good! Everything seems quite simple from the description above and is definitely simple to setup (… after 3 days of intense googling!!!). The story goes like this…

I’m getting my server (yay!) with CentOS installed on it (I requested it). When it comes to Linux distributions I like Slackware, but using CentOS is so much easier when you don’t want to configure and install everything by hand and I didn’t want to spend too much time on installing and compiling software from source, like I usually do on Slackware.

So, anyway, my CentOS is up and running and I’m about to install XEN… easy!, just type “yum install xen” and you’re done. XEN requires a new kernel, so once you reboot the machine (with the new XEN kernel) you have a working XEN environment where you can create additional virtual machines. Just to note, at least on CentOS, you don’t have to configure anything (like grub for example); yum does everything for you… so once you install XEN with yum you just restart the server and you’re done… pretty cool!

Next step: build a virtual machine… how hard could this be? Well, it turns out it’s not that simple, at least not on CentOS. First of all, I think the documentation about this is a bit lacking, especially if you’re not familiar with how Linux works and a dozen or so command-line tools. I think I’m pretty good at Linux stuff, but nevertheless, I couldn’t create a XEN virtual machine on CentOS. To keep a long story short, I was able to boot a virtual machine but I wasn’t able to install the “guest” operating system on it because I didn’t have network access from inside the virtual machine and frankly I don’t know what didn’t work and I also didn’t want to spend too much time on this.

So I went and asked Google what’s the easiest way to create a XEN virtual machine… and the answer was “use xen-tools”. I said “aha! let’s try that…” and immediately went to my CentOS to type “yum install xen-tools”. To my surprise, yum said there is no package with that name, so again I turned to Google to find out more about these xen-tools. I read now from their page: “xen-tools is a collection of simple perl scripts which allow you to easily create new guest Xen domains upon your Debian GNU/Linux host”. Damn, I’m on the wrong Linux distribution… and from looking at the examples of how to use xen-tools it seems quite simple and easy, so it’s worth trying to work with Debian on my development server.

Luckily my hosting provider has a very easy and convenient way to (re)install a new OS, so I went to my administration console and looked for Debian through the available OSes and I was very happy to find a “Debian + XEN 3.2″ OS template there… I selected that, clicked “Restore” and in about 30 minutes my new Debian + XEN 3.2 development server was back online!

What comes next… you’ll have to read it in part 2 of this post. Stay tuned!