function test_url(url) {
  // comments:  (protocol )(user&pass   ) (subdomains&domain) (TLD            )(port   ) (path & query string   )
  var regex = /^https?:\/\/(\w+(:\w+)?\@)?([\w\-]+|\.[\w\-]+)+(\.[a-zA-Z]{2,6})(:[0-9]+)?(\/[\w#!:.?+=&%@!\-\/]*)?$/i;
  return url.match(regex);
}

This file describes how to create a load-balanced virtual network of
XEN guests (virtual machines) on a XEN Debian server. It is assumed
that the XEN server and xen-tools are already installed.

Step 1: Creating the XEN virtual machine using CentOS as guest OS
-------

- Fix rinse installation of CentOS 5. Add the following lines to
/etc/rinse/centos-5.packages:

nss
nspr
python-iniparse

- Optionally, install the packages for CentOS 5 version from a new
location. Modify /etc/rinse/rinse.conf:

[centos-5]
mirror       = http://mirror.bytemark.co.uk/centos/5/os/i386/CentOS/
mirror.amd64 = http://mirror.bytemark.co.uk/centos/5/os/x86_64/CentOS/

- Create the XEN virtual machine with the following command. Modify
parameters to suit needs:

xen-create-image --hostname node01 --dist centos-5 --install-method \
rinse --size 4Gb --swap 1Gb --memory 2Gb --ip 192.168.1.101 \
--netmask 255.255.255.0 --gateway 192.168.1.1 --passwd

Step 2: Bring the new virtual machine online and open a console to it
-------

- Run the following command:

xm create /etc/xen/node01.cfg -c

Step 3: Set up the network settings on the node
-------

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce

- Setup the ipip tunnel to the load balancer:

ifconfig tunl0 <public IP of load-balancer> netmask 255.255.255.255

- Optionally, set up another public IP address on eth0:1 (to access
this node directly, which is good when developing or debugging):

On the node, type:
ifconfig eth0:1 <IP> broadcast <IP> netmask 255.255.255.255

On the gateway (XEN server / Dom0), type:
route add -host <IP> <vifX.Y>

The vifX.Y must be the one used by XEN for the virtual machine on
which the node runs.

Step 4: Install and start the HTTP server
-------

yum install httpd.x86_64
echo "node01" > /var/www/html/index.html
echo "ok" > /var/www/html/status.html
/etc/init.d/httpd start

The status.html file is used by the load balancer to determine if the
node is online or not so that it knows if it can use it.

Step 5: Configure the load balancer to use the new node
-------

On the load balancer node (lb), edit /etc/ha.d/conf/ldirectord.cf:
real=192.168.1.101:80 ipip 10

Test that the new node is load-balanced (you should see node01 after a
few refreshes, in case you have more than 1 node):
http://<public Load-balancer IP address>/

A few remarks:

• You need at least 2 public IP addresses on your server, but I would recommend 3 or more. One will be used by the server itself (to connect to Dom0), one will be used by the load balancer (your web requests will go to that IP and will be distributed among the available nodes) and at least another one should be assigned to one of the nodes to access it directly and not through the load balancer, which is very important for development and debugging of your load-balanced application.

• The load balancer in my setup is another (virtual) node of the system… a XEN guest. This is needed because the load balancer cannot be on the same machine as the router for your virtual network, which will be Dom0 (i.e.: the main server itself, on which XEN and everything else runs). Why not? Well, because if you use the same IP for both your router and your load balancer when a packet is sent from one of your end nodes to a client it will have the source IP set to the public IP of the load balancer. Now, the router will receive a packet from “outside” that needs to be routed with the same IP address as its own IP address and will say “no, no! this isn’t right” and will drop that packet as being “source martian”. Google a bit for “source martian problem” and you’ll get a clearer picture.

My first task was to create this system setup on my development server. Basically, the system has N nodes with a load-balancer in front of them. Since I only have one server, the obvious choice is to use XEN and create a virtual machine for every node of the system (including the load balancer). All nodes of the system (i.e.: XEN virtual machines) will sit on a private virtual network which is, again, created on just one real computer (my development server).

So far, so good! Everything seems quite simple from the description above and is definitely simple to setup (… after 3 days of intense googling!!!). The story goes like this…

I’m getting my server (yay!) with CentOS installed on it (I requested it). When it comes to Linux distributions I like Slackware, but using CentOS is so much easier when you don’t want to configure and install everything by hand and I didn’t want to spend too much time on installing and compiling software from source, like I usually do on Slackware.

So, anyway, my CentOS is up and running and I’m about to install XEN… easy!, just type “yum install xen” and you’re done. XEN requires a new kernel, so once you reboot the machine (with the new XEN kernel) you have a working XEN environment where you can create additional virtual machines. Just to note, at least on CentOS, you don’t have to configure anything (like grub for example); yum does everything for you… so once you install XEN with yum you just restart the server and you’re done… pretty cool!

Next step: build a virtual machine… how hard could this be? Well, it turns out it’s not that simple, at least not on CentOS. First of all, I think the documentation about this is a bit lacking, especially if you’re not familiar with how Linux works and a dozen or so command-line tools. I think I’m pretty good at Linux stuff, but nevertheless, I couldn’t create a XEN virtual machine on CentOS. To keep a long story short, I was able to boot a virtual machine but I wasn’t able to install the “guest” operating system on it because I didn’t have network access from inside the virtual machine and frankly I don’t know what didn’t work and I also didn’t want to spend too much time on this.

So I went and asked Google what’s the easiest way to create a XEN virtual machine… and the answer was “use xen-tools”. I said “aha! let’s try that…” and immediately went to my CentOS to type “yum install xen-tools”. To my surprise, yum said there is no package with that name, so again I turned to Google to find out more about these xen-tools. I read now from their page: “xen-tools is a collection of simple perl scripts which allow you to easily create new guest Xen domains upon your Debian GNU/Linux host”. Damn, I’m on the wrong Linux distribution… and from looking at the examples of how to use xen-tools it seems quite simple and easy, so it’s worth trying to work with Debian on my development server.

Luckily my hosting provider has a very easy and convenient way to (re)install a new OS, so I went to my administration console and looked for Debian through the available OSes and I was very happy to find a “Debian + XEN 3.2″ OS template there… I selected that, clicked “Restore” and in about 30 minutes my new Debian + XEN 3.2 development server was back online!

What comes next… you’ll have to read it in part 2 of this post. Stay tuned!

The goal is to have a system that’s scalable! Ideally, doubling the number of components (servers) in the system should double its performance as well. Redundancy on the other hand is not that important, at least not for this particular application. As we’ll see later on, the engine will be able to function correctly even if some of its components (temporarily) crash.

I’ve done quite a bit of research lately on building scalable and redundant systems using MySQL databases. Almost all articles on this topic mention something about replication. At some point I was tempted to use circular replication, to get (unnecessary) data redundancy, however this was not going to help me get the much more important scaling, which is very important, especially for ad servers.

Let’s start by looking at what kind of data is used by a typical ad server. I could identify 3 major categories:

1. Configuration Data
   This contains information about current campaigns, targeting parameters, CPM, target impressions, etc. The engine will mostly use this data in a “read-only” mode. In my particular case, this data comes from a completely separate admin application. The frequency at which this data changes is relatively low, at least when compared to the other 2 categories below.
2. Runtime Data
   This contains information generated during the normal operation of the ad server engine. You will find here things like counters, session data, etc. The engine will both read & write this data in almost equal proportions.
3. Reporting Data
   This contains everything that’s being logged about served ads (clicks, interactions, timers, etc). This information is mainly written to the database and otherwise not required by the ad server engine. In my particular case, this information will be picked up by the completely separate admin application and used to generate reports and adjust the configuration data.

The configuration data will be quite small in size (at least when compared with the reporting data) and will most likely be cached for super-fast access. The runtime data will also be kept in memory (as much as possible) but will also be saved in the database for persistence. The reporting data will be by far the largest in size and we’ll need to save this to the database as soon as possible (we’ll use an intermediary cache to reduce the number of WRITE operations to the database).

If I take into account one of the goals for this ad server engine (to be able to serve 1 billion ads per month) and assume that for each ad served an additional 10-15 database WRITE operations are required (I’ve exaggerated a bit here), it soon becomes obvious that scalability actually means in this case being able to cope with all those database WRITE operations.

I’ve mentioned above that almost all articles I’ve read about scaling MySQL talk about replication. Well, this is only useful when scaling READ operations. Scaling WRITE operations will only work by data partitioning (or sharding). Quick example: if you have an application with 100 million users, you most likely don’t want to keep all of those users in the same DB table because it will become unmanageable. Instead, you’ll split those users into say 20 tables each storing 5 million users. How do you determine which user goes into which table? Simple, you make this decision based on the user ID (using modulo 20) and this will make the user ID your partitioning key.

Since a picture is worth a thousand words, here’s how the ad server engine will look like. It’s important to realize that almost all the computers from the image below do not necessarily need to be physical boxes and the entire system will happily work on just one server.

Ad Server Engine Architecture

Most likely each web server and its corresponding database server (black box and gray box right above it) will actually be on the same physical server running and HTTP server and a MySQL database. The “…” means that we will be able to add more servers to increase performance (horizontal scaling).

The Load Balancer will dispatch the incoming requests evenly to all available servers. The only data shared by the servers is the configuration and runtime data. Everything else is kept locally.

If a server crashes (not likely, but still) its local data will become temporarily unavailable, but here’s the nice thing about it: it doesn’t matter (more on this below)! The data kept locally by each server is mostly reporting data which is not time sensitive. The runtime information stored in RAM on that server will also be lost, but the RAM cache will actually be replicated on at least another server and unless both servers die at the same time there will be no loss of runtime data.

So, how it all works? Below are a few scenarios and the corresponding system behavior.

• New ad request is received
  The Load Balancer will dispatch the request to one of the available web servers. The selected web server and its corresponding database server will become the “origin” for that ad request. An ID will be generated for the request which is actually composed of the servers ID and a locally unique ID generated by the DB (auto-increment field for example). Besides being a globally unique ID, the request ID will also contain the origin server’s ID, which is useful to know when subsequent requests are received that are dispatched to different servers by the Load Balancer. The origin server will process the request, update what needs to be updated and send back the result. Everybody’s happy!
• The user clicked on the previously served ad
  Again, the Load Balancer will dispatch this request to one of the available servers. The request this time will contain the request ID, because it’s a reporting/tracking request for an already served ad. The selected web server will inspect the request ID and determine which is the origin server and will use the appropriate database server to handle the request. So normally, all requests for already served ads will use the origin database keeping everything consistent, although the request itself might be processed by any of the available servers. So in this case, the selected server will contact the origin database (which is exactly the same as using its local database from a software perspective) and will process the request. Everybody’s happy!Now suppose the origin server is actually offline because it crashed right after processing and serving the initial ad request. In this case, the selected server (by the Load Balancer) will simply write whatever needs to be written to its local database (remember that all configuration and runtime information is still available). The beauty is that the selected server doesn’t need to access or read any of the local data stored on the origin server (which is down in our hypothetical scenario). The selected server will simply use its own local database and the available runtime data to process and serve the request. When the origin server will finally get back online, it could update its database with all the information he missed, but this is neither required nor important, it will practically be an implementation decision. Again, everybody’s happy (even if some servers crashed)!

This is it! If you’ve got questions or feedback, I’m looking forward to reading your comments. As I’ll go along and implement this ad server engine, I’ll write more posts about the interesting parts.

If you think unit testing might be a better solution, please stop now! In just a few words, I hate unit testing, I’ve never done it and I don’t think I ever will, unless the application I’m working on falls into one of those tiny, small and microscopic categories where unit testing actually makes sense. Writing code that validates other code that most likely will never change (at least most of it) just doesn’t make sense to me. Furthermore, in most web based applications the result of a test is actually something being displayed or rendered in a browser, so it’s much easier to just do the testing while you develop the application: write code, save, go to browser and press F5, check result and so on… quite easy!

Logging on the other hand is something I find truly useful. Not so much for testing, but for maintenance and debugging at a later stage. Once your application is set free into the wild, the real testing begins: scenarios you didn’t even thought were possible will start showing up and inevitably some bugs will surface. At this stage, it’s imperative and absolutely necessary for you to be able to  track down what is happening with your application.

I usually log almost everything. I’m using a logging system on 5 levels (below) meant to provide every bit of information about what the application is doing at any stage.

Level 1 – Errors
When an error message is generated, it means the application or a specific piece of code cannot perform the action it’s supposed to do and cannot recover or find alternative solutions to complete the action.

Level 2 – Warnings
Warnings are less severe error messages if you will. It means the application cannot perform a certain action, but could use an alternative solution. It could also mean that some features of the application will be disabled, but overall the application is still able to perform the action is supposed to perform.

Level 3 – Info Messages
Info messages are generated whenever something non-deterministic is happening. For example, the user pressed a button or an e-mail was received.

Level 4 – Debug Messages
Debug messages are generated whenever something deterministic is happening or as additional messages for Level 3 above. For example, if you have a timer to perform some action every 20 seconds, you might want to generate debug messages.

Level 5 – Trace Messages
At this level I like to log everything. Basically, every function of my application will start with a trace message being generated. When this level is enabled, I can see every function call and the complete logic flow of the application. It’s very useful for tracking exactly where a problem occurs and could reduce debugging time a lot.

To conclude, I think logging could really save the day when it comes to debugging and maintaining a large scale application (think millions of requests/users per day). Bugs will inevitably appear and when they do, logging could make a really big difference.