Long, long, looong time ago I wrote about setting up a virtual network using XEN virtual machines, with everything running on the same server. In the first part of this post I’ve mostly described my 2-day experience with making this work and now I’m simply going to copy & paste a README file I’ve created for myself to remember what exactly needs to be done to get such a setup up and running.

This file describes how to create a load-balanced virtual network of
XEN guests (virtual machines) on a XEN Debian server. It is assumed
that the XEN server and xen-tools are already installed.

Step 1: Creating the XEN virtual machine using CentOS as guest OS
-------

- Fix rinse installation of CentOS 5. Add the following lines to
/etc/rinse/centos-5.packages:

nss
nspr
python-iniparse

- Optionally, install the packages for CentOS 5 version from a new
location. Modify /etc/rinse/rinse.conf:

[centos-5]
mirror       = http://mirror.bytemark.co.uk/centos/5/os/i386/CentOS/
mirror.amd64 = http://mirror.bytemark.co.uk/centos/5/os/x86_64/CentOS/

- Create the XEN virtual machine with the following command. Modify
parameters to suit needs:

xen-create-image --hostname node01 --dist centos-5 --install-method \
rinse --size 4Gb --swap 1Gb --memory 2Gb --ip 192.168.1.101 \
--netmask 255.255.255.0 --gateway 192.168.1.1 --passwd

Step 2: Bring the new virtual machine online and open a console to it
-------

- Run the following command:

xm create /etc/xen/node01.cfg -c

Step 3: Set up the network settings on the node
-------

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce

- Setup the ipip tunnel to the load balancer:

ifconfig tunl0 <public IP of load-balancer> netmask 255.255.255.255

- Optionally, set up another public IP address on eth0:1 (to access
this node directly, which is good when developing or debugging):

On the node, type:
ifconfig eth0:1 <IP> broadcast <IP> netmask 255.255.255.255

On the gateway (XEN server / Dom0), type:
route add -host <IP> <vifX.Y>

The vifX.Y must be the one used by XEN for the virtual machine on
which the node runs.

Step 4: Install and start the HTTP server
-------

yum install httpd.x86_64
echo "node01" > /var/www/html/index.html
echo "ok" > /var/www/html/status.html
/etc/init.d/httpd start

The status.html file is used by the load balancer to determine if the
node is online or not so that it knows if it can use it.

Step 5: Configure the load balancer to use the new node
-------

On the load balancer node (lb), edit /etc/ha.d/conf/ldirectord.cf:
real=192.168.1.101:80 ipip 10

Test that the new node is load-balanced (you should see node01 after a
few refreshes, in case you have more than 1 node):
http://<public Load-balancer IP address>/

A few remarks:

• You need at least 2 public IP addresses on your server, but I would recommend 3 or more. One will be used by the server itself (to connect to Dom0), one will be used by the load balancer (your web requests will go to that IP and will be distributed among the available nodes) and at least another one should be assigned to one of the nodes to access it directly and not through the load balancer, which is very important for development and debugging of your load-balanced application.

• The load balancer in my setup is another (virtual) node of the system… a XEN guest. This is needed because the load balancer cannot be on the same machine as the router for your virtual network, which will be Dom0 (i.e.: the main server itself, on which XEN and everything else runs). Why not? Well, because if you use the same IP for both your router and your load balancer when a packet is sent from one of your end nodes to a client it will have the source IP set to the public IP of the load balancer. Now, the router will receive a packet from “outside” that needs to be routed with the same IP address as its own IP address and will say “no, no! this isn’t right” and will drop that packet as being “source martian”. Google a bit for “source martian problem” and you’ll get a clearer picture.

I finally managed to get a new development server that I’ll use to build the ad server engine I was talking about. As you (might) know from my previous post, the system is supposed to run on multiple load-balanced servers. Using this setup I believe scaling won’t be a problem in the future, but I guess I’ll just have to wait and see.

My first task was to create this system setup on my development server. Basically, the system has N nodes with a load-balancer in front of them. Since I only have one server, the obvious choice is to use XEN and create a virtual machine for every node of the system (including the load balancer). All nodes of the system (i.e.: XEN virtual machines) will sit on a private virtual network which is, again, created on just one real computer (my development server).

So far, so good! Everything seems quite simple from the description above and is definitely simple to setup (… after 3 days of intense googling!!!). The story goes like this…

I’m getting my server (yay!) with CentOS installed on it (I requested it). When it comes to Linux distributions I like Slackware, but using CentOS is so much easier when you don’t want to configure and install everything by hand and I didn’t want to spend too much time on installing and compiling software from source, like I usually do on Slackware.

So, anyway, my CentOS is up and running and I’m about to install XEN… easy!, just type “yum install xen” and you’re done. XEN requires a new kernel, so once you reboot the machine (with the new XEN kernel) you have a working XEN environment where you can create additional virtual machines. Just to note, at least on CentOS, you don’t have to configure anything (like grub for example); yum does everything for you… so once you install XEN with yum you just restart the server and you’re done… pretty cool!

Next step: build a virtual machine… how hard could this be? Well, it turns out it’s not that simple, at least not on CentOS. First of all, I think the documentation about this is a bit lacking, especially if you’re not familiar with how Linux works and a dozen or so command-line tools. I think I’m pretty good at Linux stuff, but nevertheless, I couldn’t create a XEN virtual machine on CentOS. To keep a long story short, I was able to boot a virtual machine but I wasn’t able to install the “guest” operating system on it because I didn’t have network access from inside the virtual machine and frankly I don’t know what didn’t work and I also didn’t want to spend too much time on this.

So I went and asked Google what’s the easiest way to create a XEN virtual machine… and the answer was “use xen-tools”. I said “aha! let’s try that…” and immediately went to my CentOS to type “yum install xen-tools”. To my surprise, yum said there is no package with that name, so again I turned to Google to find out more about these xen-tools. I read now from their page: “xen-tools is a collection of simple perl scripts which allow you to easily create new guest Xen domains upon your Debian GNU/Linux host”. Damn, I’m on the wrong Linux distribution… and from looking at the examples of how to use xen-tools it seems quite simple and easy, so it’s worth trying to work with Debian on my development server.

Luckily my hosting provider has a very easy and convenient way to (re)install a new OS, so I went to my administration console and looked for Debian through the available OSes and I was very happy to find a “Debian + XEN 3.2″ OS template there… I selected that, clicked “Restore” and in about 30 minutes my new Debian + XEN 3.2 development server was back online!

What comes next… you’ll have to read it in part 2 of this post. Stay tuned!

In a recent post I mentioned my next big project is to build an ad server engine. This will basically respond to ad requests and will record stats for served ads (such as impressions, clicks, custom interactions and whatever else you might want to track). This post is about the system’s design, how its different parts will interact and how it should behave in certain scenarios.

The goal is to have a system that’s scalable! Ideally, doubling the number of components (servers) in the system should double its performance as well. Redundancy on the other hand is not that important, at least not for this particular application. As we’ll see later on, the engine will be able to function correctly even if some of its components (temporarily) crash.

I’ve done quite a bit of research lately on building scalable and redundant systems using MySQL databases. Almost all articles on this topic mention something about replication. At some point I was tempted to use circular replication, to get (unnecessary) data redundancy, however this was not going to help me get the much more important scaling, which is very important, especially for ad servers.

Let’s start by looking at what kind of data is used by a typical ad server. I could identify 3 major categories:

1. Configuration Data
   This contains information about current campaigns, targeting parameters, CPM, target impressions, etc. The engine will mostly use this data in a “read-only” mode. In my particular case, this data comes from a completely separate admin application. The frequency at which this data changes is relatively low, at least when compared to the other 2 categories below.
2. Runtime Data
   This contains information generated during the normal operation of the ad server engine. You will find here things like counters, session data, etc. The engine will both read & write this data in almost equal proportions.
3. Reporting Data
   This contains everything that’s being logged about served ads (clicks, interactions, timers, etc). This information is mainly written to the database and otherwise not required by the ad server engine. In my particular case, this information will be picked up by the completely separate admin application and used to generate reports and adjust the configuration data.

The configuration data will be quite small in size (at least when compared with the reporting data) and will most likely be cached for super-fast access. The runtime data will also be kept in memory (as much as possible) but will also be saved in the database for persistence. The reporting data will be by far the largest in size and we’ll need to save this to the database as soon as possible (we’ll use an intermediary cache to reduce the number of WRITE operations to the database).

If I take into account one of the goals for this ad server engine (to be able to serve 1 billion ads per month) and assume that for each ad served an additional 10-15 database WRITE operations are required (I’ve exaggerated a bit here), it soon becomes obvious that scalability actually means in this case being able to cope with all those database WRITE operations.

I’ve mentioned above that almost all articles I’ve read about scaling MySQL talk about replication. Well, this is only useful when scaling READ operations. Scaling WRITE operations will only work by data partitioning (or sharding). Quick example: if you have an application with 100 million users, you most likely don’t want to keep all of those users in the same DB table because it will become unmanageable. Instead, you’ll split those users into say 20 tables each storing 5 million users. How do you determine which user goes into which table? Simple, you make this decision based on the user ID (using modulo 20) and this will make the user ID your partitioning key.

Since a picture is worth a thousand words, here’s how the ad server engine will look like. It’s important to realize that almost all the computers from the image below do not necessarily need to be physical boxes and the entire system will happily work on just one server.

Ad Server Engine Architecture

Most likely each web server and its corresponding database server (black box and gray box right above it) will actually be on the same physical server running and HTTP server and a MySQL database. The “…” means that we will be able to add more servers to increase performance (horizontal scaling).

The Load Balancer will dispatch the incoming requests evenly to all available servers. The only data shared by the servers is the configuration and runtime data. Everything else is kept locally.

If a server crashes (not likely, but still) its local data will become temporarily unavailable, but here’s the nice thing about it: it doesn’t matter (more on this below)! The data kept locally by each server is mostly reporting data which is not time sensitive. The runtime information stored in RAM on that server will also be lost, but the RAM cache will actually be replicated on at least another server and unless both servers die at the same time there will be no loss of runtime data.

So, how it all works? Below are a few scenarios and the corresponding system behavior.

• New ad request is received
  The Load Balancer will dispatch the request to one of the available web servers. The selected web server and its corresponding database server will become the “origin” for that ad request. An ID will be generated for the request which is actually composed of the servers ID and a locally unique ID generated by the DB (auto-increment field for example). Besides being a globally unique ID, the request ID will also contain the origin server’s ID, which is useful to know when subsequent requests are received that are dispatched to different servers by the Load Balancer. The origin server will process the request, update what needs to be updated and send back the result. Everybody’s happy!
• The user clicked on the previously served ad
  Again, the Load Balancer will dispatch this request to one of the available servers. The request this time will contain the request ID, because it’s a reporting/tracking request for an already served ad. The selected web server will inspect the request ID and determine which is the origin server and will use the appropriate database server to handle the request. So normally, all requests for already served ads will use the origin database keeping everything consistent, although the request itself might be processed by any of the available servers. So in this case, the selected server will contact the origin database (which is exactly the same as using its local database from a software perspective) and will process the request. Everybody’s happy!Now suppose the origin server is actually offline because it crashed right after processing and serving the initial ad request. In this case, the selected server (by the Load Balancer) will simply write whatever needs to be written to its local database (remember that all configuration and runtime information is still available). The beauty is that the selected server doesn’t need to access or read any of the local data stored on the origin server (which is down in our hypothetical scenario). The selected server will simply use its own local database and the available runtime data to process and serve the request. When the origin server will finally get back online, it could update its database with all the information he missed, but this is neither required nor important, it will practically be an implementation decision. Again, everybody’s happy (even if some servers crashed)!

This is it! If you’ve got questions or feedback, I’m looking forward to reading your comments. As I’ll go along and implement this ad server engine, I’ll write more posts about the interesting parts.

When it comes to testing and validating the code I write, I usually do it right after I write the actual code. As much as possible I try writing very small bits of code (a function or even half a function) and then test immediately. If the input range is fairly limited (for example a function that only accepts 1 true/false parameter), you could easily check all scenarios and determine if there’s a bug or not pretty quickly. However, if you have a lot of possible inputs for the piece of code you want to test, most times you’ll test for 2-3-4 different situations and once all issues are fixed you might conclude that everything works OK. At least this is how I do it.

If you think unit testing might be a better solution, please stop now! In just a few words, I hate unit testing, I’ve never done it and I don’t think I ever will, unless the application I’m working on falls into one of those tiny, small and microscopic categories where unit testing actually makes sense. Writing code that validates other code that most likely will never change (at least most of it) just doesn’t make sense to me. Furthermore, in most web based applications the result of a test is actually something being displayed or rendered in a browser, so it’s much easier to just do the testing while you develop the application: write code, save, go to browser and press F5, check result and so on… quite easy!

Logging on the other hand is something I find truly useful. Not so much for testing, but for maintenance and debugging at a later stage. Once your application is set free into the wild, the real testing begins: scenarios you didn’t even thought were possible will start showing up and inevitably some bugs will surface. At this stage, it’s imperative and absolutely necessary for you to be able to  track down what is happening with your application.

I usually log almost everything. I’m using a logging system on 5 levels (below) meant to provide every bit of information about what the application is doing at any stage.

Level 1 – Errors
When an error message is generated, it means the application or a specific piece of code cannot perform the action it’s supposed to do and cannot recover or find alternative solutions to complete the action.

Level 2 – Warnings
Warnings are less severe error messages if you will. It means the application cannot perform a certain action, but could use an alternative solution. It could also mean that some features of the application will be disabled, but overall the application is still able to perform the action is supposed to perform.

Level 3 – Info Messages
Info messages are generated whenever something non-deterministic is happening. For example, the user pressed a button or an e-mail was received.

Level 4 – Debug Messages
Debug messages are generated whenever something deterministic is happening or as additional messages for Level 3 above. For example, if you have a timer to perform some action every 20 seconds, you might want to generate debug messages.

Level 5 – Trace Messages
At this level I like to log everything. Basically, every function of my application will start with a trace message being generated. When this level is enabled, I can see every function call and the complete logic flow of the application. It’s very useful for tracking exactly where a problem occurs and could reduce debugging time a lot.

To conclude, I think logging could really save the day when it comes to debugging and maintaining a large scale application (think millions of requests/users per day). Bugs will inevitably appear and when they do, logging could make a really big difference.

I bet you didn’t think e-mail address validation could be that “easy”. Take a look at http://www.linuxjournal.com/article/9585 (the PHP way). I’m wondering if I should change my e-mail address to {^c\@talin*ciocov^}@somedomain.com .

My next major project will be to build an ad server engine. When I say “engine”, I’m referring only to that part of an ad server that will deal exclusively with ad delivery, basically the part that receives and responds to ad requests from publishers.

Although there’s a lot more going on in a full-featured ad server (user system, campaign management, ad targeting, billing system, rule-based system for ad delivery, etc…), the serving engine will be the most critical component since this part of the system will need to cope with a massive load (think billions of ad requests per month for example).

To measure the success of this project, I’ve defined the following objectives:

• In it’s minimal configuration, the engine should handle at least 1 billion ad requests per month. It should also handle traffic/load spikes of up to 30-40% of the normal traffic/load.
• The system should be scalable horizontally; that is, if additional servers are needed, those can be added relatively quickly and simply to the system with no software change required.
• In the event of a server being added, the system should detect the new configuration and automatically distribute part of the load to the new server.
• In the event of a server crash, all dependent servers (that is, those that relied on the crashed server) should automatically create connections to other still available servers.

As you can see, except for the first objective above, all other objectives hint at the idea of actually building a distributed, scalable and fault tolerant system in general, which can be used to run any kind of application that will be under heavy load.

During development, I will try to post as many times as possible with details about the various parts of the system, architecture and specific implementation details. Your feedback will be greatly appreciated along the way.

Loading as AS2 SWF into AS3 is possible using the Loader class, but that won’t help in case you also need to communicate with the loaded AS2 SWF. The trick is to create a wrapper AS2 SWF that will act as a proxy (or say bridge) between the host AS3 SWF and the target AS2 SWF. The bridge SWF will be loaded in the host AS3 SWF while the target AS2 SWF will be loaded inside the bridge.

Because I needed this quite a lot in one of the applications I’m working on, I’ve decided to create a class (AS2Loader) that will simplify the process of loading and communicating with ActionScript 2 SWFs from ActionScript 3. Here’s what you can do with it:

Suppose you have an AS3 SWF in which you want to load (and then communicate with) an AS2 SWF named “as2movieclip.swf”. Using the AS2Loader class you will do something like this:

import com.onsysol.as3.AS2Loader.*;

var as2ldr:AS2Loader = new AS2Loader();
as2ldr.addEventListener('AS2LoadInit', onAS2LoadInit);
as2ldr.load(new URLRequest('as2movieclip.swf'));

// this is triggered when the 1st frame of "as2movieclip.swf"
// is loaded and initialized:
function onAS2LoadInit(e:AS2LoaderEvent) {
    addChild(AS2Loader(e.target));
}

The above code will load the AS2 SWF and once loaded it will add it to the display list. Next, you can register event listeners or make function calls on the AS2 SWF. The nice thing about this is that you can also request to receive back the return value of the function calls you make (since under the hood this is using Local Connection objects the return value of the function calls you make will be delivered through AS2LoaderEvents). So let’s proceed.

Suppose our AS2 SWF dispatches an “ABC” event. The following piece of code will register an event listener for this event:

as2ldr.addEventListenerOnAS2('ABC', onABCEvent);
...
function onABCEvent(e:AS2LoaderEvent) {
    // the original AS2 event object is accessible through
    // e.data.eventObj:
    trace(e.data.eventObj.type + ' event dispatched from AS2.');
}

If you want to call a function on the AS2 SWF (eg: gotoAndPlay() or stop() or anything else that’s public) you’ll do it like this (the following code assumes there’s a function say(s:String) defined on the AS2 SWF):

as2ldr.callAS2Method('say', 'hello from AS3');

In case you need to call a function on the AS2 SWF and also get the return value there’s no problem. Assuming there’s a function sum(a, b) that returns the sum of a and b, the following code will call sum() on the AS2 and also request and process the return value.

as2ldr.getAS2MethodResult('sum', 2, 5);

// to get the return value from sum, we need to register an event
// listener on the AS2Loader object:
as2ldr.addEventListener('onAS2sumResult', onAS2SumResult);

function onAS2SumResult(e:AS2LoaderEvent) {
    trace('The result of 2 + 5 is ' + e.data.result);
}

As you can see from the above example, if you call a function and also request its return value, the AS2Loader object will dispatch an event named “onAS2[function name]Result” when the return value is available.

I’ve created a test/demo page here. You can also download the AS2Loader class and the example page.

If you’re actively doing development work and your production server is different from the development one, as it is in my case, at some point you’ll need to upload or deploy files from the development server to production. If this happens a lot during the day or even once a day, you might want to automate the deployment process.

I’ve created 2 small shell (PHP) scripts that will do just that, one using RSYNC/SSH and the other one plain FTP (you need to have ncftp installed). Here’s how to use them:

$> cd dir_to_deploy/
$> deploy now

Or, for the deployment via FTP:

$> cd dir_to_deploy/
$> ftpdeploy

It’s quite simple! Assuming you’ve installed the scripts in your path (with +x permission) you just go to the directory you want deployed and issue the above commands. You will be asked for the required information, such as host, user, pass, remote path, etc… the nice thing is that this information is saved on a per directory basis, so the next time you deploy you don’t have to enter it again, unless you want to change something, of course!

These tools were specially created to deploy applications made with My Framework, but they will work in almost any situation. You also have options to exclude certain files from deployment (like log and temporary files), but to find out how either ask me by posting a comment or read the source code.

Another tool I created that’s also available to download at the end of this post might help you if you want to minify and compress your JS files. It uses a PHP library that you can find here.  It is a very good practice to minify and compress your JS files because you save a lot of bandwidth, especially if you’re web server is configured to serve compressed Javascript for browsers that support this (almost all!).

Suppose you have a JS file named “myjsfile.js”. To minify it, simply use:

$> minifyjs myjsfile.js

The above command will create 2 new files in the same directory as the target file, named “myjsfile.min.js” and “myjsfile.min.js.gz”. You figured out what these are, right?

Download these tools here.

I’ve been doing a lot of Action Script recently and needed a good library for making flash remoting calls. If you Google a bit you’ll find a very good one (for AS3) here. I’ve started with this one, but because I needed some additional features I’ve decided to create my own version.

Specifically, I wanted the ability to specify a time limit for remoting calls and have multiple ways of getting the result back (since it’s an asynchronous call). Here are some examples.

Connect the the remote service:

import com.onsysol.remoting.*;

var service:RemoteService = new RemoteService('http://myserver.com/gateway.php', 'MyService');
service.connect();

Make a remote call to MyMethod() and receive the result via event listeners:

service.addEventListener('onMyMethodResult', onMyMethodResult);
service.addEventListener('onMyMethodFault', onMyMethodFault);
service.MyMethod('param1', 'param2', 'etc...');

function onMyMethodResult(e:ResultEvent) {
    trace(e.result);
}

function onMyMethodFault(e:FaultEvent) {
    trace(e.fault);
}

In case you don’t want to use an event listener for every remoting call you make you can simply listen for a “default” event triggered after every remoting call that doesn’t have a specific event listener registered for it. So, instead of the above, you could have:

service.addEventListener(ResultEvent.RESULT, onResult);
service.addEventListener(FaultEvent.FAULT, onFault);
service.MyMethod();

function onResult(e:ResultEvent) {
    trace('Result event received from ' + e.prop['methodName']);
    trace(e.result);
}

function onFault(e:FaultEvent) {
    trace('Fault event received from ' + e.prop['methodName']);
    trace(e.fault);
}

And finally, if you don’t want to use events at all to get the response, you could use callback/anonymous functions, like this:

var prop:Object = {
    onResultCallback: function(result:Object, prop:Object) {
        trace('do something in your function...');
    },
    onFaultCallback: function(fault:Object, prop:Object) {
    }
};
service.call('MyMethod', prop, 'param1', 'param2', 'etc...');

The default timeout is 2 seconds, but you can specify a new timeout value for all calls or on a per call basis:

// General timeout value, used for all calls that don't have a specific timeout:
RemoteService.TIMEOUT = 5000;   // 5 seconds

// Make a remoting call with timeout set to 20 seconds:
service.call('MyMethod', {timeout: 20000}, 'param1', '...');

You can handle timeouts by either registering specific event listeners (onMyMethodTimeout), using the generic event (onTimeout) or using callback/anonymous function (onTimeoutCallback) in a similar way to how a result or a fault is handled.

Download the library here.