Nicholas Piël

There has already been written a lot on the C10K problem and it is known that the only viable option to handle LOTS of concurrent connections is to handle them asynchronously. This also shows that for massively concurrent problems, such as lots of parallel comet connections, the GIL in Python is a non-issue as we handle the concurrent connections in a single thread.

In this post i am going to look at a selection of asynchronous servers implemented in Python.

Asynchronous Server Specs

Since Python is really rich with (asynchronous) frameworks, I collected a few and looked at the following features:

• What License does the framework have?
• Does it provide documentation?
• Does the documentation contain examples?
• Is it used in production somewhere?
• Does it have some sort of community (mailinglist, irc, etc..)?
• Is there any recent activity?
• Does it have a blog (from the owner)?
• Does it have a twitter account?
• Where can i find the repository?
• Does it have a Thread Pool?
• Does it provide access to a TCP Socket?
• Does it have any Comet features?
• Is it using EPOLL?
• What kind of server is it? (greenlets, callbacks, generators etc..)

This gave me the following table.

This is quite a list and i probably still missed a few. The main reasons for using a framework and not implementing something your self is that you hope to be able to accelerate your own development process by standing on the shoulders of other developers. I think it therefore is important that there is documentation, some sort of developers community (mailinglist fe)  and that it is still active. If we take this as a requirement we are left with the following solutions:

• Orbited / Twisted (callbacks)
• Tornado (async)
• Dieselweb (generator)
• Eventlet (greenlet)
• Concurrence (stackless)
• Circuits (async)
• Gevent (greenlet)
• Cogen (generator)

To quickly summarize this list; Twisted has been the de-facto standard to async programming with Python. It has an immense community, a wealth of tools, protocols and features. It has grown big and some say it reminds them of shirtless men drinking Jager-bombs complex. This is also one of the biggest reasons why people are looking elsewhere. Recently Facebook released the code of their async. approach called Tornado which is also using callbacks and recent benchmark show that it outperforms Twisted.

A common heard argument against programming with callbacks is that it can get overly complex. A programmatically cleaner approach is to use light-weight threads (imho). This can be achieved by using a different Python implementation; Stackless (such as Concurrence is using) or a plugin for regular python Greenlet (such as Eventlet and Gevent are using). Another approach is to simulate these light-weight threads with Python generators, such as Dieselweb and Cogen are doing.

This should already show that while all these frameworks provide you asynchronous concurrency they do this in each of their own ways. I want to invite you to look at these frameworks as they all have their own code gems. For example, Concurrence has a non-blocking interface to MySQL. Eventlet has a neat thread-pool, Tornado can pre-fork over CPU’s, Gevent offloads HTTP header parsing and DNS lookups to Libevent, Cogen has sendfile support and Twisted probably already has a factory doing exactly what you are planning to do next.

The Ping Pong Benchmark

In this benchmark i am going to focus on the performance of the framework to listen on a socket and write to incoming connections. The client pings the socket by opening it, the server responds with a ‘Pong!’ and closes the socket. This should be really simple but it is a pain to create something that does this in an asynchronous and non-blocking way from scratch and that is exactly the reason why we are looking at these frameworks. It is all about making our lives easier.

Ok, for this benchmark i am going to use httperf,  a high performance tool that understands the HTTP protocol. If we want httperf to play along in our Ping-Pong benchmark we have to make it understand the ‘PONG!’ response. We can do this by mimicking a HTTP server and have our server respond with:

HTTP/1.0 200 OK
Content-Length: 5

Pong!

instead of just ‘Pong!’. Also, since most default server configurations are not set up to handle a large amount of concurrent requests, we need to make a few adjustments:

• Raise the per-process file limit by compiling httperf after some adjustments.
• Raise the per-user file limit, set ‘ulimit -n 10000‘ on both server and client.
• Raise kernel limit on file handles: ‘echo “128000″ > /proc/sys/fs/file-max’.
• Increase the connection backlog, ‘sysctl -w net.core.netdev_max_backlog = 2500‘
• Raise the maximum connections with ’sysctl -w net.core.somaxconn = 250000‘

With these settings my Debian Lenny system was ready to hammer the different servers up to rates far beyond the capacity of the frameworks. I used the following command

httperf –hog –timeout=60 –client=0/1 –server=localhost –port=10000 –uri=/ –rate=400 –send-buffer=4096 –recv-buffer=16384 –num-conns=40000 –num-calls=1

And increased the rate with an interval of 100 from 400 up to 9000 requests per second for a total of 40.000 requests at each interval.

Code

What will now follow, is the implementation of the server side in the different frameworks. It should show the different approaches the frameworks take.

Twisted

Gentlemen start your reactor!

from twisted.internet import epollreactor epollreactor.install()
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor

class Pong(Protocol):
 def connectionMade(self):
 self.transport.write("HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n")
 self.transport.loseConnection()

# Start the reactor
factory = Factory()
factory.protocol = Pong
reactor.listenTCP(8000, factory)
reactor.run()

Tornado

Tornado, does not hide the raw socket interface, which makes this example more lengthy then the others.

import errno
import functools
import socket
from tornado import ioloop, iostream

def connection_ready(sock, fd, events):
    while True:
        try:
            connection, address = sock.accept()
        except socket.error, e:
            if e[0] not in (errno.EWOULDBLOCK, errno.EAGAIN):
                raise
            return
        connection.setblocking(0)
        stream = iostream.IOStream(connection)
        stream.write("HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close)

if __name__ == '__main__':
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.setblocking(0)
    sock.bind(("", 8010))
    sock.listen(5000)

    io_loop = ioloop.IOLoop.instance()
    callback = functools.partial(connection_ready, sock)
    io_loop.add_handler(sock.fileno(), callback, io_loop.READ)
    try:
        io_loop.start()
    except KeyboardInterrupt:
        io_loop.stop()
        print "exited cleanly"

Dieselweb

While this example is beautifully small, i do not really enjoy the generator approach which sprinkles ‘yield’ all over the place.

from diesel import Application, Service

def server_pong(addr):
    yield "HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n"

app = Application()
app.add_service(Service(server_pong, 8020))
app.run()

Circuits

I think the Circuit code is the most beautiful of them all, very elegent.

from circuits.net.sockets import TCPServer

class PongServer(TCPServer):

    def connect(self, sock, host, port):
        self.write(sock, 'HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n')
        self.close(sock)

PongServer(('localhost', 8050)).run()

Eventlet

The Eventlet uses a Greenlet approach.

from eventlet import api

def handle_socket(sock):
    sock.makefile('w').write("HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n")
    sock.close()

server = api.tcp_listener(('localhost', 8030))
while True:
    try:
        new_sock, address = server.accept()
    except KeyboardInterrupt:
        break
    # handle every new connection with a new coroutine
    api.spawn(handle_socket, new_sock)

Gevent

Gevent is presented as a rewrite of eventlet focussing on performance.

import gevent
from gevent import socket

def handle_socket(sock):
    sock.sendall("HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n")
    sock.close()

server = socket.socket()
server.bind(('localhost', 8070))
server.listen(500)
while True:
    try:
        new_sock, address = server.accept()
    except KeyboardInterrupt:
        break
    # handle every new connection with a new coroutine
    gevent.spawn(handle_socket, new_sock)

Concurrence

Concurrence uses the Tasklet approach, it can be run under Greenlet and under Stackless Python. In this benchmark there was not really any performance difference between the two different engines.

from concurrence import dispatch, Tasklet
from concurrence.io import BufferedStream, Socket

def handler(client_socket):
    stream = BufferedStream(client_socket)
    writer = stream.writer
    writer.write_bytes("HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n")
    writer.flush()
    stream.close()

def server():
    server_socket = Socket.new()
    server_socket.bind(('localhost', 8040))
    server_socket.listen()

    while True:
        client_socket = server_socket.accept()
        Tasklet.new(handler)(client_socket)

if __name__ == '__main__':
    dispatch(server)

Cogen

Cogen, uses the generator approach as well.

import sys

from cogen.core import sockets
from cogen.core import schedulers
from cogen.core.coroutines import coroutine

@coroutine
def server():
    srv = sockets.Socket()
    adr = ('0.0.0.0', len(sys.argv)>1 and int(sys.argv[1]) or 1200)
    srv.bind(adr)
    srv.listen(500)
    while 1:
        conn, addr = yield srv.accept()
        fh = conn.makefile()
        yield fh.write("HTTP/1.0 200 OK\r\nContent-Length: 12\r\n\r\nHello World!\r\n")
        yield fh.flush()
        conn.close()

m = schedulers.Scheduler()
m.add(server)
m.run()

Results

The first graph clearly shows at which connection rate (on the horizontal axis) the successful connection rate starts to degrade. It shows a huge difference between the best performer; Tornado with 7400 requests per second and the worst, Circuits with 1400 requests per second (which doesn’t use EPOLL). This connection rate was sustained for at least 40.000 requests. We can see that, when the hammering of the server continues beyond rates the server can handle, the performance drops. This is caused by connection errors or timeouts.

This graph shows the response time, it is clearly visible that once the maximum connection rate has been reached the overal response time starts to increase.

The last graph shows the amount of errors, ie no return of a 200 detected by httperf. We can see a correlation between the performance of the server and the returned errors at a given request rate. The performing servers return less overall errors. There is however, one exception. Cogen was able to return ALL its requests successfully no matter how hard it was hammered. It is therefore not visible in this graph. This is interesting, at 9000 requests per second it was still able to answer all requests. However, the average connection time (from socket open till socket close) was about 7 seconds meaning that Cogen was serving about 28000 concurrent connections somewhat at reduced performance but not dropping them.

Notes

This post should make it clear that Python has a rich set of options toward asynchronous programming. All tested frameworks show great performance. I mean, even Circuits results with 1300 requests per second isn’t too bad. Tornado really blew me away with its performance at 7400 requests per second. But if i had to choose a favorite i would  probably go with Gevent, i am really digging its greenlet style.

The clean Greentlet / Stackless style is really cool, especially since Stackless Python is keeping up nowadays with CPython. There was some talk on a mailing list about Gevent running on Stackless. The concurrence framework already runs on Stackless and can thus be a great option already if you are looking for specific features of Stackless Python such as tasklet-pickling.

I want to make clear that this test only shows  how these frameworks perform at a relatively simple task. It could be that when more stuff is going on in the background the results will change. However, I feel that this benchmark is a great indicator of how each frameworks handles a socket connection.

In the coming days I plan to investigate this some more. I will also check out  how these Python frameworks stack up against its equivalents in different languages, fe Ape, CometD, NodeJS. Stay tuned!

For my Msc thesis I have developed a system build in Python which does person recognition and have shown that it is possible to obtain a better recognition rate with this system than by using Google’s Picasa. I have put the source code online and will hereby announce that I will try my best to spend some time explaining how to do person and face recognition with Python.  I hope that a public announcements such as this will instantly create some public debt forcing me to actually complete this task. We shall see.

The approach described in my thesis uses a combination of pictorial cues (Eigenfaces, SIFT points, color histogram) and contextual cues (co-occurence with other persons or background). The idea behind this is really simple, in order to recognize a person we don’t even need to see the face of that specific person in most cases if we have more contextual information about the setting in which it was taken. For example, lets say you are looking at some pictures from ‘Sesame street’, when you detect Ernie there is a high probability that that other person on that same photo will be Bert. Even more so if we detect that the main component of that other persons color histogram is yellow.

The approach can be divided in three different subtasks:

1. Detecting the person and segmenting its specific region
2. Extracting the features
3. Clustering over these features

In the first task, we will detect the different people by using a face detection technique based on haarcascades. I plan on showing how to use OpenCV with Python and how you can improve its performance by combining the result of multiple haar cascades.

With the detected face we only have a certain square region within a picture which is very likely to contain a face. In order to detect the rest of the body i used a graph based segmentation technique and highly optimized the segmentation algorithm by using implementations in NumPy, Fortran and finally PyCUDA

From this segmented region we will then extract pictorial features such as a color histogram and SIFT features. With this information we can then try to extract and use our contextual information.

In the schematic overview we can see the different steps, we start with some images and segment it in regions of interest. From these regions we will then extract features to build our person models, over which we can then cluster by using  pictorial features (SIFT points and Color Histograms) and contextual features (ie, co-occurence with detected background or other persons). The code for this can already be found on BitBucket but is a bit rough, but as already said I promised to do some explaining. So keep an eye on this blog if you’re interested.

For now, I present you my collected list of references (also in BibTex format) regarding person recognition. It could be a nice starting point for anyone interested in this domain.

References

Now that the dust has somewhat settled after climategate, the consensus seems to be that it has been overblown. If you look at the timeline of events this isn’t surprising. Between the public appearance of the report and the first damning articles on the 20th there was less then a single day.  It is not that difficult to question how thorough the review of 160mb of data was.  It simply wasn’t.

It was as if some people thought they had hit gold and where aggressively searching for that specific quote within the leaked emails which would make them famous instantly. But all in all it was a bit disappointing if you where hoping to find exciting revelations. The thing that could be distilled from the e-mails was that most researchers are having strong opinions and big ego’s, but this shouldn’t really be a surprise.

It is naive to think that scientists are unbiassed, they simply aren’t. However, they are expected to backup up their views with unbiassed facts. The main argument thats left if we ignore all personal slander seems to be focused around a quote in one of the emails concerning the WMO Statement of the status of the global climate in 1999. The front page of this report shows the picture below and indicates that 1990-1999 has been the hottest decade on the record. So yes, it is an argument about a 10 year old report. It might be worth noting that a few days ago (8 dec 2009), the World Meteorological Institute came with a new press release that our current decade is the warmest on record. That information got probably lost in the heated debate.

From the leaked emails conservative news sources state that the following quote is a clear sign of manipulation of evidence:

“I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) amd from 1961 for Keith’s to hide the decline.”

But is it? In a rebuttal by the Climate Research Unit they state the following:

This email referred to a “trick” of adding recent instrumental data to the end of temperature reconstructions that were based on proxy data.

Phil Jones comments further: “One of the three temperature reconstructions was based entirely on a particular set of tree-ring data that shows a strong correlation with temperature from the 19th century through to the mid-20th century, but does not show a realistic trend of temperature after 1960. This is well known and is called the ‘decline’ or ‘divergence’. The use of the term ‘hiding the decline’ was in an email written in haste. CRU has not sought to hide the decline. Indeed, CRU has published a number of articles that both illustrate, and discuss the implications of, this recent tree-ring decline, including the article that is listed in the legend of the WMO Statement figure.

They also provide an extra graph where they show the climate reconstruction and the recent instrumental data seperately:

So, as you can see there isn’t really anything shocking to report.

It seems that our viewpoint concerning climate change seems closely linked to our position on the political spectrum. In the red corner, we have the conservatives who consider any idea where they might need to change their way of living threatening. In the blue corner we have the progressives, those who feel that change is a goal not just a method. During the first round of the climate gate boxing match we mainly heard the conservative viewpoint represented by the Telegraph, FOX news, Washington Times and lots of infuriated bloggers but now that that the round is over i think the focus will shift to a more progressive point of view. You see, wether or not climate change is happening we will have to think about how we manage our environment. We are running out of resources and we are polluting our environment . When we do not act accordingly we will end up like the easter islands.

Round 2: The need for data sharing

A positive result of this climate battle is the renewed focus on the public availability of data and methodologies. CRU claims that 95% of their data is already open to the public and that they will make the remaining 5% publicly available, which is great news.  This movement of ‘data freeing’ is a great initiative, certainly in this time of collective sharing. John Wilbanks of Science Commons says the following:

“the irony that right at the historical moment when we have the technologies to permit worldwide availability and distributed process of scientific data, broadening collaboration and accelerating the pace and depth of discovery…..we are busy locking up that data and preventing the use of correspondingly advanced technologies on knowledge”.

Making our research widely available is a great way to catalyze progress in the broadest sense, this is probably better illustrated with the next video by  Jesse Dylan.

The importance of data sharing is already recognized by the government of the USA, they have created data.gov with the purpose to increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government. It currently has over 118000 different datasets which really makes it a dataminers wetdream.

My efforts on data sharing

In an effort to not just stand along the sideline but participate in this ‘release your data’ party, I have decided to put my master thesis and its results in the public domain. For my master thesis i have implemented a system in mostly Python code which does person recognition on static images. You can compare it with what Google’s Picasa does. However, i was able to outperform Picasa in recognition rate on a few datasets. I have already released some of the source code on BitBucket and you can find a little bit more information on the Projects page.

In the next few months i am going to explain this approach in more detail and will put up my collected resources and a bibtex file. I think this will be a great start for anyone interested in machine vision and person recognition. If your interested just follow me on twitter!

Using a Content Delivery Network (CDN) is a method  to improve the performance of your website. Some of the reasons for using a CDN are:

• Placing content geographically close to the end user and thus lowering latency and increasing bandwidth.
• Increasing the amount of parallel downloads at the client by distributing over different domains
• Offload the burden on your servers
• Facilitate long term caching by using a robust source for libraries

Especially this last point is why I looked at Google’s CDN for Ajax libraries. It is a beautiful idea. When more people are using the same CDN, the cost of downloading an Ajax library can be ignored because it is very likely that the web browser will already have the library in its belly. Wonderfull!

For example, when I try to download the Prototype library everything goes well and the Google’s CDN spits the following back:

Content-Type:text/javascript; charset=UTF-8
Date:Mon, 30 Nov 2009 14:31:50 GMT
Expires:Tue, 30 Nov 2010 13:56:34 GMT

As you can see, Google tells your browser to cache the file for a full year as it should. Now, lets look at what happens when trying this with JQuery 1.3.2:

Content-Type:text/javascript; charset=UTF-8
Date:Mon, 30 Nov 2009 14:40:15 GMT
Expires:Tue, 30 Nov 2010 14:40:15 GMT

Again, everything is ok. Now, lets try a different version, JQuery 1.3:

Content-Type:text/javascript; charset=UTF-8
Date:Mon, 30 Nov 2009 14:41:42 GMT
Expires:Mon, 30 Nov 2009 15:41:42 GMT

Huh? When requesting the 1.3 version, Google is basically telling us to ‘remember it only for one hour’. This is wrong imho. When you specify 1.3, you are telling Google you want ‘the latest version in the 1.3 series‘. On the jquery.com site they are linking to the 1.3 version as well. This means that for a pagehit on the jquery website you will re download 60k of minified Jquery goodness when this file is not in your cache. A better approach would be if Google just let the client do its cache  revalidation  (which can do so by using the ‘if-modified since’ header).

But wait there is more. Wordpress, for example, adds an extra version argument to the file (?ver=<bla>). This can be handy when you want to generate a certain script or css file dynamically. And really should not be a problem with Google’s CDN, should it? Well lets see what happens when we request http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js?ver=1.3.2

Content-Type:text/javascript; charset=UTF-8
Date:Mon, 30 Nov 2009 15:00:00 GMT
Expires:Fri, 01 Jan 1990 00:00:00 GMT
Last-Modified:Mon, 23 Nov 2009 18:54:21 GMT

Holy cow, Google invented time travel!  The implications of this are pretty big, this may affect a lot of people with Wordpress blogs who where ’smart enough’ to use the Google CDN but without really testing if it worked.

Basically what this means is that http://ajax.googleapis.com isn’t really your performance safety net. You need to know exactly what you’re doing otherwise it will bite you back and you might be better off just hosting those libraries on your own site. Thus my recommendation would be to use the Google CDN but specify exactly which version of the library you are going to need and make sure you do not provide any arguments.

Yes, “Hello world!” , the first default post on a Wordpress blog. After much consideration i decided to simply use the best piece of software there is for blogging purposes. Yeah, i know it is PHP code and not Python and trust me, i tried really hard to use some Python alternative and even started working (doesn’t everybody else?) on my own blogging software written on top of Pylons, Mako and STORM, but it just doesn’t compare with WP.

Well, i must admit Zine (Python blog software) looks nice, but still, WP integrates so nice with everything else. Choosing for WP isn’t just choosing a software package, it is being part of a huge community. It is hard to compete with the zillions of PHP coders that extend Wordpress or integrate it with other platforms, no matter how much nicer your language of choice is.

The things that i where looking for, at minimum:

• Easy editing of posts, the WP webinterface rocks!
• RSS Support
• Edit with XML-RPC
• Pingbacks
• Spamfiltering, WP integrates nicely with Akismet

But with WP Plugins you easily get so much more:

• Gravatars (Globally Recognized Avatars)
• OpenID One password for all your sites
• Twitter Integration
• LinkedIn integration
• Easy integration with Google Analytics
• Client side optimization with CSS sprites and data-uri
• Robust performance through caching
• SEO optimized
• Automatic redirection of renamed pages

So, here it is my Wordpress blog, hosted behind a NGINX proxy which handles all static data and some Apache workers that handle the PHP scripts.  I must say, I feel confident that it can handle a Digg. At least, i benchmarked it to handle 3000 reqs/second without breaking a sweat.