Multi–Machine Parallel Python Benchmarks

Follow-up to Benchmarking Parallel Python Against Jython Threading (Benchmarks Take 3) from The Utovsky Bolshevik Show

Having claimed in a previous post that Parallel Python's ability to use the processing power of more than a single machine would work in its favour even when compared to the times for Jython threading, I thought I should probably look at some results to see if this is the case.

As previously, the benchmark being used is to sum all the primes beneath each multiple of 10000 between 100000 and 1000000.  The code examples can be found at http://oddbloke.uwcs.co.uk/parallel_benchmarks/

The Jython script uses Tim Lesher's cookbook recipe for a thread pool.  The Parallel Python script uses a slightly tweaked version of one of the examples on the Parallel Python site.

The two machines over which this is being tested are the University of Warwick Computing Society's servers, Backus and Codd, with Codd being used as the master server.  Both these machines have two CPUs.

The setup for the slave machine really is as easy as:

$ ./ppserver.py -p 35000 -w 2 

Once this was set up, I proceeded to test the Jython and Parallel Python scripts.  Disappointingly, the Jython script used more memory than I have available on my ulimit'ed account when running more than a single thread.  I have approximated based on the previous results I've had.

Looking solely at the numbers for Parallel Python, it seems that the speedup gained by using a second machine is significant.  It should be noted that Parallel Python's default regardless of whether or not it had the second machine available was 2 workers, so the automatic detection code is obviously sub-optimal.  It's trivial to override, so this wasn't a problem.

When this is compared to Jython's threading, it doesn't look significant but when we consider Jython's arithmetic ability and the fact that Parallel Python can continue to scale beyond this, Parallel Python begins to look better and better.  It should also be noted that, unsurprisingly, Jython uses a considerable amount more memory than CPython does.

EDIT: As pointed out in the comments, Jesse Noller has also started looking into benchmarking this sort of stuff.

: 14 Sep 2007 00:36 |  Tags: Computer Free Multiprocessor Parallel Python Software |  Comments (1) |  Report a problem