September 14, 2007

Multi–Machine Parallel Python Benchmarks

Follow-up to Benchmarking Parallel Python Against Jython Threading (Benchmarks Take 3) from The Utovsky Bolshevik Show

Having claimed in a previous post that Parallel Python's ability to use the processing power of more than a single machine would work in its favour even when compared to the times for Jython threading, I thought I should probably look at some results to see if this is the case.

As previously, the benchmark being used is to sum all the primes beneath each multiple of 10000 between 100000 and 1000000.  The code examples can be found at http://oddbloke.uwcs.co.uk/parallel_benchmarks/

The Jython script uses Tim Lesher's cookbook recipe for a thread pool.  The Parallel Python script uses a slightly tweaked version of one of the examples on the Parallel Python site.

The two machines over which this is being tested are the University of Warwick Computing Society's servers, Backus and Codd, with Codd being used as the master server.  Both these machines have two CPUs.

The setup for the slave machine really is as easy as:

$ ./ppserver.py -p 35000 -w 2 

Once this was set up, I proceeded to test the Jython and Parallel Python scripts.  Disappointingly, the Jython script used more memory than I have available on my ulimit'ed account when running more than a single thread.  I have approximated based on the previous results I've had.


1 Worker
2 Workers
3 Workers
4 Workers
8 Workers
Jython Threading
289s
~150s
N/A
N/A
N/A
Parallel CPython (1 machine)
660s
352s
353s
351s
N/A
Parallel CPython (2 machines)
N/A
185s
180s
183s
188s


Looking solely at the numbers for Parallel Python, it seems that the speedup gained by using a second machine is significant.  It should be noted that Parallel Python's default regardless of whether or not it had the second machine available was 2 workers, so the automatic detection code is obviously sub-optimal.  It's trivial to override, so this wasn't a problem.

When this is compared to Jython's threading, it doesn't look significant but when we consider Jython's arithmetic ability and the fact that Parallel Python can continue to scale beyond this, Parallel Python begins to look better and better.  It should also be noted that, unsurprisingly, Jython uses a considerable amount more memory than CPython does.

EDIT: As pointed out in the comments, Jesse Noller has also started looking into benchmarking this sort of stuff.


- One comment Not publicly viewable

  1. Jesse

    Hey Daniel – I wanted to see if you wanted to join forces testing out some of this stuff – you should have my email from this comment, feel free to drop me a line. I’ve also started testing this stuff out

    14 Sep 2007, 01:25


Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

September 2007

Mo Tu We Th Fr Sa Su
Aug |  Today  | Oct
               1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

Search this blog

Tags

Galleries

Most recent comments

  • I should note that the talk info is available at http://linux.conf.au/programme/detail?TalkID=293 an… by on this entry
  • "I X'd you a Y, but I eated it by Lamby on this entry
  • Nice. Did not know it was that easy, I had a few problems to get it working some time ago. Have you … by mats on this entry
  • You can't make progress if you just argue, I try to be constructive. Cheers! by James on this entry
  • Thanks a bunch for doing this. I think that both Guido and Bruce Eckel are right; we really need peo… by Eli on this entry

Blog archive

Loading…
Not signed in
Sign in

Powered by BlogBuilder
© MMXXII