Favourite blogs for Jeff's Irregular Movements

My favourites » All entries

June 16, 2011

Partial and Curried Functions

On with the scala show! Partially-applied functions and curried functions took me a while to get my head around, but really they’re quite simple, and partials at least are super-useful.

Partially-applied functions are just functions where you pre-bind one of the parameters. e.g.

scala> val message:(String,String,String)=>String = "Dear " + _ +", " + _ + " from " + _
message: (String, String, String) => String = <function3>

scala> message("Alice","hello","Bob")
res24: String = Dear Alice, hello from Bob

scala> val helloMessage=message(_:String,"hello",_:String)
helloMessage: (String, String) => String = <function2>

scala> helloMessage("Alice","Bob")
res25: String = Dear Alice, hello from Bob

scala> val aliceToBobMessage=message("Alice",_:String,"Bob")
aliceToBobMessage: (String) => String = <function1>

scala> aliceToBobMessage("greetings")
res27: String = Dear Alice, greetings from Bob

What’s happening here is reasonably self-explanatory. We create a function “message” which takes 3 parameters, then we create two more functions, “helloMessage” and “aliceToBobMessage” which just are just aliases to the “message” function with some of the parameters pre-filled.

Since functions are first-class objects that you can pass around just like anything else, this means you can do stuff like

scala> val times:(Int,Int)=>Int = _*_
times: (Int, Int) => Int = <function2>

scala> val times2=times(2,_:Int)
times2: (Int) => Int = <function1>

scala> (1 to 10) map times2
res38: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)

Currying is, at first sight, just a different (more complicated) way to achieve the same thing:

scala> val message:(String)=>(String)=>(String)=>String = (to:String)=>(message:String)=>(from:String)=>{ "Dear " + to +", " + message + " from " + from}
message: (String) => (String) => (String) => String = <function1>

scala> message("Alice")("Hello")("Bob")
res28: String = Dear Alice, Hello from Bob

scala> val aliceToBobMessage=message("Alice")(_:String)("Bob")aliceToBobMessage: (String) => String = <function1>

scala> aliceToBobMessage("greetings")res29: String = Dear Alice, greetings from Bob

What that giant line of verbiage at the start is doing, is creating a function which takes one string and returns a function that takes another string, which returns a function that takes another string, which returns a string. Phew.
This mapping from a function that takes n parameters, to a chain of n functions that each take 1 parameter, is known as “currying” (After Haskell Curry). Why on earth would you want to do this, rather than the much simpler partial application example above?
Several of the core scala API methods use curried functions – for example foldLeft in the collections classes

def foldLeft[B](z: B)(op: (B, A) => B): B = 
        foldl(0, length, z, op)

- here we’re exposing a curried function, and then proxying on to a non-curried implementation. Why do it like this?

It turns out that curried functions have an advantage (see update below) that partial ones dont. If I curry a function, and then bind some variables to form what is effectively the same as a partial function, I can also modify the type of the remaining unbound parameters. Example required!

scala> val listMunger:(String)=>(List[Any])=>String = (prefix:String)=>(contents:List[Any])=>prefix + (contents.reduce(_.toString +_.toString))
listMunger: (String) => (List[Any]) => String = <function1>

scala> val ints = List(1,2,3)
ints: List[Int] = List(1, 2, 3)

scala> listMunger("ints")(ints)
res31: String = ints123

scala> val strings = List("a","b","c")
strings: List[java.lang.String] = List(a, b, c)

scala> listMunger("strings")(strings)
res32: String = stringsabc

scala> val stringListMunger=listMunger("strings")(_:List[String])
stringListMunger: (List[String]) => String = <function1>

scala> stringListMunger(strings)
res33: String = stringsabc

scala> stringListMunger(ints)
<console>:11: error: type mismatch;
 found   : List[Int]
 required: List[String]
       stringListMunger(ints)

That last error is the important bit. We’ve specialized the parameter type for the unbound parameter in stringListMunger so it won’t accept a list of anything other than Strings. Note that you can’t arbitrarily re-assign the type; it has to be a subtype of the original (otherwise the implementation might fail).
OK; so now all I have to do is think of a real-world example where this would be useful…

Update Gah, I was wrong. You can do exactly the same type-specialization with a partial:

scala> val x:(Int,List[Any])=>Int = (_,_)=>1
x: (Int, List[Any]) => Int = <function2>

scala> val y:(List[Int])=>Int = x(1,_)
y: (List[Int]) => Int = <function1>

So now I still have no idea when you’d want to curry a function, rather than just leaving it with multiple arguments and partially applying when required. This blog entry suggests that it really exists to support languages like OCaml or Haskel that only allow one parameter per function – so maybe it’s only in scala to allow people to use that style if they like it. But then what’s it doing in the APIs ?


June 11, 2011

Further Scala: Implicit conversions

My attempts to learn scala continue…

Implicit conversions are a cool, if slightly unsettling (from a java programmers POV) scala feature. If I have an instance of one class, and I try and call a method on it which is defined in a different class, then if there’s an “implicit” method in scope which will convert between the two, scala will silently use it.
e.g.

scala> var x = 12
x: Int = 12

scala> x.substring(1,2)
<console>:9: error: value substring is not a member of Int
       x.substring(1,2)

scala> implicit def foo(i:Int):String={i.toString}
foo: (i: Int)String

scala> 12.substring(1,2)
res10: java.lang.String = 2

WITCHCRAFT! BURN THE COMPILER!

This lends itself to a very very useful trick; the ability to enhance classes with additional methods. Say you had a java Map class, and you wanted the ability to merge it with another Map according to some sort of merge function on the values. You’d probably do it like this:

class MergeableMap implements Map{

public MergeableMap(Map delegate){
 this.delegate = delegate
}

public Map merge(Map otherMap, ValueMergingFunction mergeFunction){
 ....
}

... delegate implementations of all Map methods here...
}

Trouble is, (a) writing all the delegate methods is tedious, and(b) every time you want to use it, you’ve got to do

MergeableMap m = new MergeableMap(myMapVariable)
m.merge(myOtherMap,...)

Implicits in scala make this a lot easier:

class MergeableMap[A, B](self: Map[A, B]) {
  def merge(m1, merger): Map[A, B] = {
... implementation here...
  }
}

implicit def map2mergeableMap[A,B](m:Map[A,B]):MergeableMap[A,B] = new MergeableMap(m)

myMap.merge(myOtherMap, myMergeFunction)
myMap.get(...)

there’s no need to implement the other delegate methods, since we can just call them on the original Map class – but when we call merge() compiler-based voodoo works out that we want a mergeable map, and swaps it in for us. Magical.


June 09, 2011

Long Ranges in Scala

So, following on from yesterday’s prime-factoring annoyance; a more basic requirement: How to iterate from 1 to ~10^17? (i.e. close to the limit of Long)

A Range won’t cut it…

scala> val big_long=12345678901234567L
big_long: Long = 12345678901234567

scala> (1L to big_long)
java.lang.IllegalArgumentException: 1 to 12345678901234567 by 1: seqs cannot contain more than Int.MaxValue elements.

How about a BigIntRange?

scala> (BigInt(1) to BigInt(big_long))
java.lang.IllegalArgumentException: 1 to 12345678901234567 by 1: seqs cannot contain more than Int.MaxValue elements.
    at scala.collection.immutable.NumericRange$.count(NumericRange.scala:229)
    

OK, how about a stream?

scala>  lazy val naturals: Stream[Long] = Stream.cons(1, naturals.map(_ + 1))
naturals: Stream[Long] = <lazy>
scala>  naturals.takeWhile(_<big_long).find(_ == -1L)
Exception in thread "Poller SunPKCS11-Darwin" java.lang.OutOfMemoryError: Java heap space

hmm… running out of options a bit now. Maybe if I could construct a Stream without using the map() call (since I suspect that’s what’s eating up the heap), or use for-comprehension with an efficient generator?
Or…hang on…

scala> var i=0L
i: Long = 0
scala> while (i < big_long){i = i+1L}
{...time passes...exceptions are not thrown...}

So, it turns out that “the java way” works. Which, I guess, is the benefit of a mixed language; you can always fall back to the tried-and-tested solutions if the clever bits fail. And of course, you can hide the clunkiness pretty well:

 object LongIter extends Iterator[Long]{
    var l:Long = 0
    def hasNext:Boolean={true}
    def next:Long={
      l=l+1L
      l
    }
  }
  object LongIterable extends Iterable[Long]{
    val iter = LongIter
    def iterator = {iter}
  }

//ugliness ends, now we can pretend that LongIterable was there all along...

   LongIterable.find(_ == 12345678901234567L)

I suspect that this doesn’t really conform to the iterator/iterable contract (obviously, hasNext isn’t implemented), but it does appear to work tolerably well. Well enough, in fact, that I’m surprised I’ve not yet found a more idiomatic syntax for creating an iterator whose next() function is some arbitrary function on the previous value.

...reads source of Iterator ...

  Iterator.iterate(0L)(_+1L).find(_ == 123456701234567L)

phew. Finally.


June 08, 2011

Learning Scala, a step at a time

So, I have decided to make a more serious effort to learn scala . It fits almost all of my ‘ideal language’ features; it’s object-oriented and it’s statically typed (like java), it’s reasonably terse (unlike java) and it has good support for a more functional style of programming, without forcing you to use it all the time. It also runs on the JVM, which is a good thing if you care about observability, manageability, and all the other stuff that ops people care about.

The downsides, as far as I’ve found so far, are that (a) the tooling is nothing like as good as it is for java. The Eclipse Scala plugin appears to be the best the current bunch, but even there, support for things like refactoring (which is kind of the point of a statically-typed language) is pretty sketchy. And (b) it’s a bit of an unknown quantity in production. Yes, there are some big, high-scale users (Twitter spring to mind), but they’re the kind that can afford to have the best engineers in the world. They could make anything scale. Maybe scala will scale just as well in a more “average” operational environment, but there doesn’t seem to be a huge amount of evidence one way or another at the moment, which may make it a hard sell to a more conservative operations group.

On with the show. The best resources I’ve found so far for learning the language are:

- Daniel Spiewak’s series Scala for Java Refugees . Absolutely brilliant series of articles that do exactly what they say on the tin – put scala firmly into the domain of existing java programmers. Which brings me to….

- Tony Morris’ Scala Exercises for beginners. I had a hard time with these, not so much because they were difficult, but because coming from a non-comp-sci background, I found it hard to motivate myself to complete exercises as dry as “implement add(x: Int, y: Int) without using the ’+’ operator”. I also found the slightly evangelical tone of some of the other blog entries (TL;DR:”All you java proles are dumb”) a bit annoying. But working through the exercises undoubtedly improved my understanding of Scala’s support for FP, so it was worth it. On the Intermediate exercises though, I don’t even know where to start

- Akka Akka is an actor framework for scala and java, so it’s not really a scala resource per se but, it does such a great job of explaining actor-based concurrency that it’s worth reading though and playing with, because you’ll come away with a much clearer idea of how to use either Akka or Scala’s own Actors.

- Project Euler – A series of maths-based programming challenges – find the prime factors of a number, sum the even fibonaci numbers less than a million, and so on. Being maths-ish, they’re a good match for a functional style of programming, so I find it helpful to start by just hacking together a java-ish solution, and then seeing how much smaller and more elegant I can make it. It’s a great way to explore both the syntax, and the Collections APIs, which are at the heart of most of scala’s FP support. It’s also a nice way to learn ScalaTest (the unit testing framework for scala), since my hack-and-refactor approach is reliant on having a reasonable set of tests to catch my misunderstandings.
It’s also exposed some of my concerns about the operational readiness of scala. I came up with what I thought was a fairly neat implementation of a prime-factoring function; obtain the lowest factor of a number (ipso facto prime), divide the original by that number, and repeat until you can’t factor any further:

 lazy val naturals: Stream[Long] = Stream.cons(1, naturals.map(_ + 1))

  def highestFactor(compound: Long): Long = {
   naturals.drop(1).takeWhile(_ < compound/ 2).find(
        (compound % _ == 0)).map(
            highestFactor(compound/_)).getOrElse(compound)
  }

(source)
– which works beautifully for int-sized numbers, but give it a prime towards the upper bounds of a Long and it runs out of heap space in the initial takeWhile (before it’s even had a chance to start recursively calling itself). It seems that Stream.takeWhile doesn’t permit the taken elements to be garbage-collected, which is counter-intuitive. I wouldn’t be at all surprised to find that I’m Doing It Wrong, but then again, shouldn’t the language be trying hard to stop me?


Java String.substring() heap leaks

Here’s an interesting little feature that we ran into a short while ago…

Suppose I have something like a Map which will exist for a long time (say, an in-memory cache), and a large, but short-lived String. I want to extract a small piece of text from that long string, and use it as a key in my map

Map<String,Integer> longLivedMap = ...
String veryLongString = ...
String shortString = veryLongString.subString(5,3);
longLivedMap.put(shortString,123);

Question: How much heap space have we just consumed by adding “abc”=>123 into our map? You might think that it would be just a handful of bytes – the 3-character String, the Integer, plus the overhead for the types. But you would be entirely wrong. Java Strings are backed by char arrays, and whilst the String.subString() method returns a new String, it is backed by the same char array as the originating String. So now the entire veryLongString char[] has a long-lived reference and can’t be garbage collected, even though only 3 chars of it are actually accessible. Rapid heap-exhaustion, coming right up!

The solution is pretty straightforward; if you want to hold a long-lived reference to a string, call new String(string) first. Something like

String shortString = veryLongString.subString(5,3);
longLivedMap.put(new String(shortString),123);

It would be counter-productive to do this on every substring, since most of the time the substring and the original source will go out of scope at the same time, so sharing the underlying char[] is a sensible way to reduce overhead.

Since you (hopefully) won’t have many separate places in your code where you’re storing long-lived references like this (just cache implementations of one sort or another), you can create the new strings inside the implementation of those classes instead. Now your caches are all light, and your heap is all right. Good Night.


April 07, 2011

31 Films In 31 Days

So I decided to watch 31 films in March. Deciding to this on the 5th meant I had to cram them in at various points, so I would advise if you do do this to either start on the 1st of the month, or pick a shorter month like February, June or Glovember. Oh, and don’t watch five Almodovar films in two days, it really messes with your perception of reality.

The films:

The Philadelphia Story (1940)
Dr Strangelove (1964)
Dark Star (1974)
Warriors (1979)
Pepe Luci Bom (1980)
Dark Habits (1983)
Blood Simple (1984)
Tie Me Up Tie Me Down (1990)
Desperado (1995)
Live Flesh (1997)
Taxi (1998)
Bend It Like Beckham (2002)
Hero (2002)
Sky Blue (2003)
Lost In Translation (2003)
Appleseed (2004)
Saving Face (2004)
Metal: A Headbanger’s Journey (2005)
Helvetica (2007)
Elite Squad (2007)
The Duchess (2008)
Gomarrah (2008)
Wall E (2008)
Sunshine Cleaning (2008)
A Prophet (2009)
Sherlock Holmes (2009)
Inglorious Basterds (2009)
Up (2009)
The Infidel (2010)
RED (2010)
Submarine (2011)

31 films, 31 days, ten conclusions.

1. European crime films are much harder hitting than American ones, but American ones have more guns.
2. Pixar make amazingly beautiful films with excellent plots.
3. Amazingly beautiful Japanese/Korean animations usually have a pile of fluffy sci fi bollocks where the plot should be.
4. Katharine Hepburn is amazing.
5. Helen Mirren + machine gun = good times.
6. Eleven of the films were subtitled, three were partially subtitled, Dark Star was so mumbly it should have had subtitles, and Wall E showed dialogue wasn’t entirely necessary anyway.
7. Documentaries about typefaces can be fun.
8. Keira Knightley isn’t that irritating when she’s allowed to be sassy.
9. Three of the films featured directors acting, although I would never voluntarily watch a film directed by Eli Roth.
10. Submarine has the best mullet ever in it.


March 16, 2011

Bone Marrow IV – The Five Year Gap

Sign up to donate marrow here – www.anthonynolan.org/

Later in 2006

The razor sharp amongst you might have noticed that much of this story so far took place in 2006. Signing up, getting told I was a partial match, and giving test samples of blood all took place over a relatively short period five years ago. And then… nothing.

I didn’t donate marrow in 2006.

After the samples were sent off I waited and waited, albeit in a very passive sort of way. As so often in the process I simply shoved it to the back of my mind, although it crept forward occasionally. Each step forward was harder to shake, harder to ignore. Signing up to a register is ultimately a very impersonal act, but getting tested to see if you’re a match to another person, there’s nothing abstract about that. Somewhere out there is a person who is sick, and though I don’t know them, I’m not so totally incapable of empathy that I can’t appreciate that theirs is a desperate state to be in.

No one I am close to has, to my knowledge, had leukaemia. People have had other cancers. It’s shit, shit and unfair, and strikes people who don’t deserve it, don’t need the pain and the danger. But I don’t need to tell you that. You know. It makes people very very ill, and both the treatment and the disease itself are harsh and painful.

And so it was that when I got the letter, I assumed the worst.

I was never officially told I was the match. There was no phone call or piece of paper saying “Congratulations, your jelly is just the flavour our party requires”. Or perhaps there was and I managed to miss it. The tendency of students to move around constantly, have a new address every few months means post is often lost or left on the floor of a house you used to live in, whilst the incumbents walk past it daily, each time promising they’ll sort it out tomorrow. I don’t know. I do know I got the follow up letter.

The follow up letter was addressed to a match. It was addressed to me and presumed I knew I was the match, that I knew I was the one with the marrow that offered the most. But it wasn’t telling me to get ready or to start preparing to produce stem cells or any of that. It was a relatively short letter which seemed to speak of a tragedy I was only peripherally aware of. It informed me that the transplant recipient was too ill to have the transplant.

Everyone I have ever told this story to reacts in the same way at this point. There’s something in those words which strikes a chord with people. The great unspoken between the lines. If you are ill enough to require a life-saving transplant, then you are really ill. If you are too ill to have a life-saving transplant, then things must be even worse than that. Without the bone marrow, went my thought process, this person will die, but they cannot have the bone marrow. Not unreasonably I assumed they had died.

Three people a day die in the UK because they need a transplant and the organs (not just bone marrow) aren’t available. It’s a statistic used in drives for organ donation because it says nothing of the personal tragedies behind the stat, but it doesn’t need to. It’s a lot of people, and some of them may well have lived long lives if they’d had a chance.

It didn’t feel good to have that letter. Neither myself nor the people at Anthony Nolan had done anything wrong, hell, no one had done anything wrong. The enemy was that blasted disease, and it seemed it had outpaced us all.

March 2011

I sometimes tell the story of the bone marrow donation which wasn’t. Ok, I tell a lot of stories, I talk a lot. But that one is one I tell because I kind of hope it might have a positive impact. It’s not particularly funny and it had a sad ending. It won’t get people into a partying mood, or raise spirits, or make good dinner conversation. But it sometimes felt right to tell it because they need those men and those non-whites to sign up. If my experience was anything to go by they need white girls too.

I told the story again in the middle of February 2011. Just another recital. Nothing too serious. Then on the evening of 28th February my phone decided to play silly buggers and refused to receive any calls. I didn’t know this until I got home from work to find it hadn’t rung and someone who had then left a message. The message started somewhat unexpectedly, “Hello, this is a message for Holly Cruise, this is [name] from the Anthony Nolan register, the bone marrow register, can you please give me a call on [number] tomorrow, thank you”.

Twice? The white girl no one wanted was needed again? For another person? Was I a really good match cos Celtic women were particularly prone to leukaemia or was it coincidence?

Naturally the next day couldn’t come soon enough, I needed to know what was going on. So I rang back and spoke to the lady from Anthony Nolan. I was needed. I was a match in 2006 (which I’d worked out) and I was needed because the patient was sick again and needed the transplant this time.

Wait. The patient was sick again. The girl (I think I heard them referred to as “she” during the conversation) had been too ill for a transplant five years earlier, but was now in need? Had she been that critically ill for five years, too sick for a transplant the whole time? Had she recovered suddenly to the point of not needing the marrow but had now slid back into a bad state? I wouldn’t wish either on anyone, although I hoped it was the latter. Five years of being critically ill is a nightmare no one deserves.

It wasn’t even a question really when the lady asked if I was still willing to go through with it. Of course I was! I thought this girl was dead. I didn’t even know her and I was delighted she wasn’t. I would definitely help out. Definitely give what I could, which in this case was more than just platitudes and wishes, it was actual, useful jelly. Was I willing to help? Yes/ Was my health still good? Yes. Was I aware of what it would involve? Yes (stabbity stabbity).

The lady said it was up to the surgeon now to set the dates. I would receive more information in time. She emailed me the information leaflet. I read it, then panicked that I’d made a mistake and that the liver ‘condition’ I was diagnosed with in 2008 might be a problem. Yeah, I’m a teetotalitarian with a liver condition. Said condition is Gilbert’s which has virtually no symptoms and rarely impacts on the lives of those who have it, but I was worried it might impact here. I needn’t have worried, it wasn’t a problem.

So now I’m just waiting for dates. I keep thinking maybe I should tell them when I’m on holiday, or that if illness on my part will cause problems that we need to avoid September/October, the months which take quite a toll on those who work at universities, as the students come from all over the world bringing as many versions of the cold/flu as they can muster. I even decided to write about it, here, because I thought it might be interesting.

I don’t know if I will get as far as donating this time. I hope so if only because I don’t want this mystery person to die. It could be months before anything happens, I’m sorry I have no more narrative than this to give you at this time. But if anything more comes up I will write about it. There might even be pictures this time, or at least pictures of the actual process rather than the slightly random ones I’ve used so far. I might use the word “stabbity” a bit more as it has proven strangely popular. Or I might not. It might go no further once more. But in any case, I want at least one of you reading this to sign up to the register. Please.

It’s the least we could all do.


March 10, 2011

QCon day 1

A lot of good stuff today, but I’m just going to jot down a couple of my favourite points:

Craig Larman talked about massive-scale Scrum-based projects. I don’t suppose I’m going
to be running (or even part of) at 1500-person dev team very much, but some of his points
are applicable at any scale:
  • The only job title on the team is “Team Member”. There might be people with specialist skills,
    but no-one can say “it’s not my job to fix that”
  • If you don’t align your dev. teams’ organisation with your customers, then Conway’s law means your
    architecture will not align with your customers either, and you won’t be able to react when their needs
    change
  • Don’t have management-led tranfoormation projects. How will you know when they’re done? Instead,
    management’s role is just to remove impediments that the dev team runs up against – the “servant-leader”
    model
Dan North spoke about how he moved from what he thought was a pretty cutting edge, agile environment
Thoughtworks, consulting to large organisations starting to become leaner/more agile) to a really agile
environment (DRW, releasing trading software 10s of times per day), and how if you have a team that
is technically strong, empowered, and embedded in the domain (i.e. really close to the users), you can do
away with many of the traditional rules of Agile. A couple that really struck me were:
  • Assume your code has a half-life. Don’t be afraid to just rewrite it, or bin it The stuff that stays in
    can get better over time, but it doesn’t have to be perfect from day 1
  • Don’t get emotionally attached to the software you create. Get attached to the capabilities you enable
    for your users
  • Remeber, anything is better than nothing.
Juergen Hoeller talked about what’s new in Spring 3.1. No amazing surprises here, but some nice stuff:
  • Environment-specific beans – avoid having to munge together different config files for system-test vs.
    pre-production vs. live, have a single context with everything defined in it (Even nicer, arguably, when you
    do it via Java Config and the @environment annotation)
  • c: namespace for constructor args. Tasty syntactic sugar for your XML, and the hackery they had to go through
    to get it to work is impressive (and explains why it wasn’t there from the start)
  • @cacheable annotation, with bindings for EHCache and GemFire (not for memcached yet, which is a bit of a surprise)
Liz Keogh talked about perverse incentives. Any time you have a gap between the perceived value that a metric
measures, and the actual value that you want to create, you make an environment where arbitrage can occur. People can’t
help but take advantage of the gap, even when they know at some level that they’re doing the “wrong thing”.
  • Focus on the best-performing parts of the organisation as well as the worst-performing. Don’t just say “This
    project failed; what went wrong”; make sure you also say “This project succeeded better than all the others; what went right?”
  • Don’t try and create solutions to organisation problems, or you’ll inevitably make perverse incentives. Instead, make
    make Systems (Systems-thinking, not computer programs) that allow those solutions to arise.
Chris Read and Dan North talked about Agile operations. Surprisingly for me, there wasn’t a great deal of novel stuff
here, but there were a couple of interesting points:
  • Apply an XP-ish approach to your organisational/process issues: Pick the single biggest drag on you delivering value and
    do the simplest thing to fix it. Then iterate.
  • Fast, reliable deploys are a big force multiplier for development. If you can deploy really fast, with low risk,
    then you’ll do it more often, get feedback faster, allow more experimentation, and generally waste less. The stuff that Dan
    and Chris work on (trading data) gets deployed straight from dev workstations to production in seconds; automated testing
    happens post-deploy

Yoav Landman talked about Module repositories. Alas, this was not the session I had hoped for; I was hoping for some
takeaways that we could apply to make our own build processes better, but this was really just a big plug for Artifactory,
which looks quite nice but really seems to solve a bunch of problems that I don’t run into on a daily basis. I’ve never needed
to care about providing fine-grained LDAP authorisation to our binary repo, nor to track exactly which version of hibernate was
used to build my app 2 years ago. The one problem I do have in this space (find every app which uses httpclient v3.0,
upgrade, and test it) is made somewhat easier by a tool like Artifactory, but that problem very rarely crops up, so it
doesn’t seem worth the effort of installing a repo manager to solve it. Also it doesn’t integrate with any SCM except Subversion,
which makes it pretty useless for us.


March 09, 2011

Bone Marrow III – The Big Needle

Need I include the disclaimer that I don’t normally write about my real life? Well, I don’t, but I don’t normally let people take my internal organs out for use elsewhere…

Sign up to the registerhttp://www.anthonynolan.org/

2006, summer and 2011, March

People are fascinated by the Big Needle. The Big Needle is a symbol. The Big Needle is a celebrity. The Big Needle is only part of the bone marrow donation process anyone seems to know about. Both in 2006 and now in 2011 it’s what I hear most often from people when I tell them what I am doing.

“Doesn’t that involve a Big Needle?”
“I hear it hurts when they use the Big Needle.”
“Wow, I wouldn’t do that, not with that Big Needle.”


Stabbity.

The Big Needle appears to be the iconic element of the process, a sharp, stabby certainty, headed straight for the hip bone of those willing to give. I knew about the Big Needle when I signed up for the process, but like all the unpleasant elements of donation I blithely ignored it because I really didn’t think I would ever be called upon to donate and thus worrying about the specifics wasn’t worth it. I do a similar thing with my legs – I have one leg longer than the other and my hips are out of alignment. A doctor once told me there was a good chance I might need a hip replacement at a relatively young age (40s?!) but they weren’t sure. They could be right, but there’s no point fretting about it if it’s not certain. And merely being on the bone marrow list wasn’t a guarantee that Big Needle would be part of my life so I ignored it.

And when I couldn’t ignore it anymore, I discovered that as with so many icons, the Big Needle had lost relevance in recent years.

Ever had tuberculous cervical lymphadenitis? Probably not, it’s more commonly known as scrofula, and is the sort of disease which modern historians get to read about and modern doctors don’t get to treat very often. When it does pop up it gets beaten down with antibiotics. In the old days the cure was more elaborate, the monarch of the day was required to touch the sufferers. Scientific advances ensure that we don’t need Elizabeth II to high five the contagious, and funnily enough scientific advances have resulted in the Big Needle losing some influence in the world of bone marrow extraction.


“I cannot believe I have to touch you scrubbers. Helen Mirren wouldn’t stand for this shit.”

I found this out courtesy of the booklet Anthony Nolan sent me. It was full of happy smiling pictures of donors and recipients. How can anyone refuse a process where the outcome is glossy smiling photos, complete with inspirational captions? Ok, I’m being flippant, but these booklets will have been market tested and designed to perfection, they are precision tooled to make the donor feel less apprehensive about the process, to banish Big Needle.

There are two ways of donating.

I didn’t know this. The other way was even presented as the preferred method. Above Big Needle. The fearsome beast wasn’t even first choice, it was sat on the medical substitute’s bench, kicking its heels and periodically running up and down the side of the hospital, waiting to be called on. A plan B.

Plan A was much more futuristic. The donor would be injected with human growth hormone to make their marrow overproduce stem cells. The spongy mass of the marrow is less important than the tiny parts of marrow yet to come. After a few days the recipient would travel to a hospital where they would be hooked up to a machine, one tiny needle in each arm, with blood drawn out via one tube, filtered through a machine to remove the stem cells, before being put back into the donor’s body via the second tube in the other arm. I may or may not have made this up, but I came away convinced the process involved a centrifuge, with the prospect that my blood would be spun around, with stem cells flying out of it into the collection point, before being returned. The idea that my vital fluids might enjoy a ride as part of the process sounded fun.

The Big Needle was for when this didn’t work. No problem, I thought, if it happens I’ll just produce lots of stem cells. I’ll sit at home, producing really hard. In 2006 I figured I could study in the library, but produce stem cells at the same time. Essays and stem cells. Not a problem, and you don’t even have to footnote the stem cells. Now, in 2011, I’ll produce them while I work, or while I stand outside as the fire alarms go off, a strangely frequent event at work right now.

And hey, even if the Big Needle was needed, there was general anaesthetic for that.

stabbyneedle
Actually, the Big Needle is pretty big when you put it like that…

Knowing the procedures made them less scary. Is this what my ancestors felt the first time they found a dead boa constrictor and thought “Wow, I know it’s eaten half the tribe, but look, it’s just a tube with a silly face, that’s not scary”. Mind you, my ancestors ended up in Ireland, one of only two countries worldwide with no native snake species, so perhaps that’s a bad example.

Whatever, even if it did involve the Big Needle the equation was simple: stabby needle-induced hip pain for a week is less painful than dying of leukaemia. There is no other way to look at this. If I was a match I would do it, no matter what the short term pain. The person who needed marrow needed it more than I needed a pain free week.

But I would probably whine about the pain anyway. I’m not a saint.

And in the ebbing days of summer 2006, the last really good summer I remember us having in Britain, I waited to see if that one-in-four chance was going to come in. Was I the match? Would I meet the Big Needle or have to sit in the library producing stem cells?


March 08, 2011

"Designing Software, Drawing Pictures

Not a huge amount of new stuff in this session, but a couple of useful things:

The goal of architecture: in particular up-front architecture, is first and foremost to communicate the vision for the system, and secondly to reduce the risk of poor design decisions having expensive consequences.

The Context->Container->Component diagram hierarchy:

The Context diagram shows a system, and the other systems with which it interacts (i.e. the context in which the system operates). It makes no attempt to detail the internal structure of any of the systems, and does not specify any particular technologies. It may contain high-level information about the interfaces or contracts between systems, if
appropriate.

The Container diagram introduces a new abstraction, the Container, which is a logical unit that might correspond to an application server, (J)VM, databases, or other well-isolated element of a system. The container diagram shows the containers within the system, as well as those immediately outside it (from the context diagram), and details the
communication paths, data flows, and dependencies between them.

The Component diagram looks within each container at the individual components, and outlines the responsibilties of each. At the component
level, techniques such as state/activity diagrams start to become useful
in exploring the dynamic behaviour of the system.

(There’s a fourth level of decomposition, the class diagram, at which we start to look at the high-level classes that make up a component, but I’m not sure I really regard this as an architectural concern)

The rule-of-thumb for what is and what isn’t architecture:

All architecture is design, but design is only architecture if it’s costly to change, poorly understood, or high risk. Of course, this means that “the architecture” is a moving target; if we can reduce the cost of change, develop a better understanding, and reduce the risk of an element then it can cease to be architecture any more and simply become part of the design.


March 07, 2011

Five things to take away from Nat Pryce and Steve Freeman's "TDD at the system scale" talk

  • When you run your system tests, build as much as possible of the environment from scratch.
    At the very least, build and deploy the app, and clear out the database before each run
  • For testing assemblies that include an asynchronous component, you want to wrap
    your assertions in a function that will repeatedly “probe” for the state you want
    until either it finds it, or it times out. Something like this
       doSomethingAsync();
       probe(interval,timeout,aMatcher,anotherMatcher...);    

    Wrap the probe() function into a separate class that has access to the objects you
    want to probe to simplify things.

  • Don’t use the logging APIs directly for anything except low-level debug() messages, and maybe
    not even then. Instead, have a “Monitoring” topic, and push structured messages/objects onto
    that queue. Then you can separate out production of the messages from routing, handling, and
    persisting them. You can also have your system tests hook into these messages to detect hard-to-observe state changes
  • For system tests, build a “System Driver” that can act as a facade to the real system, giving
    test classes easy access to a properly-initialised test environment – managing the creation and
    cleanup of test data, access to monitoring queues, wrappers for probes, etc.
  • We really need to start using a proper queueing provider

Bone Marrow II – Twelve Small Pots

Once again I must protest that I’m not keen on blogging about my actual life but after the first of these posts it turned out a friend had made it to this stage too, so maybe it’s more common than I first thought. I dunno, it’s never been the topic of any polite dinner party conversation I’ve been part of to talk about marrow donation, although I’ve been to about two polite dinner parties in my life (once by mistake) so it could be common.

Do not lose sight of the importance of this linkhttp://www.anthonynolan.org/

2006, a few weeks later

I’m scared of donating blood. Yes, I know this is about the other donation, the one with the HUGE needles, and I know donating blood involves small needles. I have no issue with needles. I wasn’t the kid in my secondary school who reacted to their BCG jab by turning green and passing out. I used to watch the injections as they went into my arm, fascinated by the vanishing liquid. No, it’s not the mechanics of blood donation which bother me.

No one knows what causes migraines for sure. Like all migraine sufferers I know roughly what triggers them, but how extreme stress or not-so-extreme food deprivation translate into blindness, splitting headaches and projectile vomiting is a mystery. All a doctor has ever been able to say to me is that it might be related to chemical levels in the blood.

So far, despite my massive clumsiness I’ve never lost a significant amount of blood in any incident. Even the time I sliced my thumb in half (yes, I am an idiot, I know this) it wasn’t a bloody as it could have been. Well, it didn’t cause my blood-phobic housemate to pass out, so it can’t have been too bad. It wasn’t Ichi The Killer levels. The upshot of this rambling is to say that I’ve never donated blood because I’m worried it might set off migraines. It’s irrational, it probably won’t, but I don’t know. Perhaps once I’ve given marrow I should make it a target to give blood within six months to prove it. Who knows.

Thumb and Blade
The aftermath of my thumb slicing incident. Blood all cleaned (except the stuff we couldn’t clean so we put furniture over it so the landlord wouldn’t notice.

But my other worry is I don’t have enough blood. And this is where I find myself back in the summer of 2006.

When people need bone marrow donations their family is screened first. It makes sense, you’re more likely to find a match with close relatives than random strangers* like us names bobbling around on the Anthony Nolan register. But sometimes it doesn’t work out and then it’s pot luck. To the register. To sign up we gave small pots of blood, not enough to run more than a couple of tests. So when a person in need of a transplant comes to the register they tend to test a few partially matching people more closely to find the best match.

It had been only a few months since I had left the students’ union after signing up. I’d already put it to the back of my mind and can you blame me? I was a white female. They didn’t need white females, they’d said so. Had there been any men or any non-white people I would have been sent away. I’d have gone home to Leamington and wondered why I still had no feeling in my thumb (the sliced in half one, the feeling still isn’t more than 40% today, five years later). I’d have watched TV, done my essay, eaten. I wouldn’t have been one of the four people on the register that were a rough match to an ill person somewhere in Britain.

I laughed when I found out I was a partial match. “They said I wasn’t needed!” I told blood-phobic housemate, “And here I am, a partial match”. It was a one in four chance I might be the best match, which are good odds if you’re betting on horses and bad odds if you’re betting on football (yeah, I worked in a bookies for a while).


Like these but twelve of them.

To find the best match involves tests. Tests require blood. I was sent a box of twelve small pots, like the first small pot from signing up day. Only this time there was twelve of them. I took my box of goodies to the doctor’s clinic to be filled and the nurse laid them out. They had different coloured tops. They looked like pieces from a strange boardgame which only served to make it even more surreal as the nurse put a needle in my arm and filled them one by one until all were fu… no, wait. The last three or so weren’t full. We couldn’t fill them. She prodded my arm once or twice but for the last three (look away squeamish folk) my blood merely dripped in, rather than leaping forward with enthusiastic abandon as it had done initially. The nurse smiled and assured me it was normal, and maybe it was. Maybe twelve pots is a lot to ask. But my irrational side was in its element, “Look! You can’t possibly donate blood, you don’t have enough for yourself! How could someone as pale as you possibly have enough blood”. I am very pale it’s true. Even other Irish people think I’m pale. I tried to ignore the irrational voice, especially as the nice, rational nurse said it was quite normal not to fill all twelve pots.

My blood was posted to London to be prodded and probed. I didn’t have a migraine. I didn’t feel any change at all, which at least reassured me that no one was performing any voodoo on my platelets which was nice of them. So I went back to work. It was summer and time to work my arse off because that’s the lot of many students these days. Work work work whilst they bleed you dry. At least I’d been bled (slightly) dry for a slightly better cause.

And now it was time to wait and see who was the best match…

*Reading about family matches and donations I found the fascinating case of Karen Keegan, an American who needed a kidney transplant. When her children were tested to see if they were matches the results said they weren’t even her children. It turned out she’s a chimera, one person with two sets of DNA made when two fraternal twins fuse in the womb to grow as one person. Her womb and her blood were produced from different embryos but lived in one body. Biology is interesting.


March 04, 2011

Ideas For New Bank Holidays

Writing about web page http://www.bbc.co.uk/news/uk-12640636

The government wants to move the May Day bank to either St George’s day a week or so before, or to ‘Trafalgar Day’ in October. Now there are some who might suggest that targeting the May Day holiday, a strongly socialist and pagan linked date, rather than the other May bank holiday is a bit ideological. Perhaps it isn’t. but it ignores one simple and undisputable fact – we don’t get enough days off compared to the rest of Western Europe.

The TUC have complained about this for years, saying in 2001 that with a European average of 10.8, Britain still only had 8, a number which would have been part of the European average, thus dragging it down.

So why not let us have another bank holiday without ditching an existing one? Some suggestions:

Pankhurst Day, 14th or 15th July

“Votes for my homegirls or else!”

Emmeline Pankhurst’s birthday. Why not pick a suffragette? Was her struggle, and those of the 50% of the population she fought alongside and for, any less significant than that of Nelson at Trafalgar? The uncertainty of her birth date means we can choose, shall we synchronise with France or strike out on our own? Plus mid-July is a good bet for some half decent weather.

Armada Day, 18th August

“Michel Pez said the weather would be lovely, what an idiot!”

Why Trafalgar Day? Why not another battle? Most WWI and WWII events were over longer periods of time, or are rather more international in flavour, so why not pick a military event which really sums Britain up – the Spanish Armada whose defeat was a combination of British blood mindedness and the weather. Mostly the weather. 18th August is the New Calendar date of Elizabeth I’s “heart and stomach of a king” speech, which is one of the best in British history. The downside is it would be quite close to the existing August holiday.

Reform Act Day, 7th June

George Cruickshank’s depiction of the Peterloo Massacre, a significant event in the history of democracy in Britain (and a possible contender for a bank holiday in its own right).

We don’t really do revolution in Britain, do we? The one time we did, overthrowing Charles I and the ructions that lead to Cromwell taking over, it was reversed within a few decades. Britain has long been the country of incremental changes, so why not celebrate one of the most major incremental changes? The Reform Act 1832 wasn’t perfect, it left a lot of people still disenfranchised (women and poor people mostly), but it was a significant step away from the still borderline feudal world we lived in, towards a slightly more democratic future. We could celebrate the day by going to Old Sarum and pelting effigies of crooked politicians with eggs.

First Association Football International Day, 30th November

Scotland’s 1872 team, great moustaches.

If we want major events that happened in the last three months of the year, currently underserved by bank holidays, forget Guy Fawkes Day (might upset Catholics), Halloween (might upset non-pagans/people who hate costumes and having their houses egged), or my birthday (apparently I am not significant enough, humph). What we need is something which celebrates Britain’s biggest cultural contribution to the world. Which celebrates British innovation and ability to apply rules to anything, even something which essentially used to be an excuse for a fight. Which celebrates the way England and Scotland just love to get one over on each other. The first ever official football international match, a 0-0 draw between England and Scotland. Considering the game involved men with amazing names like Cuthbert Ottaway and Reginald de Courtenay Welch, it seems rude not to celebrate it. (Wales and Northern Ireland can have a day off on the day of the first Wales vs Northern Ireland match).

Any other similarly genius/insane suggestions?


March 03, 2011

Bone Marrow I – Signing That Register

I don’t normally blog about my personal life for the simple reason that my personal life is really rather dull. But at the moment it isn’t. It’s not extraordinary, but it’s not dull, and I think it needs to be told because what I’m currently doing could and should be done by others. Maybe even you.

This is the (sort of) diary of a (potential) bone marrow donor.

These are the people you need: http://www.anthonynolan.org/

2006, one afternoon

Campus universities are a great source of bodily fluids it would seem. No, get your mind out of the gutter; I am referring to substances like blood and marrow. At one stage during my time at the University of Warwick it was a weekly occurrence to see the blood donation lorry parked in the grounds waiting for passing students to drop in. A worthy cause, it almost seemed a shame that they would come on Tuesdays as many of the sports teams played on Wednesdays and I knew of more than one sporty student who wanted to donate but didn’t want to do so on a Tuesday – “I can’t be weak for the match!”. You would sometimes also see members of Warwick Pride protesting at the ban on blood donations by men who have had sex with men, often accompanied by a groups of lesbians who would go and give their blood, whilst commenting loudly that their male friends couldn’t. It was a good way to protest.


From www.warwick.ac.uk

It wasn’t just the blood people who came to visit. Anthony Nolan would come too.

Anthony Nolan is a charity that supports research into leukaemia, and which runs a register for bone marrow donors. As with all such charities it relies on volunteers to join its register and be matched up with sufferers of leukaemia and other blood disease who are in need of a transplant. Interestingly, in light of what I observed about the blood donation truck at Warwick, Anthony Nolan allow gay men to join the register and donate with the same restrictions as heterosexuals. As one Anthony Nolan representative said “If they are practising safe sex with a trusted partner then we are more than happy to accept them.”

And so it was one afternoon that Anthony Nolan nurses took over a couple of rooms in the students’ union at Warwick.

Students make a great target market for such drives. One could flatter the altruistic nature of students by saying their increased social awareness and tendency towards idealism makes it likely that they’d sign up for this sort of thing, to positively intervene in the lives of others. One could also point out that they have a lot of free time and a tendency to be distracted by anything which looks vaguely interesting and unusual, like a queue with some nurses at the end of it. For me, it was a combination of the two.

su
Warwick Students’ Union, image from Wikipedia

There was a queue of people in the students’ union building. I was on a free afternoon (in the sense that I probably should have been in the library somewhere, reading something important and relevant and historical) and with mates. A queue leading to a room which wasn’t usually occupied was an invitation to go and join it. Is there any act more British than joining a queue just because it’s there? Who knows, I’m only half British so maybe I should have only half joined it.

It turned out to be a queue to sign up for the Anthony Nolan bone marrow register. At this point we could have wandered off again, in search of queues with less bloodletting at the end of them. But we didn’t. Call it that student conscience or whatever, but we decided a bit of queuing, form filling and blood sample giving was a good use of our time.

By this stage it was late in the afternoon, and the nurses were anxious that they would have to leave soon. One stuck her head out of the door to the room and announced to the queue that they were finishing soon, and that they would be prioritising certain groups of people.

Men and ethnic minorities.

She explained briefly that there weren’t enough of either on the register, and that they were wanted for this reason. From their website I later discovered the following:

It is a priority for us to recruit more male donors because men can generally provide greater volumes of blood stem cells than female donors. This helps faster engraftment and the reconstitution of the immune system post-transplant. If there is a choice of donor for a patient in most cases a male donor will be preferred.

Ethnic origin is important when matching donors and patients. The ‘markers’ that are tested when searching for a suitable stem cell donor are genetically inherited and often unique to a particular race. A patient in need of a transplant is more likely to discover a suitable donor amongst groups of people who share a similar genetic history to them. In practice this means that an African-Caribbean patient, for example, has the greatest opportunity of finding a donor within his or her own ethnic community. There are still too many patients in the UK from black and ethnic minority communities for whom we are unable to find a compatible donor.

The nurse looked up and down the queue. A queue made up entirely of white females. Resigned to the fact she couldn’t get any more of the rarer groups that day she indicated for the next girl to come into the room. Jokingly I turned to my mate and wondered if being Celtic (as in I have three Irish grandparents, not that I am a football team) was enough of an ethnic minority to make a difference. Probably not we concluded. There’s so many British people with an Irish grandparent that it must be pretty common. How else do you explain the 1994 Irish World Cup squad?


Ray Houghton scores for Ireland against Italy at 1994 World Cup. He was born in Scotland. Watch the glorious video here.

The rest of the process was speedy. In we trooped, one by one, to give a small sample of blood and fill in some forms to say we weren’t injecting heroin into our eyes and having unsafe sex with as many hookers as we could (even though the University of Warwick is located in Coventry, it’s hardly close to any particularly mean streets). And we left. It didn’t feel particularly significant. The whisperings about how they get marrow (“a HUGE needle”) didn’t really seem that pressing. The lady had said they weren’t really looking for white females and we were all white females. They wouldn’t need us, surely?

Surely?


February 24, 2011

Solaris IGB driver LSO latency.

Yes, it’s another google-bait entry, because I found it really difficult to find any useful information about this online. Hopefully it’ll help someone else find a solution faster than I did.

We migrated one of our applications from an old Sun V40z server, to a newer X4270. The migration went very smoothly, and the app (which is CPU-bound) was noticeably faster on the shiny new server. All good.

Except, that when I checked nagios, to see what the performance improvement looked like, I saw that every request to the server was taking exactly 3.4 seconds. Pingdom said the same thing, but a simple “time curl …” for the same URL came back in about 20 milliseconds. What gives?
More curiously still, if I changed the URL to one that didn’t return very much content, then the delay went away. Only a page that had more than a few KBs worth of HTML would reveal the problem

Running “strace” on the nagios check_http command line showed the client receiving all of the data, but then just hanging for a while on the last read(). The apache log showed the request completing in 0 seconds (and the log line was printed as soon as the command was executed).
A wireshark trace, though, showed a 3-second gap between packets at the end of the conversation:

23    0.028612    137.205.194.43    137.205.243.76    TCP    46690 > http [ACK] Seq=121 Ack=13033 Win=31936 Len=0 TSV=176390626 TSER=899836429
24    3.412081    137.205.243.76    137.205.194.43    TCP    [TCP segment of a reassembled PDU]
25    3.412177    137.205.194.43    137.205.243.76    TCP    46690 > http [ACK] Seq=121 Ack=14481 Win=34816 Len=0 TSV=176391472 TSER=899836768
26    3.412746    137.205.243.76    137.205.194.43    HTTP    HTTP/1.1 200 OK  (text/html)
27    3.412891    137.205.194.43    137.205.243.76    TCP    46690 > http [FIN, ACK] Seq=121 Ack=15517 Win=37696 Len=0 TSV=176391472 TSER=899836768

For comparison, here’s the equivalent packets from a “curl” request for the same URL (which didn’t suffer from any lag)

46    2.056284    137.205.194.43    137.205.243.76    TCP    49927 > http [ACK] Seq=159 Ack=15497 Win=37696 Len=0 TSV=172412227 TSER=898245102
47    2.073105    137.205.194.43    137.205.243.76    TCP    49927 > http [FIN, ACK] Seq=159 Ack=15497 Win=37696 Len=0 TSV=172412231 TSER=898245102
48    2.073361    137.205.243.76    137.205.194.43    TCP    http > 49927 [ACK] Seq=15497 Ack=160 Win=49232 Len=0 TSV=898245104 TSER=172412231
49    2.073414    137.205.243.76    137.205.194.43    TCP    http > 49927 [FIN, ACK] Seq=15497 Ack=160 Win=49232 Len=0 TSV=898245104 TSER=172412231

And now, it’s much more obvious what the problem is. Curl is counting the bytes received from the server, and when it’s got as many as the content-length header said it should expect, the client is closing the connection (packet 47, sending FIN). Nagios, meanwhile, isn’t smart enough to count bytes, so it waits for the server to send a FIN (packet 27), which is delayed by 3-and-a-bit seconds. Apache sends that FIN immediately, but for some reason it doesn’t make it to the client.

Armed with this information, a bit more googling picked up this mailing list entry from a year ago. This describes exactly the same set of symptoms. Apache sends the FIN packet, but it’s caught and buffered by the LSO driver. After a few seconds, the LSO buffer is flushed, the client gets the FIN packet, and everything closes down.
Because LSO is only used for large segments, requesting a page with only a small amount of content doesn’t trigger this behaviour, and we get the FIN immediately.

How to fix? The simplest workaround is to disable LSO:

#  ndd -set /dev/ip ip_lso_outbound 0

(n.b. I’m not sure whether that persists over reboots – it probably needs adding to a file in /kernel/drv somewhere). LSO is beneficial on network-bound servers, but ours isn’t so we’re OK there.

An alternative is to modify the application code to set the TCP PSH flag when closing the connection, but (a) I’m not about to start hacking with apache’s TCP code, (b) it’s not clear to me that this is the right solution, anyway.

A third option, specific to HTTP, is just to use an HTTP client that does it right. Neither nagios nor (it seems) pingdom appear to know how to count bytes and close the connection themselves, but curl does, and so does every browser I’ve tested. So you might just conclude that there’s no need to fix the server itself.


February 16, 2011

Deploying LAMP websites with git and gitosis

Requirement of the day: I’m providing a LAMP environment to some other developers elsewhere in the organisation. They’ll do all the PHP programming, but I’ll look after the server, keep it patched, upgrade it when necessary, and so on.

At some point in the future, we’ll doubtless need to rebuild the server (new OS version, hardware disaster, need to clone it for 10 other developers to each have their own, etc…), so we want as little configuration as possible on the server. Everything should be built from configuration that lives somewhere else.

So, the developers basically need to be able to do 3 things:
– update apache content – PHP code, CSS/JS/HTML, etc.
– update apache VHost config – a rewrite here, a location there. They don’t need to touch the “main” apache config (modules, MPM settings, etc)
– do stuff (load data, run queries, monkey about) with mysql.

Everything else (installation of packages, config files, cron jobs, yada yada) is mine, and is managed by puppet.

So, we decided to use git and gitosis to accomplish this. We’re running on Ubuntu Lucid, but this approach should translate pretty easily to any unix-ish server.

1: Install git and gitosis. Push two repositories – apache-config and apache-htdocs

2: As the gitosis user, clone the apache-config repository into /etc/apache2/git-apache-conf, and the apache-htdocs repository into /usr/share/apache/htdocs/git-apache-htdocs

3: Define a pair of post-receive hooks to update the checkouts when updates are pushed to gitosis.
The htdocs one is simple:

cd /usr/share/apache2/git-apache-htdocs && env -i git pull --rebase

the only gotcha here is that because the GIT_DIR environment variable is set in a post-receive hook, you must unset it with “env -i” before trying to pull, else you’ll get the “fatal: Not a git repository: ’.’” error.

The apache config one is a bit longer but hopefully self-explanatory:

cd /etc/apache2/git-apache-conf && env -i git pull --rebase && sudo /usr/sbin/apache2ctl configtest && sudo /usr/sbin/apache2ctl graceful"

Add a file into /etc/apache/conf.d with the text “Include /etc/apache2/git-apache-conf/*” so that apache picks up the new config.

We run a configtest before restarting apache to get more verbose errors in the event of an invalid config. Unfortunately if the config is broken, then the broken config will stay checked-out – it would be nice (but much more complex) to copy the rest of the config somewhere else, check out the changes in a commit hook, and reject the commit if the syntax is invalid.

And that’s it! make sure that the gitosis user has rights to sudo apachectl, and everything’s taken care of. Except,of course, mysql – developers will still need to furtle in the database, and there’s not much we can do about that except for making sure we have good backups.

You might be wondering why we chose to involve gitosis at all, and why we didn’t just let the developers push directly to the clone repositories in /etc/apache2 and /usr/share/apache2/htdocs. That would have been a perfectly workable approach, but my experience is that in team-based (as opposed to truly decentralised) development, it’s helpful to have a canonical version of each repository somewhere, and it’s helpful if that isn’t on someone’s desktop PC. Gitosis provides that canonical source
Otherwise, one person pushes some changes to test, some more to live, and then goes on holiday for a week. Their team-mate is left trying to understand and merge together test and live with their own, divergent, repo before they can push a two-line fix.
More experienced git users than I might be quite comfortable with this kind of workflow, but to me it still seems a bit scary. My years of CVS abuse mean I like to have a repo I can point to and say “this is the truth, and the whole truth, and nothing but the truth” :-)


Oracle locks

Follow-up to oracle lock – look it again from Oracle/Java/Others

Step 1


create table test (id number)
insert into test values (1);

Before commit or rollback, check the v$lock table.
Two locks:
First one’s lock type is TM (DML enqueue), Lock mode is 3 – row-x(SX), Request is 0 (none)
and
Second one’s lock type is TX (Transaction enqueue), Lock mode is 6 – exclusive (X), Request is 0 (none).

Check v$locked_object table. It displays the similar information as well

Step 2
In the same session

update test set id = 2

The locks in v$lock and v$locked_objects show same information as step 1.

Step 3
Open a new session

update test set id = 3

Now check the v$lock table
The lock associated with the first session is still same: two locks.
But the “block” column in the “TX” lock is 1 , means it is blocking another session.
The “block” column in “TM” lock is still 0.

Session 2 has two locks now:
TM lock, Lmode is 3 (SX), request is 0
TX lock, LMode is 6 (X), request is 6 (X – exclusive X)

Check the v$locked_object

The row attached to session 1 is same.
A new row attached to session 2 has same (locked_mode, object_id), but XIDUSN, XIDSLOT, XIDSQN are all zero.

Conclusion

v$locked_object is very useful. It displays 1) user 2) object_id information 3) blocker & waiter

how to find out locked object in v$lock?

Other tables

- dba_waiters: including waiters and blocker, mode_held & mode_requested
- dba_blockers: just one column holding_session.
- dba_lock_internal: Very slow to access
- v$access : Very slow to access


January 10, 2011

Is Babel A Twit Or A Menace?

Writing about web page http://www.guardian.co.uk/football/2011/jan/09/ryan-babel-howard-webb-manchester-united-liverpool

It seems to be a bit of trend at the moment for footballers to launch their toys directly out of the pram (do not pass go, do not blackmail your club for £200,000 a week) about refereeing decisions affecting Manchester United.

Now obviously in some cases these tweets are just inane and gormless. Wojciech Szczesny, a man who desperately prays every night that he’s at Arsenal because he’s capable of breaking their crap goalkeeper tendency rather than being another exponent of it, tweeted after the West Brom game about Gary Neville’s foul “How can you not get frustrated with decitions [sic] like that going ALWAYS Man Utds way?! Its a clear pen and sending off!”. Bless, you cannot really blame him for having completely failed to notice that mere days earlier United had had to settle for a draw after a Birmingham goal which involved a foul, a handball and a marginal offside decision from Brum. Or the many decisions which have gone against United in recent games with Chelsea. It’s really not been a particularly good 18 months for lovers of the ‘United get all the ref decisions’ conspiracy.

Anyway, so Ryan Babel’s the latest to have a go, tweeting several tweets (thus expending more effort on whining than he has done playing for Liverpool since they signed him) including this good-idea-poor-execution picture of Howard Webb MBE.


He’d still have been better than half the Liverpool players out there on Sunday.

Now the question is, are the FA right to charge him with one of their arcane rules about saying things and being interesting? They didn’t pick on Szczesny, although as he has to play behind Koscielny and Squillaci you have to wonder if this was the FA thinking “He’s suffered enough, leave it”.

It’s tricky, cos Twitter’s just a run of ephemeral ramblings, does it matter what people say on there? But on the other hand no one likes it when players rush up and surround the ref, is Twitter another way to harass the official, albeit one where the can’t respond with a quick red? Plus lots of people were, rightly, outraged by Glen Johnson’s dreadful remarks about Paul Merson on his Twitter page, so is Babel’s comment really harmless banter or should it slapped down?

It’s a hard one – I hate the media managed, say nothing nature of players, but at the same time I also hate the way they surround and harass refs. Is the post match social media thought-blowout an extension of the pitch or not?


January 04, 2011

Cocktails For Teetotalitarians

My friend Dawn of 101 Wankers fame and awesomeness is considering giving up alcohol for the entirety of 2011! This is either the best idea ever or the most crazed scheme ever, more crazed than invading Russia or leaving the economy in the hands of George Obsorne. Oh.

Anyway, with this in mind, I wish to offer my 26 years of teetotalitarian experience with these exciting, non-alcoholic cocktails.


That’s not a natural colour for a drink, surely?

Mixer Surprise

Ingredients:
Whatever soft drinks are available at the party you are at
Other people who are drinking

To make Mixer Surprise arrive at the party early and pour yourself a cup of mixer. And another. And another. Feel free to mix it up a bit, have a glass of lemonade, then a glass of coke, then some fizzy orange. Repeat until all the mixers are gone. This will surprise the alcohol drinkers who will then drink even faster until they are drunk at 11pm and pass out in a heap whilst you spend the evening going to the toilet every 15 minutes.

Alcopop Emulator

Ingredients:
A soft drink or fruit drink (depending on whether you want it fizzy or not)
16kg sugar
Cough sweets
11kg sugar
Food colouring – neon if possible
21kg sugar
A subscription to a Hoxton fanzine

First take the soft drink or fruit drink and add some of the sugar slowly, being sure to mix it in. Melt the cough sweets over an open fire, mix them with sugar, and add it to the mixture. Add some more sugar, then introduce the food colouring slowly, mixing it with sugar as you go along. Finally make a label using pictures of twats and words from the Hoxton fanzine. Drink until the sugar absorbs all your internal organs and you vomit your innards over you skinny jeans.

Squash Roulette

Squash – any flavour
Water
Clear lemonade
Sugar infused water (lots of sugar)
Salt water
Water with a drop of fish oil it it

Pour out a glass each of water, salt water, sugar water, fish water, and clear lemonade. Turn your back whilst a mate adds squash to each then mixes them around. How many do you dare drink? Vary amounts of each type for added drama.

Tequila Sub

Ingredients:
Grape juice
Tabasco
Cabbage (boiled)
Lemon juice
Celery
Gummi bear

We all know what tequila represents. A chance to torture yourself, your tastebuds and your stomach in search of a mythical good time. No, all tequila ever induces is a headache, a burnt throat, a slight vommy feeling and a sense of mystery as to why the faff with the salt and the lemon. For the none alcoholic version, mix grape juice and tabasco with lemon juice and a hint of celery (healthier than salt) and glug it down in one. Then take a fistful of cabbage to simulate that just hocked up your dinner feeling and celebrate with a gummi bear instead of the worm at the bottom of the bottom. Repeat until amnesia.

Water

Ingredients:
Tap
Intricate system of sewage, water transportation and processing which represents arguably one of the greatest achievements of the Victorian era
An ISA

Pour yourself a glass of water. Drink. Repeat. Invest money you have saved from this free venture in an ISA which will yield a paltry rate of interest due to economic crisis but which will stand as a monument to your good intentions. Don’t buy bottled water because this will make you a twat and an environment trashing one at that.


December 30, 2010

Albums Of 2010 – 1–10

Ok, what’s this here? Two joint number 1 albums? Yeah, I honestly couldn’t call it between them, so different are they in style, place, time and arms-in-the-air-ability, that it was impossible to find a way to compare them in order to establish which, if either was better. So there you go. Just know that both are reasons to conclude that 2010 was a good year for music, no matter how much the charts made us cry hot salty tears of pain. Bah.

=1. Arcade Fire - The Suburbs


This is one of many options you can choose unless you’re mental and want to get them all.

Too obvious? Sod that, yeah they might be in almost every top ten of the year, in every magazine and newspaper, but far from being a sign of everyone being unimaginative and boring, it’s for the simple reason that this really is a stormer of an album. And an album it is. I honestly think that the best albums have, at most, 12 tracks worth having. This has 16, and 15 are worthy of a place in an album which works as a cohesive work, one which should be listened to in order and in one sitting. It’s something of a rarity these days, when skip functions and albums packed with ‘singles’ rather than a flowing set of songs are the norm. I’m as guilty as the next person when it comes to finding albums and listening to just the tracks I like best, but this one positively demands to be listened to in order. It’s also classic Arcade Fire, more like their superlative debut than the bitty and slightly dissatisfying Neon Bible which had individually good songs but lacked the feeling of a complete piece. The Suburbs has big choruses in unexpected places, the brief bellow of “Now I’m ready to start” or the jaunty loping “Sometimes I can’t believe it/I’m moving past the feeling”. Less frantic than earlier albums, it unfolds neatly, changing gears but always full of heart, perhaps too much of a slow burner for those seduced by the punch of the first two albums, but worth the time and effort. Lovely.

=1. Robyn - Body Talk


Robyn is the first pop star made of car air freshers.

At a time when every American pop and r’n’b star seems to have decided to put a bit of the Euro-pop thing in their records, it takes an actual European from the nation which pretty much owns pop, Sweden, to do it better than anyone else. Robyn’s grand experiment, releasing music as it was ready rather than to anyone else’s schedule has paid off in style. Body Talk is a pop record which pop fans can use against any accusations levelled at it. Shallow and plastic? How can you argue with the way she stalks and cries her way across the dancefloor during ‘Dancing On My Own’. Empty and meaningless? Try telling me that ‘Call Your Girlfriend’ isn’t packed with years of experience and insight into that most difficult of actions, breaking up with someone. Formulaic collaborations with bored rappers in a bid for credibility? ‘You Should Know Better’ is probably more a case of Robyn giving Snoop Dogg a credibility kick than him bringing her one. It’s also one of the funniest songs of the year. That’s what this album’s all about, the songs are either sad ones with arms-in-the-air moments, or smart arse ones which genuinely smart lyrics. Messing with the form when it suits (the metronomic ‘Don’t Fucking Tell Me What To Do’ isn’t far removed from a psychedelic drone record) or just doing something simple (‘Hang With Me’, ‘Indestructible’, most of the others). Plus it contains the lyric “I gotta lotta automatic booty applications” which is some sort of twisted genius. No really. It is.

3. Warpaint - The Fool


Ooooh, a mystical magic eye.

There’s little more tedious than guitar-bored asserting that girls can’t play guitar as well as boys. Give them this album. Give them the sparkling riffs, the intricate interplay between the guitars and bass. Give them Warpaint, who’ve melded the reverb soaked, fret climbing, late-night jangle of Interpol with the harmonious-but-tough girly vocals of Au Revoir Simone. But with better drumming than either. They’re the sort of band that make a mockery of the “radio edit” versions of singles. Yeah, so their songs are long and often take a while to unfold, but this is music for those who have time to invest in songs with layers and intricacies. Beautiful songs. Songs to lose yourself in. The guitar riff which kicks in part way through the eponymous ‘Warpaint’ the outro to the slow burning ‘Undertow’, the vocal interplay during ‘Composure’, these all give the impression of band who like to take their time in coming up with something special.

4. 65daysofstatic - We Were Exploding Anyway


Wait, neither of you are in the band! Get a room!

They’ve been mixing postrock guitars with techno rhythms and synths for a while now, but 65daysofstatic might just have done something very retro – making a truly great third album. Remember when bands were allowed three albums to get it right? Well it might not be the industry norm anymore, but this is evidence that it maybe still should be. Enormous tunes come crashing through the record; so confident are the band that they feel able to leave an excellent Robert Smith (yes, of The Cure) collaboration, ‘Come To Me’, towards the end of the record, and indeed the last track, ‘Tiger Girl’ might well be the best of the bunch, a thumping techno undercarriage bringing stabs of guitar along with it. Postrock without the slow build (the quiet bits) might sound like it has robbed the music of the essential contrast which gives it heart, but 65daysofstatic have found a way around this, via the dancefloor.

5. Foals - Total Life Forever


It’s important indie bands learn how to swim in time for the next downpour at Leeds fest.

If there was one complaint you could make about Foals’ pretty damn good first album Antidotes it’s that it was a bit cold, a bit remote, a bit hard to relate to emotionally. Enter Total Life Forever which is warmer, more ambitious, more elaborate and more intricate. Just more. Unafraid to let songs build from quiet, almost silent openings (‘Spanish Sahara’) Foals have taken all the elements of postrock which 65daysofstatic have discarded, and applied them to their jerky indie template. The result is lush guitars, bouncy rhythms, and expansive sonics. Oh, and nice vocal harmonies everywhere. And it does a fantastic line is endings. I love a song with a great ending, one where the whole song has been building towards that last minute or so. It explains my love of postrock in general. Total Life Forever has the best endings of any album this year. If we were judging albums solely on the last minute of each track then this would be top – ‘Black Gold’, ‘What Remains’, ‘Spanish Sahara’, all end in awesome crescendos of sound. If only this tiny review had a comparably good ending. But it doesn’t. Bah.

6. Crystal Castles - Crystal Castles II


Emo kid finds no sympathy from the dead.

My mate Bun hates this lot. He likes the Eagles and other beardy guitar bands, so it’s not really a surprise. And his main complaint, that Crystal Castles trade in ear bleeding noise, aren’t quite as fair on this album as their were on the first one. Yes, two of the first three tracks – ‘Doe Deer’ and ‘Fainting Spells’ – do sound like the first album’s main formula of throwing an Atari down the stairs onto a banshee, but it’s the track they sandwich, ‘Celestica’, which is more typical. It is, surprisingly, beautiful, ethereal, floaty… it’s not what you’d expect after listening to ‘Alice Practice’. It’s a good formula too, bringing dreamy, crackling synths, but still keeping some of the menace of previous records. ‘Year Of Silence’ perfectly sets the mood, taking one of Sigur Ros’s more upbeat vocals and layering them over a martial synth thud. It’s all still as unintelligible as ever – Alice certainly sounds like she’s singing “This is your potato” on ‘Baptism’, a track which sounds like something the KLF would make today – but if that puts you off then you’d probably prefer the clear diction of the Eagles, and that’s no fun.

7. Sleight Bells - Treats


Bet you didn’t know cheerleaders grew on trees.

Sleigh Bells could have been a sweet, melodic lo-fi band with a cutesy sounding lead singer, in the vein of the many boy-girl bands around at the moment peddling whispy but jaunty tunes. They could but for one thing – apart from Alexis Krauss’s cheeky, chirpy vocals, everything is loud to a point which would cause sonic nerds to collapse into gibbering heaps. That guy complaining that the age of the Loudness War http://en.wikipedia.org/wiki/Loudness_war is causing music to become distorted and painful? He will hate this. So will most people, but it’s actually a thing of beauty, noise twisted in on itself until it because pure pop again. Kind of like what Crystal Castles did on their first album, only with electric guitars and a singer who wants to lure you in, not batter you over the head and rip your limbs off. Oh yeah, and then there’s ‘Rill Rill’ where they show they can do a quiet but beautifully tuneful number, as if to prove that they know what you’re thinking – “All noise and no tunes? No, we have tunes, you just have to come to them, we won’t serve them up to you so easily”. Worth every sore ear.

8. LoneLady - Nerve Up


“Where’s the feckin’ light switch?”

If there’s any downside to the ever expanding cloud of success which has enveloped The xx it’s that they might have occupied too much of the same space as LoneLady to give her a chance of getting heard more widely. Sparse instrumentation, lyrics loaded with abstract emotional punches, and the voice of a vulnerable angel, ingredients which might sound familiar to those who follow the Mercury Music Prize winners, but Julie Campbell is no copyist. LoneLady constructs her songs with just a few pieces, a voice, a guitar or two, some drums. No bass, no fripperies. Just her sparkling cross of minimalistic indie, PJ Harvey and the terse atmospherics of many of the historic totems of Manchester music. Yes, it seems trite to bring her hometown into it, but this really is the most Manchester album released in quite some time, if by Manchester we mean the forward looking Manchester of the early 80s, not the retrogressive lad-rock bilge of the 90s. Deceptively danceable, music made for Julie Campbell by Julie Campbell which conveniently manages to be awesome for those who aren’t Julie Campbell.

9. Caribou - Swim


More mystic bollocks. Seriously, mystics have multicoloured bollocks, check it out next time one flashes you.

Ah, sad dance music. Kind of. Is this really dance music? Sure it can be danced to, but it’s almost a surprise to find it’s not from DFA such is that label’s formula of sad vocals, thumping bass, and arms in the air like you just do care in evidence here. That Dan Snaith decided to give away opener ‘Odessa’ as a free download earlier this year is impressively ballsy as it’s a strong contender for best song of the year and frankly if people won’t pay for something this brilliant then we should abandon money and live in a hole in the ground. Whilst listening to ‘Odessa’, natch. It’s all fairly similar in a sense, hypnotic and rhythmic, songs loop around themselves and wind their way to wistful places where bits of percussion float in and out of hearing on waves of sound. Yes, I know this makes me sound like a wanker, but it really is that good. Songs about divorce and upheaval never sounded so good.

10. Kelis - Fleshtone


Kelis takes her defeat to Janelle Monae in Hat Of The Year stoically.

Hey up, Kelis has discovered dance music. She’s got the chops for it, or specifically the voice for it, her husky tones languidly poured over some good quality dance. This isn’t David Guetta teaming up with some second division American r’n’b diva to do a big poo in your ear canal, this is at times understated, at times thumping tunes for waving your arms around to. There are songs addressed to her newborn which don’t make you wanna throw up (‘Acapella’ although I’ve heard people reckon it’s about God), and even some old school Kelis shouting (‘Emancipate Yourself’). ‘4th Of July’ is certified sultry dancefloor banger, even giving pounding house piano a good going over. Funny how it took an American to show European dance music how to occupy the dancefloor without being diabolical, although I’m not sure it’s quite enough to erase memories of ‘I’ve Got A Feeling’, damn you Black Eyed Peas!


Albums Of 2010 – 11–20

11. Marina And The Diamonds - The Family Jewels


Do you think she thinks she’s sexy? Is she?

Self referencing in the first song on your debut album? That takes some stones, and Marina Diamandis would appear to have those. Diamonds in fact. Whilst not the best pure pop album of the year (it’s two, maybe three tracks too long), this is a good debut made by someone unafraid to be both pop and smart. Clever clever wordplay, some of Kate Bush’s best vocal tics, and a great way with a song, she’ll wind up some, but for fans of pop with personality this is a definite album of the year (in years where Robyn doesn’t release records).

12. Delphic - Acolyte


Defy the smoking ban, get dropped from a building.

You know what, I couldn’t be bothered being clever clever and not referencing Manchester when talking about LoneLady, and I’m not going to try here. This sounds like a modern take on New Order. I bloody love New Order. And in parts I bloody love this. Yeah, like other albums here, it’s a bit too long and could lose a track or two, but my goodness they have some bangers up their sleeves. The only weak point is the slightly nondescript vocals, but when so many of the songs make you wanna swoosh your arms around why complain?

13. Yeasayer - Odd Blood


Horrible 80s graphics are horrible.

Confession, I really didn’t wanna like Yeasayer. They looked like twats and were pretentious as hell in interviews. But they’ve also made an album which dared to take all the worst bits of 80s music – slap bass, bad drum production, over emoting vocals – and make a good album from it. Loping and louche, it doesn’t seem to care how cheesy it is in parts, which is pretty much how you should revive the 80s, shamelessly.

14. Fan Death - Womb Of Dreams


Hmm, elephants do have very long pregnancies, it’s true.

It’s not often that NME’s habit of saying all new bands sound like x + y (and yes I am aware that I’ve already done an x + y comparison in this post) but they actually managed to get it spot on with Fan Death = Florence And The Machine + Hercules And Love Affair. It’s disco night down the goth club, and all the girls in floaty lace dresses are singing paeans to lost or impossible love (one song’s about Jesus and Pilate’s wife, but not in a blasphemous way) over stomping disco beats and stabby disco strings. Tastefully groovy.

15. Holly Miranda - The Magician’s Private Library


Why are you sleeping? We have Stella Artois for you.

Girl sings swoonsome modern folk songs as walls of sounds ebb and flow? Possibly, but it’s also quicker to say girl sings TV On The Radio songs. Ok, so Ms Miranda wrote these herself, but with Dave Sitek’s producer paws all over, this is a TV On The Radio album in many ways. What Holly Miranda has going for her is a willingness to let Sitek be a dictator and infuse her songs with a hum that is at times sad-happy (‘Forest Green, Oh Forest Green’, apparently not about the non-league football club), sad-sexy (‘No One Just Is’, the album highlight with its eastern strings), and sad-sad (pretty much everything else). She has a great voice and some great friends.

16. LCD Soundsystem - This Is Happening


James Murphy can also defy gravity.

Why mess with the formula when you’ve cracked it? Sure, this isn’t as good as its two predecessors, but I would be using the fingers of just one hand if I had to list albums from this century which are better than Sound Of Silver. Instead James Murphy serves up more of the same where the same is brilliant dance music, wry lyrics, and occasional doses of dancefloor heartbreak where hearts are broken not by partners but by the sheer elation of waving your arms in the air until you have to leave. And with that LCD Soundsystem leave us. Sniff.

17. Two Door Cinema Club - Tourist History


Generic indie band 2010 imagery #1 – a bloody cat.

TDCC don’t do anything revolutionary, and if their influence list includes anything beyond Franz Ferdinand, The Futureheads and Bloc Party then they’re lying. But they just so happen to write really good songs. Simple 00s style indie, it is probably easy to hate if you don’t rate the hit-hat friendly rhythms and wiry guitars, but I like that sort of thing when it’s good, and this is. ‘I Can Talk’ is one of the songs of the year, fact.

18. School Of Seven Bells - Disconnect From Desire


Runic bollocks. No really, they look like bollocks.

Swooshy guitars and punchier percussion than last time around, this isn’t a great leap forward, more a purposeful step in the direction of yet more lush soundscapes and indeed Lush soundscapes (mid 90s indie, not lovely soaps, duh). And if it occasionally sounds less like those mystical shoegaze and more like Donna Lewis’s ‘I Love You Always Forever’ http://www.youtube.com/watch?v=O7NV52UApGY (I’m looking at you ‘I L U’) then that’s grand, a lazy summer afternoon in album form. Nice.

19. Skream - Outside The Box


FACE!

Have I got this wrong? Am I supposed to like Magnetic Man more? Well I don’t. I do like that album, but this one’s just a bit more engaging, especially the superb run of tracks at the end taking in ‘Listening To The Records On My Wall’ and ‘Finally’, the latter being one of the best things La Roux has done. Ok there’s no Katie B, but she’s only on a couple of Magnetic Man’s tracks anyway.

20.Beach House – Teen Dream


Zebras, Elton John style.

Bare with me on this as I admit now I might well be wrong. This could be much better than this placing. I came to it late, but it’s lovely, all dreamy but with hints of steel underneath. And Victoria Legrand has a great voice, which raises it where sometimes it feels like it might float away. So yeah, perhaps with more time I’d have placed this higher up. Sorry. Despite all evidence to the contrary I can’t listen to everything. Sadface. (A same conundrum exists with Janelle Monae’s album, which also sounds quite fab but which I need more time with).


December 29, 2010

Albums Of 2010 – 21–30

21. La Roux and Major Lazer – Lazerproof


Ripping off Iron Man is the quickest way to good artwork.

Huh, a remix album mashing up La Roux’s debut with Major Lazer’s arsenal of sounds, raps and tunes? Well yes, and it’s better than the sum of its parts. No really, sticking together the two has produced mashups like ‘Keep It Fascinating’ which are less shrill but still tense and nervy. Plus it can be downloaded for free – http://sub.maddecent.com/lazerproof/

22. Blood Red Shoes – Fire Like This


That’s a lot of hair gel.

In pretty much the same way as their debut, Fire Like This shows Blood Red Shoes only really have one trick, but it’s a good one. Frantic guitars and drums, you wonder what they have to do which is so urgent that everything has to be played at breakneck speed. Is the oven on? No idea, but the wide eyed mayhem of ‘Don’t Ask’ and ‘Heartsink’ rush along with glee regardless.

23. KT Tunstall – Tiger Suit


Washed out photo = srs bsns.

Confession: I’ve been hoping for this album for a good while. Y’see ever since that performance on ‘Later…’ I’ve suspected that Ms Tunstall has had a genuinely good album in her. Not a whimsical female singer-songwriter-fest as with both her previous albums. She’s got a great voice, and seemed in touch with the idea of looping and fiddling with her acoustic sounds. So rejoice, here’s an album which is 66.67% what I’ve been waiting for! Ok, it tails off a bit into mimsying, but the opening barrage of tracks – ‘Uumannaq Song’, ‘Push That Knot Away’ and ‘Fade Like A Shadow’ in particular – are punchy and danceable whilst still being largely acoustic. Nice one growly-voice girl.

24. M.I.A. – /\/\ /\ Y /\


M.I.A. is watching you watching Newport on repeat.

Hmm, it’s a bit like the LCD Soundsystem record again, listening to this I cannot help but think “The previous albums were better”. Still, in ‘XXXO’ she has an electro pop anthem which Gaga, Rihanna et al would kill for, and the balls to leave it unattended on an album of shouty noise pop (‘Born Free’) and unexpected shifts in gear. It’s disjointed but worth the effort. ‘Space’ is pure beauty.

25. Hundred In The Hands – Hundred In The Hands


Did you let a child fingerpaint over your photo again?

It happened again! Another late one, but this is lovely stuff, a little bit Fan Death, a little bit Warpaint, a little bit Beach House, it’s girly voiced indie but with a wide ranging remit and a strong melodic touch. And lastfm justifies its existence with one recommendation. I’m listening right now, in fact.

26. Everything Everything – Man Alive


Oh shit, that fox is about to get run over!

Why so low? I was a bit disappointed with this album. Having caught them live a couple of times and been very very impressed with the rush of ideas and silly vocals, and then hearing the brill singles ‘Suffragette Suffragette’ and ‘My Keys, Your Boyfriend’, it was a bit of a shame that the album seemed to lack… something. Hard to say what really. Individually most of the songs are strong, ‘Photoshop Handsome’ is hilarious, so it’s not a bad album. Oh well, best just see them live again.

27. Janelle Monae – The ArchAndroid


Hat of the year.

As I mentioned, I think this is probably a lot better then its placing suggests but I am, for shame, a late comer to it. Forgive me. On the other hand it means I do get to listen to it now, with its rich imagery and captivating variety of styles. ‘Cold War’ is great too. Give me time. For now be reassured that it sounds good on first listen and we should all stick to it. It’s about robots, forcrissakes, it cannot be bad.

28. Fight Like Apes - The Body Of Christ And The Legs Of Tina Turner


With a title like that the cover could have been totally shit and it wouldn’t have mattered…

I didn’t really like this when I first heard it, it felt like a disappointment after the debut. My main complaint was that all the songs sounded like two tracks off that debut – ‘Tie Me Up With Jackets’ and ‘Battlestations’. Of course all I needed to do was remember that those two tracks are ace, and an album of sound-a-likes, whilst not as good as the debut, was still good enough for me. And that’s the best album title of the year, nay ever. Ever.

29. Jonsi – Go


Technicolour dandruff.

In which he… sings in English. It feels so wrong, the elves shouldn’t be singing in the same tongue as Liam Gallagher. Ah well, here’s a bunch of songs which sound like Sigur Ros speeded up, right down to being slightly higher pitched than those roaring postrock guitars. It’s all very sparkly, if a little lightweight, but good fun all the same. Put it on your ipod when you so to Mt Doom to dispose of troublesome rings.

30 Salem – King Night


Possibly trying too hard.

It’s hard to listen to more than a few tracks without the dark sludginess leaving you in need of some light, but if for no other reason than the title track this is worth dipping your toe into the darkness for. Vocals rinsed through a million evil effects, everything slowed down to a lurch. Embrace the darkness, if only for a track or two at a time. Rarr.


December 09, 2010

A Letter To My MP

I just wrote a letter to my Liberal Democrat MP, John Leech, who today voted against the tuition fee bill. I am glad he did, glad he kept his promises, and heartened that in his speech on the matter – he recognised that education can benefit the country as well as the individual, and that large amounts of debt are a bad thing. I wrote the letter for two reasons, one to communicate my pleasure that my representative in Parliament did as he said he would in order to get my vote, and secondly because I want to know where he goes from here. I want to know what those Lib Dems who kept their promises are thinking. I don’t know if I will get a response, but I thought it worth a try.

Dear John,

First of all I wish to congratulate you on sticking your election promise, and recognising that the raising of tuition fees in conjunction with the massive cuts to university funding will harm this country on many levels, leaving the young of today facing obscene amounts of debt, hurting our institutions’ abilities to deliver world class teaching and research, and damaging the knowledge economy which remains probably the only thing this country truly excels at now the financial sector has been shown up as a house built on false promises.

It is encouraging also to see that not all politicians will sacrifice their promises, and by extension their electorate, at the first sniff of power.

However I would like to ask a question – how can you continue to be a part of a party some of whose leaders and members have shown themselves to be spineless and treacherous? As things stand, despite my admiration for your stance, I do not feel able to vote for the Liberal Democrats ever again. The trust is gone, and whilst individuals such as yourself have shown that there are those who will keep their promises, I feel distinctly uncomfortable that my vote in a way helped contribute to the passing of a policy which I think is wrong by giving the Liberal Democratsthe clout in Parliament to form this coalition with the Conservatives which has clearly turned the heads of a large number of your colleagues.

How do you intend to proceed knowing your leader is new widely, and in my view correctly, viewed as a two-faced liar? What reassurances, if any, can you offer your constituents about your future stances and that of the Liberal Democrats? If this party is asking you to vote against what you think is right, and you are willing to defy your leadership on a matter they deem to be crucial, can you see yourself continuing as a Liberal Democrat MP?

I do respect what you have done on this matter, which why I feel able to write this letter to you as I think your vote shows you are one of the Liberal Democrats still willing to listen to the people who elected them in the first place.

Kindest regards,
Holly


December 07, 2010

The Rules Of Cricket

My dear friend and lover of such things as knitting, ATP and cricket, Ms Dawn has been following the Ashes with me on my short-thought page (www.twitter.com/Hollyzone) and would like me write up a guide to the rules of cricket seeing as my explanations on the short-thought page were so enlightening or something. So here are the rules of cricket.
 
By “cricket”, I do of course mean International Test Cricket! This is the longest form of cricket and can take up to five days to complete a match, after which there will always be a winner, even if that winner is the weather and not one of the team playing. Other forms of cricket include:
  • County Cricket – look at the word “county”, it’s like “country” but small, and this applies in County Cricket which is only allowed to last up to four days, unlike International Test Cricket’s five, because countries are bigger than counties and deserve more attention (warning, do not point out that the county of West Yorkshire is bigger than actual countries like Lichtenstein or the powers that be will leak all your embarrassing photos to Wikileaks).
  • One Day Cricket – this lasts one day, except it actually only last half a day because 24 hour cricket would be hard on the cricketers who are clearly made of less organised and stern stuff than, say, racing drivers who can manage 24 hours at Le Mans.
  • 20/20 Cricket – this form of cricket is hated by purists, and is rather like 5-a-side football except without the bunching and people running into walls.
     
    The rules of International Test Cricket are pretty easy to follow.

The Pitch

The pitch is big and green. This is because it is covered in grass. The grass serves two purposes, it is cheap, and it makes hilarious stains on the cricketers’ white clothes so you can identify which cricketer has done the most spectacular diving/sliding catches.

In the middle is the Main Bit. This is where most of the stuff happens. At each end of the Main Bit is a pile of twigs with a small and very important twig placed on top. The teams will take it in turns to defend these twig piles. If the very important top twig falls off bad things will happen.

Cricket appears to involve breaks for tea. I am unsure if this means a cuppa or an evening meal. It also has a lunch break. These are clearly signs of civilisation.

There are two teams. They each take it in turns it to throw balls and hit the balls with sticks.


From the BBC – the Blimey Bouncing Cricket people.

The cricketers with sticks each stand in front of a twig pile. This means only two at a time get to be on the pitch and shows the importance of patience in the modern cricketer, although these days the other members of the team are probably playing FIFA or Halo on their X Boxes. This seems plausible as sometimes the TV camera will pan to the side of the pitch and you’ll see three of them sat there, pretending the rest have nipped inside for a pie and will be back any minute.

The other team then spread out across the whole pitch. There lots of names to describe the bits of the pitch, like Silly Mid Off, which sound so obviously made up that you suspect it’s all a joke. Suffice to say if they think the stick man has strong arms they will stand far away, whereas weaklings they will stand near. Sometimes they will make nasty comments to the stick man. This is called sledging which makes sense as sledging, luge, bobsleigh and skeleton are the sports most likely to induce swearing in those who practice them (“I’m sliding down a hill at 100mph, chin first…. SHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIT!”).

One of the spread out team gets to throw the ball at the stick man. If he takes a very long run up he is a Fast Bowler even though his throws will take much longer than a short run up, or Slow Bowler, and so the game will not proceed very fast after all. There is a good explanation for this but it involves physics and the Large Hadron Collider.

The Scoring System

  1. You get 1pt for a mad, arm flailing, run from one set of upright twigs to the other. If you run back again you get 2pts, and so on. In theory you can run infinitely, but the throwing team don’t normally allow this unless they have been bribed by a dodgy betting syndicate (this only happens in about 30% of matches).
  2. You get 4pts if you express your disgust at the commercialisation of the sport by hitting the ball into an advertising board.
  3. You get 6pts if you give demonstrate a socialist attitude and liberate the balls from the grips of the authorities by hitting it to a pleb.
  4. You can also score 1pt if the thrower has a Victorian style attack of the vapours and hurls the ball in the wrong place, like America.


This man did not score enough points and nor did his friends. Sadface.

Unlike in rounders you don’t score a half rounder if the thrower guesses your height wrong and throws the ball at your knee/face.

Out

The stick men continue until they are out. There are over eight million ways to get out but these are the recognised ones. The other ways are kept secret for fear knowledge of them could destroy the world as we know it.

  1. Knocking the very important top twig off the twig pile. If you hit it with your stick or the other team hit it with their ball then you are deemed not sufficiently caring of the twigs to continue. This happens a lot when people run with their arms flailing wildly but the ball reaches their destination pile of twigs first. This is because balls travel faster than humans so one must deceive the ball and convince it one is not going to run rather than risk a flat race.
  2. When a field cricketer catches a ball hit by the stick man then the stick man is out as it’s probably not safe to hit a ball so close to a person, someone could get hurt.)
  3. LBW – no one knows what the letters mean but they are believed to relate to the practice of putting one’s leg before the wicket and letting the ball hit it. The leg is not a stick and should not be used as one. Stop it. Now.
  4. Hitting the ball twice with your stick will get you out. Kind of like volleyball, but fewer tiny shorts.
  5. Taking more than three minutes to get ready to face the thrower. This is common at junior level as it helps weed out youngsters who would be better off playing football as they will take over three minutes to gel their hair, adjust their snood and have an affair with a team mate’s wife.
  6. If the stick man handles the ball they are out. Why they would do this when they have a nice stick is unclear but it takes all sorts. People who do this should be redirected to tennis where handling the balls is acceptable and the balls are fluffy and nicer to touch.


It is not a requirement to look like you’ve shat yourself when you catch a ball.

Winning

When a team is all out, i.e. ten out of eleven of them have managed to fanny it up by doing one of the things above, then the other team gets to use the sticks. Sometimes a team with a really big score might decide to let the other team bat without all getting out – this is called Declaring and is basically the stick team saying “Your turn now” but in a really condescending way.

Each team gets two goes unless time runs out in which case the sun goes supernova and we all die. The team with the most points wins. This isn’t normally the end as there will probably be several games in the series and it’s important to win at least two of these, although five is best. If you win all five this is a Whitewash which is good because it means you can get all the grass stains out of your clothes.

These are the rules of cricket. Now go forth and care about it, at least for as long as England are any good.


November 29, 2010

Ferrero Rocher's New Advertising Campaign

I should work in advertising… or not…