January 15, 2007

Solaris SMF and HAProxy won't play nicely

Following on from the problems we had with mod_jk a while back, we’ve been using HAProxy as a replacement balancer on our newer apps, and for the most part it works very well. However, I’ve come up against what seems to be an irreconcilable difference between HAProxy and Solaris SMF, which is making me think about re-evaluating loadbalancers.

SMF manages services with what it terms a ‘contract’ . A contract is basically a process, or group of processes, which SMF will look after for you. If one fails, it will restart it automatically, and it will make sure that when your app stops, all the process in the contract stop, and so on.

Now, HAProxy has one feature which is unusual amongst load-balancers. Once you’ve started an HAProxy instance, you can’t modify it’s configuration. So you can’t drop one server out of a running instance and replace it with another (of course, it can automatically detect a dead server and remove that, but that’s a different problem). Instead, what you can do is tell the running instance to stop listening on it’s TCP port, but finish processing any active requests. You can start a new instance, with a new configuration, as soon as the old one has stopped listening, and the old process will run on in the background until such time as it completes all it requests; then it will exit.

Of course, this totally doesn’t work with SMF. SMF can’t cope with the idea that a process that was once part of a contract is no longer part of the contract. So, if you try to restart HAproxy, SMF will send the appropriate kill signal to tell the old instance to die, but it won’t start the new instance until the old one goes away. No use at all.

So we have 2 options:

1) Remove haproxy from SMF, and just use the traditional init-script approach. Not a bad idea, but we lose the ability to have the process auto-restarted if it dies in the night

2) Use another proxy – perlbal, pen, or even apache mod_balance.

Ho hum. Time to download and play…

UPDATE 7/11/2009: 2 years on, and we’re still happily using haproxy with SMF, just not using soft-restarts. Here’s a blog entry with a bit more detail


- 2 comments by 1 or more people Not publicly viewable

  1. willy tarreau

    Hi,

    I’m always interested in getting feedback on haproxy. What does SMF monitor precisely ? Does it just monitor the parent process ? If this is the case, it is possible to write a shell script to start and restart haproxy, so that SMF will always see the script running. Right now, I’m not aware of anyone using SMF with haproxy, so I have no trivial solution. It is generally used with legacy init scripts. If your concerns are only about the risk of having the process die in the night, I may try to reassure you a little bit by the fact that I never encountered a single crash in about 5 years of production on about 50 machines, some of which having been running the same process non-stop between 1 and 3 years along.

    I know that I need to implement hot-reconfiguration, but this is really tricky because when the process starts, it drops its privileges and chroots. So it cannot reach the config file anymore, nor bind to privileged ports. So the hot-reconf mechanism would require a full-privileged parent process which would spawn children with the new config. While not too much complicated, it adds new complexity in that you always have to manage several processes even when only one would be needed (most situations). Another solution might be to keep everything in the same process and make it try to reload the conf and bind to new ports, then either accept to change its conf, or refuse it depending on success or failure. I’m still thinking about those solutions, but what I’m seeking above all is not to turn existing simple setups into complicated things.

    Best regards,
    Willy

    28 Jan 2007, 07:44

  2. Chris May

    Hi Willy, thanks for taking the time to comment!

    First I guess I should point out that actually we’ve done neither option (1) or option (2) above! In the end we decided that we liked and trusted haproxy too much to want to go to the effort of choosing another load-balancer, and moving away from SMF for just one app was too much of a fiddle! Instead, what we do when we want to do hot-reconfiguration is to run up two instances of haproxy, and use apache mod_rewrite (our requests go through apache first, then haproxy, then on to a java app. server) to switch between the two haproxies – so we never have to hot-reconfigure an haproxy instance, just swap the apache config round and apachectl graceful

    As to the specific questions, as far as I know, SMF just monitors the parent – so, as you say, a wrapper script might do the trick. The auto-restarting is nice, though as you say we’ve never had an haproxy instance crash. But we like SMF for all the other features (consistent interface for all services, RBAC integration instead of sudo, depenency management between services, and so on).

    Hot reconfiguration of a running process, a la apachectl graceful (or squid -k reconfigure ) would certainly be a nice-to-have for haproxy, but the lack of it isn’t so much of a disaster as I first thought!. Actually what would be really cool (I have no idea how hard to implement ) would be a feature like perlbal has, where you can telnet into a management port on the proxy and reconfigure it that way.

    28 Jan 2007, 11:32


Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

Most recent entries

Loading…

Search this blog

on twitter...


    Tags

    Not signed in
    Sign in

    Powered by BlogBuilder
    © MMXIX