Odd Solaris error of the day

I consolidated a bunch of java web-apps onto Solaris zones recently, and started seeing a small number of errors like these in the apache logs:

 [error] (125)Address already in use: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (*) failed

- which would result in apache serving an HTTP 503 error.

Each application uses an apache front-end, which connects to java via mod_proxy_http and mod_rewrite.

After digging a bit, I discovered the cause of the error, which is a solaris kernel bug: 6670527 “Simultaneous connect() issued from multiple shared stack zones fails with EADDRINUSE”

we have a single connection hash for the single stack. If we are using INADDR_LOOPBACK(127.0.0.1) in all address specifications and the server in question is on a static port, the only unique member of the tuple is the local port. This can be duplicated in ephemeral port selection, hence the failure.

There’s a kernel patch, and a neat workaround if you don’t want to wait for a reboot. It’s based on the fact that the problem only affects the loopback address, so we can use the zone’s own IP as follows:

First, define an alias “localzone” for your zone’s local IP address in /etc/hosts:

12.34.56.78 foo-zone localzone

Now, in your rewrite rules, use “localzone” where you would previously have used “localhost” or 127.0.0.1. If you move to another server, you need only ensure that the “localzone” alias exists there too, rather than having to change a load of references in your apache config.

There’s one small downside to this; you now need to go through host-name resolution for every proxy call. So a small overhead, and you have one more moving part to go wrong. I’ve been bitten before by bugs in solaris’s nscd (it can block in the face of badly-behaved DNSes, causing all lookups to stall) so I’d steer away from this for really performance-critical sites, and apply the patch instead.

Chris May : 16 Dec 2009 13:39 |  Comments (0) |  Report a problem