The joys of tracking down performance problems
So, after a week of struggling on and off with performance problems, I finally nailed it last night.
Sometimes you just need to look at things from a different direction. I'd been really concentrating on garbage collection tuning, hibernate sql tweaking and investigating caches. I did find small problems in all of those areas, but nothing that could really explain why things were going quite so wrong.
So, having eliminated those, I started looking at the access logs again…and there it was. Staring me right in the face was several requests in a row for a particular page in blogs that we'd unlinked months ago because we knew it was slow. But there were the requests anyway. Even though it was unlinked and blocked by a robots.txt and there had been no requests to it for months, suddenly at the start of February, lots of requests started coming in.
It turns out that the Google image search had indexed it a long time ago and finally someone was finding images on that page and requesting them a whole bunch of times.
Moral of the story: Don't just unlink something, disable it completely!
On the plus side, I've learnt a hell of a lot more about tuning this kind of app in the process, but still a frustrating time none the less.