All entries for February 2005
February 28, 2005
In an E-Lab team meeting the other week, we talked about how we communicate with the outside world and let our customers and our colleagues know what is going on within IT Service.
We don't seem to be very good at letting everyone in the University know what we are up to. Sure, we sent out the odd newsletter or emails to departmental IT contacts and keep websites of varying quality/detail for most projects, but there isn't really an up-to-date single point of contact for details of upcoming projects in IT Services.
To someone like me, the obvious answer is a blog of course. Makes so much sense you might think. In a fit of enthusiasm, I created blogs.warwick.ac.uk/itsprojects/ and wrote a brief introduction as to what to expect to find there.
That was 3 weeks ago. Today I de-activated it (with no entries). It never got off the ground because I realised (or more accurately, John and other senior management realised) that you just can't get people to write yet more stuff about their projects. Maybe we're doing our colleagues an injustice by not even asking them, but it just didn't seem worth it.
I had intended to ask various people from around ITS to write something about a project once or twice a month for the blog. If this was staggered out, there would be new content several times a week with a bit of luck.
It should have been obvious really, if you look around at our project pages and our other attempts at "what is everyone up to" type pages, they are not kept as up to date as they should be (I'm as guilty as everyone).
I've still got this lovely image of everyone subscribed by RSS or viewing a feed on the ITS homepage of regular project updates from across the department, but everyone is just toooo busy. Should I have given up so easily? Was it a bad idea in the first place? Who knows.
February 22, 2005
Follow-up to Garbage collection and Hibernate performance tuning from Kieran's blog
Some of the changes I was making to the GC settings were not really making the different I thought they should be. More or less every minute on the dot, there was a full GC, which is far too often.
We had known for a while that there is a default full GC interval of 1 minute for RMI GC.
The above will set the intervals to be an hour. However…it turns out that everywhere we had it set we had "gcinterval" without the capital I :(
Change the variable to be the capital I and all of a sudden the per minute full GC's are gone! Sigh.
Analysing the throughput with the very handy GCViewer showed that it had gone from a poor 86% up to 97%, with full GC's probably only every 5 minutes or more.
Now that the stupid RMI interval problem is fixed, I should be able to more reliably tune the GC even further.
February 21, 2005
I'm sorry, Maggie's just too cute. I promise, no more dog photos!
Follow-up to Garbage collection and Hibernate performance tuning from Kieran's blog
- System home page:71 sql queries
- Individual blog home page:120 sql queries
- System home page: 24 sql queries (15% faster load times…very approximate)
- Individual blog home page: 73 sql queries (25% faster load times…very approximate)
Turns out the old system homepage was still generating the hot topics and most recent entries, but not actually using them.
February 17, 2005
We've been experiencing some performance problems recently with BlogBuilder. Occasionally we've been having periods of up to 10 seconds where BlogBuilder has completly stopped responding. It didn't take too long to work out that this was the the garbage collection (GC) in the JVM. Only problem is…how do you fix it?
1) First things first, analyse the problem.
After a bit of faffing around I found that the easiest way to get some information about what is going on with GC is to set the following flag when you start up your application, in my case JBoss.
This will print out some nice stats such as these:
0.000: [Full GC 31662K->1547K(3114944K), 0.1660533 secs]
46.941: [Full GC 302615K->20372K(3114944K), 0.8736640 secs]
107.826: [Full GC 237781K->33703K(3114944K), 0.9595437 secs]
168.812: [Full GC 997600K->68269K(3114944K), 2.0233878 secs]
230.857: [Full GC 1045518K->85888K(3114944K), 1.9429575 secs]
278.238: [GC 1072896K->102427K(3114944K), 0.3734575 secs]
You can just look at these and see how long garbage collection is taking, in my case around 1 or 2 seconds per full GC. The problem comes once the application has been running for a while.
84564.203: [Full GC 1178937K->867583K(3114944K), 11.3700822 secs]
84618.071: [GC 1854591K->897218K(3114944K), 0.5409472 secs]
84635.640: [Full GC 1253982K->904732K(3114944K), 11.9113516 secs]
84697.271: [GC 1891740K->937871K(3114944K), 0.5528494 secs]
84707.583: [Full GC 1125289K->935547K(3114944K), 12.2230228 secs]
84760.543: [GC 1922555K->960498K(3114944K), 0.4644272 secs]
84779.833: [Full GC 1589435K->981758K(3114944K), 14.6843600 secs]
84854.553: [Full GC 1919262K->970964K(3114944K), 13.7111592 secs]
At a glance we can see things are going wrong, but to really get a good overview of what is happening, HPjTune comes to the rescue. This small java app analyses these logs and prints out some nice graphics for you.
As can be seen from this, the GC duration rises steadily and gets very spikey occasionally as well. From other information jTune provides, you can also find out the memory that is getting used up and freed up on each GC.
There are two main possible solutions to sorting out GC problems. These are:
- Use a better GC algorithm
- Reduce the memory used and number of objects created by your application
Using a better GC algorithm.
Tuning Garbage Collection with the 1.4.2 Java[tm] Virtual Machine tells you pretty much everything you need to know. The conclusion is that if we want GC to stop blocking, we need to use a parallel GC algorithm. This will only work on a multi-processor box though (which is ok because blogs does run on one), but it is difficult to test because we don't have a comparable test machine.
Reduce the memory used
The less memory there is used, the less memory the GC algorithm has to work through to clean it up. Simple. We use up a lot of memory in BlogBuilder because we have a very large second level Hibernate cache (using EHCache). I'm beginning to think it is too big and too long lived. Getting to the root cause of why we need such a big cache lead me to look a bit closer at some logs. It turns out BlogBuilder is pumping out a lot more SQL than it needs.
BlogBuilder is generating SQL inefficiently -> We need big caches so we don't re-run the SQL -> We get long GC periods -> We get bad performance.
It turns out you just can't get around not fixing the initial performance bottle necks, the caches will save you to a certain degree, but once you get a lot of data (and with 3000 blogs, 20000 entries and 40000 comments, we have a lot of data), you need to go back to basics and get the individual pages running efficiently.
3) Improve Hibernate SQL efficiency
BlogBuilder already has all collection relationships set as lazy so that data is only ever loaded when it is needed (this is a huge performance gain), but even so, when the data has to be loaded, there is no getting around the fact that some SQL is going to be issued.
The one optimisation I'd glaringly missed when first developing BlogBuilder was the batch-size settings for collections. If we have a permissions object which has two groups on it, staff and students, then Hibernate might issue three sql queries, one for the permission and one for each of the two groups. In practice, Hibernate is quite clever and will generally just issue two sql queries, one for the permission and then just a single query to get both groups. Good.
However, what setting batch-size does for you is instructs Hibernate to be even more aggressive in optimising the sql.
Now, say you have three permissions objects and each has two groups, you might normally have to issue either nine or six sql statements. If you set batch-size="10" on the groups relationship, Hibernate will try and load all groups across all three permissions in one go, meaning just four queries. Brilliant!
Using this and a couple of other methods, I reduced the number of queries for a single un-cached page down to one third of the original number.
At the end of the day, you must track down performance problems to their root cause, don't just work around them with a higher level fix. Hopefully this will mean that I can be less reliant on the caching for good performance, so I can reduce the cache size and timeouts, leading to better GC performance.
February 15, 2005
Facing the danger that this will become a "Look at my cat/dog/car" type blog head on…I will briefly mention how a lazy and selfish man like me has managed with a dog for the first week.
After three agonising nights of 3/4 hours of Maggie demanding to be let out of the conservatory at night and us ignoring her as best we can, we finally caved in. On night four, she got the run of the whole of downstairs, thus proving that it only takes 3 days for a dog to train their new owners. Oops.
Walking out of the door at 7:20 every morning to take her for a walk has been a shock to the system, I can tell you. I wonder if she appreciates my frozen ears? Doubtful.
Now we've had her for over a week, she knows that we love her and find her cute, which is giving her a little bit too much confidence. Yesterday I went home at lunch time to find that she had managed to open the door to our bedroom and then click it shut behind her. In any other room she would have got upset, but spying a comfy bed she proceeded to pass the morning stomping and sleeping on our bed :(
Door closed, dog back downstairs, Steph comes home at 5 and where is she? You guessed it…locked in our bedroom again. Doh!
I don't make her sound that great, but she is really. She is generally so relaxed and happy if a little shy still. I could not really imagine how a rescue dog could be better. I guess you've got to remember that most dogs come into a rescue centre because of their owners, not because of the dog.
Now I've just got to work through one of my new dog training books to get a bit more respect from the little angel. (Dog Training by Bruce Fogle seems pretty good if you are a bit of a beginner like me)
February 06, 2005
After months of thinking, we finally decided to get a dog. Having both grown up with dogs, Steph and I had really started missing living with dogs.
We looked at the two local rescue centres, the Dogs Trust (Kenilworth) and the RSPCA (Coventry). I'm sure the RSPCA do a great job overall, but I was really disappointed by the facilities they had. The Dogs Trust on the other hand was amazing. They have recently had a complete rebuild with room for 250 dogs to live in comfort whilst waiting for their new homes.
After just 2 visits, we managed to find a dog suitable for us. Wandering around looking at the dogs and trying to find ones that we liked and were suitable for our lifestyle was not getting us anywhere. At the suggestion of one of the staff (thanks Jo), they matched us up to Smudge, a 6 year old Border Collie. We immediately took to her and although she was pretty indifferent to us (more interested in what was going on around her than in us in particular), we could tell that the potential was there for her to be the right dog for us.
A rehoming talk and a home visit later and she was ours. We picked her up at lunch time today.
So, introducing Maggie…
She's very relaxed and quite happy to just lie down and have a nap, but still a little unsure of us. I'm sure in a few days she'll really start to connect with us and trust us.
I'm just dreading tonight, will she, won't she bark? Fingers crossed.
February 04, 2005
February 01, 2005
Writing about web page http://blog.case.edu/jms18/2005/01/21/checking_out_other_universitys_blog_hosting_services
Ahhhh, it's nice when people notice :)
Jeremy Smith from Case Western Reserve University runs down a little review of different University blogging systems.
I'll quote him just in case you can't be bothered to read it.
Well, we've hit the cream of the crop, it seems. Not only does this system use existing University credentials, but it is integrated with their Single Sign On system (something we, at Case, don't have). They have a blog directory and a "planet" site. And, if you noticed in the directory, they have blogs listed in there for courses and faculty divisions and services.
They have tons of documentation in FAQs, About pages, Tours, and a Glossary.
I wasn't able to play with it, but it seems they developed a piece of software called SiteBuilder (screenshot) that does all of the heavy lifting and handles the editing of a blog's templates and styles.
All and all, this is great stuff.
5 out of 5, no doubt.
He doesn't get it exactly right about how it works, but he is close enough.
Turns out that CASE themselves also now have a blogging system, that's another one for the list.