All entries for July 2004

July 30, 2004

Power of the profiler

With our new server up and running, it was time this week to start thinking a bit more about the performance of blogbuilder.

With only 50/60 blogs and 1300 entries, performance isn't really a big problem at the moment, some pages would be slow occasionally, but nothing too bad. However, we have to plan for the big start of year influx of students (hopefully all wanting a blog).

In my little test scenario, I am imagining 1000 blogs with between 0 and 30 entries on each in the first term. There really is no way of knowing if these figures are realistic or not, but it will have to do as a starting point. Maybe we'll only have 100 blogs, maybe we'll have 10000.

So, having loaded up the new database with 1000 blogs and 15000 entries, I set about trying to render a page on the new server. 30 seconds and an OutOfMemoryException later I knew I had some work to do :)

I use Hibernate for the persistence in blogbuilder and that does a really good job of hiding away the tediousness of writing SQL, however, if you don't get your relationships right and let it cascade too far through the object graph, things can get out of hand. On the very first request of the home page, it tried to load more of less the entire database…oops. This was partly because the test data was linked to just 10 users with 100 blogs each rather than 1000 users with a blog each…but still…not good.

Until no I'd avoided lazy-loading collections in hibernate because I didn't need them, the dataset was just not big enough to pose any problems…not any more. So, by adding lazy-loading to every collection in the system, the performance was hugely improved, but only after changing to using Spring's OpenSessionInViewFilter which enables objects to keep loading even in jsps. It's not the most elegant approach, but it is very efficient.

Having fixed that nasty problem, at least pages were loading now, but not particularly fast. Time for the profiler.

With JProfiler you can really easily profile JBoss or any java application. My main concern here was processor time rather than memory now.

It turns out that just 3/4 areas needed tweaking.

  1. The stats box on the front page counting the number of blogs and entries was doing "select *" rather than a "select count" type of query, so although Hibernate had everything cached, it still have to go through 1000 blogs. Not any more.
  2. The comment counts in the hover-tips on the front page were also very slow. I've not completely got to the bottom of this, but as it is not the most important feature in the world, it is just gone for now.
  3. The calendar widget on aggregate pages was very inefficient for large numbers of entries. On an individual blog where there are likely to be more days than entries, say 31 days and just 10 entries, it is more efficient to just get all the entries in that range and mark the days with entries on. However, on an aggregate of potentially 1000's of entries in a month, this just doesn't work! So, on an aggregate, each day is checked and counted instead of the entries.
  4. The last bottleneck which was still keeping big pages down to the 4/5 seconds rendering mark was the Textile-HTML markup rendering. On an individual entry, Textile-HTML might only take 0.1s, but when you have 20/30 entries on an aggregate page all with a couple of thousand characters, that can really add up. So, time to cache the rendered mark up, after all, it doesn't change often once you've created your entry. So as a result, the database now contains both a Textile version of your entry and an HTML version, thus letting Hibernate deal with the caching for me, without me having to implement a caching layer on top of the Textile renderer.

I was surprised by some of the finding of the profiler, they were certainly not the first places I would have looked. It certainly saved me a hell of a lot of time. It also reiterates to me what I read a while back.

Do not worry about performance until you have to, because you just never know where your bottlenecks are going to be.

Obviously you don't want to go doing anything stupid, but it's not worth wasting your time tweaking something that is never going to be a problem.

All of that adds up to a blog home page in front of a database with 1000 blogs and 15000 entries rendering in 0.4s
I'll be doing load tests next week to see how it manages under some heavy user load.

XMLHttpRequest for live page updates

Writing about web page

John D found a really neat feature of someones blog yesterday, Andy Budd's blog has a live textile preview function. As you type your textile markup into the textarea, below it shows what it will really look like without having to submit the form. How clever is that?

I thought he might have implemented a textile parser in javascript, but it turns out he is using a little know bit of javascript called the XMLHttpRequest. I'll not to into detail here, but basically it means that on any javascript event, clicking a button, typing into a field, even mousing over something, you can fire off a call to an XML web service and get the results back live without submitting the page.

Live search (try typing something into the search field on the right)
Live textile preview

Possibilities are endless, only question is…how does it scale? I can't imagine it can be all that efficient to send a web request for each character typed into a comment field!

Neat none-the-less.

July 24, 2004

My morning in a Ferrari

If anyone likes fast cars, they just have to try something like this one day! It's great, you fork out some cash and you get to blat it around a race track in a Ferrari Testarossa…sounds like a good deal to me.

At about 9am this morning I was at Prestwold Hall just outside Loughborough, along with about 40 other people, looking forward to our various driving experiences. My little day out was a 100 Red Letter Day event I got for Christmas, but some of these guys here were paying 400+ for a whole day event that involves getting to drive about 4/5 super cars, including a Diablo, 550, Noble, Spider….gits.

My 100 gave me 2 laps around the 2 mile track in a little Audi A3 2.0FSi followed by 4 laps in the 175mph Ferrari Testarossa. Even the Audi was quick, especially when you get the right racing lines around the very open and quick track.

Then came the Ferrari. Wow! I have never been in a car that quick before. After revving it way too high to get out of the pits (scared of stalling), I quickly got the hang of the gears/clutch and raced up the first straight. In the blink of an eye, the first lap was over, adrenaline pumping, my hands shaking.

At the start of the long straight on the third lap, I was starting to get confident and really went for it, overtaking a Dodge Viper that was just getting up to speed out of the pits before hitting 120mph at the end of the straight and slamming on the brakes and drifting the car beautifully around the corner….damn these things can handle!

It was a real shame to finish, it was such a rush. If I had the money, I would go again for the whole day event, that must be amazing.

More pictures are in my Ferrari Gallery

July 20, 2004

University provided iPods

Writing about web page,1412,64282,00.html?tw=wn_tophead_3

Some people have just too much money!

Duke University are going to be providing 1650 iPods to freshmen when they arrive in August. The project is to see if having easy access to digital information, especially audio on the go is useful in an academic context.

The school will preload the 20-GB iPods (retail price, $300) with freshman-orientation information, an academic calendar and even the Duke fight songs before handing them out to the incoming class Aug. 19.

Fight songs?

Students also will be able to use the devices to download course content, recorded lectures, foreign language lessons, audio books and music from a special Duke website modeled after iTunes. The school will supply voice recorders for some classes, enabling students to record notes while working in the field.

Recording lectures for playback anytime and anywhere…nice

With the iPods, students will be able to pause, rewind and play audio lessons as often as they like.

Good for us slow learners :)

As you can see, quite a project they've got cooking. Sounds very similar to the PDAs that Dartmouth College provided to hundreds of students for free a couple of years ago.

No doubt Duke are getting help from Apple on this, but it is still estimated that the project will cost around $500,000. Dartmouth got help from HP for their PDAs. Can any UK universities compete with this kind of spending and corporate sponsorship? How good is our relationship with IBM now-a-days? I know they helped out with equipment for CompSci. Perhaps they might be interested in more sponsorship/donations in the future. Fingers crossed.

New blog design

Paul has done a great job of converting one of Hannah's designs into reality.

I couldn't work out how to do this one initially but Paul realised it could easily enough be done with tiling together background images in the right places and aligned correctly to get the nice jagged edges.

The advantages of blog builders all CSS layouts is that it only takes one CSS file to completely change the look and feel of someones blog. There is no custom HTML done at all for the different designs.

It might be nice further down the line to let people edit their own CSS if they are feeling brave enough, but in the shorter term we might let people tinker with a few designs features of their blog such as which parts of the left-hand column to show, font sizes/styles

Might also be nice to see how it would work if we put the current left-hand column over onto the right for some designs.

Surprise…more decorating

Surprise, surprise. Not loads of work done this weekend because of distracting (if fun) nights out on Friday and Saturday. However, got the 3rd bedroom finished at last (Well, almost)

It's a bit bright I know, but it needed it as it is the darkest room in the house otherwise. Should be getting our carpets in this week so we can finally properly unpack. I can't believe we've been moved in for 4 weeks already!

Steph tried to make up a little list of things that we've still got to do, it goes something like this:

  1. Fix/replace some floorboards in bedroom 2, bedroom 3, landing and bathroom (people who put in central heating before didn't put back floorboards properly…losers)
  2. Hang final new door
  3. Paint all doors :(
  4. Enlarge openning to loft (it is tiny)
  5. Put boards down in loft
  6. Move lights in the 3 bedrooms into the middle of the rooms (Paul informed me that a lot of old houses are like this where the lights are next to the windows so as not to cast a shadow whilst you get undressed. Very annoying as they look really silly)
  7. Possibly fit larger water tank in loft following newly enlarged hatch, current one it tiny, possibly the cause of our occasional lack of hot water…need to look into this one further

They are most of the little jobs. There are of course the big 3!

  1. Kitchen extension into garage and new fitted kitchen
  2. New bathroom
  3. Complete renovation of the garden

God what as summer I'm going to have.

July 16, 2004

University of Waterloo blogs

Writing about web page

Came across this blog page when searching for university blogs on google.

Looks like a nicely polished site when you first arrive with a smart looking flash into, but once you enter, you realise that there are just 3 blogs. It appears that Waterloo are using them for publicity purposes. Comments appear to be turned off, so they really are slimmed down versions of real blogs.

I wonder how successful they have been? Should we perhaps choose a small group of student bloggers and get them started before the start of year and give them pride of place as "featured bloggers", using them as examples to new students?

Perhaps get them to write specifically about what it's like arriving at University for the first time and their experiences in the first few weeks. Obviously we'd hope this is just the sort of thing that lots of people will write about, but it might be nice to hand pick some good writers who'd be happy to update maybe several times a day.

July 15, 2004

Transparency filter in CSS

Writing about web page

Paul pointed this out to me today, he's been using it for a while in a few web pages, but I never knew you could do this quite so easily!

Hopefully we can work in some nice blog designs using this nifty technique.

July 13, 2004

Polystyrene fetish

Writing about web page

This google search for "polystyrene fetish" does not seem to reveal the old owners of my house, but I'm sure they were really into it.

There is just too much polystyrene covering and ceiling tiles….yuck!

Before and after shots of bedroom 3 after tonights decorating. Can't wait to get this last bedroom done.

Da Vinci review

4 out of 5 stars
What a great book!

July 2004

Mo Tu We Th Fr Sa Su
Jun |  Today  | Aug
         1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31   


Search this blog

Most recent comments

  • One thing that was glossed over is that if you use Spring, there is a filter you can put in your XML… by Mathew Mannion on this entry
  • You are my hero. by Mathew Mannion on this entry
  • And may all your chickens come home to roost – in a nice fluffy organic, non–supermarket farmed kind… by Julie Moreton on this entry
  • Good luck I hope that you enjoy the new job! by on this entry
  • Good luck Kieran. :) by on this entry


Not signed in
Sign in

Powered by BlogBuilder