Performance issues
It's not been a fun day today. Partly related to the massive code changes we've made to blogs recently, we've had some nasty performance problems today.
Hopefully the bugs with the release and the performance issues are now resolved. Unfortunately my task today was made harder by some very bad timing. At around the same time, we had 4 search engines indexing blogs and a huge number of images requests coming from external sites. This made diagnosing the performance problems all the more difficult. As a temporary measure, I've had to block the search engine crawlers and requests for our images that are from non-Warwick pages.
What this means is that until we lift this restriction (which will hopefully be very soon), you won't be able to see images from your galleries embedded in non-Warwick pages and our latest content won't show up in the search engines quite so quickly.
To give an idea of the size of this problem, here are some stats:
- 160,000 requests for user uploaded images a day
- Only 6% are requested from on campus!
- Only 60% of images are embedded in Warwick pages, the other 40% are linked to from other websites
- Our top external referrers are myspace.com and Google images
- Top 5 image searches in the first few days of Feb: kate beckinsale, arctic monkeys, evolution motorcycle trousers,hell, mafia
- 30% of all images requets are for images from just 10 blogs
- The top 2 individual images alone count for 7% of all image hits (we have almost 80,000 images)
- In the first week or so of Feb we served out 1.3m images which is 33GB of data
Now then, as you can imagine, that's a fair few hits. The problem I have with it is that the performance for our staff and students is degraded because of a massive number of external requests. I like the fact that Warwick Blogs rates highly in Google…but with that ranking comes a lot of unwanted traffic.
Obviously we'll try and resolve the performance issues and try and allow these requests to start flowing again. The problem is that we are not just serving images statically as we are doing single sign-on checks and permission checking and resizing of images on the fly. These are all problems that can be fixed and optimised, but it just goes to show that with systems like this, you never can tell where the bottlenecks will be until you hit them.
09 Feb 2006 23:23
| 
Tags: BlogBuilder Performance
| 
Comments (12)
| 
Report a problem
12 comments by 4 or more people
[Skip to the latest comment]Yay for learning experiences! =)
09 Feb 2006, 23:51
Talking about blogbuilder performance: this week I've seen it happen twice that all entries on the latest entries page have the same screwed up format, e.g. bold and italic, or strikethrough and italic. A page refresh makes it all ok, and I don't care too much, but you asked for any suspicious behaviour in a previous entry.
10 Feb 2006, 00:13
Which are the top 10 image blogs? Which were the top 2 images?
10 Feb 2006, 00:32
Leslie Contarini
Yea, you guys need to hurry up and fix it. My myspace looks like crap and I didn't spend hours uploading my pictures to this blog to have to do it again.
10 Feb 2006, 12:43
John Dale
Unfortunately, it's possible that you did. We aren't committing, either now or in the future, to supporting serving images from out of WB to blogs or other web pages elsewhere. If you want to blog or maintain a presence elsewhere then you should host your images and other content elsewhere too.
10 Feb 2006, 18:15
I giggled when I saw the title of this. I apologise. :/
The hell?!?
Since the early days of blogging I've been wondering about the effects of external image requests on performance. If you were to prevent linking of WB images, then firstly how (un)poplular would this prove and secondly just how beneficial would be on the servers?
13 Feb 2006, 17:52
Update: Crawlers are indexing again as of a couple of days ago.
15 Feb 2006, 21:01
General problem – my comments is not always being updated. With me it's acknowledging some comments but not others that I write.
20 Feb 2006, 19:06
Now that we have deployed a fix for a Hibernate bug that was causing deadlocks under load and everything seems healthier, I've allowed external image requests to come back in again.
21 Feb 2006, 09:19
Leigh Robinson
I can't find any 'bug list' or somewhere to post any bugs/quirks – so sorry for highjacking this post.
I am experiencing the good ole' "inserted value too large for column" exception when I am attempting to update the description of a gallery. The length of my description does not seem excessive :
If the limitation is by design some validation would be better than scaring people with the exception!
21 Feb 2006, 15:31
Thanks Leigh, we'll get that fixed.
22 Feb 2006, 11:59
Mathew Mannion
Hi Leigh,
The new version of BlogBuilder (which should be released sometime within the next week) has this issue fixed, and we have also increased the limit on the size of a gallery description from 255 characters up to a whopping 4000 characters, just for those people who can't get enough of those gallery descriptions.
02 Mar 2006, 16:20
Add a comment
You are not allowed to comment on this entry as it has restricted commenting permissions.