September 15, 2008

Providing JSON data views onto database data in search

As part of working on Warwick's search engine, in the revamp we've been trying to leverage a lot of data that extends above and beyond just web pages and blog entries. This means we've been indexing a lot of data that is then stored in a database and refreshed periodically (typically, every 5 minutes). This is then included from a number of sources, such as our interactive map of campus and the XMPP/Jabber bot that we run at (try adding it to your Warwick IM account!) - but it all comes down to exposing the data in the end.

But hey, it's all about sharing, so knock yourself out with the data...

Search Results

You can get search results for sitebuilder or blogs (or both) from the endpoint, for example at (replace the sitebuilder with blogs, or remove the indexSection altogether to get amalgamated results).

Example usage: A live search, or search results from the XMPP bot.

Asking about search results

Free seats in workareas

ITS provides a service at that shows where free seats are in workareas, and where the workareas are.

Example usage: Showing where there are free seats in work areas near to your department; results in interactive map; embedded results in search results; querying the Ask bot:

Free seats in workareas

Live Bus Tracking

Travel West Midlands have started to track the positions of buses in relation to some bus stops in real time (presumably by some radio or satellite system) and are publishing the information to customers at their mobile website. For the stops in and around campus, this is useful information to some, so we've indexed the information (only including those buses that are being tracked in real time).

Example usage: A live tracking page; embedded in search results; results in interactive map; querying the Ask bot:

Getting bus timetables from the Ask bot

What's on?

By amalgamating results from a number of sources (the Students' Union, Warwick Arts Centre, and the internal and external News and Events pages) into a single database, it's easy to query information from it and return a fairly comprehensive set of information about what's on on a particular day.

Example usage: Providing a feed of what's on today on campus; showing what's on in relevant searches; Ask bot results:

Asking what

Where is this room?

Thanks to Emily in e-lab's tireless work collecting and collating the data, we also now have a database of information on what rooms there are on campus and where they are - 3018 rooms in total! As a former student myself, I've experienced the frustrating of receiving a timetable with rooms on and not knowing where they are. Hopefully this will be a help!

The JSON feed requires a query parameter, for example:


Example usage: Linking room data to a map; results on interactive map; injection into search results:


What this exercise has taught me, if nothing else, is that everything would be a whole lot easier if every bit of data provided me an API to access it!

- 9 comments by 2 or more people Not publicly viewable

[Skip to the latest comment]
  1. Let me guess – current students only?

    16 Sep 2008, 20:01

  2. Mathew Mannion

    To what, exactly? :) Search is completely open – as is all the information you see here. The Warwick IM service is only available for current members of the University, but you should be able to add if you use any Jabber/XMPP client (including Google Talk, although the Google Talk client doesn’t allow you to add non-GTalk users so you’d have to use something like Pidgin)

    16 Sep 2008, 20:11

  3. Phil Wilson

    Excellent stuff. We have some jabber bots internally at Bath ( but nothing exposing our public information. I’ll see if I can put something similar together tomorrow!

    Also, just to confirm that I can use the askbot from my own Jabber account!

    16 Sep 2008, 21:19

  4. Mathew Mannion

    Cheers Phil, we also have a Bamboo reporting bot on server (as well as a bot that reports problems with our Nagios sensors) but HungryBot looks like it might be something we could consider adding. We’ve been looking at the possibility of adding useful PUSH data (you have a lecture in 10 minutes, your 9am has been cancelled because your lecturer is in the Maldives, there is a gunman on campus; that kind of thing) personalised for students/staff. However, there are political ramifications with actually getting to the data before we can move on to that kind of thing :)

    16 Sep 2008, 21:38

  5. Phil Wilson

    If I do have a small request, it’d be that outgoing messages from ask@ are returned in the same type as the incoming.

    i.e. if I open a chat window and send “help”, I don’t want to get an event window (i.e. a normal XMPP message stanza) with the response, I want it to be in my chat window. Assuming you’re using the Smack client libraries, I think it’s just a matter of setting type=”chat” on the Message object (from my memory at least!).

    With regards to pushful data, do eLabs or other warwick groups have user-centred (rather than emergency) SMS tools? Feel free to email me back if you want to take this out of the public eye ;)

    21 Sep 2008, 18:28

  6. Mathew Mannion

    Interestingly, when using Pidgin/Adium, ask messages do respond directly in the message window – bizarre! I’ve added message.setType( to the code and have redeployed the bot – can you let me know whether you still get the same behaviour now? Is this the behaviour with the Smack client?

    We have investigated (and partially developed) an SMS system for pushing useful messages out, but it was sidetracked due to changing priorities, if I recall correctly. You’d have to talk to someone higher up to get the full story I think, since I didn’t work on it myself :) For the meantime, we are using JanetTxt for our emergency SMSing.

    21 Sep 2008, 21:28

  7. Mathew Mannion

    Of course, I mean Spark, not Smack :)

    21 Sep 2008, 21:38

  8. Phil Wilson

    I was using Psi at the time. Seems to work now, thanks!

    22 Sep 2008, 10:38

  9. Mathew Mannion

    You can always add me ( if you want to discuss further :)

    22 Sep 2008, 10:42

Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

<about />


I’m a Web Developer in e-lab, part of IT Services at the University of Warwick.

<input type="search" />

<ol id="recentComments">

  • First commit was around 12 months ago, but there was a long period where it was put on the back burn… by Mathew Mannion on this entry
  • So, how long did it take?! by Phil Wilson on this entry
  • Hi Matthew, having a problem putting favourites into folders, it only seems to let me put them in on… by Rupert Elder on this entry
  • I wrote one entry in Chinese. It published the content as lots of questions marks. by Hongfeng Sun on this entry
  • I'm sure there's a setting somewhere that'd filter it out. by Nick on this entry

<ol id="archive">


Am I still fat?

Not signed in
Sign in

Powered by BlogBuilder