All entries for Tuesday 26 June 2018

OMERO deployment – the aftermath

In our previous blog post, I have presented some detail on the process we went through to get OMERO up and running. Of course, that is only a part of the job: after it's up and running, we need to make sure it's working properly, that it can "talk" to existing data, that it's being updated and that people are actually using it. This is still very much a work in progress, we are slowly getting there. In this post, I will just go through some points that are important for a live OMERO install.

- Integration with existing data

Most of the PIs around here are (at least to some degree) committed to adopting OMERO moving forward. Storage for our install is using our brand new, petabyte-scale server. The future is more or less taken care of; but how about the past? People have data in other servers. They have their ways of doing things, and changing those mean a bit of extra work. Inertia is a hell of a thing. The big question becomes: how do we deal with that and get people on board?

There are a couple of avenues we have been exploring. The first one is using an the in-place import feature of OMERO. This allows for existing files to be imported into OMERO without needing to physically upload them to our storage server. In practical terms, it means we can import the data from other storage servers without creating duplicates. People can see and use their older data in OMERO without needing to do any transferring.

There are downsides, of course. The main one is that it either requires "freezing" the imported data on their actual locations (making the files read-only, for example) or regularly checking for changes on the files locations and then deleting and reimporting the changed data. The latter option is not great (deleting and reimporting means we would get rid of annotations, attachments and so on), and the former option limits the data we can actually import to "dead" datasets, i.e. datasets where data will not change or be moved.

Currently, we are not doing any of that. It turns out that our storage server does not want to play nice with OMERO. In-place importing relies on symlinks to point to the existing data, and somewhere in that process we are hitting a snag. We're still working together with IT services to get that one sorted. In the future, our plan is to do such imports on a user-by-user basis, explaining clearly that the imported data will become read-only and that they won't be able to move it any longer.

- Maintaining an OMERO server

This will be a very short section. Other than troubleshooting the eventual issue, maintenance consists basically of updating OMERO itself and the systems on the server machines. The latter is managed by our good friends in IT Services; the former is a fairly painless process on our end. The instructions on the OMERO website are very good, but also very comprehensive. For minor updates, it consists mostly on replacing the binaries and moving over custom scripts.

- There are still issues...

We are still sorting things out. Our main issue for the moment is adoption: our shiny server is not being very heavily used. As I previously mentioned, there are all sorts of explanation for that: inertia is probably the big thing. To deal with that, we're now testing a solution to import data from microscope computers automatically into OMERO (blog post on that soon!). Not only that will mean people don't need to book time in the microscopes only to transfer their data to a storage server, but it will also mean that, since their data is already in OMERO, the entry barrier is much lower.

Another issue is that some of our older microscope computers just cannot do OMERO at all. The OMERO client requires a Java version that just cannot be installed on those computers (I've recently learned what happens when you try to force a Linux system to use a newer glibc version than it's supposed to...). This will most certainly be mitigated by auto importing data into OMERO, so it's not a huge deal.

In general, running an OMERO server has been a pretty smooth process. Other than a few snags (certain file formats tend to crash the server...), our main obstacle is just getting people to use it. It will take time and it might take some work, but I'm convinced we'll get there!

Erick Martins Ratamero : 26 Jun 2018 15:29 |  Comments (0) |  Close comments |  Report a problem