Unplanned downtime
Blogs, together with all other IT systems in the university, have been unavailable for much of today. Early this morning there was a fire in an electricity sub-station in Earlsdon. This caused a high voltage switch on the University's inner power loop to blow, which took out power to some buildings on campus including the main ITS machine room.
When the power was switched back on it blew an electrical circuit board in the ITS machine room. Electricians worked throughout today to replace this component and power has now been restored to the room.
So far the campus network, internet connection, web services (SiteBuilder, Blogs, Forums, etc.) and most of the core components that allow customers to log on to their workstations have been restored. The SquirrelMail email service for students has also been restored.
The GroupWise email service remains unavailable as is the ResNet service and the FARGO data server. Work will continue on restoring these and other systems throughout the weekend.
John Dale
17 Mar 2006 18:55
| 
Tags: ITS
| 
Comments (65)
| 
Report a problem
65 comments by 5 or more people
[Skip to the latest comment]ResNet seems to be working again. At least, i assume the fact that I'm writing this from my room in Tocil means that ResNet is working and this isn't a giant hallucination.
17 Mar 2006, 19:09
Likewise David (although I am also Tocil). I assume Tocil is ResNet and not main campus internet?
17 Mar 2006, 19:11
John Dale
Excellent; I presume this means that the restoration work is going faster than anticipated.
17 Mar 2006, 19:33
In terms of groupwise. We appear to be back to normal service. That being the lovely page cannot be displayed service.
17 Mar 2006, 19:53
Do you know if the UPS worked as intended, and things got fried when they plugged in the mains again? It's implied by the newsflash, but it's not clear.
17 Mar 2006, 19:58
Well ResNet worked for a bit then stopped working again… Seems fine this morning though!
18 Mar 2006, 10:23
ResNet is up. Groupwise is still down.
18 Mar 2006, 11:07
Access to the ResNet has now been restored for all users.
If you have any problems please contact the Student Computing Helpdesk on 024 765 75000 (or 75000 internally).
18 Mar 2006, 13:15
Ironic really. 2 minutes after my last comment ResNet falls over again.
18 Mar 2006, 13:40
It makes me sad when WB goes offline, glad its back now! Although still feel slightly lost without groupwise (how dependent on email am I?!) Thanks for letting us know whats going on, John.
18 Mar 2006, 14:10
IT Services now predicting anything up to a week for Groupwise to be reinstated – this is getting beyond ridiculous now, surely?
18 Mar 2006, 16:36
The reason you may have experience temporary connection issues with the ResNet was due to the fact it was being reconfigured and tested by the networks team.
As for Groupwise, the newsflash will be updated as we receive updates. Also please consider migrating to SMail, as this service is up and running. You can migrate at link
18 Mar 2006, 16:46
All those who bemoan groupwise – change to Squirrel! It rarely goes wrong and works quickly and efficiently. It's also simple to change – just click here and follow the info on screen. People really should be using a client such as Outlook/Outlook Express/Thunderbird to manage their e-mails instead of a web-based system too – you are then not space limited (except by your hard disk) and you can access e-mails you've already looked at faster and you don't need to be online.
18 Mar 2006, 16:51
Groupwise provides me with endless amusement. I'm staying put for now.
18 Mar 2006, 16:56
I use Outlook Express as my e-mail client, but it's completely redundant when it can't connect to the server you're trying to access in the first place.
18 Mar 2006, 17:20
Thomas – in that case, you have even less of a case for switching from Groupwise to Squirrel! You should be re-instated with service the next day on the Squirrel system.
18 Mar 2006, 17:21
So, if I migrate to Squirrel now,will I still be able to access groupwise later for old emails, addresses, etc? 'Migrating' just means forwarding any future emails, rather than transferring all data from groupwise??? Sorry, I'm technologically incompetent!
18 Mar 2006, 19:39
No Catherine – I forgot to make this point. Any information you've stored in groupwise will be lost with the transfer I believe (unless the I.T. bods suggest otherwise)
18 Mar 2006, 20:04
From what I read on the insite news, all this has been caused by a power surge after the cut-out frying some of the equipment. Did it not occur to anyone at I.T. services to have surge protection on the equipment. Even most basic P.C. users at home have surge protected extension leads. I am just glad I changed to google mail after the dramatic decline in groupwise service quality with the change over.
18 Mar 2006, 20:24
Richard – I received this comment from Chris May yesterday on my blog:
"It was the distribution board that goes between the UPS, the mains, and the ITS machine room. There are literally hundreds of boxes in there, so as you can imagine it's a fairly beefy bit of electrical kit (about which I know very little)
There was a power cut, the UPS kicked in, and the distribution board went up in a cloud of smoke, taking the machine room with it.
It took estates about 6 hours or so to source a new one and wire it up. As of now (17:30 ish) power, network, and the basics (DNS, DHCP) are back, the other services will come back when they can. The good news is that stuff which lives in university house (like blogs) was largely unaffected, so it's not as much of a nightmare as it has been in the past."
I don't know if it was a surge that caused it to fail, but I think it sounds like dodgy equipment, and I don't think pointing the finger at ITS is fair unless you know otherwise.
18 Mar 2006, 20:54
Catherine,
Your GroupWise mailbox will still be accessible until the end of next term, however all new emails will be delivered to your Smail mailbox.
18 Mar 2006, 21:05
Just to confirm then, Paul – I've made the switch to Squirrel – but I'll still be able to access all the e-mails on Groupwise yet to be received from this period of downtime before the end of the academic year?
18 Mar 2006, 21:24
David Metcalfe
I am just a little suspicious that Squirrel only shows me emails from other Warwick addresses. I tried sending myself an email from my hotmail account earlier this morning and it still hasn't shown up in my Warwick inbox according to Squirrel. Is this the case for everyone or should I take this latest affront personally?
18 Mar 2006, 22:47
Thomas,
Yes thats correct. Any emails sent since friday will be queued for delivery, so you will be able to collect these via POP3/IMAP through Outlook Express (using gwmail.warwick.ac.uk) or webmail. Incidentally, dont forget to update the incoming mail server in Outlook Express, smail.warwick.ac.uk is the Squirrel server you should connect to. So, just to clarify, any emails sent after the switch has been made (usually in the early hours of the morning) will arrive in your Squirrel mailbox.
If you have any problems give us a call on x73737 (024 765 73737)
18 Mar 2006, 23:06
Peter Dunn
Why is Groupwise proving more difficult to fix than all the other systems?
Peter Dunn
18 Mar 2006, 23:45
My Groupwise thingy is still buggered, at least it says 'unable to connect with groupwise server agent' or something. I really would like to check my emails, as my groupwise thingy keeps breaking and it's getting on my nerves a bit.
19 Mar 2006, 12:32
Groupwise is working for me now.
19 Mar 2006, 12:38
Peter Dunn
Still not working for me – just the "Unable to communicate with GroupWise WebAccess Agent" on the web version that we come to see so often over the years we have had Groupunwise.
Peter
19 Mar 2006, 13:12
With regards to David Metcalfe's post, I'm facing exactly the same problem – I've sent test e-mails to my Warwick account (now supposedly switched to Squirrel) and haven't received any of them. Is there currently a delay in switching accounts from Groupwise to Squirrel, are these mails still heading to the Groupwise system, are they being lost completely or what?
19 Mar 2006, 14:12
Ian Liverton
The change over occurs at around 3am the morning after you switch, so if you sent emails before that you won't receive them. Also ensure you're testing from a reliable account – I've regularly had delays of a few days receiving emails from hotmail.
19 Mar 2006, 14:31
Ian Liverton
I should have said they'll continue to turn up in groupwise till 3am, not that you won't receive them.
Also i would imagine that having had mail relays that pass around in excess of 100000 messages a day down for the best part of 24 hours, they're getting hammered pretty hard today so there may well be a delay in sending and receiving through them
19 Mar 2006, 14:40
I've sent test messages to check if the migration has worked, and they haven't turned up in groupwise or squirrel. One from my Warwick account to itself, and one from my Yahoo account.
19 Mar 2006, 16:36
I've yet to receive anything other than the odd blog e-mail into my squirrel account, and a lot of blog reminder messages are bouncing in my account too… I suspect that all is not well just yet.
19 Mar 2006, 16:42
mark
this really is a disgrace, if ITS were a commercial outfit they'd be being sued left, right and centre. Maybe the email stuff should be farmed off to a private company that is actually vaguely competent – I for one am waiting for a very important email about a job and if it falls through because i didnt get the damn email then I am going to be furious.
19 Mar 2006, 16:48
Ian Liverton
Has anybody phoned ITS to explain their problems? I know they've just closed but if you suspect that emails are not being delivered it might be worth a call. If they don't know there's anything wrong – they can't help you! Everybody just seems to assume that they know what the problems are.
19 Mar 2006, 17:08
I don't use my Warwick e-mail account for anything important (such as job related) for precisely that reason. I just can't trust it (or at least I couldn't before I switched to Squirrelmail.) I don't think Novell or ITS would really treat me missing a job interview as a major concern and I certainly wouldn't expect any compensation to be forthcoming. I've just got a place on a course when I finish at Warwick this summer, and all my e-mails for the application process – which has gone on for 3 months – have gone through Hotmail, which has been flawless.
19 Mar 2006, 17:08
Richard Johnson
Maybe it was faulty equipment but as Mark says, ITS are a disgrace. I changed my email account after I couldn't access an email containing DNA sequence data for a project. Because of that, it put me days behind and I didn't finish my project to how I would have liked to. They're never very helpful when you try and get things sorted either, I rang up the other day to try and solve a problem with my girlfriend's laptop (which was bought from warwick) connecting to the library hotspots. I was told that they didn't know much else I could do other than try restarting it!!
19 Mar 2006, 17:12
Has anyone threatened ITS with outsourcing?
Such threats have produced impressive improvements in many an IT department's service.
19 Mar 2006, 20:02
Tim
Well, the networking side of resnet was outsourced to data integration and was brought in house. Presumably after the problems during the first few weeks?
19 Mar 2006, 20:23
Apparently, outsourcing the mail would actually be more expensive that it is now.
19 Mar 2006, 20:59
Chris May
George: Unless I'm misremembering it, during the planning phase for the groupwise replacement project, ITS offered to outsource email, but the university turned it down on cost grounds. So it would hardly be a "threat" :-)
IME, IT outsourcing in public sector & similar organisations is more often than not regarded a failure, if not for the accountants then certainly for the end users. Where have these 'impressive improvements' of which you speak been sighted?
19 Mar 2006, 20:59
Mathew Mannion
Just out of interest, is there any date on when the Unix-delivered email will be working? I get mine delivered there and then use .forward to forward it to my GMail (with my limited knowledge of anything)... Also, has all my email I got since Friday been lost? :(
19 Mar 2006, 21:07
Ian Liverton
In theory email should be queued by the sending mail server until the recipient comes back online (usually up to 5 days), which is why I predicted the servers will have got hammered today as the x000 incomming messages left over from friday are coming through.
19 Mar 2006, 21:43
nick
When I try to log into Groupwise i get…
Unable to communicate with GroupWise WebAccess Agent
..is this what you mean by normal service resuming?
19 Mar 2006, 22:13
nick
I get
Can't connect to the Java Servlet Engine. Contact the administrator of this site.
This morning – a new message, but it still means that i cant access groupwise
20 Mar 2006, 07:47
That's pretty much what I mean by normal service resuming. But I'm sure someone on here is payed enough to disagree with me :p
20 Mar 2006, 08:43
John
On Friday morning, a freind sent me a hilareous picture of a monkey sat on the toilet. I didn't recieve this piece of comedy magic until 8am Monday and for that reason, my weekend was just a tiny bit less joyfull. I for one will be throwing the bookat ITS for their total lack of proffessionalism.
20 Mar 2006, 08:47
Chris May wrote:
I was thinking of Walsall council – they were threatened with outsourcing and were consequently so motivated that they improved performance to where they were far better than any outsourcing contender. Sometimes it's just a matter of being bothered to look for and copy best practice from elsewhere.
Chris May also wrote:
Is the problem that the University is being too cheapskate? A common problem with British business culture – btw isn't the University in the private sector? – its disfunction and arrogance reminds me of private sector monopoly.
20 Mar 2006, 09:52
Ian Liverton
From my last call to the helpdesk. due to the hammering they got yesterday, and the unvailability of groupwise the mail relays have filled up, so they need to pass this mail on before any more can be sent or received, so no mail in or out of camparse for a while….
20 Mar 2006, 10:41
Well, my test e-mails haven't showed up yet, but I have received actual genuine e-mails in my Squirrel Mail account now, so I assume things are back close to normal now.
20 Mar 2006, 14:11
I haven't been able to access my groupwise for days and days…=o(
20 Mar 2006, 19:24
Groupwise has been resurrected now, I think – I got past log-in for the first time in five days.
Squirrel now seems to be picking up all mail – some are without content though – what's that about?
20 Mar 2006, 21:11
Ian Liverton
Afraid I can now log into both my groupwise accounts and mail is slowly chugging through from the weekend. If you still have problems check again, then call ITS.
20 Mar 2006, 21:29
Adam Westbrook
I'm getting most of my emails through, if with a bit of a delay, but a worrying number are arriving blank
i.e. the actual body of text isn't there, just an empty box. Anyone know why this is, and if it can be fixed? Getting a blank email is as bad as getting no emails at all. :(
21 Mar 2006, 15:05
No idea – although I've heard this off a few people.
Maybe worth a call to the helpdesk.
21 Mar 2006, 15:27
groupwise isnt working again. is it just me, or are other people not being to connect to it either? I dont see why they think finalists are worthless and must be prioritised last. Most of are waiting for job offers or interview emails and stuff, and email not working for days is soooo irritating. Anyone have any idea when it will work again?
21 Mar 2006, 15:43
It isn't just you, I can't connect to it either, been trying for a few hours.
21 Mar 2006, 15:50
Cat
I just called the helpdesk about groupwise, as I kept getting the message 'unable to communicate… – they suggested typing in the url, link (i have the login page set as my homepage) and it worked!
21 Mar 2006, 16:12
Hey, that worked for me too, thanks a lot Cat! Now I can finally send that email…
21 Mar 2006, 16:18
Chris May
Please don't use http://webmail.warwick.ac.uk . Use https://webmail.warwick.ac.uk instead (that's https, not http) , that way you won't be broadcasting your user id and password to the entire internet. The helpdesk really ought to know that by now…
21 Mar 2006, 21:36
…
I did ask some years back why traffic wasn't simply redirected to a https connection, but the answer was "there's probably some reason".
Go figure.
21 Mar 2006, 21:39
Ian Liverton
Even RaW automatically redirects you. It probably just cos its external wwritten stuff they don't mess with
21 Mar 2006, 23:07
usually one of those links worked for me but today it's really not working. when will they go back to normal? it's so frustrating.
28 Mar 2006, 13:21
Kam
Well i confess i am becoming increasingly frustrated with this email system. I stupidly use warwick email for all things important such as for jobs correspondence (just because it sounds better then Krazy kam@ something unprofessional.com) and now i regret it. I am expecting important news and just can't access my account. HELP ME PLEASE!
29 Mar 2006, 18:25
Mark
You should have stuck with Krazy kam @ something unprofessional.com…. it suits you.
30 Mar 2006, 09:44
Add a comment
You are not allowed to comment on this entry as it has restricted commenting permissions.