November 28, 2007

Datatastrophe

Writing about web page http://media.www.utahstatesman.com/media/storage/paper243/news/2007/11/26/CampusNews/Blackboards.Shutdown.Statewide-3115226.shtml

Golly. From the Utah State University Student Newspaper

Due to corrupt data in the statewide database Blackboard Vista, the system shut down, causing thousands of faculty and students at USU to lose all information submitted between Tuesday, Nov. 13 and Friday, Nov. 16.

This data is apparently irrecoverable, and at least one academic there is quoted as saying that he will have to re-mark (and his students will have to resubmit) several hundred papers, and he will have to resubmit thousands of grades into Blackboard. The system administrators plan to have a mirror server in place “by the spring term”.

Additionally, the Chronicle reports that administrators believe that the crash was caused by “a problem with a network port”. Yikes.


- 6 comments by 4 or more people Not publicly viewable

[Skip to the latest comment]
  1. Mathew Mannion

    They didn’t have a backup? Good grief.

    28 Nov 2007, 10:14

  2. There is remarkably little bile and outrage about this at the moment – at least on the blogs that I monitor. It seems that Utah has a history of failures with Blackboard and people are resigned to it. Maybe it’s time to re-evaluate the systems they have in place?

    28 Nov 2007, 11:49

  3. Simon Harper

    Wow. On one hand I guess it’s a case of ‘there for the grace of God go I’, but it sounds as if it’s an unusually large cock up. You can imagine the conversation: “You mean to tell me that the last useable backup is from Tuesday? but today is Friday!...”

    I put all my data on a CD and send it to myself in the post, just to be on the safe side ;-)

    28 Nov 2007, 15:38

  4. John Dale

    Absolutely. But if, as it sounds like, it was a “creeping corruption” problem, then I do sympathise; it’s a very hard problem to recover from, because even if your backup routine is meticulous, once the corruption gets into the backups, you’re stuffed and have no choice but to roll back to before the corruption started.

    28 Nov 2007, 17:10

  5. Simon Harper

    I must say I do have sympathy with anyone who finds themselves in this situation, especially as I’ve been there myself. It’s been a good few years now (fingers crossed!) but the lesson learned there is that backups are not enough. We now have automatic restore, and simple tests of the restored data, included in the backup routine.

    29 Nov 2007, 09:23

  6. ez

    From the articles, the port on the switch in the SAN was bad and not throwing errors. It wasn’t until it started acting up again on Nov 26th and restarted that it started throwing errors.

    If you backup a data file with corrupted data causing issues in a table, then its not of any use to you. They went to their most recent backup without the corrupted Also, they apparently had some bad archive logs so they could not roll forward from the backup. Its a DBA’s worst nightmare.

    29 Nov 2007, 14:46


Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

Search this blog

Tags

Blog archive

Loading…

Most recent comments

  • I'm looking for two authors/books from the 1970s or early 80s. The first was set in England. All I r… by Leo on this entry
  • I'm looking for two authors/books from the 1970s or early 80s. The first was set in England. All I r… by Leo on this entry
  • I am trying to find a book about a grandfather who tells his grandson that if he imagines hard enoug… by hilary woolf on this entry
  • Hi Looking for a series of books in which the main character was a knight. The knight was either a f… by Ely McKenna on this entry
  • I'm trying to track down my favourite children's book from the early 1970s (pub. 1970 – 1973?) about… by Ally Holloway on this entry
Not signed in
Sign in

Powered by BlogBuilder
© MMXXI