For a couple of centuries before its destruction by fire, the Royal Library at Alexandria was the hub of European civilisation.
Its hundreds of thousands of scrolls, covering every aspect of classical science and culture, were not all gained honestly: there are stories that ships calling into the port were required to lend all scrolls on board to the library for them to be copied and returned. When its end came, the Mediterranean lost a unique research facility, and many previously seminal works were lost to oblivion.
As we entered the New Year in 2009, those who had built their ideas and experiences into blogs at JournalSpace.com over the last six years were shocked to discover that it had suddenly gone the way of those invaluable scrolls.
Apparently, the system administrator who was responsible for designing and maintaining the site and its SQL server had not been backing up the database that made up the content of the blogs. When confronted with a charge that he had been stealing from the company, it is claimed that he overwrote the database on his way out of the door.
There are many lessons to be learned from JournalSpace.com’s disaster, and few of us are completely free of the flaws that were their undoing.
Their site ran on Mac OS X Server, using a RAID mirror configured over two large hard disks. As soon as the problem was manifest, both drives were sent to a data recovery firm, but they discovered that the database had been overwritten rather than damaged, and was thus completely unrecoverable. Attempts to recover blogs using facilities such as the Internet Wayback Machine failed, because the site blocked such archiving, although some bloggers were able to use Google caches to reconstruct their writings.
RAID mirroring is not backup, but then neither is any single technique, nor any software product. Backup is a thorough, in-depth, deliberate, risk-assessed strategy that is detailed in a policy and implemented in standard procedures. One of the commonest questions asked by novice system administrators (and users who lack expert supervision) is what backup software they should use, when they have not thought through their backup policy or procedures.
Still more shocking was the discovery by the management of JournalSpace.com that their database was not backed up, only after it had been blown away. They appeared to have delegated business-critical details to a single employee, without ever checking that they had performed test restorations, for instance.
Every business that depends on computers for data storage and processing must bring those processes into its day-to-day management, not simply trust them to some techie(s) who are left to get on with it on their own. Even when organisations do have sound policies and practices, it is common for backups to be made less frequently than intended, or for them to prove inadequate when used to restore vital data. If management does not engage in periodic tests of backups by restoration and recovery, these will not come to light until data has been lost, for example after a crash.
A more general lesson from both the loss of the Royal Library at Alexandria and JournalSpace.com’s disaster is that valuables are best distributed.
The Internet, and civilisation as a whole, has high resilience because assets are distributed. Sites and sources are mirrored on different continents, so that when several undersea cables become mysteriously severed, you can still download information and software from an alternative location. Even artists with very small outputs, like Vermeer, have works spread across many different galleries, so that if any one were completely destroyed his work would not disappear with it.
Whilst comparing JournalSpace.com to the Royal Library at Alexandria is perhaps inflating the importance of a few thousand blogs, these concerns extend to Cloud computing. In theory, a Cloud service can do all the right things, with multiple geographically-separated data centres, multiple redundancy, frequent archiving, and more. To match that level of data security would be prohibitively expensive in even medium-sized organisations.
But in practice you, as a client, are unlikely to be able to check that your service provider is doing all the right things, and that when the library catches fire or the disgruntled and dishonest employee leaves, your data will be left unscathed.
JournalSpace.com is not the first such site to vanish after disaster (Diary-X did so in early 2006, for example), and it will sadly not be the last.
In June 2014, Codespaces.com was lost without trace after it was hacked. This time the problems arose from a deliberate attack, in which an intruder took over the site’s control panel and demanded money.
Its Cloud storage was not hacked, but before its operators could regain control, most of the site had been blown away by the intruder. As they had no independent backup of what was stored on Amazon’s Cloud servers, using a single set of controls the attacker was able to delete almost all the data.
Let’s hope that this sort of event never happens to a Cloud service itself.
Updated from the original, which was first published in MacUser volume 25 issue 05, 2009.