What do you mean by backups?

Here's the problem.  Different people mean different things, ranging from:

  • an ad hoc copy of the data on a DVD in the bottom draw of the researcher's file cabinet,
  • regular snapshots copied to another machine; e.g. a departmental file server, or
  • a full "backup solution" which manages the snapshots and allows you to retrieve or restore individual files and directories from old snapshots.

Then there are a number of parameters that determine whether your backups are adequate to your needs.

  • Are the backups automated, or do they rely on the user (or an administrator) doing things by hand?  (A manual system is typically less reliable ...)
  • How frequently is a backup taken?  Hourly? Daily? Weekly?  Bear in mind that any data that is written to storage after your last backup won't be recoverable.
  • Is the backup copy on a different machine?  In a different machine room?  In a different building?  On a different site?
  • Are the backup copies online or offline?  How long will it take to retrieve data from a backup?
  • How long are backups kept?

The bottom line is that a different people ... and different kinds of data ... need different levels of backing up. And the "better" your backups are / need to be, the more it is likely to cost (someone!) to implement.

Do I need to worry about it?

Of course you do!

There are many ways that you could get into a situation where you need a backup copy:

  • Disk failure: It is hard to get a reliable estimate of MTBF (mean time between failure) for a typical disk drive, but it is generally believed to significantly shorter than the numbers given by the manufacturers.  (And those numbers are typically for total failure ... not for sector loss. ) This can be mitigated by RAID disk storage, but RAID is not bomb-proof:
    • If a RAID system loses multiple discs at the same time, or serially and people didn't notice (seen that!), then data can be lost irrecoverably.
    • The RAID controller or the file store entire could die.  The situation is usually recoverable, but it could take a long time.
  • Environmental issues: for example, UQ had a fire in an ITS server room not too long ago.
  • Unauthorised access: for example, a hacker with an anti-science agenda, or ... ransomware
  • Accidental damage: somebody (yourself, a co-worker, or your IT service provider) accidentally deletes a bunch of files.

The only case where you don't need to worry about backups is when your data has no value to anyone!

Whose responsibility is it?

Ultimately, it is yours!  At least, it is your responsibility to decide on what level of backup your data requires, and your responsibility to make the necessary arrangements. 

Are my Backups actually working?

One trap that people often fall into, is to set up their backups and assume that they are working.  Then, a few months or years later when they need to restore some files, they discover that the backups have stopped working ... or that they never worked in the first place ... and that the files that they need cannot be restored.

There are two things you can (and should) do to avoid this:

  1. Monitor your backups.  Ideally, try and get it to email you (or something) when a backup run fails.  If you can't do that, then you need to regularly check the logs or the backup system status, or whatever.
  2. Check that you can restore files.  Pretend that you have lost a file or set of files, and go through the procedure of actually restoring it.  You definitely need to do this when you set up your backup regime, and after it has (supposedly) bedded down, and it is a good idea to do it periodically after that ... in case there is a problem that your monitoring is missing.

Backups versus Archiving

Archiving is qualitatively from backup. 

  • Backups are "insurance" against accidental data loss due to software / hardware / environmental factors ... or user error (aka "fat fingering").  Backups have a limited useful lifetime, and are generally kept for a relatively short period of time (e.g. a few months).
  • Archiving is about creating and retaining offline copies with the intention that they be locatable and retrievable over long time-scales. Indeed, a common use-case is to archive the data so that the online copy can be deleted to free up disc space for other things.

Where to put your Backups

There are a variety of places that you could put backups for your virtuals.

  • You can use NeCTAR Object Storage (aka Swift) as discussed here.
  • You can use a cloud storage provider, though you will most likely need to pay real money for this if your backup needs are significant
  • You can use services provided by your local or university-wide IT services.
  • You can use a external drive or DVD attached to your PC or laptop.  This is not really a sustainable or scalable solution, for a variety of (obvious) reasons.
Ideally, you want a recent copy of your backups to be off-site, or at least in a different building to the primary online version.  This is to guard against the possibility of a building fire or similar disaster destroying both your online and backup copies.