The problem we are trying to solve

While we tend to think of NeCTAR Instances as living "in the cloud", in fact at any point in time a NeCTAR Instance is tied to a specific computer in a specific data centre.  Normally, we don't care about this.  However, sometimes it matters.  For example, a virtual may need to be moved because the computer needs to be maintained, or because the virtual needs to be nearer some stored data (e.g. an RDSI collection).

We call the process of moving a NeCTAR virtuals "migration". Unfortunately, the NeCTAR OpenStack infrastructure cannot do migration of Instances automatically. Instead, migration of NeCTAR virtuals needs to be done manually, and even then there are some important caveats.

(It should be mentioned, that there is potential alternative to migrating. If you know for sure that you don't need to keep anything in a given NeCTAR Instance, the simplest approach is to simply Terminate it, and then launch and set up a brand new Instance in the new place. But the problem is that Terminating an Instance is final.  There is no going back.)

Migrations using snapshots

Let us assume that we want to migrate an Instance from one place to another.  The simple way to do this is take a snapshot of the Instance at its old location, and then launch a new Instance from the snapshot in the new location.

In more detail, the steps for doing a migration (safely) are as follows:

  1. Make sure that you have an up-to-date backup of anything important on the Instance.
  2. Shut the Instance down.  (Don't Terminate it yet!  Only Terminate the old Instance when you are sure thay the migration worked.)
  3. Create a Snapshot of the shut-down Instance.
  4. Launch a new Instance from the Snapshot.
  5. Check that the new Instance is working, and that it contains all of the files, etc from the old Instance.
  6. When you are absolutely sure, you can Terminate the old Instance.  You can also delete the Snapshot that you used for doing the migration.

 

That sounds simple doesn't it.  Unfortunately, there are few issues that can complicate ... or derail the process.

Most importantly, if you mess up the migration procedure or overlook one of the caveats, it may fail result in a migrated Instance that doesn't work properly, or that is missing data.  You need to be careful, and take the recommended precautions above.

Caveats

Here are some of the issues.

  1. The process above requires sufficient quota (for instances, VCPUs, memory, etc) to be able to have the old and new Instances in existence at the same time.  This may be more than your current project resources.  If you are in this situation, you need to ask for more resources temporarily.
    For QCIF (Queensland) users, just visit the QRIScloud support page and raise a ticket, and we will expedite a temporary quota increase for you. But the flipside is that we will expect to take the extra quota back once you have finished the migration.
  2. When you launch a new Instance while the existing Instance still exists (running or shut down), the new Instance will have a new IP address.  (Even if you Terminated the old instance first (risky!!), it is bit of a lottery as to whether you would get the same IP address for the replacement Instance.)
  3. If you have proprietary software license keys that are tied to MAC addresses, you will need to get new keys. NeCTAR OpenStack does not allow you to specify the MAC address of a new Instance.
  4. NeCTAR Instance Snapshots DO NOT include the state of the Instance's ephemeral volume.  Typically, that means that the simple migration process won't transfer anything in your "/mnt" file system (i.e. the "/dev/vdb" device) across to the migrated instance.  (That is why it is called "ephemeral"!)
  • If your Instance uses NeCTAR Volume Storage, there are two things to note:
    • The Volumes may need to be snapshotted and moved separately from the Instance.  (Volumes typically can only be attached to Instances in the same availability zone.
    • You must detach any Volumes from your old Instance before terminating it.  (There is a rather nasty bug in OpenStack that causes serious problems if you attempt to Terminate an Instance while it has Volumes attached. Recovering your volume requires the NeCTAR Node operation staff to manually patch a database, which is a messy and potentially risky procedure.)

 

Migration with ephemeral storage

We mentioned above that a Snapshot of an Instance does not include the Instance's ephemeral storage / file system.

In fact, the NeCTAR OpenStack infrastructure doesn't provide any way to copy the ephemeral "disk".  So how do you deal with it?  Basically, there are 3 options:

  • This simple option is to simply ignore it.  Generally speaking, it is a bad idea to use the ephemeral file system to hold the sole copy of anything. If you have taken this advice to heart, then it shouldn't matter if the data in the Instance's "/mnt" file system was wiped.
  • The second option is to do the migration as above (up to but not including the final Terminate!!) and then do the following:
    • Start up the old Instance
    • Use "scp", "rsync" or similar Linux tools to copy the files from "/mnt" on the old Instance to "/mnt" on the new instance.  For example, here's how to do it if the files are owned by your service account at both ends, and you are currently logged into the "old" instance:
      $ scp -r /mnt/* nnn.nnn.nnn.nnn:/mnt
      where nnn.nnn.nnn.nnn is the IP address of the other system.
    • Do the final Shutdown and (ultimately) Terminate of the old Instance.
  • The final option is to create a backup of the contents of "/mnt" before you start the migration process, and save it to "somewhere else".  (That "somewhere else" be a NeCTAR Object Storage, RDSI Collection Storage, ... storage at your home institution, or even the hard-drive of your laptop or workstation (if that is also backed up).  Then, after you have completed the migration, retrieve and restore the backup onto "/mnt" on the new Instance.