... dealing with NeCTAR's "IPs are not forever" problem

The Problem

IP addresses are the network addresses used to identify computers / systems in order to connect to them over a network.  You typically use them directly (e.g. as in the URL"http://130.102.154.157/foo") or via a name in the Dynamic Name Service (DNS); e.g. "http://zeus.rcc.uq.edu.au/foo".

The problem we are trying to address here is that NeCTAR does not provide a long-term stable naming mechanism for virtuals.  Specifically:

  • When you Terminate a NeCTAR instance, its IP address is likely to be reassigned to a new instance that is owned by someone else.
  • If you then Launch a new instance to replace the old one, you cannot request the original instance's IP address, and the chances are that you will get a different one.

The net result is that when you need to terminate and recreate a NeCTAR instance, any person or service that relies the old IP address for your service will need to be informed of the new one.

Is it really as bad as that?  Yea, pretty much!

  • If you are quick and lucky, you may get your old IP address when you launch the replacement instance.  But you can't count on it.
  • The NeCTAR node administrators may be able to change a NeCTAR instance's IP address, but it is difficult. The problem is even more difficult if your instance's old IP address has already been reassigned to some other user's instance.

The normal way to deal with IP address reassignment is to use a stable DNS name for your service, and update it whenever your server's IP changes.  People who are using your service's DNS name won't notice that the IP address has changes. Unfortunately, NeCTAR does not provide a DNS for instances with the right characteristics.

So what can you do about it?

With a bit of planning, and some help from your local IT support, you can largely mitigate the problem, using an external DNS entry.

The basic idea is that you ask your local IT support people to create a DNS record for your NeCTAR instance with a memorable name in your local institutional nameserver. This is the address that you give out to the users of your virtual; e.g. in the form of a URL. The DNS record for your memorable name needs to be a CNAME (canonical name) which refers to the dynamic DNS name that NeCTAR provides for your instance.  The dynamic DNS name in turn will have a DNS record (in the NeCTAR DNS servers) that gives your instance's actual IP address.

How does that help?  Suppose that you have to terminate and relaunch your NeCTAR instance, and that after the relaunch the IP address has changed.  Rather than telling everyone they need to use a new IP address, you ask your local IT support (the people who created your CNAME) to change the entry to point to the dynamic name for the new instance.  Once this change has propagated through the DNS services, people will be able to use your service's memorable name to connect to the new instance.  This may take a few hours ... depending on the TTL (time to live) of the CNAME record.  But we can do better than that.  (Read on.)

A planned IP address change

The previous scenario assumed that the IP change needed to be done without any prior notice.  We then had a period during which DNS lookups would fail, or they would send the user to the wrong server.

If you know well ahead of time that you are going to need to change the IP address, you can do the following:

  1. Do a DNS lookup to find out the TTL for your CNAME record (e.g. see example below).
  2. Ask your IT support to change the TTL on your CNAME record to (say) a couple of minutes.  This has to happen at least 6 hours before the switch over.
  3. Ahead of time (if possible) launch the new instance and set it all up.
  4. At the appointed time:
    1. turn your current instance's services off
    2. transfer remaining data to the new instance as required
    3. start the services on the new instance
    4. get the local IT support people to change the CNAME to point to the new instance, and to set the TTL to the normal value.
  5. Finally, at your leisure you can complete the decommissioning of the old instance and Terminate it.

If you plan this carefully (and your local IT can do the DNS changes in a timely fashion), your service will be out of action for a minimum time.

Example DNS records

The following is the output from a DNS lookup for the "Zeus" instance which we have set up with a stable DNS name, as described above.

$ nslookup -debug zeus.rcc.uq.edu.au
Server:        130.102.128.43
Address:    130.102.128.43#53

------------
    QUESTIONS:
    zeus.rcc.uq.edu.au, type = A, class = IN

    ANSWERS:
    ->  zeus.rcc.uq.edu.au
    canonical name = vm-130-102-154-157.qld.nectar.org.au.
    ttl = 3600
    ->  vm-130-102-154-157.qld.nectar.org.au
    internet address = 130.102.154.157
    ttl = 3506

    AUTHORITY RECORDS:
    ->  qld.nectar.org.au
    nameserver = ns1-qh2.melbourne.nectar.org.au.
    ttl = 20674
    ->  qld.nectar.org.au
    nameserver = ns1-np.melbourne.nectar.org.au.
    ttl = 20674

    ADDITIONAL RECORDS:
    ->  ns1-np.melbourne.nectar.org.au
    internet address = 115.146.83.251
    ttl = 333
    ->  ns1-qh2.melbourne.nectar.org.au
    internet address = 115.146.81.251
    ttl = 145
------------
zeus.rcc.uq.edu.au    canonical name = vm-130-102-154-157.qld.nectar.org.au.
Name:    vm-130-102-154-157.qld.nectar.org.au
Address: 130.102.154.157

In this case, the "rcc.uq.edu.au" namespace is managed by UQ's ITS, so asked them to create the CNAME record for us.  You could also purchase a domain name in a suitable domain from a commercial DNS hosting service, and create the CNAME record there.

References