Preamble

There is lots of material on the Opscode site about how to use Chef in its various forms. The new Chef Quickstart Guide is a good place to start learning.  But this page is for people who are impatient, and just want to get on with it.  The assumption is that you have a basic understanding of Linux system administration, and that you have sufficient access to run "root" privilege commands using "sudo".

Berkshelf provides a better way of managing your Chef cookbooks than the vanilla Chef toolchain; e.g. using "knife cookbook site" to install community cookbooks.  Berkshelf requires a bit more setting up, but it is worth it.

Warning!!

Whenever you do things using Chef (or indeed, any other system administration tool!) there is the potential for causing irretrievable damage to the system you are administering.

Advice

  • When using "sudo", be careful, and make sure you understand what you are doing before you do it.  There is no magic "undo" button!!
  • Try things out first on a test machine or virtual. For example, use your NeCTAR trial project to create a "throw away" virtual, or use Vagrant to spin up a virtual on your own desktop.
  • Before doing something potentially dangerous on a production virtual (i.e. one that you can't afford to trash), make sure that you have an up-to-date backup to restore from, or a snapshot to revert to.

Getting started with Chef Solo & Berksfile

This is a version of the "Getting started with Chef Solo" procedure modified for use with a Berkshelf; i.e. a chef-repo that uses a Berksfile to configure its external cookbook dependencies. These instructions are suitable for Chef 11.x and Berkshelf 3.x.  (The chef-repo will be a clone of "nectar-cookbooks/chef-solo-repo" from GitHub.)

(Note: a Chef + Berkshelf installation takes ~90Mb of disk (typically on the "/" partition), and the installation process may require 150+Mb more.  If this is an issue for you, see "Juggling Disk Space" below.)

The Quick Way

The easy way to install Chef and Berkshelf on a fresh NeCTAR virtual is to cut-and-paste the following into the "Post-installation script" text-box when you launch your instance.

#!/bin/sh
wget -O - https://github.com/nectar-cookbooks/chef-solo-repo/raw/master/scripts/bootstrap-nectar.sh \
| /bin/sh > /root/bootstrap.out 2>&1

(For the "chef-dk" version, change "bootstrap-nectar.sh" to "bootstrap-nectar-chefdk.sh".)

These fetch and execute a "bootstrap" script from the "nectar-cookbooks" project, and then run it. When you SSH to your newly launched instance, you can "sudo tail -f /root/bootstrap.out", "top" and "ps" to check on the bootstrap progress.  When it is finished, you chould have a "chef-repo" installed ready to go.  The last step of the bootstrap is to run the "setup" recipe.

You can use the same scripts to bootstrap Chef + Berkshelf and a chef-repo into an existing NeCTAR instance.

The Long Way

Following instructions will install the latest production Chef release, and then add Berkshelf 3.0 to it.  It you would prefer to use a Chef-DK release (see the Chef-DK download page) you need to install "git", download and install the Chef-DK package for your platform... and then skip to step #4.

  1. Install the prerequisites

    sudo yum install curl git gcc gcc-c++ ruby-devel tar autoconf

    or

    sudo apt-get update
    sudo apt-get install curl git build-essential ruby-dev
  2. Install the latest version of the Chef client tools:

    sudo bash -c "curl -L https://www.opscode.com/chef/install.sh | bash"
  3. Install Berkshelf:

    sudo /opt/chef/embedded/bin/gem install --no-rdoc --no-ri berkshelf
    sudo ln -s /opt/chef/embedded/bin/berks /usr/local/bin
  4. Create a directory for doing Chef Solo work.

    sudo mkdir /var/chef-solo
    sudo chown $USER /var/chef-solo
  5. Clone the master Git repository.  (This needs to be a "chef-repo" with a "Berksfile" in the top directory.  For example https://github.com/nectar-cookbooks/chef-solo-repo.git)

    cd /var/chef-solo
    git clone https://github.com/nectar-cookbooks/chef-solo-repo.git
    cd chef-solo-repo
  6. Create a node definition by copying the example file provided:

    cp solo/sample-node.json mynode.json
  7. Use Berkshelf to install the dependent cookbooks (as specified in the Berksfile) into the "chef-repo/cookbooks" directory ... which is where our "solo/solo.rb" says to look.

    berks vendor cookbooks

    (The "vendor" command is new in Berkshelf 3. For Berkshelf 2, run "berks install --path cookbooks" instead.)

  8. Run the Chef recipes using chef-solo:

    sudo chef-solo -c solo/solo.rb -j mynode.json

Some tips on using Chef and Berkshelf

Berksfile and Vendoring

When you ran "git clone" to create your "chef-repo-solo" file, one of the files that you fetched was the Berksfile.  This file is essentially a directory of cookbooks, and where Berkshelf can get them from.  Then, when you ran "berks vendor cookbooks" the first time, Berkshelf fetched the latest versions of cookbooks into your shelf (in your home directory) and then created a "vendor" directory called "cookbooks" in the current directory. The "cookbooks" directory will then contain a snapshot of the cookbooks as they were when you "vendored" your Berkshelf.  (Please excuse the terminology...)

When you then run "chef-solo" using the "solo/solo.rb" config file supplied, it will look for the cookbooks in the vendor directory.

(The alternative approach is to use the cookbooks directly from your cookbook, put explicit version constraints into your "Berksfile", and/or check your "Berksfile.lock" file into version control.)

Updating your vendored cookbooks

Suppose that you want to refresh the "cookbooks" snapshot to pick up the latest versions of the cookbooks that you are using. Here's what you do if you are using Berkshelf version 3:

cd /var/chef-solo/chef-solo-repo
berks update
rm -rf cookbooks
berks vendor cookbooks

The "berks update" command fetches the latest versions of the cookbooks mentioned in your Berksfile (and their dependencies) subject to any version constraints.  These are place in your berkshelf directory.

The "rm" and "berks vendor cookbooks" commands then rebuild the vendored snapshot from the berkshelf.

If you don't tweak the Berksfile, this procedure will give you the HEAD version from the "master" branch of the respective Git repositories.  This assumes that the person doing cookbook maintenance follows "best practice", and only pushes properly tested changes to the "master" branch.  If there are problems, you can modify the Berksfile to pull from a different branch (or different repo), or to use a specific version.

You can also do a selective update; e.g. "berks update omero" will only update the "omero" cookbook and its dependencies.

If you are using Berkshelf version 2, then the commands are as follows:

cd /var/chef-solo/chef-solo-repo
berks update
berks install --path cookbooks

Updating your "chef-solo-repo" directory

From time to time, we add new cookbooks to the "nectar cookbooks" collection.  When we do this, we need to create new entries in the master Berksfile.  We might also make changes to the other files in the "chef-solo-repo" tree.

To pick up these changes, you should just be able to do a "git pull" from the master Git repository for "chef-solo-repo".  If you have tinkered with Berksfile or other files that are under version control, you will need to commit the changes (locally) before the "git pull" will succeed.

Juggling Disk Space

As noted above, a Chef + Berkshelf installation takes a significant amount of disk space, and the installation process takes even more.  Here are a couple of tricks that will help if your "/" partition doesn't have enough space.  (This assumes that you are using the bootstrap scripts.)

  • The Chef and Berkshelf installables are actually installed in either "/opt/chef" or "/opt/chefdk" (depending on which version you use).  You can get the package installer to put it somewhere else by using a symlink.
  • The bootstrap script downloads the chef or chefdk installer into the current directory.  So make sure your current directory is on a file system with enough space.
  • Berkshelf will cache the versions of the cookbooks that you use in your "~/.berkshelf" directory.  If you are doing cookbook development, that could build up to a large size. You could move and symlink it.
  • You could also move and symlink "/var/chef" and "/var/chef-solo", but I wouldn't recommend it.
Filed under: , ,