1. Installing

This section describes what you need to install Charliecloud and how to do so.

Note that installing and using Charliecloud can be done as a normal user with no elevated privileges, provided that user namespaces have been enabled.

1.1. Build and install from source

1.1.1. Using release tarball

We provide tarballs with a fairly standard configure script. Thus, build and install can be as simple as:

$ ./configure
$ make
$ sudo make install

If you don’t have sudo, you can:

• Run Charliecloud directly from the build directory; add $BUILD_DIR/bin to your $PATH and you are good to go, without make install.
• Install in a prefix you have write access to, e.g. in your home directory with ./configure --prefix=~.

configure will provide a detailed report on what will be built and installed, along with what dependencies are present and missing.

1.1.2. From Git checkout

If you obtain the source code with Git, you must build configure and friends yourself. To do so, you will need the following. The versions in most common distributions should be sufficient.

• Automake
• Autoconf

Create configure with:

$ ./autogen.sh

1.1.3. configure options

Charliecloud’s configure has the following options in addition to the standard ones.

1.1.3.1. Feature selection

By default, all features that can be built will be built and installed. You can exclude some features with:

option	don’t build or install
--disable-html	HTML documentation
--disable-man	man pages
--disable-tests	test suite
--disable-ch-grow	ch-grow unprivileged builder and ch-tug

You can also say --enable-FOO to fail the build if FOO can’t be built.

1.1.3.2. Dependency selection

Some dependencies can be specified as follows. Note that --without-FOO is not supported; use the feature selectors above.

--with-python

Shebang line to use for Python scripts. Default: /usr/bin/env python3.

--with-sphinx-build

Path to sphinx-build executable. Default: the sphinx-build found first in $PATH.

--with-sphinx-python

Path to Python used by sphinx-build. Default: shebang of sphinx-build.

1.1.3.3. Less strict build

By default, Charliecloud builds with CFLAGS including -Wall -Werror. The principle here is that we prefer diagnostics that are as noisy as practical, so that problems are identified early and we can fix them. We prefer -Werror unless there is a specific reason to turn it off. For example, this approach identified a buggy configure test (issue #798).

Many others recommend the opposite. For example, Gentoo’s “Common mistakes” guide advises against -Werror because it causes breakage that is “random” and “without purpose”. There is a well-known blog post from Flameeyes that recommends -Werror be off by default and used by developers and testers only.

In our opinion, for Charliecloud, these warnings are most likely the result of real bugs and shouldn’t be hidden (i.e., they are neither random nor without purpose). Our code should have no warnings, regardless of compiler, and any spurious warnings should be silenced individually. We do not have the resources to test with a wide variety of compilers, so enabling -Werror only for development and testing, as recommended by others, means that we miss potentially important diagnostics — people typically do not pay attention to warnings, only errors.

That said, we recognize that packagers and end users just want to build the code with a minimum of hassle. Thus, we provide the configure flag:

--enable-buggy-build

Remove -Werror from CFLAGS when building.

Don’t hesitate to use it. But if you do, we would very much appreciate if you:

1. File a bug explaining why! We’ll fix it.
2. Remove it from your package or procedure once we fix that bug.

Please do not use this option routinely, as that hides bugs that we cannot find otherwise.

1.2. Install with package manager

Charliecloud is also available using a variety of distribution and third-party package managers.

Maintained by us:

• Generic RPMs downloadable from our releases page.
• Spack; install with +builder to get ch-grow.
• Fedora/EPEL; check for availabile versions with {yum,dnf} list charliecloud.

Maintained by others:

• Debian
• Gentoo
• NixOS
• SUSE and openSUSE

Note that Charliecloud development moves quickly, so double-check that packages have the version and features you need.

Pull requests and other collaboration to improve the packaging situation are particularly welcome!

1.3. Dependencies

Charliecloud’s philosophy on dependencies is that they should be (1) minimal and (2) granular. For any given feature, we try to implement it with the minimum set of dependencies, and in any given environment, we try to make the maximum set of features available.

This section documents Charliecloud’s dependencies in detail. Do you need to read it? If you are installing Charliecloud on the same system where it will be used, probably not. configure will issue a report saying what will and won’t work. Otherwise, it may be useful to gain an understanding of what to expect when deploying Charliecloud.

Note that we do not rigorously track dependency versions. We update the minimum versions stated below as we encounter problems, but they are not tight bounds and may be out of date. It is worth trying even if your version is documented to be too old. Please let us know any success or failure reports.

Finally, the run-time dependencies are lazy; specific features just try to use their dependencies and fail if there’s a problem, hopefully with a useful error message. In other words, there’s no version checking or whatnot that will keep you from using a feature unless it truly doesn’t work in your environment.

1.3.1. User namespaces

Charliecloud’s fundamental principle of a workflow that is fully unprivileged end-to-end requires unprivileged user namespaces. In order to enable them, you need a vaguely recent Linux kernel with the feature compiled in and active.

Some distributions need configuration changes. For example:

• Debian Stretch needs sysctl kernel.unprivileged_userns_clone=1.
• RHEL/CentOS 7.4 and 7.5 need both a kernel command line option and a sysctl. RHEL/CentOS 7.6 and up need only the sysctl. Note that Docker does not work with user namespaces, so skip step 4 of the Red Hat instructions, i.e., don’t add --userns-remap to the Docker configuration (see issue #97).

Note: User namespaces always fail in a chroot with EPERM. If configure detects that it’s in a chroot, it will print a warning in its report. One common scenario where this comes up is packaging, where builds often happen in a chroot. However, like all the run-time configure tests, this is informational only and does not affect the build.

1.3.2. Supported architectures

Charliecloud should work on any architecture supported by the Linux kernel, and we have run Charliecloud containers on x86-64, ARM, and Power. However, it is currently tested only on x86_64 and ARM.

Most builders are also fairly portable; e.g., see Docker’s supported platforms.

1.3.3. Details by feature

This section is a comprehensive listing of the specific dependencies and versions by feature group. It is autogenerated from the definitive source, configure.ac.

Listed versions are minimums, with the caveats above. Everything needs a POSIX shell and utilities.

The next section contains notes about some of the dependencies.

1.3.3.1. Building Charliecloud

required:

• C99 compiler

documentation:

• sphinx-build ≥ 1.2.3
• sphinx-build Python
• “docutils” module ≥ 0.14
• “sphinx-rtd-theme” module ≥ 0.2.4

1.3.3.2. Building images via our wrappers

with Buildah:

• Buildah ≥ 1.11.2

with ch-grow:

• enabled
• Python shebang line
• Python in shebang ≥ 3.4
• “lark-parser” module ≥ 0.7.1
• “requests” module ≥ 2.6.0
• ch-run

with Docker:

• Docker ≥ 17.03
• mktemp

1.3.3.3. Managing container images

build from Dockerfile with ch-build:

• at least one builder
• access to an image repository

pack images from builder storage to tarball:

• at least one builder
• tar

pack images from builder storage to SquashFS:

• at least one builder
• mksquashfs ≥ 4.2

Note: Pulling/pushing images from/to a repository is currently done using the builder directly.

1.3.3.4. Running containers

ch-run:

• user+mount namespaces

unpack image tarballs:

• tar

ch-mount and ch-umount SquashFS images:

• SquashFUSE ≥ 0.1.100

inject nVidia GPU libraries:

• nvidia-container-cli ≥ 1.0.0
• nVidia libraries & executables present

1.3.3.5. Test suite

basic tests:

• test suite enabled
• ch-run
• Bats ≥ 0.4.0
• Bash ≥ 4.1
• wget ≥ 1.11

recommended tests with tarballs:

• basic tests
• any builder above
• access to Docker Hub or mirror
• pack images with tar
• unpack images with tar

recommended tests with SquashFS:

• recommended tests with tar
• pack images with SquashFS
• mount/unmount SquashFS images

complete test suite:

• recommended tests with SquashFS
• documentation
• ShellCheck ≥ 0.6.0
• generic sudo

1.3.4. Notes on specific dependencies

This section describes additional details we have learned about some of the dependencies. Note that most of these are optional. It is in alphabetical order by dependency.

1.3.4.1. Bash

When Bash is needed, it’s because:

• Shell scripting is a lot easier in Bash than POSIX shell, so we use it for scripts applicable in contexts where it’s very likely Bash is already available.
• It is required by our testing framework, Bats.

1.3.4.2. Bats

Bats (“Bash Automated Testing System”) is a test framework for tests written as Bash shell scripts.

Upstream Bats is unmaintained, but widely available. Both version 0.4.0, which tends to be in distributions, and upstream master branch (commit 0360811) should work. There is a maintained fork called Bats-core, but the test suite currently does not pass with it; see issue #582. Patches welcome!

1.3.4.3. Buildah

Charliecloud uses Buildah’s “rootless” mode and ignore-chown-errors storage configuration for a fully unprivileged workflow with no sudo and no setuid binaries. Note that in this mode, images in Buildah internal storage will have all user and group ownership flattened to UID/GID 0.

If you prefer a privileged workflow, Charliecloud can also use Buildah with setuid helpers newuidmap and newgidmap. This will not remap ownership.

To configure Buildah in rootless mode, make sure your config files are in ~/.config/containers and they are correct. Particularly if your system also has configuration in /etc/containers, problems can be very hard to diagnose.

1.3.4.4. C compiler

We test with GCC. Core team members use whatever version comes with their distribution.

In principle, any C99 compiler should work. Please let us know any success or failure reports.

Intel icc is not supported because it links extra shared libraries that our test suite can’t deal with. See PR #481.

1.3.4.5. Docker

1.3.4.5.1. Security implications of Docker

Because Docker (a) makes installing random crap from the internet really easy and (b) is easy to deploy insecurely, you should take care. Some of the implications are below. This list should not be considered comprehensive nor a substitute for appropriate expertise; adhere to your moral and institutional responsibilities.

• Docker equals root. Anyone who can run the docker command or interact with the Docker daemon can trivially escalate to root. This is considered a feature.

  For this reason, don’t create the docker group, as this will allow passwordless, unlogged escalation for anyone in the group. Our scripts expect to run Docker as sudo docker.

  Also, Docker runs container processes as root by default. In addition to being poor hygiene, this can be an escalation path, e.g. if you bind-mount host directories.

• Docker alters your network configuration. To see what it did:

  $ ifconfig    # note docker0 interface
  $ brctl show  # note docker0 bridge
  $ route -n
  

• Docker installs services. If you don’t want the Docker service starting automatically at boot, e.g.:

  $ systemctl is-enabled docker
  enabled
  $ systemctl disable docker
  $ systemctl is-enabled docker
  disabled
  

1.3.4.5.2. Configuring for a proxy

By default, Docker does not work if you have a proxy, and it fails in two different ways.

The first problem is that Docker itself must be told to use a proxy. This manifests as:

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
Pulling repository hello-world
Get https://index.docker.io/v1/repositories/library/hello-world/images: dial tcp 54.152.161.54:443: connection refused

If you have a systemd system, the Docker documentation explains how to configure this. If you don’t have a systemd system, then /etc/default/docker might be the place to go?

The second problem is that programs executed during build (RUN instructions) need to know about the proxy as well. This manifests as images failing to build because they can’t download stuff from the internet.

The fix is usually to set the proxy variables in your environment, e.g.:

export HTTP_PROXY=http://proxy.example.com:8088
export http_proxy=$HTTP_PROXY
export HTTPS_PROXY=$HTTP_PROXY
export https_proxy=$HTTP_PROXY
export ALL_PROXY=$HTTP_PROXY
export all_proxy=$HTTP_PROXY
export NO_PROXY='localhost,127.0.0.1,.example.com'
export no_proxy=$NO_PROXY

You also need to teach sudo to retain them. Add the following to /etc/sudoers:

Defaults env_keep+="HTTP_PROXY http_proxy HTTPS_PROXY https_proxy ALL_PROXY all_proxy NO_PROXY no_proxy"

Because different programs use different subsets of these variables, and to avoid a situation where some things work and others don’t, the Charliecloud test suite will fail if some but not all of the above variables are set.

1.3.4.6. image repository access

FROM instructions in Dockerfiles and image pushing/pulling require access to an image repository and configuring the builder for that repository. Options include:

• Docker Hub, or other public repository such as gitlab.com or NVIDIA’s NCG container registry.
• A private Docker-compatible registry, such as a private Docker Hub or GitLab instance.
• Filesystem directory, for builders that support this (e.g., ch-grow).

1.3.4.7. “lark-parser” Python package

PyPI has two incompatible packages that provide the module lark, “lark-parser” and “lark”. You want “lark-parser”.

1.3.4.8. Python

We use Python for scripts that would be really hard to do in Bash, when we think Python is likely to be available.

1.3.4.9. ShellCheck

ShellCheck is a very thorough and capable linter for shell scripts. In order to pass the full test suite, all the shell scripts need to pass ShellCheck.

While it is widely available in distributions, the packaged version is usually too old. Building from source is tricky because it’s a Haskell program, which isn’t a widely available tool chain. Fortunately, the developers provide pre-compiled static binaries on their GitHub page.

1.3.4.10. Sphinx

We use Sphinx to build the documentation; the theme is sphinx-rtd-theme.

Minimum versions are listed above. We currently use Sphinx 1.8 for building what’s on the web because there are bugs problematic for us in 2.x, e.g. bad spacing in lists.

If you’re on Debian Stretch or some version of Ubuntu, installing with pip3 will silently install into ~/.local, leaving the sphinx-build binary in ~/.local/bin, which is often not on your path. One workaround (untested) is to run pip3 as root, which violates principle of least privilege. A better workaround, assuming you can write to /usr/local, is to add the undocumented and non-standard --system argument to install in /usr/local instead. (This matches previous pip behavior.) See Debian bugs 725848 and 820856.

1.3.4.11. SquashFS

The SquashFS workflow requires SquashFS Tools and/or SquashFUSE. Note that distribution packages of SquashFUSE often provide only the “high level” executables; the “low level” executables have better performance. These can be installed from source on any distribution.

1.3.4.12. sudo, generic

Privilege escalation via sudo is used in the test suite to:

• Prepare fixture directories for testing filesystem permissions enforcement.
• Test ch-run’s behavior under different ownership scenarios.

(Note that Charliecloud also uses sudo docker; see above.)

1.3.4.13. Wget

Wget is used to demonstrate building an image without a builder (the main test image used to exercise Charliecloud itself).

1.4. Pre-installed virtual machine image

This section explains how to create and use a single-node virtual machine with Charliecloud and all three builders pre-installed. This lets you use Charliecloud on Macs and Windows, and/or obtain a pre-configured Charliecloud environment without installing anything.

You can use this CentOS VM either with Vagrant or with VirtualBox alone. Various settings are specified, but in most cases we have not done any particular tuning, so use your judgement, and feedback is welcome.

Warning

These instructions provide for an SSH server in the virtual machine guest that is accessible to anyone logged into the host. It is your responsibility to ensure this is safe and compliant with your organization’s policies, or modify the procedure accordingly.

1.4.1. Import and use an ova appliance file with plain VirtualBox

This procedure imports a .ova file into VirtualBox and walks you through logging in and running a brief Hello World in Charliecloud. You will act as user charlie, who has passwordless sudo.

The Charliecloud developers do not distribute a .ova file. You will need to get it from your site, a third party, or build it yourself with Vagrant using the instructions in the Contributor’s guide.

Prerequisite: Installed and working VirtualBox. (You do not need Vagrant to use the .ova, only to create it.)

1.4.1.1. Configure VirtualBox

1. Set Preferences → Proxy if needed at your site.

1.4.1.2. Import the appliance

1. Download charliecloud_centos7.ova, or whatever your site has called it.
2. File → Import appliance. Choose charliecloud_centos7.ova and click Continue.
3. Review the settings.
   • CPU should match the number of cores in your system.
   • RAM should be reasonable. Anywhere from 2GiB to half your system RAM will probably work.
   • Tick Reinitialize the MAC address of all network cards.
4. Click Import.
5. Verify that the appliance’s port forwarding is acceptable to you and your site: Details → Network → Adapter 1 → Advanced → Port Forwarding.

1.4.1.3. Log in and try Charliecloud

1. Start the VM by clicking the green arrow.

2. Wait for it to boot.

3. Click on the console window, where user charlie is logged in. If the VM “captures” your mouse pointer, type the key combination listed in the lower-right corner of the window to release it. You can avoid capture by clicking the title bar instead of window content.

4. Change your password. You must use sudo because you have passwordless sudo but don’t know your password.

   $ sudo passwd charlie
   

5. SSH (from terminal on the host) into the VM using the password you just set. Accessing the VM using SSH rather than the console is generally more pleasant, because you have a nice terminal with native copy-and-paste, etc.

   $ ssh -p 2222 charlie@localhost
   

6. Build and run a container:

   $ ch-build -t hello -f /usr/local/src/charliecloud/examples/serial/hello \
              /usr/local/src/charliecloud
   $ ch-builder2tar hello /var/tmp
   57M /var/tmp/hello.tar.gz
   $ ch-tar2dir /var/tmp/hello.tar.gz /var/tmp
   creating new image /var/tmp/hello
   /var/tmp/hello unpacked ok
   $ cat /etc/redhat-release
   CentOS Linux release 7.3.1611 (Core)
   $ ch-run /var/tmp/hello -- /bin/bash
   > cat /etc/debian_version
   8.9
   > exit
   

Congratulations! You’ve successfully used Charliecloud. Now all of your wildest dreams will come true.

Shut down the VM at your leisure.

Possible next steps:

• Follow the tutorial.
• Run ch-test (Note that the environment variables are already configured for you in this appliance.)
• Configure /var/tmp to be a tmpfs, if you have enough RAM, for better performance.

1.4.2. Build and use the VM with Vagrant

This procedure builds and provisions an idiomatic Vagrant virtual machine. You should also read the Vagrantfile in packaging/vagrant before proceeding. This contains the specific details on build and provisioning, which are not repeated here.

Prerequisite: You already know how to use Vagrant.

1.4.2.1. Caveats and gotchas

In no particular order:

• While Vagrant supports a wide variety of host and virtual machine providers, this procedure is tested only on VirtualBox on a Mac. Current Vagrant versions should work, but we don’t track specifically which ones. (Anyone who wants to help us broaden this support, please get in touch.)
• Switching between proxy and no-proxy environments is not currently supported. If you have a mixed environment (e.g. laptops that travel between a corporate network and the wild), you may want to provide two separate images.
• Provisioning is not idempotent. Running the provisioners again will have undefined results.
• The documentation is not built. Use the web documentation instead of man pages.
• Only the most recent release of Charliecloud is supported.

1.4.2.2. Install Vagrant and plugins

You can install VirtualBox and Vagrant either manually using website downloads or with Homebrew:

$ brew cask install virtualbox virtualbox-extension-pack vagrant

Sanity check:

$ vagrant version
Installed Version: 2.1.2
Latest Version: 2.1.2

You're running an up-to-date version of Vagrant!

Then, install the needed plugins:

$ vagrant plugin install vagrant-disksize \
                         vagrant-proxyconf \
                         vagrant-reload \
                         vagrant-vbguest

1.4.2.3. Build and provision

To build the VM and install Docker, Charliecloud, etc.:

$ cd packaging/vagrant
$ vagrant up

By default, this uses the newest release of Charliecloud. If you want something different, set the CH_VERSION variable, e.g.:

$ CH_VERSION=v0.10 vagrant up
$ CH_VERSION=master vagrant up

Then, optionally run the Charliecloud tests:

$ vagrant provision --provision-with=test

This runs the Charliecloud test suite in standard scope.

Note that the test output does not have a TTY, so you will not have the tidy checkmarks. The last test printed is the last one that completed, not the one currently running.

If the tests don’t pass, that’s a bug. Please report it!

Now you can vagrant ssh and do all the usual Vagrant stuff.