This section describes what you need to install Charliecloud and how to do so.
Note that installing and using Charliecloud can be done as a normal user with no elevated privileges, provided that user namespaces have been enabled.
We provide tarballs with a
configure script. Thus, build and install can be as
$ ./configure $ make $ sudo make install
If you don’t have sudo, you can:
Run Charliecloud directly from the build directory; add
$PATHand you are good to go, without
Install in a prefix you have write access to, e.g. in your home directory with
configure will provide a detailed report on what will be built and
installed, along with what dependencies are present and missing.
If you obtain the source code with Git, you must build
friends yourself. To do so, you will need the following. The versions in most
common distributions should be sufficient.
pip3package installer and its
This script has a few options; see its
Note that Charliecloud disables Automake’s “maintainer mode” by default, so
the build system (Makefiles,
configure, etc.) will never automatically
be rebuilt. You must run
autogen.sh manually if you need this. You can
also re-enable maintainer mode with
configure if you like, though this
is not a tested configuration.
configure has the following options in addition to the
220.127.116.11. Feature selection:
By default, all features that can be built will be built and installed. You can exclude some features with:
ch-imageunprivileged builder & image manager
logging to syslog (see individual man pages)
You can also say
--enable-FOO to fail the build if
18.104.22.168. Dependency selection:
Some dependencies can be specified as follows. Note only some of these support
--with-FOO=no, as listed.
Whether to link with
If not specified: Look for
libsquashfusein standard install locations and link with it if found. Otherwise disable internal SquashFS mount, with no warning or error.
yes: Look for
libsquashfusein standard locations and link with it if found; otherwise, error.
libsquashfuselinking and internal SquashFS mounting, even if it’s installed.
libsquashfuseinstall prefix: Link with
libsquashfusefound there, or error if not found, and add it to
ch-run’s RPATH. (Note this argument is not the directory containing the shared library or header file.)
Note: A very specific version and configuration of SquashFUSE is required. See below for details.
Shebang line to use for Python scripts. Default:
sphinx-buildexecutable. Default: the
sphinx-buildfound first in
Path to Python used by
sphinx-build. Default: shebang of
22.214.171.124. Less strict build:
Please do not use this option routinely, as that hides bugs that we cannot find otherwise.
By default, Charliecloud builds with
-Werror. The principle here is that we prefer diagnostics that are as noisy
as practical, so that problems are identified early and we can fix them. We
-Werror unless there is a specific reason to turn it off. For
example, this approach identified a buggy
configure test (issue #798).
Many others recommend the opposite. For example, Gentoo’s “Common mistakes”
guide advises against
-Werror because it causes breakage that is
“random” and “without purpose”. There is a well-known blog post
from Flameeyes that recommends
-Werror be off by default and used by
developers and testers only.
In our opinion, for Charliecloud, these warnings are most likely the result of
real bugs and shouldn’t be hidden (i.e., they are neither random nor without
purpose). Our code should have no warnings, regardless of compiler, and any
spurious warnings should be silenced individually. We do not have the
resources to test with a wide variety of compilers, so enabling
-Werror only for development and testing, as recommended by others,
means that we miss potentially important diagnostics — people typically do not
pay attention to warnings, only errors.
That said, we recognize that packagers and end users just want to build the
code with a minimum of hassle. Thus, we provide the
Don’t hesitate to use it. But if you do, we would very much appreciate if you:
File a bug explaining why! We’ll fix it.
Remove it from your package or procedure once we fix that bug.
126.96.36.199. Disable bundled Lark package:
This option is minimally supported and not recommended. Use only if you really know what you are doing.
Charliecloud uses the Python package Lark for parsing Dockerfiles and image references. Because this package is developed rapidly, and recent versions have important features and bug fixes not yet available in common distributions, we bundle the package with Charliecloud.
If you prefer a separately-installed Lark, either via system packages or
pip, you can use
./configure --disable-bundled-lark. This
excludes the bundled Lark from being installed or placed in
tarballs. It does not remove the bundled Lark from the source directory; if
you run from the source directory (i.e., without installing), the bundled Lark
will be used if present regardless of this option.
Bundled Lark is included in the tarballs we distribute. You can remove it and
./autogen.sh --rm-lark --no-lark. If
you are starting from a Git checkout, bundled Lark is installed by default by
./autogen.sh, but you can prevent this with
The main use case for these options is to support package maintainers. If this is you and does not meet your needs, please get in touch with us and we will help.
Charliecloud is also available using a variety of distribution and third-party package managers.
Maintained by us:
Maintained by others:
Note that Charliecloud development moves quickly, so double-check that packages have the version and features you need.
Pull requests and other collaboration to improve the packaging situation are particularly welcome!
Charliecloud’s philosophy on dependencies is that they should be (1) minimal and (2) granular. For any given feature, we try to implement it with the minimum set of dependencies, and in any given environment, we try to make the maximum set of features available.
This section documents Charliecloud’s dependencies in detail. Do you need to
read it? If you are installing Charliecloud on the same system where it will
be used, probably not.
configure will issue a report saying what will
and won’t work. Otherwise, it may be useful to gain an understanding of what
to expect when deploying Charliecloud.
Note that we do not rigorously track dependency versions. We update the minimum versions stated below as we encounter problems, but they are not tight bounds and may be out of date. It is worth trying even if your version is documented to be too old. Please let us know any success or failure reports.
Finally, the run-time dependencies are lazy; specific features just try to use their dependencies and fail if there’s a problem, hopefully with a useful error message. In other words, there’s no version checking or whatnot that will keep you from using a feature unless it truly doesn’t work in your environment.
Charliecloud’s fundamental principle of a workflow that is fully unprivileged end-to-end requires unprivileged user namespaces. In order to enable them, you need a vaguely recent Linux kernel with the feature compiled in and active.
Some distributions need configuration changes. For example:
Debian Stretch needs sysctl
RHEL/CentOS 7.4 and 7.5 need both a kernel command line option and a sysctl. RHEL/CentOS 7.6 and up need only the sysctl. Note that Docker does not work with user namespaces, so skip step 4 of the Red Hat instructions, i.e., don’t add
--userns-remapto the Docker configuration (see issue #97).
Note: User namespaces always fail in a chroot with
configure detects that it’s in a chroot, it will print a warning in
its report. One common scenario where this comes up is packaging, where builds
often happen in a chroot. However, like all the run-time
tests, this is informational only and does not affect the build.
Charliecloud should work on any architecture supported by the Linux kernel, and we have run Charliecloud containers on x86-64, ARM, and Power. However, it is currently tested only on x86_64 and ARM.
Most builders are also fairly portable; e.g., see Docker’s supported platforms.
We want Charliecloud to work with any C99/POSIX libc, though it is only tested with glibc and musl, and other libc’s are very likely to have problems. (Please report these bugs!) Non-glibc libc’s will currently need a standalone libargp (see issue #1260).
This section is a comprehensive listing of the specific dependencies and
versions by feature group. It is auto-generated from the definitive source,
Listed versions are minimums, with the caveats above. Everything needs a POSIX shell and utilities.
The next section contains notes about some of the dependencies.
188.8.131.52. Building Charliecloud¶
extended glob patterns in –unset-env
ch-run internal SquashFS mounting:
“docutils” module ≥ 0.14
“sphinx-rtd-theme” module ≥ 0.2.4
184.108.40.206. Building images¶
with Buildah (warning: not tested; see issue #990):
Buildah ≥ 1.11.2
Python shebang line
Python in shebang ≥ 3.6
“requests” module ≥ 2.6.0
ch-image using build cache:
Git ≥ 2.28.1
Docker ≥ 17.03
220.127.116.11. Managing container images¶
build from Dockerfile:
at least one builder
access to an image repository
pack images from builder storage to tarball:
at least one builder
pack images from builder storage to SquashFS:
at least one builder
Note: Pulling/pushing images from/to a repository is currently done using the builder directly.
18.104.22.168. Running containers¶
run SquashFS images:
manual mount with SquashFUSE ≥ 0.1.100
internal mount with libsquashfuse
inject nVidia GPU libraries:
nVidia libraries & executables present
22.214.171.124. Test suite¶
basic tests, all stages:
test suite enabled
any builder above
access to Docker Hub or mirror
Bats ≥ 0.4.0
Bash ≥ 4.1
more complete tests:
ShellCheck ≥ 0.6.0
DOT ≥ 2.30.1
git2dot ≥ 0.8.3
recommended tests, tar-unpack mode:
recommended tests, squash-unpack mode:
pack/unpack SquashFS images
recommended tests, squash-mount mode:
recommended, squash-unpack mode:
internal SquashFS mounting
This section describes additional details we have learned about some of the dependencies. Note that most of these are optional. It is in alphabetical order by dependency.
When Bash is needed, it’s because:
Shell scripting is a lot easier in Bash than POSIX shell, so we use it for scripts applicable in contexts where it’s very likely Bash is already available.
It is required by our testing framework, Bats.
Bats (“Bash Automated Testing System”) is a test framework for tests written as Bash shell scripts.
Upstream Bats is unmaintained, but widely available. Both version 0.4.0, which tends to be in distributions, and upstream master branch (commit 0360811) should work. There is a maintained fork called Bats-core, but the test suite currently does not pass with it; see issue #582. Patches welcome!
Charliecloud uses Buildah’s “rootless” mode and
storage configuration for a fully unprivileged workflow with no sudo and no
setuid binaries. Note that in this mode, images in Buildah internal storage
will have all user and group ownership flattened to UID/GID 0.
If you prefer a privileged workflow, Charliecloud can also use Buildah with
newgidmap. This will not remap
To configure Buildah in rootless mode, make sure your config files are in
~/.config/containers and they are correct. Particularly if your system
also has configuration in
/etc/containers, problems can be very hard
126.96.36.199. C compiler¶
We test with GCC. Core team members use whatever version comes with their distribution.
In principle, any C99 compiler should work. Please let us know any success or failure reports.
icc is not supported because it links extra shared libraries
that our test suite can’t deal with. See PR #481.
188.8.131.52. image repository access¶
FROM instructions in Dockerfiles and image pushing/pulling require
access to an image repository and configuring the builder for that repository.
We use Python for scripts that would be really hard to do in Bash, when we think Python is likely to be available.
ShellCheck is a very thorough and capable linter for shell scripts. In order to pass the full test suite, all the shell scripts need to pass ShellCheck.
While it is widely available in distributions, the packaged version is usually too old. Building from source is tricky because it’s a Haskell program, which isn’t a widely available tool chain. Fortunately, the developers provide pre-compiled static binaries on their GitHub page.
We use Sphinx to build the documentation; the theme is sphinx-rtd-theme.
Minimum versions are listed above. Note that while anything greater than the minimum should yield readable documentation, we don’t test quality with anything other than what we use to build the website, which is usually but not always the most recent version available on PyPI.
If you’re on Debian Stretch or some version of Ubuntu, installing with
pip3 will silently install into
~/.local, leaving the
sphinx-build binary in
~/.local/bin, which is often not on
your path. One workaround (untested) is to run
pip3 as root, which
violates principle of least privilege. A better workaround, assuming you can
/usr/local, is to add the undocumented and non-standard
--system argument to install in
/usr/local instead. (This
pip behavior.) See Debian bugs 725848 and 820856.
184.108.40.206. SquashFS and SquashFUSE¶
The SquashFS workflow requires SquashFS Tools to create SquashFS archives.
To mount these archives using
ch-run’s internal code, you need:
development files, which are probably available in your distribution, e.g.,
libfuse3-dev. (Build time only.)
fusermount3executable, which often comes in a distro package called something like
fuse3. This is typically installed setuid, but Charliecloud does not need that; you can
chmod u-sthe file or build/install as a normal user.
SquashFUSE v0.1.105 or later (we need the
libsquashfuse_llshared library). This must be installed, not linked from its build directory, though it can be installed in a non-standard location.
Without these, you can still use a SquashFS workflow but must mount and unmount the filesystem archives manually. You can do this using the executables that come with SquashFUSE, and the version requirement is much less stringent.
libfuse2 development files are available but those for
libfuse3 are not, SquashFUSE will still build and install, but the
proper components will not be available, so Charliecloud’s
configure will say it’s not found.
220.127.116.11. sudo, generic¶
Privilege escalation via sudo is used in the test suite to:
Prepare fixture directories for testing filesystem permissions enforcement.
ch-run’s behavior under different ownership scenarios.
(Note that Charliecloud also uses
sudo docker; see above.)
Wget is used to demonstrate building an image without a builder (the main test image used to exercise Charliecloud itself).