Jupyter Containers

The Jupyter team maintains the Jupyter Docker Stack ³ with corresponding public container images on DockerHub ⁴ and Quay ⁵. These container images can be use on any compute-cluster.

Login to a cluster submit node and start a JupyterLab container:

apptainer run docker://jupyter/base-notebook

The JupyterLab service daemon will print log information to the Terminal, including the connection address and a so called access token described in the next section.

Stop JupyterLab by pressing CTRL-C once you have finished working.

stdout

# Information from the logs of the JupyterLab service
Or copy and paste one of these URLs:
    http://lxbk0725:12345/lab?token=598fb99446ce46996e9a74ea28bea3d2c7fd2591be4431f6
    http://127.0.0.1:12345/lab?token=598fb99446ce46996e9a74ea28bea3d2c7fd2591be4431f6

ssh -vv -N -L localhost:12345:lxbk0725.gsi.de:12345 virgo.hpc.gsi.de
# …logs will be print to the terminal …use ctrl-c to close

Build a Container

The following section implies that you want to add Jupyter to an existing container. Alternatively you could use the Jupyter Docker Stack as foundation and build from there by adding your components on top.

To integrate JupyterLab into your Apptainer container, you’ll typically extend an existing configuration that includes your working environment. Following assumes you have Python 3 available in our container and uses a Python virtual environment ⁷ to install JupyterLab.

Adding JupyterLab to an Apptainer Container Definition File ⁸:

apptainer.def

%post
# …add in an appropriate place in the post-section
mkdir /app
cd /app
python3 -m venv venv
. venv/bin/activate
# install JupyterLab including some extensions
pip3 install \
        jupyter \
        jupyterlab-spellchecker \
        jupyterlab-git \
        jupyterlab-lsp \
        'python-lsp-server[all]'

%environment
export JUPYTERLAB_PORT=54321

%runscript
. /app/venv/bin/activate
exec jupyter lab --no-browser --ip 0.0.0.0 --port $JUPYTERLAB_PORT

%startscript
. /app/venv/bin/activate
exec jupyter lab --no-browser --ip 0.0.0.0 --port $JUPYTERLAB_PORT

The example above prepares the use of apptainer instance ⁹ to manage your JupyterLab installation:

Apptainer, also allows you to run containers in a “detached” or “daemon” mode where the container runs a service. A “service” is essentially a process running in the background…

Following environment variables is used to set the service port:

Build the container and copy the image to shared storage:

1apptainer build apptainer.sif apptainer.def

# Make sure to store the container image on persistent storage for example Lustre
export LUSTRE_HOME=/lustre/$(id -ng)/$USER
export APPTAINER_CONTAINERS=$LUSTRE_HOME/containers
2cp apptainer.sif $APPTAINER_CONTAINERS/jupyterlab.sif

1

Build the Apptainer container image from the definition file ¹⁰

2

Copy the JupyterLab container images to your Lustre directory

Start the container instance on the current host:

1export APPTAINERENV_JUPYTERLAB_PORT=12345
2apptainer run $APPTAINER_CONTAINERS/jupyterlab.sif

1

Overwrite the default port configuration with an environment variable

2

Start your JupyterLab instance on the local node

Start a JupyterLab instance on a cluster node:

1export LUSTRE_HOME=/lustre/$(id -ng)/$USER ; cd $LUSTRE_HOME
2srun --nodes=1 --ntasks-per-node=1 --time=01:00:00 --pty bash -i

# …once your allocation has been granted
3export APPTAINER_CONFIGDIR=$LUSTRE_HOME/.apptainer
4apptainer instance start $LUSTRE_HOME/jupyterlab.sif jupyterlab

1

Configure your working directory, typically on shared storage

2

Allocate an interactive session on a compute node with srun

3

Make sure the Apptainer configuration path is located on writable storage

4

Start your JupyterLab instance on the compute node

Read the access token from the log-files of your JupyterLab instance:

>>> cat $APPTAINER_CONFIGDIR/instances/logs/$(hostname)/$USER/jupyterlab.err \
        | grep -o 'http.*lab?token.*' | sort | uniq
http://127.0.0.1:54321/lab?token=041ded5856f8bfe5202c5118027016f990390729ff7d0433
http://lxbk0724:54321/lab?token=041ded5856f8bfe5202c5118027016f990390729ff7d0433

Using Apptainer

Apptainer ¹ enables users to interact with containers transparently. Execute programs inside a container (as if running on the host). Redirect I/O, use command pipes, pass arguments, access files, sockets and ports. Apptainer is updated regularly ² and most Linux distributions have packages available in their repositories. On Fedora for example, install Apptainer using the official apptainer package ³:

# install apptainer
sudo dnf install -y apptainer

# verify functionality
apptainer run docker://alpine

Users of MacOS and Windows can execute Linux containers in a virtual environment on their computer. It is not necessarily required to have Apptainer installed on your machine. Many HPC systems will have the apptainer command pre-installed, including the capability to build container images on the cluster nodes. However it is often preferable to use container images in the local environment as well, and copy them on-demand to a target HPC system. This approach ensures that you work with the exact same environment under all circumstance.

Container Images

# Set the path to a directoy storing container images...
export APPTAINER_CONTAINER=$LUSTRE_HOME/containers

# ...download a container image from a container registry
apptainer pull $APPTAINER_CONTAINER/jupyter.sif docker://quay.io/datascience-notebook:latest

Interact with Containers

The Jupyter datascience-notebook container image downloaded in the previous section includes a Python environment supporting the NumPy package ⁷ for scientific computing. Users interact with container images ⁸ using a selection of Apptainer sub-commands. The sub-command exec executes a specified command within a container. Below a very simple Python program utilizing the NumPy package:

#!/usr/bin/env python

import numpy as np

array = np.array([
    [3, 7, 1],
    [10, 3, 2],
    [5, 6, 7]
])

print(np.sort(array, axis=1))

# Implement the NumPy program above in an example file...
$EDITOR example.py

# ...make the example file an executable program
chmod +x example.py

# Execute the Python program in the downloaded container image
apptainer exec $APPTAINER_CONTAINER/jupyter.sif ./example.py

The shell sub-command starts an interactive shell within a container:

>>> apptainer shell $APPTAINER_CONTAINER/jupyter.sif         
Apptainer> ipython
Python 3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:40:35) [GCC 12.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: 

Build Containers

Create a new container image ⁹ with the help of a container definition file (or “def file” for short). The following example builds a container image for the GNU Hello ¹⁰ software, based on a definition file called hello.def:

BootStrap: docker
From: quay.io/fedora/fedora:latest

%labels
Maintainer John Snow <j.snow@example.com>

%post
version=2.12.1
archive=hello-$version.tar.gz

dnf install -y wget tar gzip gcc make
dnf clean all

wget https://ftp.gnu.org/gnu/hello/$archive
tar xvzf $archive -C /opt
rm $archive
cd /opt/hello-$version

./configure
make
make install

%runscript
/usr/local/bin/hello

Run the following command providing the container image name and the definition file as arguments to build a container image:

apptainer build $APPTAINER_CONTAINER/hello.sif hello.def

Test the functionality of the container image by executing the hello application in the container:

>>> apptainer run $APPTAINER_CONTAINER/hello.sif                
Hello, world!

>>> apptainer exec $APPTAINER_CONTAINER/hello.sif hello --version | head -n1
hello (GNU Hello) 2.12.1

# ...from DockerHub
BootStrap: docker
From: fedora:latest

# ...from Quay (RedHat)
BootStrap: docker
From: quay.io/rockylinux/rockylinux:8

%setup
    touch /file1
    touch ${APPTAINER_ROOTFS}/file2

%files
    /file1
    /file1 /opt

%environment
    export LISTEN_PORT=12345
    export LC_ALL=C

%post
    apt-get update && apt-get install -y netcat
    NOW=`date`
    echo "export NOW=\"${NOW}\"" >> $APPTAINER_ENVIRONMENT

%runscript
    echo "Container was created $NOW"
    echo "Arguments received: $*"
    exec echo "$@"

%startscript
    nc -lp $LISTEN_PORT

%test
    grep -q NAME=\"Ubuntu\" /etc/os-release
    if [ $? -eq 0 ]; then
        echo "Container base is Ubuntu as expected."
    else
        echo "Container base is not Ubuntu."
        exit 1
    fi

%labels
    Author alice
    Version v0.0.1

%help
    This is a demo container used to illustrate a def file that uses all
    supported sections.

BootStrap: docker
From: quay.io/fedora/fedora:latest

%post

dnf install -y @development-tools \
      gcc-c++ gcc gcc-gfortran git git-delta gnupg2 \
      python3 python3-pip python3-setuptools python3-boto3 \
      psmisc rsync tmux tree wget unzip \
      bat curl fd-find fzf findutils \
      hostname iproute netcat neovim make patch \
dnf clean all

# ...build the container image
apptainer build fedora-base.sif fedora-base.def

Bootstrap: localimage
From: fedora-base.sif

Container Cache

By default the container build cache is located in the directory $HOME/.apptainer. Details about the container build environment are available in the Apptainer User Guide ¹⁷. Pay attention to the storage consumed by the container cache, since the amount of available space may be limited depending on your environment.

# show storage capacity used by the cache
apptainer cache list

# ..detailed view
apptainer cache list -v

# clean up everything
apptainer cache clean

The environment variable listed in the table on top are used to configure the paths to the container caches. Following example stores all container artifacts on volatile storage in the /var/tmp directory by setting both variables:

# create a user working directory
pushd $(mktemp -d /var/tmp/$USER-apptainer-XXXXXX)

# locate all artifacts within this directory
export APPTAINER_TMPDIR=$PWD
export APPTAINER_CACHEDIR=$PWD

This is useful in make sure that all build-artifact don’t linger in your environment.

Test Sandboxes

A sandbox provides a writable (ch)root directory to interactively work with a container image. This is obviously not a reproducible method to build an image, but it is helps testing during implementation of definition files:

# ...create a container within a writable directory
apptainer build --fix-perms --sandbox rootfs/ docker://quay.io/rockylinux/rockylinux:8

# ...make changes within the container
apptainer shell --writable --fakeroot --home $PWD rootfs/

# ...build a new container from the sandbox
apptainer build apptainer.sif rootfs/

Do not forget to delete the rootfs directory afterwards, since these can consume multiple Gigabytes of storage.

Docker Compatibility

Container build and runtime tools have a complex interrelationship and overlap of functionality. The most widely used container tools outside of HPC are Docker ¹⁸ and Podman ¹⁹, which are mostly compatible to each other. Chances are high that you will encounter projects providing a Dockerfile ²⁰ as container definition. Consider the example below for GNU Hello introduced previously:

FROM quay.io/fedora/fedora:latest

ARG version=2.12.1
ARG archive=hello-$version.tar.gz

RUN dnf install -y wget tar gzip gcc make
RUN dnf clean all

RUN wget https://ftp.gnu.org/gnu/hello/$archive
RUN tar xvzf $archive -C /opt
RUN rm $archive

WORKDIR /opt/hello-$version

RUN ./configure
RUN make
RUN make install

ENTRYPOINT /usr/local/bin/hello

The notation above differs significantly from an Apptainer definition file. In order to build a container image from a Dockerfile, use the build sub-command ²¹ of Podman. Please consult the corresponding documentations for more details.

# Build a container image using a `Dockerfile` in the working directory...
podman build -t hello .

# ...and execute the container image to test its functionality
podman run --rm -it localhost/hello:latest

For the purpose of this article it is interesting to understand the compatibility of Apptainer and Docker container images. Apptainer uses a different container format SIF, as discussed above. Fortunately it is easy to convert between both formats:

# Create an OCI container archive...
podman push localhost/hello:latest oci-archive:/tmp/hello-latest.tar

# ...and use this archive to create a SIF container image
apptainer build hello.sif oci-archive:/tmp/hello-latest.tar

The two commands above use an standardized container image format called OCI Image Layout ²² as intermediate step during conversion. There is a lot more details to the conversion then described here. We leave it to the reader to continue to investigate this.

The problem

However, some experiments, such as ALICE at CERN, have, for better or worse, designed their computing workflows in such way that the computing payload requires to connect to the outside world during runtime.

Since the collaboration with ALICE is too important to be subject to the cluster’s network policy, an exception needed to be implemented.

A solution needs to fulfil at least the following requirements

• The outgoing connections from ALICE computing payloads MUST be permitted
• Any outgoing connection from other processes MUST fall into the default routing workflow
• Outgoing connections MUST be NATed for internet-routability.
• The mechanism MUST be transparent for ALICE as well as for other users of the cluster
• The mechanism SHOULD be efficient
• The mechanism SHOULD be fault-tolerant

The solution

The following is an extremely simplified illustration of the default network situation around the cluster.

There is only one router, and it each worker node has configured it as its default gateway.

That router sends the packets it received to its default gateway, heading for the firewall, which drops packets attempting to connect to the outside.

The solution is a mechanism which consists of two parts.

#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1  inr.ruhep
201 alinat

[Unit]
Description=Routing Configuration
Requires=network-online.target
Wants=systemd-networkd-wait-online.service sssd.service
After=network-online.target sssd.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=-/usr/sbin/iptables -t mangle -D OUTPUT -p tcp -m owner --uid-owner aliprod  -j MARK --set-mark 1
ExecStart=-/usr/sbin/iptables -t mangle -D OUTPUT -p tcp -m owner --uid-owner alise    -j MARK --set-mark 1
ExecStart=-/usr/sbin/iptables -t mangle -D OUTPUT -p tcp -m owner --gid-owner alice    -j MARK --set-mark 1
ExecStart=/usr/sbin/iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner aliprod  -j MARK --set-mark 1
ExecStart=/usr/sbin/iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner alise    -j MARK --set-mark 1
ExecStart=/usr/sbin/iptables -t mangle -A OUTPUT -p tcp -m owner --gid-owner alice    -j MARK --set-mark 1
ExecStart=-/usr/sbin/ip rule del fwmark 1 table alinat
ExecStart=-/usr/sbin/ip r flush table alinat
ExecStart=/usr/sbin/ip rule add fwmark 1 table alinat
ExecStart=/usr/sbin/ip r add 10.20.0.0/16 dev ib0 table alinat
ExecStart=/usr/sbin/ip r add 10.20.1.0/24 dev ib0 table alinat
ExecStart=/usr/sbin/ip r add 10.10.24.0/24 via 10.20.0.1 dev ib0 table alinat
ExecStart=/usr/sbin/ip r add 140.181.2.0/24 via 10.20.0.1 dev ib0 table alinat
ExecStart=/usr/sbin/ip r add 140.181.60.0/24 via 10.20.0.1 dev ib0 table alinat
ExecStart=/usr/sbin/ip r add default via 10.20.3.227 dev ib0 table alinat
ExecStop=-/usr/sbin/iptables -t mangle -D OUTPUT -p tcp -m owner --uid-owner aliprod  -j MARK --set-mark 1
ExecStop=-/usr/sbin/iptables -t mangle -D OUTPUT -p tcp -m owner --uid-owner alise    -j MARK --set-mark 1
ExecStop=-/usr/sbin/iptables -t mangle -D OUTPUT -p tcp -m owner --gid-owner alice    -j MARK --set-mark 1
ExecStop=/usr/sbin/ip rule del fwmark 1 table alinat
ExecStop=/usr/sbin/ip r flush table alinat

[Install]
WantedBy=multi-user.target

[root@alinat1 ~]# cat /etc/sysctl.d/10-alinat.conf 
net.ipv4.ip_forward = 1

[root@alinat1 ~]# nft list ruleset
table inet table1 {
    chain input {
        type filter hook input priority filter; policy accept;
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        iifname "ib0" oifname "eth9" accept
        iifname "eth9" oifname "ib0" accept
    }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}
table ip tablenat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        oifname "eth9" masquerade
    }

    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
    }
}

Conclusion

This setup has proven to have an excellent specificity and sensitivity, which is to say there have not been any observed false positives or false negatives w.r.t. which packets get routed to our NAT router vs the normal default gateway.

Secondly, it is extremely efficient. Even when saturating the 10GbE link there is hardly any noticable CPU usage on the NAT router.

Also, it is reliable and probably the most fault-tolerant component in our ALICE setup.

There is slight operational room for improvement on the worker node side. In case the InfiniBand network goes down and up again, the IP address disappears and all routes with it. Currently we do not automatically recover from that fault, but we detect it and manually need to restart route.service. This could theoretically be further automated.

The GSI contribution to ALICE computing in Run 3

GSI is one of the ALICE Grid sites and provides computing and storage resources (several PB and several thousands cores) to the ALICE experiment. The amount and mode of integration of these resources in the global framework of the experiment results from the computing requirements ² and is reviewed on a yearly bases to account for the current needs and possibilities. As anticipated above the data throughput from the detector has drastically increased from Run 2 to Run 3 leading to big changes in the ALICE data processing model in particular to anticipate as much as possible the calibration and reconstruction just after the read out, before storing and archiving the data. This has also implied a reorganization of the role played by the Grid sites, in particular to minimize the displacement of data and to maximize the proximity of computing jobs to their input data. In Run 2 the GSI site has been a Tier 2 in the ALICE Grid, meaning a site at the second level in the Grid hierarchy, where the first level – Tier 1 – includes some big computing facilities (GridKA in the case of Germany) and the upper level – Tier 0 – are the central facilities very closed to the detector itself. As a Tier 2 site GSI resources have been used in the context of the Run 2 computing model for central simulations, central analysis and user analysis jobs. In the data processing of Run 3 the role of the different sites is more specialized and GSI becomes an Analysis Facility (AF), whose focus is in being able to process a large amount of data in a short time (up to 5 PB in 12 hours, or equivalently 100 GB/s) providing fast analysis validation an cut tuning.

How are GSI resources integrated in the ALICE Grid

As the other big High Energy Physics experiments at the LHC, ALICE generates data volumes whose storage and processing require to be distributed on many sites. The ALICE Grid middleware JAliEn ³ allows to represent world-wide spread resources under a single interface transparently to the user, who sees a single namespace for files and a single resource manager for computing jobs. For this to happen each Grid site maintains dedicated services and servers which make the storage and computing resources at the site (Storage Element and Computing Element) centrally accessible.

Some technical challenges and our solutions

The aim of the middleware is to link local resources and central services transparently for the users and painlessly for the local and central admins. Nevertheless we had some issues to be solved, mostly related to the peculiarity of providing resources on shared, non-dedicated systems (our storage instance of Lustre and our cluster Virgo). Hereafter we mention a couple of these issues and describe how we solved them.

Container Images

Ready to use container images are provided to the cluster on CVMFS in the directory /cvmfs/phelix.gsi.de/sifs/. The latest images are stored in the development dev/ sub-directory for validation from the user community. After validation, images are moved to the production prod/ sub-directory, replacing the previous versions preserved for reference in the old/ sub-directory. The directory structure looks similar to the following example:

>>> tree /cvmfs/phelix.gsi.de/sifs
.
├── cpu
│   ├── dev
│   │   ├── rlx8_ompi5_ucx_cma.sif
│   │   ├── rlx8_ompi5_ucx_dask.sif
│   │   ├── rlx8_ompi5_ucx_gcc13_cma.sif
│   │   ├── rlx8_ompi5_ucx_gcc13.sif
│   │   └── rlx8_ompi5_ucx.sif
│   ├── old
│   └── prod
│       ├── rlx8_ompi_ucx_gcc12.sif
│       └── rlx8_ompi_ucx.sif
└── gpu
    ├── dev
    │   └── rlx8_rocm-5.7.1_warpx.sif
    ├── old
    └── prod
        ├── rlx8_rocm-5.4.6.def
        ├── rlx8_rocm-5.4.6.sif
        ├── rlx8_rocm-5.4.6_warpx.def
        ├── rlx8_rocm-5.4.6_warpx.sif
        ├── ubuntu-20.04_rocm-5.4.2_picongpu.def
        ├── ubuntu-20.04_rocm-5.4.2_picongpu.sif
        ├── ubuntu-20.04_rocm-5.4.2_warpx.def
        └── ubuntu-20.04_rocm-5.4.2_warpx.sif

The file names for container image reflect the base Linux distribution, and compiler and library versions. For example RockyLinus 8 with the default compiler GCC 8.5:

• /cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx.sif
• /cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_cma.sif
• /cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_dask.sif

Similar but with a recent GCC 13.2:

• /cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_gcc13.sif
• /cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_gcc13_cma.sif

Getting Started

In order to use a container image of your choice, login to the (non-VAE) cluster submit nodes. These nodes grant access to the bare-metal environment enabling users to launch custom container instance:

# ...login to a (bare-metal) submit node
ssh $USER@virgo.hpc.gsi.de

# ...selecting a RockyLinux 8 container with GCC 8.5.0
export APPTAINER_CONTAINER=/cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx.sif

# ...bind storage directories to the container
export APPTAINER_BINDPATH=/lustre/rz/<username>/,/cvmfs
export OMPI_MCA_io=romio341

# ...run an application within a container instance
echo "." | srun --export=ALL -- $APPTAINER_CONTAINER epoch3d

#!/bin/bash

echo "." | srun  --export=ALL  ./epoch3d.sh

#!/bin/bash

export APPTAINER_CONTAINER=/cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_cma.sif
export APPTAINER_CONFIGDIR=/lustre/rz/$USER/apptainer 

export OMP_NUM_THREADS=1
export OMPI_MCA_io=romio341

export EPOCH_EXE=epoch3d

nb_instances=`flock -x /tmp apptainer instance list $SLURM_JOB_ID | grep $SLURM_JOB_ID | wc -l`

if [ $nb_instances -eq 1 ]
then
        echo "Instance already created:  join ..." $nb_instances $SLURMD_NODENAME  
        apptainer exec instance://$SLURM_JOB_ID $EPOCH_EXE 
else
        apptainer instance list $SLURM_JOB_ID | grep $SLURM_JOB_ID | wc -l
        echo "Instance not created:  create it ..." $nb_instances $SLURMD_NODENAME
        flock -o -x /tmp \
              apptainer instance start  -B /lustre -B /cvmfs $APPTAINER_CONTAINER $SLURM_JOB_ID
        apptainer exec instance://$SLURM_JOB_ID $EPOCH_EXE
fi

>>> apptainer exec /cvmfs/phelix.gsi.de/sifs/cpu/rlx8_ompi_ucx.sif bash -l  
Centos system profile loaded ...
Apptainer> python3 --version
Python 3.6.8
Apptainer> python3
Python 3.6.8 (default, Feb 21 2023, 16:57:46) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import matplotlib
>>> import sdf
...

GPU Containers

A dedicated container images provides the latest WarpX 23.11 used with the cluster gpu partition. The image is built on top of the standard RockyLinux container adding latest Radeon Open Compute (ROCm) ¹⁵ from AMD. The image has been tested on the AMD MI 100 GPUs available on the gpu partition. The gpu_scripts/ sub-directory in the pp-container repository provides example batch scripts to launch a batch job on GPUS:

#!/bin/bash

# Define container image and working directory
export CONT=/cvmfs/phelix.gsi.de/sifs/gpu/dev/rlx8_rocm-5.7.1_warpx.sif
export WDIR=/lustre/rz/<username>/gpu/warpx

# Define I/O module for openMPI 
export OMPI_MCA_io=romio341

# Definie apptainer external filesystems bindings 
export APPTAINER_BINDPATH=/lustre/rz/<username>/,/cvmfs

# Executable with dimentionality and corresponding input deck file.
srun --export=ALL -- $CONT warpx_2d  $WDIR/scripts/inputs/warpx_opmd_deck

Install Software

Install additional packages and software using the provided container images as foundation. Build software within a container environment by stating a container instance for an infective session. The apptainer> prompt indicates the containerized environment. Load your Bash profile by running the bash command (note that your home-directory is automatically mounted to the container):

# ...set an environment variable with the path for the container image to use
export APPTAINER_CONTAINER=/cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_cma.sif

# ...start the container mounting shared storage (if required)
apptainer exec $APPTAINER_CONTAINER -B /lustre -B /cvmfs bash -l
Apptainer> 

# ...load your Bash profile
Apptainer> bash
[<username>@lxbk1130 /lustre/rz/<username>]$

You can check for example which version are available within the environment:

>>> g++ --version
g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

>>> ompi_info | grep ucx
  Configure command line: '--prefix=/usr/local' '--with-pmix=/usr/local' '--with-libevent=/usr' '--with-ompi-pmix-rte' '--with-orte=no' '--disable-oshmem' '--enable-mpirun-prefix-by-default' '--enable-shared' '--without-verbs' '--with-hwloc' '--with-ucx=/usr/local/ucx' '--with-lustre' '--with-slurm' '--enable-mca-no-build=btl-uct' '--with-cma=no'
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.0)
                 MCA pml: ucx (MCA v2.1.0, API v2.1.0, Component v5.0.0)

Installation additional software to Lustre shared storage and use it with a batch script similar to:

export APPTAINER_CONTAINER=/cvmfs/phelix.gsi.de/sifs/cpu/dev/rlx8_ompi5_ucx_cma.sif
export APPTAINER_BINDPATH=/lustre/rz/<username>/,/cvmfs
export SLURM_WORKING_DIR=/lustre/rz/<username>/warpx
export OMPI_MCA_io=romio321

srun --export=ALL \
      -- apptainer exec -B /lustre -B /cvmfs $APPTAINER_CONTAINER \
      $my_executable $options

Apptainer Container

Depending on the target hardware different container images are recommended:

• For CPU workloads JupyterLab ⁵ containers are the best choice in terms of options available, e.g. frameworks like TensorFlow, Scipy, Spark or languages like Julia or R. Please refer to the official documentation ⁶ in order to see which container may be better suited for your application.
• For GPU workloads it is preferable to use container images provided by either the AMD Infinity Hub ⁷ or the NVIDIA Registry ⁸ for the respective hardware.

A simple array example which uses NumPy:

#!/usr/bin/env python

import numpy as np

array = np.array([
    [3, 7, 1],
    [10, 3, 2],
    [5, 6, 7]
])
print(array)
print()

# Sort the whole array
print(np.sort(array, axis=None))

# Sort along each row
print(np.sort(array, axis=1))

# Sort along each column
print(np.sort(array, axis=0))

Pull the JupyterLab data-science container from Docker Hub:

apptainer pull docker://jupyter/datascience-notebook:latest

Test the container interactively with the previous script:

apptainer exec datascience-notebook.sif numpy_array.py

Refer to the container section in the Virgo User Guide for instructions on how to submit an container as batch job on the compute cluster.

Conda-Forge

Alternative to a container it is possible to install a standalone Python installation using Conda-Forge, a community-driven installer for Python. Note that Anaconda and Miniconda should not be used on the cluster infrastructure due its licence!

# Download the installer...
curl -OL https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh

# ...set a prefix environment variable
APP_DIR="~/opt/conda"
# ...run the install passing the prefix as option
/bin/sh Miniforge3-Linux-x86_64.sh -b -p ${APP_DIR}

# ...load the installation into your shell environment
source ${APP_DIR}/etc/profile.d/conda.sh

# ...modifies your shell prompt and show (base) as prefix
conda activate

# ...print information about Conda
conda info

# ...remove Conda from your shell environment
conda deactivate

# ...search for packages
conda search $package_name

# ...install a package with a specified version...
conda install pytorch=2.0.0=cpu_py39he4d1dc0_0

# create a new environment with basic options
conda create -n $env_name

# switch to the newly created environment
conda activate $env_name

# install packages within the environment
conda install $package_name

# list installed packages
conda list

# remove an environment
conda env remove -n $env_name

Virtual Environments

Python virtual environments are useful if you do not want to maintain a complete Python installation yourself, but rather a small subset of Python packages. The virtual application environments provide Python as loadable Spack package:

# Load the Python interpreter...
spack load python target=$(spack arch -t)

# ...and create a virtual environment
python -m venv myEnv

• Initialize the environment: source venv/bin/activate (the environment can be deactivated simply with: deactivate)
• Installation of additional modules can be done simply with pip install.

Note that the installation directory (if not specified otherwise) will always be $HOME/myEnv, so your Linux home directory (/u/$USER) will be used as default installation path which poses a problem when a job is launched via Slurm since the home directory is not available on the batch farm.

If you decide to use a directory on Lustre to deploy Python virtual environments, we will kindly ask you to avoid having dozens of “big” frameworks installed (e.g. TensorFlow, PyTorch, etc.) at the same time. This would affects the performance of the shared storage for all users. Please read the Lustre best practices section in the User Guide for more details. Consider to use containers as described above instead.