Overview
HPC Infrastructure
This website describes the High-Performance Computing (HPC) infrastructure built by the GSI IT Department. The HPC facility is built for physics simulation and experiment data analysis. The infrastructure is available to all scientists associated with GSI and/or FAIR experiments.
Cluster Computing, Data Analysis, Data Simulation
The Quickstart section in the user guide presents a short list of steps to access the HPC infrastructure.
Introduction
The Virgo cluster has replaced its predecessor Kronos after being in production from 2015 until end of 2020. The system was made available to selected users as pre-production in June 2020 and entered production in August 2020. The entire cluster infrastructure is located in the Green Cube compute center (see Figure 1) at GSI Darmstadt. All applications on Virgo run in Linux containers and allow for a complete separation of host platform from user workload. This enables the continuous upgrade of the cluster without interference for users.
Virgo provides Virtual Application environments (VAEs) based on containers. VAEs enable multiple different user environments on a single host platform. The cluster computing group supports a selection of VAEs for common use-cases. All users can build and run their own custom containers as well.
The capabilities of the workload management system have been adapted to allow flexible partitioning of resources. This is particularly useful in order to dynamically orchestrate groups of compute nodes upon requests for online-computing by different scientific experiments. The following document introduces the reader to the software and hardware platform composing the Virgo cluster infrastructure, and illustrates very basic examples how to use this resource for scientific computing.
Clusters
Two sibling clusters complement the Virgo infrastructure. Hydra is a dedicated GPU cluster optimized for accelerator workloads. Orion serves as a test cluster where updates and configuration changes are validated before being rolled out to the Virgo production environment.
| Cluster | Description |
|---|---|
| Virgo | Main production cluster for physics simulation and experiment data analysis |
| Hydra | Dedicated GPU cluster optimized for accelerator workloads |
| Orion | Test cluster used to validate updates before rolling them out to Virgo |
Upgrades
With the migration to Virgo 4.0, GPU resources have been consolidated into a dedicated cluster named Hydra, allowing the HPC team to optimize its Slurm configuration specifically for GPU workloads.
The following table lists upgrades to Virgo since it came into production in 2020:
| Date | Version | Base OS | Slurm | Lustre | CVMFS | Container Engine | CUDA | ROCm |
|---|---|---|---|---|---|---|---|---|
| 2020/03 | 1.0 | CentOS 7.7 | 18.04 | 2.12 | 2.6 | Singularity 3.4 | ||
| 2020/07 | 1.1 | CentOS 7.8 | 18.04 | 2.12 | 2.7 | Singularity 3.6 | ||
| 2021/02 | 1.2 | CentOS 7.9 | 18.04 | 2.12 | 2.8 | Singularity 3.7 | 11 | 4.1 |
| 2022/02 | 2.0 | RockyLinux 8.6 | 21.08 | 2.12 | 2.9 | Apptainer 1.1 | 11.4 | 5.3 |
| 2023/10 | 2.1 | RockyLinux 8.8 | 21.08 | 2.15 | 2.11 | Apptainer 1.2 | 11.4 | 5.3 |
| 2024/03 | 3.0 | RockyLinux 8.9 | 23.11 | 2.15 | 2.11 | Apptainer 1.3 | 6.0 | |
| 2026/03 | 4.0 | RockyLinux 9.7 | 25.11 | 2.15 | 2.13 | Apptainer 1.4 |