How to Build an Apptainer Pre-Installed Rocky Linux 8 Custom Image for AWS ParallelCluster

Yoshiaki SendaJanuary 17, 2024

The last article showed how to use AWS ParallelCluster 3.8.0 with Rocky Linux 8. This article shows you how to use AWS ParallelCluster with an Apptainer pre-installed Rocky Linux 8 custom image. This method can also be applied to installing other software. Let’s get started.

Create Apptainer installation component

To customize your image, create “component” apptainer.yaml with the following contents:

name: Apptainer Installation
description: This is a component to show how to install Apptainer PR
schemaVersion: 1.0

phases:
  - name: build
    steps:
      - name: apptainerInstallation
        action: ExecuteBash
        inputs:
          commands:
            - |
              set -v
              # Ensure EPEL repository is available
              dnf install -y epel-release
              # Install Apptainer
              dnf install -y apptainer

Store the yaml file to your S3 bucket:

$ aws s3 cp apptainer.yaml s3://ysenda

Create a component using the aws imagebuilder create-component command:

$ aws imagebuilder create-component \
    --region ap-northeast-1 \
    --name apptainer \
    --semantic-version 1.0.0 \
    --platform Linux \
    --uri s3://ysenda/apptainer.yaml

Sample output:

{
    "requestId": "*******************",
    "clientToken": "********************",
    "componentBuildVersionArn": "arn:aws:imagebuilder:ap-northeast-1:****:component/apptainer/1.0.0/1"
}

We will use this componentBuildVersionArn value in the next section.

Build Apptainer pre-installed Rocky Linux 8 custom image

Create rocky8-apptainer.yaml with the following contents. The only difference from the last article is adding “Components” to the “Build” section.

Region: ap-northeast-1
Image:
  Name: Rocky Linux 8.8 Custom AMI for PCluster 3.8.0
  RootVolume:
    Size: 100
Build:
  ParentImage: ami-069b05da577cf3215
  InstanceType: c5.4xlarge
    Components:
    - Type: arn
      Value: arn:aws:imagebuilder:ap-northeast-1:****:component/apptainer-pr/1.0.0/1

Build Apptainer pre-installed Rocky Linux 8 custom image for AWS ParallelCluster using the pcluster build-image command. It takes about an hour.

$ pcluster build-image --image-id rocky8-apptainer \
    --image-configuration rocky8-apptainer.yaml

{
  "image": {
    "imageId": "rocky8-apptainer",
    "imageBuildStatus": "BUILD_IN_PROGRESS",
    "cloudformationStackStatus": "CREATE_IN_PROGRESS",
    "cloudformationStackArn": "arn:aws:cloudformation:ap-northeast-1:*****:stack/rocky8-apptainer/*****",
    "region": "ap-northeast-1",
    "version": "3.8.0"
  }
}

Once the Apptainer pre-installed image is successfully built:

$ pcluster describe-image --image-id rocky8-apptainer | grep -i status

"imageBuildStatus": "BUILD_COMPLETE",
        "key": "parallelcluster:build_status"

Check AMI ID. We use this ID at the next step.

$ pcluster describe-image --image-id rocky8-apptainer --query 'ec2AmiInfo.amiId'

"ami-******"

The next step is building an HPC cluster using AWS ParallelCluster with the Apptainer pre-installed image.

Build HPC cluster with Apptainer pre-installed Rocky Linux 8 image using AWS ParallelCluster

Create a config file for the Apptainer pre-installed Rocky Linux 8 HPC Cluster using the pcluster configure command.

$ pcluster configure --config rocky8-apptainer-cluster.yaml

After the config file is generated, add CustomAmi at line 4. We use the AMI ID value we got at the previous step. The following is an example rocky8-apptainer-cluster.yaml file.

Region: ap-northeast-1
Image:
  Os: rocky8
  CustomAmi: ami-******
HeadNode:
  InstanceType: t2.xlarge
  Networking:
    SubnetId: subnet-******
  Ssh:
    KeyName: yoshisenda-tokyo
  LocalStorage:
    RootVolume:
      Size: 100
Scheduling:
  Scheduler: slurm
  SlurmQueues:
  - Name: c5n9
    ComputeResources:
    - Name: c5n9xlarge
      Instances:
      - InstanceType: c5n.9xlarge
      MinCount: 0
      MaxCount: 16
      Efa:
        Enabled: true
    Networking:
      PlacementGroup:
        Enabled: true
      SubnetIds:
      - subnet-******
  - Name: c5n18
    ComputeResources:
    - Name: c5n18xlarge
      Instances:
      - InstanceType: c5n.18xlarge
      MinCount: 0
      MaxCount: 16
      Efa:
        Enabled: true
    Networking:
      PlacementGroup:
        Enabled: true
      SubnetIds:
      - subnet-******

Build a Rocky Linux 8 HPC Cluster:

$ pcluster create-cluster \
    --cluster-configuration rocky8-apptainer-cluster.yaml \
    --cluster-name rocky8-apptainer-cluster \
    --region ap-northeast-1

{
  "cluster": {
    "clusterName": "rocky8-apptainer-cluster",
    "cloudformationStackStatus": "CREATE_IN_PROGRESS",
    "cloudformationStackArn": "arn:aws:cloudformation:ap-northeast-1:******",
    "region": "ap-northeast-1",
    "version": "3.8.0",
    "clusterStatus": "CREATE_IN_PROGRESS",
    "scheduler": {
      "type": "slurm"
    }
  }
}

The HPC Cluster with Apptainer pre-installed Rocky Linux 8 image has been successfully built by using AWS ParallelCluster! 🎉

$ pcluster describe-cluster --cluster-name rocky8-apptainer-cluster | grep -i status
"cloudFormationStackStatus": "CREATE_COMPLETE",
  "computeFleetStatus": "RUNNING",
  "clusterStatus": "CREATE_COMPLETE",

and SSH-ing to head node:

$ pcluster ssh --cluster-name rocky8-apptainer-cluster -i ~/.ssh/yoshisenda-tokyo.pem

Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu Jan  4 16:08:19 2024
[rocky@ip-10-0-0-101 ~]$

Create Intel MPI benchmark container

The cluster we just built uses an EFA-enabled instance c5n.9xlarge. Create a container definition file imb_efa.def for Intel(R) MPI Benchmarks that includes EFA-enabled MPI with the following contents.

Bootstrap: docker
From: rockylinux/rockylinux:8.8

%post
    dnf groupinstall -y 'Development Tools'
    dnf install -y git curl 

%post
    curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.25.1.tar.gz
    tar -xf aws-efa-installer-1.25.1.tar.gz
    cd aws-efa-installer
    ./efa_installer.sh -y --skip-kmod

%post
    export LD_LIBRARY_PATH=/opt/pmix/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/slurm/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/efa/lib64:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/openmpi/lib64:$LD_LIBRARY_PATH
    export PATH=/opt/amazon/efa/bin:$PATH
    export PATH=/opt/amazon/openmpi/bin:$PATH

%post
    cd /usr/src
    git clone https://github.com/intel/mpi-benchmarks.git
    cd mpi-benchmarks/src_c
    make all

%runscript
    export LD_LIBRARY_PATH=/opt/pmix/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/slurm/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/efa/lib64:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/openmpi/lib64:$LD_LIBRARY_PATH
    export PATH=/opt/amazon/efa/bin:$PATH
    export PATH=/opt/amazon/openmpi/bin:$PATH
    /usr/src/mpi-benchmarks/src_c/IMB-MPI1 "$@"

Build container using Apptainer:

apptainer build imb_efa.sif imb_efa.def

Create a sample job script job.sh with the following content:

#!/bin/bash
#SBATCH --partition=c5n9
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1

#FI_LOG_LEVEL=Debug \
PMIX_MCA_gds=hash \
srun --mpi=pmix imb_efa.sif -msglog 3:28 PingPong

Submit job:

$ sbatch job.sh

Conclusion

This article showed you (1) how to build an Apptainer Pre-installed Rocky Linux 8 Custom Image for AWS ParallelCluster, (2) how to build an HPC Cluster with Apptainer pre-installed Rocky Linux 8 image, and (3) how to build an EFA-enabled Intel MPI Benchmarks container using Apptainer.

The benefit of this approach, using Apptainer for your software installation, is once you have built containers of your applications and push those containers to the registry, you can pull containers whenever and/or wherever you need. Separation of this HPC clusters deployment and applications installation brings you more flexibility and agility when you use cloud compute resources.

How to use container registry with Apptainer and how to use container registry with AWS ParallelCluster will be covered in the next article.

Happy New Year!

Appendix

OSU Micro-benchmarks

Container definition file for OSU Micro-benchmarks:

Bootstrap: docker
From: rockylinux/rockylinux:8.8

%post
    dnf groupinstall -y 'Development Tools'
    dnf install -y git curl wget

%post
    curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.25.1.tar.gz
    tar -xf aws-efa-installer-1.25.1.tar.gz
    cd aws-efa-installer
    ./efa_installer.sh -y --skip-kmod

%post
    export LD_LIBRARY_PATH=/opt/pmix/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/slurm/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/efa/lib64:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/openmpi/lib64:$LD_LIBRARY_PATH
    export PATH=/opt/amazon/efa/bin:$PATH
    export PATH=/opt/amazon/openmpi/bin:$PATH

%post
    cd /usr/src
    wget http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.6.2.tar.gz
    tar zxvf ./osu-micro-benchmarks-5.6.2.tar.gz
    cd osu-micro-benchmarks-5.6.2
    ./configure CC=mpicc CXX=mpicxx
    make -j $(nproc)

%runscript
    export LD_LIBRARY_PATH=/opt/pmix/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/slurm/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/efa/lib64:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/opt/amazon/openmpi/lib64:$LD_LIBRARY_PATH
    export PATH=/opt/amazon/efa/bin:$PATH
    export PATH=/opt/amazon/openmpi/bin:$PATH
    /usr/src/osu-micro-benchmarks-5.6.2/"$@"

Sample job script for OSU Micro-benchmarks:

#!/bin/bash
#SBATCH --partition=c5n9
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1

#FI_LOG_LEVEL=Debug \
PMIX_MCA_gds=hash \
srun --mpi=pmix osu-mb.sif mpi/pt2pt/osu_bw