Athena on Lima

Introduction

Lima is another containerisation technology that can be used to run Athena (and which works on Apple silicon macs).

An advantage over Docker is that it is very lightweight, and (potentially) a lot faster. It either uses qemu, or can use the native macOS virtualisation technology (which has significant performance advantages, but is at the time of writing is still considered experimental).

Getting started

First, you need to get Lima:

brew install lima

Next, you need to start a container. The recommended approach is to use the centrally hosted configuration

limactl start https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml --name=alma9

You can use limactl list to see the status of your container:

NAME     STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
alma9    Running    127.0.0.1:57323    qemu      aarch64    4       4GiB      100GiB    ~/.lima/alma9

You now have a fully functioning version of almalinux 9, with lots of useful packages installed, and in particular a fully functioning CVMFS setup inside the container. To access it, you can do:

limactl shell alma9

And once inside the container, you can do the usual:

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
asetup Athena,main--dev4LCG,latest

For Apple silicon machines, dev4LCG is currently required in order to get the correct build of Athena and externals - but soon main will work too.

Now you can clone Athena, compile packages and run as usual. By default, you will have access to /tmp/lima on both the host and the container. So, for example, you could clone Athena there.

Moving on from the default configuration

Adding a case sensitive volume

One problem you might run into is that macOS is by default case-insensitive, whilst Linux (certainly as used by ATLAS) is case-sensitive. This can cause subtle problems, so you are therefore advised to create a case-sensitive disk image. To do this, open Disk Utility on the host (your mac) and click the + Volume button at the top. You should choose “APFS (Case-sensitive)” for the Format. Be aware that this volume is dynamic and will grow with the content. You should be careful to not fill up your host’s disk.

Now we need to tell Lima that it should grant access to this folder. To do this, quit out of the VM and stop it, the edit the configuration file:

limactl stop alma9 
limactl edit alma9 

You should uncomment the following lines:

#  - location: /Volumes/Lima
#    writable: true

Lima will ask you if you want to start the instance. Choose yes, and use limactl shell alma9 to get back in. You should now be able to see the new volume:

cd /Volumes/Lima/
git clone ssh://git@gitlab.cern.ch:7999/atlas/athena.git
cp athena/Projects/WorkDir/package_filters_example.txt package_filters.txt
mkdir build
cd build
cmake -DATLAS_PACKAGE_FILTER_FILE=../package_filters.txt ../athena/Projects/WorkDir
# etc

Improving performance

By default you will get (typically) 4 cores and 4Gib. If you have a powerful machine and would like something more beefy, you can stop the container and again edit this default configuration. This time, uncomment these lines:

# How many cores to give the VM
#cpus: 6
# How much memory to give the VM
#memory: 16GiB

(Obviously tweak these depending on what your machine can handle). If you restart, you should now be able to see your changes:

~ % limactl list
NAME     STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
alma9    Running    127.0.0.1:57816    qemu      aarch64    6       16GiB     100GiB    ~/.lima/alma9

Changing virtualisation framework and filesystem access

One final thing you can change is to uncomment this line:

#vmType: "vz"

This will switch from using qemu (the default) to macOS Virtualization.Framework and virtiofs. This is much faster (git clone is ~3 times quicker), but it is still considered experimental.

This will require a complete re-install of the VM, so you will need copy the file locally, edit it, and then delete any existing VM limactl delete alma9 (or just create a new one).

curl https://gitlab.cern.ch/emoyse/Scripts/-/raw/master/Lima/almalinux9.yaml > myalmalinux.yaml
# Edit the yaml file
limactl start ./myalmalinux.yaml --name=myalma9

Known problems

Currently the VM seems to freeze when compiling AthenaExternals. This might be related to cvmfs running inside the container, and is under investigation.

At least one hard crash has been seen using the vz option.

Debugging

In case there are problems, have a look at the Lima documentation but for some logs to look at are

$HOME/.lima/<INSTANCE>/serial.log
/var/log/cloud-init-output.log (inside the guest)
/var/log/cloud-init.log (inside the guest)

You can also start with the --debug flag:

limactl --debug start alma9