Athena on Lima

Last update: 08 Feb 2024 [History] [Edit]

Introduction

Historically the Virtual Machines recommended for use in ATLAS have included Parallels and VirtualBox. However, Parallels is not free, and if you are using a newer Mac, Virtual box does not yet support Apple Silicon.

Lima is another virtualisation technology that can also be used to run Athena (and which also works on Apple silicon macs).

The main advantages over Docker are that it is potentially a lot faster, and is (arguably) also easier to setup.

Internally Lima either uses qemu, or it can use the native macOS virtualisation technology (which has significant performance advantages).

Getting started

First, you need to get Lima:

brew install lima --HEAD

Tip In versions of Lima prior to v0.19.0 there was a serious known problem with Lima (see known issues below).

Next, you need to start a container. The recommended approach is to use the centrally hosted configuration

limactl start https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml --name=alma9

Tip At this point you can edit the configuration (see below) to change the number of cores, memory, etc. Please also look at the known issues section below.

You can use limactl list to see the status of your container:

NAME     STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
alma9    Running    127.0.0.1:57323    qemu      aarch64    4       4GiB      100GiB    ~/.lima/alma9

You now have a fully functioning version of almalinux 9, with lots of useful packages installed, and in particular a fully functioning CVMFS setup inside the container. To access it, you can do:

limactl shell alma9

And once inside the container, you can do the usual:

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
asetup Athena,main,latest

Now you can clone Athena, compile packages and run as usual. By default, you will have access to /tmp/lima on both the host and the container. So, for example, you could clone Athena there (but bear in mind that it is still faster to compile directly within the VM and not on a folder shared with the host, so another option is to use /home/<your name> and ssh into the VM).

So to quickly recap, all you need to do to have a fully functioning Athena installation on your mac is to do:

brew install lima {--HEAD}
limactl start https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml --name=alma9
limactl shell alma9

Known issues

  • Concurrency bugs in the default Linux Kernel 5.14 issued with AlmaLinux 9.2 (see this comment on the UTM ticket for details). This can be mitigated by using a newer kernel (see here) and it is strongly recommended that you do this.
  • Another issue that has been reported is that Lima will not start, complaining e.g. 
    Waiting for the essential requirement 1 of 3: "ssh"
    • A solution, suggested in this comment, is to disable the guest machine (and ssh) from sleeping, by typing the following inside the guest: 
      sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
    • (if you cannot get into the guest, try doing a factory reset with limactl factory-reset from the host)

Older issues

In versions of Lima prior to v0.19.0 there was serious issue with xfs corruption and hanging virtual machines on Apple Silicon (this does not only affect Lima, but other virtualisation solutions too). You can read more about it on Lima issue #1957 and UTM issue #4840, but the short version is there seem to be two causes:

  • Further bugs related to virtualisation of Linux on Apple Silicon (they do not appear with e.g. OpenBSD and do not seem to appear with x86).

It seems like one workaround is to disable caching. This was included in v0.19.0.

Moving on from the default configuration

Adding a case sensitive volume

One problem you might run into is that macOS is by default case-insensitive, whilst Linux (certainly as used by ATLAS) is case-sensitive. This can cause subtle problems, so you are therefore advised to either work entirely within the VM (which is by far the fastest and most stable option and is the recommended way to work) or to create a case-sensitive disk image to share.

If you really do need to do this, open Disk Utility on the host (your mac) and click the + Volume button at the top. If you do this, you should choose “APFS (Case-sensitive)” for the Format. (Be aware that this volume is dynamic and will grow with the content i.e. you should be careful to not fill up your host’s disk).

Now we need to tell Lima that it should grant access to this folder. To do this, quit out of the VM and stop it, the edit the configuration file:

limactl stop alma9 
limactl edit alma9

You should uncomment the following lines:

#  - location: /Volumes/Lima
#    writable: true

Lima will ask you if you want to start the instance. Choose yes, and use limactl shell alma9 to get back in. You should now be able to see the new volume:

cd /Volumes/Lima/
git clone ssh://git@gitlab.cern.ch:7999/atlas/athena.git
cp athena/Projects/WorkDir/package_filters_example.txt package_filters.txt
mkdir build
cd build
cmake -DATLAS_PACKAGE_FILTER_FILE=../package_filters.txt ../athena/Projects/WorkDir
# etc

Improving performance

By default you will get (typically) 4 cores and 4Gib. If you have a powerful machine and would like something more beefy, you can stop the container and again edit this default configuration. This time, uncomment these lines:

# How many cores to give the VM
#cpus: 6
# How much memory to give the VM
#memory: 16GiB

(Obviously tweak these depending on what your machine can handle). If you restart, you should now be able to see your changes:

~ % limactl list
NAME     STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
alma9    Running    127.0.0.1:57816    qemu      aarch64    6       16GiB     100GiB    ~/.lima/alma9

You can also achieve the same effect from the standard template by e.g.

limactl start --vm-type vz --cpus 6 --memory 20 almalinux9.yaml --name myalma9

or by choosing to edit the configuration when you start the VM.

Changing virtualisation framework and filesystem access

One final thing you can change is to uncomment this line:

#vmType: "vz"

(Or start with --vm-type vz)

This will switch from using qemu (the default) to macOS Virtualization.Framework and virtiofs. This is much faster (git clone is ~3 times quicker), but (as of v0.18) is still considered experimental.

Warning This will require a complete re-install of the VM, so you will need copy the file locally, edit it, and then delete any existing VM limactl delete alma9 (or just create a new one).

curl https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml > myalmalinux.yaml
# Edit the yaml file
limactl start ./myalmalinux.yaml --name=myalma9

Enable X graphics

Enable X graphics export from the VM to the host with the following steps.

(1) Edit the Lima VM configuration to enable the display:

video:
  display: default

(2) After login to the VM with limactl shell alma9 create an account with:

sudo passwd <USERNAME>

(3) Outside the VM, find the ssh port <SSHPORT> number for the currently running VM:

limactl list

(4) Then login to the VM with:

ssh -X -Y <USERNAME>@127.0.0.1 -p <SSHPORT>

New Linux kernel

In case of problems with the file system or vz virtualization it might be worth updating to the latest stable mainline linux kernel from the standard AlmaLinux9 linux kernel 5.14. The EL9 kernel repository is at elrepo.org.

To update interactively use:

sudo yum update # Optional for additional updates over default AlmaLinux 9.2
sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo yum install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm
sudo dnf config-manager --set-enabled elrepo-kernel
sudo yum install kernel-ml
sudo yum update

After these steps, reboot the VM.

Debugging

In case there are problems, have a look at the Lima documentation but for some logs to look at are

$HOME/.lima/<INSTANCE>/serial.log
/var/log/cloud-init-output.log (inside the guest)
/var/log/cloud-init.log (inside the guest)

You can also start with the --debug flag:

limactl --debug start alma9

Accessing the VM from outside

You can either access the VM by exposing a display (see above), logging in via limactl shell, or directly by ssh-ing into the VM. To do this, you need to find the ssh port number by typing limactl list.

You can also use the built-in ssh functionality of some IDEs, such as Visual Studio Code. Please see the our ssh VSCode documentation here or the VSCode documentation for more details.

If you do want to use VSCode, you can simplify the experience by using an auto-generated ssh config file. To do this, you need to have the Remote - SSH extension installed and then go to ‘Settings->Extensions->Remote - SSH’ and look for ‘Config File’ (see screenshot below). To this you then add the absolute file path to your Lima VM’s ssh config i.e. /Users/emoyse/.lima/alma9/ssh.config.

Now you can click on the connection icon in the bottom left of VSCode and choose ‘Connect to Host…’ and you should see your VM listed. Click on it and you should be able to connect easily to the VM from now on.