How to update an atlasexternals package in Athena

• General remarks at the start
• Build AthenaExternals and Athena step by step
  • Prepare the local build area
  • Update atlasexternals
  • Build atlasexternals changes locally
  • Commit and push atlasexternal changes to gitlab
  • Athena test build with atlasexternal changes
  • Run Athena ctests
  • Upload new tarball files to EOS
  • Open the atlasexternals MR
  • Open the Athena MR

General remarks at the start

• Instructions about how to build a release are at: How to build a release
• Recommend access to a build machine, with e.g. 64 cores and 128 GB RAM - otherwise the building and testing will take several hours. On a CERN 64 CPU core build machine AthenaExternals takes about ~7 min to build and Athena takes about ~90 min using gcc13 and without ninja.
• Constant interactive access to the build machine is nice to have since AFS access during the build to the AtlasAuthentication package is recommended. Otherwise setup a AFS token that does not get lost when logging out of the build machine.
• A Release build requires about 40 GB of local disk space, a RelWithDebInfo build needs about 250 GB of disk space.
• Build on a local file system, since AFS is too slow and does usually not have enough space.

Build AthenaExternals and Athena step by step

In the subsequent commands replace <username> with your actual login username.

Prepare the local build area

bash
ssh powerfulmachine.cern.ch
mkdir /build1/<username>/onnx_20240827
cd /build1/<username>/onnx_20240827
setupATLAS -q
lsetup git

Update atlasexternals

Checkout the code from gitlab:

bash
git clone https://gitlab.cern.ch/<username>/atlasexternals.git
cd atlasexternals
git remote add upstream https://:@gitlab.cern.ch:8443/atlas/atlasexternals.git
git fetch upstream
git checkout -b onnx_20240827 upstream/main

Edit atlasexternals/External/onnxruntime/CMakeLists.txt (gitlab link) and update the onnxruntime version URLs of the 3 packages and corresponding md5sums - the URLs are at https://github.com/microsoft/onnxruntime/releases - in a later subsection the update of the new files on http://cern.ch/atlas-software-dist-eos/externals/onnxruntime/ is explained.

Build atlasexternals changes locally

asetup none,gcc13,cmakesetup --cmakeversion=3.29.5
cmake -DCMAKE_BUILD_TYPE=Release -S ./atlasexternals/Projects/AthenaExternals/ -B ./onnx-build
cmake --build ./onnx-build/ --target Package_onnxruntime

Commit and push atlasexternal changes to gitlab

git add atlasexternals/External/onnxruntime/CMakeLists.txt
git commit -m "Update to onnxruntime v1.19.0"
git push origin

A new merge request into the main branch of atlasexternals will only be opened at a later step when it has been verified that all works as expected.

Athena test build with atlasexternal changes

Since the code changes are now in gitlab, let’s do an Athena test build with the new onnxruntime code in atlasexternals:

git clone https://gitlab.cern.ch/atlas/athena.git
export AtlasExternals_URL=https://gitlab.cern.ch/<username>/atlasexternals.git
export AtlasExternals_REF=origin/onnx_20240827
./athena/Projects/Athena/build_externals.sh -c -t Release > external.log 2>&1  &
### wait until this build is finished
./athena/Projects/Athena/build.sh -cmi -t Release > build.log 2>&1 &

Check external.log and subsequently build.log for the strings error: , Error 1 and Error 2 to spot any problems. The builds take about take about 7 min and 90 min respectively on a 64 core machine.

If all goes well and there are no compilation problems, then proceed to the next step.

Run Athena ctests

In case you’d like to run the ctests of the local full athena build:

asetup Athena,25.0.16 --releasepath=/build1/<username>/onnx_20240827/build/install/
cd build/build/Athena
ctest -j32

With this configuration the CITests are not executed. These can be enabled by adding -x -DATLAS_ENABLE_CI_TESTS=TRUE to the command line of build.sh in the previous step.

Upload new tarball files to EOS

As mentioned in a earlier step, the new onnxruntime tarballs have to be uploaded to the EOS webserver. Open an ATLINFR ticket and ask the expert to copy these files to the EOS webserver. Before opening the ticket, copy the tarballs:

ssh lxplus.cern.ch
mkdir $HOME/public/onnxruntime
cd $HOME/public/onnxruntime
wget wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.0/onnxruntime-linux-aarch64-1.19.0.tgz
wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.0/onnxruntime-linux-x64-1.19.0.tgz
wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.0/onnxruntime-linux-x64-gpu-1.19.0.tgz

Open a new ATLINFR ticket as done at https://its.cern.ch/jira/browse/ATLINFR-5496.

Open the atlasexternals MR

Open the MR with the changes in atlasexternals: https://gitlab.cern.ch/atlas/atlasexternals/-/merge_requests/1143.

The MR will be reviewed, merged and a new atlasexternals tag has be created eventually at https://gitlab.cern.ch/atlas/atlasexternals/-/tags by an expert.

Open the Athena MR

If the new atlasxternals tag has been made, this new tag has to be included into athena. An expert or you will open a subsequent MR to update athena to the new atlasexternals tag like e.g. https://gitlab.cern.ch/atlas/athena/-/merge_requests/72199.

With the update of onnxruntime it is very likely that the reference file outputs of several FTAG variables in the reconstruction and/or derivation CITests will change, so the finial MR will require some additional attention and discussions with experts and domain coordinators.