How to build a release

• Introduction
• Build externals and Athena together
  • The build scripts
  • Setup and build
    • Using Ninja for the build
  • Running Athena
  • Running unit tests
• Building the externals alone
  • Without build recipe modification
  • With build recipe modifications
  • Building packages on top of the modified external(s)
• Building Athena alone
  • Using the build script
  • Performing the build by hand
    • Building Athena by hand, on top of local externals
• Building Athena packages with private Gaudi build

Introduction

In addition to the usual centrally produced releases and nightlies it is of course possible to build the release yourself. And for certain developments, this is probably the best way to go — you build the entire Athena code stack yourself, make your changes, compile and run your tests. Of course this requires a powerful machine (don’t try this on lxplus!) and even so will take several hours the first time (subsequent rebuilds should usually be much faster). It will also require a significant amount of space - how much exactly will depend on what you do, but for e.g. a full rebuild of Gaudi, atlasexternals and Athena, you should probably allow about 100 GB.

To start with, please make sure you have read the information on releases and nightly builds, so you understand a bit how it all fits together.

Be aware that we depend on the LCG releases, so we’re assuming that you have cvmfs installed. If not, it’s still possible to install from e.g. RPMs but this is not yet covered in these instructions.

For ATLAS people, you can also check the instructions on the reconstruction software twiki.

Note that these instructions refer to building the projects out of the main branch of athena. Building releases out of other/older branches can be a little different. (Though the overall setup for a full build is generally kept stable.)

Normally you won’t want to compile exactly what was already in a nightly, but rather a version with modifications. So make sure to have a look at building the externals and building athena below.

Build externals and Athena together

The build scripts

For the nightlies, the central build machines do the following:

asetup none,gcc131,cmakesetup
./athena/Projects/Athena/build_externals.sh
./athena/Projects/Athena/build.sh

i.e. they use asetup to get a compiler, then call a script to checkout, build, install and package the externals, and finally another script to build, install and package Athena.

You can have a look at the general build scripts under Build/AtlasBuildScripts, and the Project specific scripts are found in the Projects directory. For example, the Athena scripts are under Projects/Athena. The main scripts that matter here are build_externals.sh (which checks out and builds the AthenaExternals project out of the appropriate version of the atlasexternals repository according to Projects/Athena/externals.txt), and build.sh which builds Athena.

The scripts come with extensive help messages (see --help) about their numerous command line options.

Setup and build

To build externals and Athena (without building any RPM packages) on an “lxplus-like machine” you can go to a fresh directory and do the following:

setupATLAS
lsetup git
git clone https://gitlab.cern.ch/atlas/athena.git
asetup none,gcc131,cmakesetup
./athena/Projects/Athena/build_externals.sh -c
./athena/Projects/Athena/build.sh -acmi

For older gcc62-based releases, you should instead do: asetup none,gcc62,cmakesetup --gccversion=6.2.0binutils

In some cases, you may need to provide a specific cmake version: asetup none,gcc83,cmakesetup --cmakeversion=3.14.7

You should refer to the list of command line options for the meaning of the recommended options given above. These turn off the build of RPM packages, and make sure that the build would stop right away on errors. (As opposed to how we perform “nightly builds”, which continue on errors as long as they can.)

A full Athena build needs access to CVMFS, AFS and EOS at the time of writing.

• CVMFS is a very strong requirement. It’s not impossible to build Athena without it, but it certainly requires expert level knowledge about our build system.
• AFS is used during the build to pick up authentication files that are only visible to members of the ATLAS experiment. If these files are not readable during the build, the build results will not contain these authentication files. (For many software development purposes this is okay. Most simulation/reconstruction jobs do not need such direct access to ATLAS databases.)
• EOS is (currently) used as the source of Geant4 data files. Which, at the time of writing, are needed already during the build. If you don’t have access to EOS, your best bet is to set:

export G4PATH=/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-el9-gcc13-opt/Geant4

You can find the appropriate path for the nightly/branch that you want to build locally, like:

asetup Athena,main,latest
echo $G4PATH
/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-el9-gcc13-opt/Geant4

On appropriate systems you may need to explicitly mount EOS, like:

export EOS_MGM_URL=root://eosatlas.cern.ch
/usr/bin/eos -b fuse mount eos

Depending on your EOS installation, after kinit and mounting EOS, you may need to also do:

eosfusebind -g

Using Ninja for the build

CMake and the build scripts will make use of GNU Make for the build by default. In order to use some other build system generator, you can pass extra arguments to CMake through the build scripts.

The most useful alternative build system to use is Ninja, a lightweight piece of software, which is much more efficient at dealing with the hundreds of thousands of files and tens of thousands of build targets that we use while building Athena.

To use it, first make sure that the ninja executable is available in your runtime environment. In an “lxplus-like environment” you should do this with:

export PATH=/cvmfs/sft.cern.ch/lcg/contrib/ninja/1.11.1/Linux-x86_64/bin:$PATH

With it set up, you can tell the Athena (or AthenaExternals) build to use Ninja with:

-x "-G Ninja"

If you use the full build for “iterative development”, you should definitely make use of Ninja. And once the first build is done, perform the additional builds like:

./athena/Projects/Athena/build.sh -ami -x "-G Ninja"

Running Athena

In order to use the above, in a fresh terminal you would do:

setupATLAS
asetup Athena,<VERSION> --releasepath=<YOURBUILDAREA> --siteroot=/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-el9-gcc13-opt

If needed, you can help Athena find additional files that may be missing from your local build (as discussed above) like:

export DATAPATH=$DATAPATH:/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-el9-gcc13-opt/atlas/offline/ReleaseData/v20

Running unit tests

To run the unit tests, use the following environment setup:

setupATLAS
asetup Athena,<VERSION> --releasepath=<YOURBUILDAREA> --siteroot=/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-el9-gcc13-opt
cd <YOURBUILDAREA>/build/Athena/
ctest

Building the externals alone

If you want to test a new version of an external (like Gaudi, Geant4, Acts, etc.), it is often enough to only build a select number of packages on top of an existing nightly, instead of embarking on a full project build. In such cases it is generally enough to only build the external in question, and not the full AthenaExternals project.

The following will specifically talk about building an alternative version of Gaudi, but the same techniques can be used for all other externals as well. Specific instructions for building against a locally modified version of Gaudi can be found in this section.

Without build recipe modification

In case the new/alternative version of the external does not need modified build instructions, you can make use of the build_externals.sh script directly. Like:

./athena/Projects/Athena/build_externals.sh -b $PWD/gaudi-build -x -DATLAS_GAUDI_TAG="vXXrY.ZZZ" -k Package_Gaudi

You need to check exactly which CMake cache variable(s) you should override in the appropriate package in atlasexternals/External to build the version of the external that you want.

With build recipe modifications

In case you need to modify the build recipe of the external in question, please refer to the instructions given in the atlasexternals repository. But in general, you will be able to build your (modified) package like:

asetup none,gcc131,cmakesetup
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -S ./atlasexternals/Projects/AthenaExternals/ -B ./gaudi-build
cmake --build ./gaudi-build --target Package_Gaudi
DESTDIR=./gaudi-install cmake --install ./gaudi-build

Once you’re happy with your modifications to atlasexternals, you can test your feature branch’s build using the build scripts of the athena repository with:

export AtlasExternals_URL=https://gitlab.cern.ch/<username>/atlasexternals.git
export AtlasExternals_REF=origin/<branch_name>
./athena/Projects/Athena/build_externals.sh -b $PWD/gaudi-build -k Package_Gaudi

Building packages on top of the modified external(s)

To now build a select number of packages on top of a full nightly build, but using your locally installed external(s), you can do the following:

• Prepare a package selection file as described here;
• Set up your build like:

setupATLAS
asetup Athena,main,latest
mkdir build
cd build/
export CMAKE_PREFIX_PATH=<EXTERNAL_INSTALL_DIR>:$CMAKE_PREFIX_PATH
cmake -DATLAS_PACKAGE_FILTER_FILE=../package_filters.txt ../athena/Projects/WorkDir/

Note that the directory that you need to point to with CMAKE_PREFIX_PATH depends on whether you used build_externals.sh, or you built the external yourself by hand.

• When using build_externals.sh, you need to do the following wrt. to the “build directory” specified for the script:

export CMAKE_PREFIX_PATH=$PWD/gaudi-build/install/AthenaExternals/<VERSION>/InstallArea/<ARCHITECTURE>:$CMAKE_PREFIX_PATH

• When building the external “by hand” according to the above instructions, the path would have to be set like:

export CMAKE_PREFIX_PATH=$PWD/gaudi-install/AthenaExternals/<VERSION>/InstallArea/<ARCHITECTURE>:$CMAKE_PREFIX_PATH

Building Athena alone

Once the externals are built correctly, or in case you just want to use the externals from an existing nightly / release, you still have some options for how to perform a full Athena build.

Using the build script

To build Athena on top of the externals from the nightly, you can use the same recipe as shown earlier, as long as you are in the correct environment. Like:

setupATLAS
asetup AthenaExternals,main,latest
./athena/Projects/Athena/build.sh -acm -x "-G Ninja"

Note that the installation step was skipped in this case, since most of the time you don’t actually need it when building Athena. Also note that for incremental builds you should later on just execute:

./athena/Projects/Athena/build.sh -am

I.e. you shouldn’t re-run the CMake configuration on every build attempt. Not unless your code modifications explicitly require it.

Performing the build by hand

An even better setup can be to just build Athena “completely by hand”. For interactive code development this results in a much better user experience.

In this setup you would set up your build like:

setupATLAS
asetup AthenaExternals,main,latest
mkdir build
cd build/
cmake -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE ../athena/Projects/Athena/
cmake --build .

Note the -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE argument. If you use it, code editors like VS Code will be able to find all headers when looking at C++ sources.

Building Athena by hand, on top of local externals

If you want to build Athena “by hand” (directly executing the build commands as shown above), and want to do it on top of a full local build of the externals performed by build_externals.sh, then the most convenient setup to use is:

setupATLAS
asetup none,gcc131,cmakesetup
source ./athena/Projects/Athena/build_env.sh -b <EXTERNAL_INSTALL_DIR>
mkdir build
cd build/
cmake -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE ../athena/Projects/Athena/
cmake --build .

Building Athena packages with private Gaudi build

Sometimes it is necessary to build a subset of Athena packages on top of a locally modified Gaudi version.

The build needs to be done in two separate SSH sessions (to avoid mixing Gaudi/Athena environments) and you will end up creating a directory hierarchy like the following in your working area:

- ${GAUDI_INSTALL}
   - Gaudi    # Gaudi sources
   - build    # Gaudi build area
   - inst     # installed Gaudi libraries
- athena      # Athena sources
- build       # Athena build area

The first step is to find the currently used version of Gaudi and LCG. In case of building against the main branch check Projects/Athena/build_externals.sh where you will find e.g.:

ATLAS_EXTRA_CMAKE_ARGS=(-DLCG_VERSION_NUMBER=104
                        -DLCG_VERSION_POSTFIX="d_ATLAS_19"
                        -DATLAS_GAUDI_SOURCE="URL;https://gitlab.cern.ch/atlas/Gaudi/-/archive/v38r2.000/Gaudi-v38r2.000.tar.gz;URL_MD5;739b777252581eb13dd159532ea19985"

Then use this information to configure a standalone build of Gaudi. Note that a few components need to be disabled in the Gaudi build (taken from ExternalProject_Add in the official externals build configuration):

# for convenience
export GAUDI_VERSION=v38r2.000
export LCG_VERSION=LCG_104d_ATLAS_19

# setup environment (choose release as appropriate)
setupATLAS && lsetup git
asetup AthenaExternals,main,latest

# create build aread and checkout Gaudi
mkdir ${GAUDI_INSTALL} && cd ${GAUDI_INSTALL}
mkdir build inst
git clone https://gitlab.cern.ch/atlas/Gaudi.git -b ${GAUDI_VERSION}
cd build

# build
export CMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}:/cvmfs/sft.cern.ch/lcg/views/${LCG_VERSION}/${LCG_PLATFORM}
cmake ../Gaudi -DCMAKE_INSTALL_PREFIX=../inst/${LCG_PLATFORM} -DGAUDI_USE_GPERFTOOLS:BOOL=FALSE -DGAUDI_USE_DOXYGEN:BOOL=FALSE -DGAUDI_USE_INTELAMPLIFIER:BOOL=FALSE -DGAUDI_USE_JEMALLOC:BOOL=FALSE -DBUILD_TESTING:BOOL=FALSE
make install

The above uses the ATLAS mirror (atlas/Gaudi) of the official Gaudi repository (gaudi/Gaudi). It contains the official Gaudi tags (with postfix .000) and optionally tags with back-ports of Gaudi patches (e.g. .001). You can also use the above recipe to build against the HEAD of the official Gaudi repository by replacing of the URL in the git clone.

In a new shell, you can now use this build of Gaudi in a regular sparse build of Athena:

setupATLAS
asetup Athena,main,latest
export Gaudi_DIR=${GAUDI_INSTALL}/inst/${LCG_PLATFORM}
mkdir build && cd build
# edit your package_filters.txt
cmake -DATLAS_PACKAGE_FILTER_FILE=package_filters.txt ../athena/Projects/WorkDir
make
source ./${LCG_PLATFORM}/setup.sh