How to build a release

Last update: 05 Jul 2023 [History] [Edit]

Introduction

In addition to the usual centrally produced releases and nightlies it is of course possible to build the release yourself. And for certain developments, this is probably the best way to go — you build the entire Athena code stack yourself, make your changes, compile and run your tests. Of course this requires a powerful machine (don’t try this on lxplus!) and even so will take several hours the first time (subsequent rebuilds should usually be much faster). It will also require a significant amount of space - how much exactly will depend on what you do, but for e.g. a full rebuild of Gaudi, atlasexternals and Athena, you should probably allow about 100 GB.

To start with, please make sure you have read the guide about the releases and the nightlies, so you understand a bit how it all fits together.

Be aware that we depend on the LCG releases, so we’re assuming that you have cvmfs installed. If not, it’s still possible to install from e.g. RPMs but this is not yet covered in these instructions.

For ATLAS people, you can also check the instructions on the reconstruction software twiki.

Note that these instructions refer to building the projects out of the main branch of athena. Building releases out of other/older branches can be a little different. (Though the overall setup for a full build is generally kept stable.)

Normally you won’t want to compile exactly what was already in a nightly, but rather a version with modifications. So make sure to have a look at building the externals and building athena below.

Build externals and Athena together

The build scripts

For the nightlies, the central build machines do the following:

asetup none,gcc112,cmakesetup
./athena/Projects/Athena/build_externals.sh
./athena/Projects/Athena/build.sh

i.e. they use asetup to get a compiler, then call a script to checkout, build, install and package the externals, and finally another script to build, install and package Athena.

You can have a look at the general build scripts under Build/AtlasBuildScripts, and the Project specific scripts are found in the Projects directory. For example, the Athena scripts are under Projects/Athena. The main scripts that matter here are build_externals.sh (which checks out and builds the AthenaExternals project out of the appropriate version of the atlasexternals repository according to Projects/Athena/externals.txt), and build.sh which builds Athena.

The scripts come with help messages about their numerous command line options. At the time of writing these are as follows:

[bash][atspot01]:atlassoftwaredocs > ./athena/Projects/Athena/build_externals.sh -h
Script building AthenaExternals.

Usage: build_externals.sh [options]
Options:
 -t: Build type, by default set to 'RelWithDebInfo'
 -b: 'Main build directory' to use for the code checkout and build
 -f: Force rebuild of externals from scratch, otherwise if script
     finds an external build present it will only do an incremental
     build
 -c: Build the externals for the continuous integration (CI) system,
     skipping the build of the externals RPMs.
 -x: Extra cmake argument(s) to provide for the build(configuration)
     of all externals needed by Athena.
 -k: Extra 'make tool' argument(s) to provide to the build tool.

If a build directory is not given, the default is '../build'
relative to the athena source directory.
[bash][atspot01]:atlassoftwaredocs >

[bash][atspot01]:atlassoftwaredocs > ./athena/Projects/Athena/build.sh -h
Script building Athena.

Usage: build.sh [options]
Options:
  General flags:
    -t: The (optional) CMake build type to use. (default: RelWithDebInfo)
    -b: The (optional) build directory to use.
    -x: Custom argument(s) to pass to the CMake configuration.
    -k: Custom argument(s) to pass to the 'make tool'.
    -a: Abort on error.
  Build step selection:
    -c: Execute the CMake step.
    -m: Execute the make/build step.
    -i: Execute the install step.
    -p: Execute the CPack step.

If none of the c, m, i or p options are set then the script will do
all steps necessary for a nightly build. Otherwise only the enabled
steps are run - it's your reponsibility to ensure that precusors are
in a good shape.

If a build directory is not given, the default is '../build'
relative to the athena/ source directory.
[bash][atspot01]:atlassoftwaredocs >

Setup and build

To build externals and Athena (without building any RPM packages) on an “lxplus-like machine” you can go to a fresh directory and do the following:

setupATLAS
lsetup git
git clone https://gitlab.cern.ch/atlas/athena.git
asetup none,gcc112,cmakesetup
./athena/Projects/Athena/build_externals.sh -c
./athena/Projects/Athena/build.sh -acmi

Alert At the time of writing the compiler we use in production is GCC 11.2. You have to be explicit with setting it up, gcc11 would set up the latest GCC 11.X version, which at the time of writing is GCC 11.3. Which is not fully compatible with all of our code at the moment.

Alert For older gcc62-based releases, you should instead do: asetup none,gcc62,cmakesetup --gccversion=6.2.0binutils

Alert In some cases, you may need to provide a specific cmake version: asetup none,gcc83,cmakesetup --cmakeversion=3.14.7

You should refer to the list of command line options for the meaning of the recommended options given above. These turn off the build of RPM packages, and make sure that the build would stop right away on errors. (As opposed to how we perform “nightly builds”, which continue on errors as long as they can.)

A full Athena build needs access to CVMFS, AFS and EOS at the time of writing.

  • CVMFS is a very strong requirement. It’s not impossible to build Athena without it, but it certainly requires expert level knowledge about our build system.
  • AFS is used during the build to pick up authentication files that are only visible to members of the ATLAS experiment. If these files are not readable during the build, the build results will not contain these authentication files. (For many software development purposes this is okay. Most simulation/reconstruction jobs do not need such direct access to ATLAS databases.)
  • EOS is (currently) used as the source of Geant4 data files. Which, at the time of writing, are needed already during the build. If you don’t have access to EOS, your best bet is to set:
export G4PATH=/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-centos7-gcc11-opt/Geant4

You can find the appropriate path for the nightly/branch that you want to build locally, like:

[bash][lxplus722]:~ > asetup Athena,main,latest
Using Athena/24.0.6 [cmake] with platform x86_64-centos7-gcc11-opt
        at /cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-centos7-gcc11-opt/2023-07-04T2101
Unchanged: COOL_ORA_ENABLE_ADAPTIVE_OPT=Y
[bash][lxplus722]:~ > echo $G4PATH
/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-centos7-gcc11-opt/Geant4
[bash][lxplus722]:~ >

On appropriate systems you may need to explicitly mount EOS, like:

export EOS_MGM_URL=root://eosatlas.cern.ch
/usr/bin/eos -b fuse mount eos

Depending on your EOS installation, after kinit and mounting EOS, you may need to also do:

eosfusebind -g

Using Ninja for the build

CMake and the build scripts will make use of GNU Make for the build by default. In order to use some other build system generator, you can pass extra arguments to CMake through the build scripts.

The most useful alternative build system to use is Ninja, a lightweight piece of software, which is much more efficient at dealing with the hundreds of thousands of files and tens of thousands of build targets that we use while building Athena.

To use it, first make sure that the ninja executable is available in your runtime environment. In an “lxplus-like environment” you should do this with:

export PATH=/cvmfs/sft.cern.ch/lcg/contrib/ninja/1.10.0/Linux-x86_64:$PATH

With it set up, you can tell the Athena (or AthenaExternals) build to use Ninja with:

-x "-G Ninja"

If you use the full build for “iterative development”, you should definitely make use of Ninja. And once the first build is done, perform the additional builds like:

./athena/Projects/Athena/build.sh -ami -x "-G Ninja"

Running Athena

In order to use the above, in a fresh terminal you would do:

setupATLAS
asetup Athena,<VERSION> --releasepath=<YOURBUILDAREA> --siteroot=/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-centos7-gcc11-opt

If needed, you can help Athena find additional files that may be missing from your local build (as discussed above) like:

export DATAPATH=$DATAPATH:/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-centos7-gcc11-opt/atlas/offline/ReleaseData/v20

Running unit tests

To run the unit tests, use the following environment setup:

setupATLAS
asetup AthenaExternals,<VERSION> --releasepath=<YOURBUILDAREA> --siteroot=/cvmfs/atlas-nightlies.cern.ch/repo/sw/main_Athena_x86_64-centos7-gcc11-opt
cd <YOURBUILDAREA>/build/Athena/
ctest

Building the externals alone

If you want to test a new version of an external (like Gaudi, Geant4, Acts, etc.), it is often enough to only build a select number of packages on top of an existing nightly, instead of embarking on a full project build. In such cases it is generally enough to only build the external in question, and not the full AthenaExternals project.

The following will specifically talk about building an alternative version of Gaudi, but the same techniques can be used for all other externals as well.

Note that to build externals without using atlasexternals is also possible, but is not in the scope of this page. ATLAS people can for instance refer to this twiki for how to develop Gaudi code like that.

No build recipe modification

In case the new/alternative version of the external does not need modified build instructions, you can make use of the build_externals.sh script directly. Like:

./athena/Projects/Athena/build_externals.sh -b $PWD/gaudi-build -x -DATLAS_GAUDI_TAG="vXXrY.ZZZ" -k Package_Gaudi

You need to check exactly which CMake cache variable(s) you should override in the appropriate package in atlasexternals/External to build the version of the external that you want.

With build recipe modifications

In case you need to modify the build recipe of the external in question, please refer to the instructions given in the atlasexternals repository. But in general, you will be able to build your (modified) package like:

asetup none,gcc112,cmakesetup
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -S ./atlasexternals/Projects/AthenaExternals/ -B ./gaudi-build
cmake --build ./gaudi-build --target Package_Gaudi
DESTDIR=./gaudi-install cmake --install ./gaudi-build

Once you’re happy with your modifications to atlasexternals, you can test your feature branch’s build using the build scripts of the athena repository with:

export AtlasExternals_URL=https://gitlab.cern.ch/<username>/atlasexternals.git
export AtlasExternals_REF=origin/<branch_name>
./athena/Projects/Athena/build_externals.sh -b $PWD/gaudi-build -k Package_Gaudi

Building packages on top of the modified external(s)

To now build a select number of packages on top of a full nightly build, but using your locally installed external(s), you can do the following:

  • Prepare a package selection file as described here;
  • Set up your build like:
setupATLAS
asetup Athena,main,latest
mkdir build
cd build/
export CMAKE_PREFIX_PATH=<EXTERNAL_INSTALL_DIR>:$CMAKE_PREFIX_PATH
cmake -DATLAS_PACKAGE_FILTER_FILE=../package_filters.txt ../athena/Projects/WorkDir/

Note that the directory that you need to point to with CMAKE_PREFIX_PATH depends on whether you used build_externals.sh, or you built the external yourself by hand.

  • When using build_externals.sh, you need to do the following wrt. to the “build directory” specified for the script:
export CMAKE_PREFIX_PATH=$PWD/gaudi-build/install/AthenaExternals/<VERSION>/InstallArea/<ARCHITECTURE>:$CMAKE_PREFIX_PATH
  • When building the external “by hand” according to the above instructions, the path would have to be set like:
export CMAKE_PREFIX_PATH=$PWD/gaudi-install/AthenaExternals/<VERSION>/InstallArea/<ARCHITECTURE>:$CMAKE_PREFIX_PATH

Building Athena alone

Once the externals are built correctly, or in case you just want to use the externals from an existing nightly / release, you still have some options for how to perform a full Athena build.

Using the build script

To build Athena on top of the externals from the nightly, you can use the same recipe as shown earlier, as long as you are in the correct environment. Like:

setupATLAS
asetup AthenaExternals,main,latest
./athena/Projects/Athena/build.sh -acm -x "-G Ninja"

Note that the installation step was skipped in this case, since most of the time you don’t actually need it when building Athena. Also note that for incremental builds you should later on just execute:

./athena/Projects/Athena/build.sh -am

I.e. you shouldn’t re-run the CMake configuration on every build attempt. Not unless your code modifications explicitly require it.

Performing the build by hand

An even better setup can be to just build Athena “completely by hand”. For interactive code development this results in a much better user experience.

In this setup you would set up your build like:

setupATLAS
asetup AthenaExternals,main,latest
mkdir build
cd build/
cmake -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE ../athena/Projects/Athena/
cmake --build .

Note the -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE argument. If you use it, code editors like [VS Code]((/guides/vscode) will be able to find all headers when looking at C++ sources.

Building Athena by hand, on top of local externals

If you want to build Athena “by hand” (directly executing the build commands as shown above), and want to do it on top of a full local build of the externals performed by build_externals.sh, then the most convenient setup to use is:

setupATLAS
asetup none,gcc112,cmakesetup
source ./athena/Projects/Athena/build_env.sh -b <EXTERNAL_INSTALL_DIR>
mkdir build
cd build/
cmake -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE ../athena/Projects/Athena/
cmake --build .