As well and Online and Offline data processing, Athena is frequently used in Calibration workflows that run at Tier0 in the so-called “Prompt Calibration Loop” on data fresh from the detector. Historically, this has often been done by patching code installed at the Tier0 directly, which then gets compiled and run on top of Athena releases taken from cvmfs.
Issues with local compilation based workflows
While this is no doubt a convenient working model which guarantees that code updates can be applied quickly, it very frequently leads to undesirable consequences, especially if a) code updates are not collected in an appropriate release branch promptly after being made, b) the base Athena release used at Tier0 is not updated frequently to track the latest developments, and c) appropriate testing is not done to see that the code runs and gives consistent output for the latest releases. This can lead to the following outcomes:
- The code may simply no longer run in current releases due to incompatibilities; in certain cases it may become mandatory to move to a more current release due to serious bugs or infrastructure updates and so in this case the workflow would be completely broken.
- The code has not been reviewed, meaning that there can be deviations from best practice, coding guidelines, or incompatibilities with ongoing developments which may not be discovered until very much later when they are much more difficult to solve.
- At the point when the code is re-integrated the changeset may be huge, leading to a very hard task for the person who opens the Merge Request, as well as for the shifters who need to review it.
- Updates may be in conflict with (or undo) code quality updates made in releases, discovered via static code checkers such as Coverity, or from searching for problematic patterns in the code base.
- The lack of consistent versioning may lead to problems in reproducibility, as it may not be possible to easily go back to a previous configuration that was run.
For these reasons, workflows making use of any local updates to Athena releases is strongly discouraged.
Calibration Workflow Guidelines
When developing or updating a calibration workflow, the following points should be observed:
- Athena forks associated with special group accounts (
attrtcal,atlidali, etc) should be used like a “normal” user fork, i.e. that it is used to collect developments in branches which are then frequently integrated back into an appropriate release branch (normally the Tier0 production branch) via Merge Request as soon as complete so that they can be integrated into the next release
- This implies that: In general, updates should be picked up by moving to a new release in the series, and not by relying on local compilations of patched code. In general operation of the workflow, no Athena code should be compiled locally. Local patches should only be used when it is operationally critical, and the necessary update is not yet available in a release (and should recompile only the absolute minimum possible number of packages). Code updates will be available in a new release at the very latest after a week, and typically this can be much shorter as if needed releases can be created from any given nightly build of Athena by contacting the Prompt Reconstruction Coordinators (PROC), so in the most urgent cases the new release can be provided the next day for critical issues.
- No filepaths in Athena code should refer to files from
/afs directories by default, especially those that may be restricted to users of specific calibration accounts. These may not be accessible to all users, and so this causes problems for testing. The default filepaths for any files required to run a workflow test should refer to files in CVFMS and should be readable to all ATLAS users.
- Frequently, calibration workflows will include production of histograms/plots, scripts for launching of jobs with various configuration sets,
.txt, .dat, or .json input files, etc. The Athena repository should be used only for the Athena code base. It should not contain:
- Standalone
ROOT code/macros
- Data files of any kind (
.root, various text-based formats, etc)
- Scripts that simply launch Athena jobs with a variety of options passed via CLI steering options
- “Backup” copies representing the previous versions of code
- These should either be stored in a separate
git project/repository (in the case of ROOT code/macros and job scripts), on CVFMS storage (in the case of data files), or not at all (code backups, which are obviated by version control anyway).
- Tests should be added to run in ART (and also be monitored regularly) which ensure that the necessary code continues to run, and produce consistent output in all relevant release branches.
If you feel that these guidelines will not work for your particular case, please contact the ATLAS software coordinators, who will help you to come up with an appropriate working model.