This page contains information about the workflows and procedures a release coordinator is concerned with. It is assumed that you are familiar with the general workflow for ATLAS Offline software development as summarised in the Workflow Quick Reference. Having an overview of the tasks of the software review shifters as well as knowing basics of git is also helpful.
Before accepting new MRs, you should check the status of the latest nightly. The first priority of a release coordinator should be to make sure that the release is in good shape, and obviously existing problems should usually be fixed before adding more MRs (though as release coordinator you should always feel free to use your judgement).
The status of the nightly builds can be checked in two places:
The latter contains information on ART build jobs and other detailed information for the different build steps. Information on the nightly (and CI) build infrastructure are available on this twiki page.
For each nightly build check:
Any problem seen in the unit tests will very likely also affect the CI jobs. To help the review shifters please mark any relevant Jira tickets with the “CI” label. This will make the issue appear on the CI Status Board.
First check on Jenkins if the nightly finished or it is still ongoing. If it is still ongoing and it is already afternoon check that it is not stuck on something. Just by checking the console log file should give you a good idea if the nightly is progressing (select the nightly you are interested in and then click on the “Console” icon).
If the nightly finished and the status of the build is a Blue ball - the nightly should be available on CVMFS.
If the nightly finished and the Status is Red ball - the build failed for some reason. Now you need to identify the exact reason. Every nightly is executing the following steps so while investigating check the steps one by one and see where there is a problem:
The logfile is copied under: /cvmfs/atlas-nightlies.cern.ch/repo/sw/master/2018-07-15T2059/master__Athena__x86_64-slc6-gcc62-dbg__2018-07-15T2059__1531720274.ayum.log
All the information can be found inside the “Console” log file for the given nightly.
Note that issues with git and copying the RPMs to EOS are IT infrastructure related issues. If there were git problems you can restart the nightly. If there were RPM to EOS copy problems check with Atlas.Release to redo the copy.
In git, different releases are represented by different branches of the
atlas/athena git repository. One
subtle point is that what code we build from the branch is controlled
by the project specific scripts that live in the
of the branch (e.g., here
master branch). You are, however, the release coordinator
for all projects built from a branch.
Upon creation, merge requests will be tagged automatically by the CI system with a label indicating their target branch. In addition, the CI system also adds a label indicating the stage and result of the software review process. Changes approved by the software review shifters are labeled as review-approved. Therefore, you should be able to select all ready-to-be-accepted merge requests from the GitLab merge request overview page. Simply filter the list of merge requests by selecting review-approved and your release branch name from the dropdown menu for labels.
Full rebuilds need to be triggered after updates where an incremental rebuild may not catch all changes properly, such as a LCG version update. Such updates should be performed in late evenings, with advance notification emailed to ATLAS robot to provide CI system administrators enough time to schedule cleaning local disks of build machines (to ensure builds are from scratch).
The CI job state is depicted by a small icon on the right (e.g. green checkmark for passed, red cross for failed, blue circle for running). This status sometimes does not get updated correctly (e.g. a CI job is indicated as still running while it has already finished). Therefore, please do not rely on this icon. If in doubt, check the discussion tab of this MR for the latest CI summary comment from the ATLAS robot.
Before accepting any merge request, please have a quick look at the description and the discussion on GitLab merge request page. They may contain some more detailed information about the nature of this merge request and possibly also the relation with other merge requests (e.g. dependencies between MRs). You should also make sure that all discussions were resolved.
Once you are sure that you want to merge these changes into your release, you can accept the merge request by pushing the green “Merge” button.
Please do not use the command line merging procedure as described on GitLab.
Due to the fact that updates of the CI pipeline status has been found unreliable in the past, release coordinators are discouraged from using the “Merge When Pipeline Succeeds” feature.
If you want to undo the changes introduced by an (already accepted) merge request, you have to go the GitLab webpage for this merge request. There is an orange button labeled Revert. Clicking on this button opens a new dialog where you should choose the target branch to be the same as the target branch of the initial merge request. Please leave the checkbox “Open a new merge request” checked. Confirm the settings by clicking on the green Revert button. This will open a new merge request undoing the changes introduced by the faulty merge request and the usual CI jobs and the review process start automatically.
Once you are satisfied as the release coordinator that a nightly release is good enough to be deployed to the production CVMFS server then you need to do the following steps:
See here for instructions on how to setup tunnelling using browser plugins.
Log in with username “jobrest”. If you do not know the password, ask Alex Undrus.
Click “Build with parameters” on the left hand sidebar. Fill in the details on the screen where:
masterand so on
AnalysisBaseetc and then click “Build”
Repeat the above steps for any other platforms you may need (e.g. the
dbg builds typically
have slightly different timestamps).
You now have two choices: you can either open a ticket, or you can start the copy yourself. Both processes are explained here.
As with the copy RPM steps above, you need to log into Jenkins as “jobrest” and start the copy script with the following Jenkins link (tunnelling is necessary from outside CERN at present).
You need to wait for the RPM copy to be complete before starting the copy to cvmfs.
Click “Build with parameters” on the left hand sidebar. Fill in the details on the screen where:
numbered_releaseshould be the new release e.g.
cmtconfigis the build configuration e.g.
Then click “Build” to start the script. Check the console to make sure nothing has gone wrong.
Multiple copies (e.g. for opt and dbg) can be started in parallel.
Create a Jira ticket (Task) in ATLINFR
requesting that the appropriate nightly release should be promoted to
a full numbered release (the Component is
CVMFS Release Installation).
You need to give the name of the
git branch, the Project that was built and the Datestamp of the
Every project already knows what release candidate it is, which is stored in the
version.txtfile of the project’s directory (like this). You will update this number later on in this procedure.
It is impossible to install a build as a different numbered production release from the
version.txtcandidate number when the nightly was built. e.g., if the release candidate number is 21.0.21 it is not possible to install this as 21.0.22 or 18.104.22.168 or 21.0.21p1.
Create the tag for the release in the main repository. From the front
page of the main repository click
New tag. Enter the
Tag name, which should always be
release/A.B.X[.Y], i.e., the actual numbered release version; the
should be the nightly build tag that is being used, which has the
In case you have built multiple platforms for the same release use the nightly stamp of the primary platform (e.g.
gcc8-opt). In practice this should not matter as the nightly tags for different platforms should all point to the same git commit if started around the same time.
Then enter an informative message about the reason for deploying this particular release that will be useful to others. An example might be the following:
You should also generate the changelog using this script (this is likely to be moved soon, so please check these instructions agaim if it is no longer there). The script is documented, but you run it in the
athena directory and give it two parameters, the tag of the release you are building, and the tag of the nightly used to build it. For example:
git clone ssh://firstname.lastname@example.org:7999/atlas/athena.git cd athena prepare_release_notes.py release/22.0.32 nightly/master/2021-04-16T2101
You can also create the tag locally in any clone of the repository and push it to
atlas/athena. It’s not recommended unless you really know what you are doing.
Finally, now that a nightly has been promoted to a numbered release
the release candidate number needs to be updated to be the next number
in the release series for this branch + project. The easiest way to do this
is via the simple editor in GitLab itself. Navigate to the correct
branch and projects’
version.txt file, e.g.,:
Just then click on the
Edit button and switch to the new version
number. The commit message can be quite simple, describing the update
now that a previous nightly has been promoted.
All Projects (except
AthAnalysis) should be kept in sync. So if, for example, you update the version of
22.0.21you should also update
Only people with push rights to the branch can do this directly. You should have these rights as a release coordinator. (Lesser mortals need to make a merge request.)
To set up a production branch, first make the base release (as above) so there is a clear branching point, and so the release numbering makes sense. Next, make a branch from this release, naming it descriptively e.g. 22.0-mc20. Another acceptable option would be to name it after the base release e.g.
You should then setup permissions for this this branch allowing only the branch managers to push and merge (look for
Protected branches under
Repository Settings) and have a look at other protected branches for examples).
Next, it is very important to update the release candidate version (recall that in order to make make a release from a nightly, the nightly needs to know the release it will become) e.g. if you branched from
22.0.41, you should immediately change the
version.txt in all relevant projects (i.e. all, except
Finally, do not forget to update the README.md with details about the new branch (ideally in all branches, but certainly in the branch and master).
In productions branches we will need to make patch or point releases, e.g. 22.214.171.124
The procedure should be exactly the same as above, namely:
After the release has been built, please announce the release by sending out an email:
mailto: email@example.com, hn-atlas-recoIntegration@cern.ch, firstname.lastname@example.org, hn-atlas-releaseKitAnnounce@cern.ch subject: Release Athena,22.0.42 Dear all, This is to let you know that the release Athena,22.0.42 has been built from the nightly Athena,master,2021-08-23T2101 and is in the process of being distributed to CVMFS. The JIRA ticket can be found here: https://its.cern.ch/jira/browse/ATLINFR-4208 The release notes are available at: https://gitlab.cern.ch/atlas/athena/-/releases/release%252F22.0.42 Regards, myName
The CI compiles code-changes in the MRs incrementally. Occasionally, the build nodes end up in a bad state and need full rebuilds to recover. Updates to the externals frequently create this problem. This can be achieved by scheduling the
clean-build-dir jobs on
atlas-sit-ci.cern.ch after logging in via SSO. The ongoing CI jobs will be allowed to complete before the cleanup. All newcoming CI jobs will be waiting in the queue until the
clean-build-dir jobs are completed on all slave nodes.
Locate the job
clean-build-dir and select Build with Parameters. The default parameters should be fine: