A Continuous Integration (CI) service was introduced to the ATLAS software development workflow as part of the git migration. This forces all changes on any development branch to be automatically built and tested as a first step of the code review. Changes in the code can then be immediately validated - rather than full testing needing to wait for code being included in a nightly build - which significantly reduces test feedback latency for developers.
For ATLAS offline software the CI service is provided by a hosted instance of the Jenkins platform. CI jobs on Jenkins are automatically triggered upon the submission of a new merge request and for any subsequent commits within the same request. The Jenkins/GitLab interaction is coordinated by a functional account (the ATLAS robot) which publishes job result information to the merge request discussion and sets the appropriate status labels:
Jenkins Service
If you are within the CERN network you can directly access the Jenkins instance for ATLAS here (if you are outside CERN, please follow the remote login instructions) and you should see the following page:
The job CI-MERGE_REQUEST is triggered from the initiation of (or changes to) a merge request. This is a multi-step job containing the following sequence of jobs:
On the left hand side of the page you can see the build history of CI-MERGE_REQUEST. Selecting a build job (e.g. 1548) that is in progress:
You can view all the jobs as part of CI-MERGE_REQUEST. In the example above four have already completed and the build job (CI-gcc-build) is underway. By clicking on the console output you can watch the job as it progresses:
Troubleshooting
For the majority of the merge requests you will see a build success message posted into the merge request discussion panel by the ATLAS robot and have limited interaction with Jenkins itself. If the code submitted as a merge request has been tested correctly by the developer it is reasonably expected that the CI build should in general succeed. Any failures observed therefore need to be understood and (if possible) addressed before a merge request is approved.
At this time the suite of tests run on the CI build is still under development and failures may indicate a problem with the build system and not with the underlying code. See the Examples section for failures of this type.
For example, you may see a failure of this kind reported by ATLAS robot in the merge request discussion:
As you can see above the build logs are posted to the NICOS webpage for further inspection. This is generally the better option to explore and share first due to the limited access to Jenkins outside of CERN. It should then be possible to identify the reasons for the build failure by inspecting the logs available at the advertised link. Once you have identified the error it is good practice to post the error snippet and the NICOS direct link into the merge request discussion.
If the problem is not obvious then this may be due to the build system itself and requires a further look through the job details on the Jenkins instance:
One issue may be due to an excessive delay waiting for the test to complete or the build job getting stuck for some reason. As a shifter you can restart the build jobs if necessary - please see section #4 of the git merge request shifter FAQ for further details.