Git-ATLAS

Last update: 03 Sep 2018 [History] [Edit]

git-atlas is the go-to git extension to handle the ATLAS git repositories. It is meant to be a wrapper around some git commands but not to replace your usage of standard git commands.

It includes several commands: init-config, init-workdir, addpkg and rmpkg, and finally list-pkg.

Setup

To use the git-atlas script, you simply need to run

lsetup git

which will also provide you with a more recent version of git than the default for most distributions.

In case you want to try bleeding edge features, you’ll need to add the following to your *rc file.

source /afs/cern.ch/user/k/karolos/public/git-tools/setup.sh

Options

> git atlas
Usage: git atlas init-config [CERN login] [--apply]
   or: git atlas init-workdir [-p pkgList [--]] [-b branch] [-g group] baseRepo [localPath]
       pkgList can be a file containing package paths, or a list of packages
       To save disk space, baseRepo may be a local workdir of the same project
Options:
-b, --branch  	Branch you want to check out (for init-workdir)
-g, --group   	GitLab group to use instead of 'atlas'
-p            	Packages for sparse-checkout, provided via either
              	file, or list terminated by option, -- or repo source (baseRepo)

The following flavors are availble from within a clone/checkout:
   or: git atlas list-pkg [--all]
   or: git atlas addpkg|rmpkg PackageName

Global options:
-h, --help    	This help
-v, --verbose 	Verbose output (level 2)
-V, --version 	Display version
-q, --quiet   	Quiet output (verbosity level 0)
-a, --all     	Set 'all' flag for the commands who need it
-n, --dry-run 	Dry run: do not execute any command

init-config

This is the command that will set you up for using git within ATLAS. It allows setting up settings such as user.name and user.email (these are necessary to be able to commit to a repository) as well as some ATLAS-specific items such as athena: as a shortcut to https://:@gitlab.cern.ch:8443/atlas/athena.git.

To get started, simply run

git atlas init-config

The script will then lookup your email and name in the CERN LDAP database (credentials needed outside CERN/lxplus), assuming your CERN user name is the same as that of your system. If it’s not, you may add it as the first argument: git atlas init-config karolos. Below is the output you’d get by default.

# [git-atlas init-config] No CERN login provided: assuming karolos

# [git-atlas init-config] Fetching ATLAS configuration items

# [git-atlas init-config] The following commands will be executed when setting the --apply option:
git config --global atlas.user karolos
git config --global url."https://:@gitlab.cern.ch:8443/atlas/athena.git".insteadOf athena:
git config --global user.name "Karolos Potamianos"
git config --global user.email "karolos.potamianos@cern.ch"

Note that you may have less options in case you’ve already setup some of these options.

Now let’s call the script with the --apply option:

git atlas init-config karolos --apply

This will result in the relevant items being added to your git configuration file.

# [git-atlas init-config] Fetching ATLAS configuration items

# [git-atlas init-config] Applying configuration settings
+ git config --global atlas.user karolos
+ git config --global url.https://:@gitlab.cern.ch:8443/atlas/athena.git.insteadOf athena:
+ git config --global user.name 'Karolos Potamianos'
+ git config --global user.email karolos.potamianos@cern.ch

Note: if you see the following message in the output

# [git-atlas init-config] Your (global) git user email is not in the .cern.ch domain.
    Assuming you did that on purpose. But I'll use your CERN email when checking out an ATLAS repository

don’t worry. This means that you’ve already setup your name and email in git. We’ll leave it like that but we’ll set the committer email for your clone to your CERN email address.

Checking that everything is OK

After you’ve run with the --apply option, you can confirm that all is at it should be using the following command

git atlas init-config

When you’ve already executed the script (or did the adjustments manually), you should then see the following output:

# [git-atlas init-config] No CERN login provided: assuming karolos

# [git-atlas init-config] Fetching ATLAS configuration items

# [git-atlas init-config] You're all set: nothing to add to git config

init-workdir

The init-workdir command is used to setup your working copy of the ATLAS Offline Git repository. Your first workdir will be a clone from a remote repository, https://:@gitlab.cern.ch:8443/atlas/athena.git, and will contain a full copy of all the repository contents. Your subsequent working directories should start off your first workdir in order to save disk space, so as to point to it using symbolic links. This section describes how to achieve this.

To perform your first clone of athena, you can run

git atlas init-workdir https://:@gitlab.cern.ch:8443/atlas/athena.git atlasoff-base -p PixelCabling

OR, in case you are not on lxplus and have no Kerberos credentials

git atlas init-workdir https://:@gitlab.cern.ch:8443/atlas/athena.git atlasoff-base -p PixelCabling

whose full output at the time of writing is

# [git-atlas init-workdir] Creating new working directoy at atlasoff-base:
    * switching to branch master of https://:@gitlab.cern.ch:8443/atlas/athena.git
    * using your fork at https://:@gitlab.cern.ch:8443/karolos/athena.git as the repository to push to
    * you didn't request any package to be checked out
Initialized empty Git repository in /tmp/karolos/atlasoff-base/.git/
remote: Counting objects: 235721, done.
remote: Compressing objects: 100% (99737/99737), done.
remote: Total 235721 (delta 128385), reused 235561 (delta 128339)
Receiving objects: 100% (235721/235721), 161.19 MiB | 31.87 MiB/s, done.
Resolving deltas: 100% (128385/128385), done.
From https://gitlab.cern.ch:8443/atlas/athena
 * [new branch]      19.2       -> origin/19.2

[...]

 * [new tag]         release/21.0.8 -> release/21.0.8

# [git-atlas init-workdir] Adding some packages needed by CMake: AtlasCMake, Athena
    Note: this requirement will be removed in the future.
Branch master set up to track remote branch master from origin.
Already on 'master'

# [git-atlas manage-pkg] Called with action 'addpkg' and package name 'PixelCabling'
InnerDetector/InDetDetDescr/PixelCabling/ added to sparse checkout

Already on 'master'
Your branch is up-to-date with 'origin/master'.

This cloned a copy of https://:@gitlab.cern.ch:8443/atlas/athena.git (using your Kerberos credentials) into atlasoff-base as a sparse checkout of the master branch, only checking out the package PixelCabling (but you’ll still have a complet copy of the repository inside atlasoff-base/.git).

You can see this running

find atlasoff-base -name CMakeLists.txt

and seeing that you only have the file atlasoff-base/InnerDetector/InDetDetDescr/PixelCabling/CMakeLists.txt.

Working directory without specifying packages

In case you don’t know what package you want to checkout yet (or want to use list-pkg to get a full list of items), you can run

git atlas init-workdir https://:@gitlab.cern.ch:8443/atlas/athena.git atlasoff-base

whose output is as above, except the end

# [git-atlas init-workdir] You haven't specified a list of packages:
    * use git atlas addpkg from your checkout to add some

Initial checkout at /tmp/karolos/atlasoff-base
Use git atlas addpkg to add a package.

You then need to cd to atlasoff-base and run git atlas addpkg PackageName.

Using repository aliases

You can use a shortcut notation (made available to you by init-config) to clone a repository:

git atlas init-workdir athena: atlasoff-base master -p PixelCabling

You can create custom aliases using the following command:

git config --global url."FULL_URL".insteadOf SHORT_NAME:

and replacing FULL_URL and SHORT_NAME as appropriate. Note that the colon is compulsory. You can then run

git clone SHORT_NAME:

or

git atlas init-workdir SHORT_NAME: LOCAL_FOLDER

Adding more working copies

Now let’s make another workdir using another package:

git atlas init-workdir atlasoff-base atlasoff-beamspot -p InDetBeamSpotExample

This time, the command ran much faster and you got an output like

# [git-atlas init-workdir] Creating new working directoy at atlasoff-beamspot:
    * switching to branch master of https://:@gitlab.cern.ch:8443/atlas/athena.git
    * BUT using the .git local repository from atlasoff-base
    * using your fork at https://:@gitlab.cern.ch:8443/karolos/athena.git as the repository to push to
    * checking out the following packages: InDetBeamSpotExample
Already on 'master'
Your branch is up-to-date with 'origin/master'.

# [git-atlas init-workdir] Adding some packages needed by CMake: AtlasCMake, Athena
    Note: this requirement will be removed in the future.
Already on 'master'
Your branch is up-to-date with 'origin/master'.

# [git-atlas manage-pkg] Called with action 'addpkg' and package name 'InDetBeamSpotExample'
InnerDetector/InDetExample/InDetBeamSpotExample/ added to sparse checkout

Already on 'master'
Your branch is up-to-date with 'origin/master'.

Initial checkout at /tmp/karolos/atlasoff-beamspot
Use git atlas addpkg to add a package.

You can note that the script detected that atlasoff-base was already a checkout and told you it will be using .git from atlasoff-base, but that the remotes have been set to https://:@gitlab.cern.ch:8443/atlas/athena.git (pull and push) and https://:@gitlab.cern.ch:8443/atlas/athena.git (pull only).

Warning Do not remove the atlasoff-base folder as subsequent working directories point to it

Let’s create yet another working copy

git atlas init-workdir atlasoff-beamspot atlasoff-PixelGeoModel -p PixelGeoModel PixelCabling

Which yields

# [git-atlas init-workdir] Detected that atlasoff-beamspot is not a base checkout (i.e. .git is already symlinked)
      Using the source of the link (in case you delete the working copy): /tmp/karolos/atlasoff-base/

# [git-atlas init-workdir] Creating new working directoy at atlasoff-PixelGeoModel:
    * switching to branch master of https://:@gitlab.cern.ch:8443/atlas/athena.git
    * BUT using the .git local repository from /tmp/karolos/atlasoff-base/
    * using your fork at https://:@gitlab.cern.ch:8443/karolos/athena.git as the repository to push to
    * checking out the following packages: PixelGeoModel PixelCabling
Already on 'master'
Your branch is up-to-date with 'origin/master'.

# [git-atlas init-workdir] Adding some packages needed by CMake: AtlasCMake, Athena
    Note: this requirement will be removed in the future.
Already on 'master'
Your branch is up-to-date with 'origin/master'.

# [git-atlas manage-pkg] Called with action 'addpkg' and package name 'PixelGeoModel'
InnerDetector/InDetDetDescr/PixelGeoModel/ added to sparse checkout


# [git-atlas manage-pkg] Called with action 'addpkg' and package name 'PixelCabling'
InnerDetector/InDetDetDescr/PixelCabling/ added to sparse checkout

Already on 'master'
Your branch is up-to-date with 'origin/master'.

Initial checkout at /tmp/karolos/atlasoff-PixelGeoModel
Use git atlas addpkg to add a package.

The command understood that atlasoff-beamspot is not a base clone, thus it did use the its base atlasoff-base. We also provided two packages to add: PixelGeoModel and PixelCabling, which were both added.

Tip You are free to remove atlasoff-beamspot because it is not a base checkout, but you should still not remove atlasoff-base.

addpkg, rmpkg and list-pkg

Once you’ve set your workdir up using init-workdir, you can also edit the packages to be checked out using

git atlas addpkg PackageName

where /PackageName/ is going to be searched for in the list of available files, i.e. PixelCabiling will checkout InnerDetector/InDetDetDescr/PixelGeoModel/ but InnerDetector will checkout all packages under InnerDetector/.

To remove a package, use

git atlas rmpkg PixelCabling

To list the packages you have checked out, use

git atlas list-pkg

If you don’t remember a package name, you can use

git atlas list-pkg --all PackageName

where PackageName is the regular experssion that’s going to be provided to grep -i. For example,

> git atlas list-pkg Pixel
# [git-atlas list-pkg] Listing all known ATLAS packages (i.e. paths with CMakeLists.txt) matching 'Pixel'
InnerDetector/InDetCalibAlgs/PixelCalibAlgs
InnerDetector/InDetConditions/PixelConditionsData
InnerDetector/InDetConditions/PixelConditionsServices
InnerDetector/InDetConditions/PixelConditionsTools
InnerDetector/InDetConditions/PixelCoralClientUtils
InnerDetector/InDetDetDescr/PixelCabling
InnerDetector/InDetDetDescr/PixelGeoModel
InnerDetector/InDetDigitization/PixelDigitization
InnerDetector/InDetEventCnv/PixelRawDataByteStreamCnv
InnerDetector/InDetG4/PixelG4_SD
InnerDetector/InDetMonitoring/PixelMonitoring
InnerDetector/InDetRecTools/InDetTestPixelLayer
InnerDetector/InDetRecTools/PixelToTPIDTool
InnerDetector/InDetValidation/PixelRTT

You can then addpkg the package you want. Note that using regular experssion on addpkg was not added on purpose. We might reconsider this for the future. In the meanwhile, you can maybe checkout the parent folder in case you need a collection of packages.

Advanced usage

For more complex setups, you can provide a text file with the list of packages (absolute path form the root) and pass that as an argument to init-workdir instead of a package name.

> cat myPkgList.txt
InnerDetector/InDetDigitization/PixelDigitization
InnerDetector/InDetEventCnv/PixelRawDataByteStreamCnv
InnerDetector/InDetG4/PixelG4_SD
> git atlas init-workdir atlasoff-beamspot atlasoff-MyPkgList -p myPkgList.txt

# [git-atlas init-workdir] Detected that atlasoff-beamspot is not a base checkout (i.e. .git is already symlinked)
      Using the source of the link (in case you delete the working copy): /tmp/karolos/atlasoff-base/

# [git-atlas init-workdir] Creating new working directoy at atlasoff-MyPkgList:
    * switching to branch master of https://:@gitlab.cern.ch:8443/atlas/athena.git
    * BUT using the .git local repository from /tmp/karolos/atlasoff-base/
    * using your fork at https://:@gitlab.cern.ch:8443/karolos/athena.git as the repository to push to
    * cheking out the items listed in myPkgList.txt:
   InnerDetector/InDetDigitization/PixelDigitization
   InnerDetector/InDetEventCnv/PixelRawDataByteStreamCnv
   InnerDetector/InDetG4/PixelG4_SD
Already on 'master'
Your branch is up-to-date with 'origin/master'.

# [git-atlas init-workdir] Adding some packages needed by CMake: AtlasCMake, Athena
    Note: this requirement will be removed in the future.
Already on 'master'
Your branch is up-to-date with 'origin/master'.
Already on 'master'
Your branch is up-to-date with 'origin/master'.

Initial checkout at /tmp/karolos/atlasoff-MyPkgList
Use git atlas addpkg to add a package.
> find atlasoff-MyPkgList/ -name CMakeLists.txt
atlasoff-MyPkgList//InnerDetector/InDetDigitization/PixelDigitization/CMakeLists.txt
atlasoff-MyPkgList//InnerDetector/InDetEventCnv/PixelRawDataByteStreamCnv/CMakeLists.txt
atlasoff-MyPkgList//InnerDetector/InDetG4/PixelG4_SD/CMakeLists.txt

This mode also allows using regular expressions in the sparse-checkout file.