Migrating code with Merges and Cherry-Picks
Merge operations in git are powerful as they take into account the full history of both the source and target branches of the merge in order to automate (as much as is possible) the process of combining the independent sets of changes in both branches. However, merge operations act on the repository as a whole and our repository holds our entire codebase - this implies the merging of every package. Branches which remain very similar to each other, e.g. 21.0 (used for Tier 0 offline builds) and 21.1 (used for Point 1 online builds) can be merged into each other but this remains an operation for the Release Coordinators as it affects all packages.
The other main git mechanism for distributing changes, a Cherry-Pick, isolates the diff applied by a single commit (in our case, a single Merge Request) and then tries to apply the same diff to another branch. It can only succeed automatically if the lines which are being modified by the Merge Request were initially in the same state in both branches.
If two branches have diverged from each other sufficiently such that they can never practically be merged together, then Cherry-Picks remain the only option. However if there are then a large number of individual Merge Requests which require migrating, Cherry-Picking them one-by-one can also be impractical.
Migrating code with Pseudo-merges
git-package-pseudomerge.py is a script which provides an easier solution to this problem. It guides the user through the process of updating a select list of packages from one branch into another. It was written with the goal of simplifying the migration of changes from release 21 branches to main in 2018-19.
First asetup a recent release (e.g. asetup Athena,main,latest) and change directory to your git checkout. Note that the procedure will work with both full and sparse clones of athena. However, a full clone is recommended (a sparse clone will not be very sparse any more by the end of the procedure).
Step 1: Perform the merge operation
You should specify the source and target branches, which package(s) to migrate, and that you wish to perform --stage 1 of the operation. Here we demonstrate using two packages, TrigSteering and TriggerMenu:
asetup Athena,main,latest
# cd to your existing athena directory, or create a new one with
git clone https://:@gitlab.cern.ch:8443/${USER}/athena.git
cd athena
git remote add upstream https://:@gitlab.cern.ch:8443/atlas/athena.git
# Run stage one
git-package-pseudomerge.py \
--packages Trigger/TrigSteer/TrigSteering Trigger/TriggerCommon/TriggerMenu/ \
--source upstream/21.0 \
--target upstream/main \
--stage 1
A merge operation will be performed but not committed, all packages except for your specified ones will be reset on the target branch. At this point git status will be run and will display a list of conflicted files in your package(s) which could not be automatically merged, you need to resolve all conflicts.
Fix merge conflicts and compile
There are many good guides on the internet to resolving git merge conflicts but in essence you are looking in all conflicted files for markers which look like >>>>>>>, ======= and <<<<<<<. These markers denote the code as it is in the target branch and the source branch.
In the conflicting blocks, the same lines of code will have been edited independently in both the source and target branches hence requiring manual conflict resolution. Remove the conflict markers and leave the code in the correct state then run git add on the file to resolve the conflicted status.
Another frequently occurring conflict is the case of obsolete files having been deleted in the target branch:
deleted by us: Trigger/TriggerCommon/TriggerMenu/cmt/requirements
In this case the conflict can be resolved via git rm Trigger/TriggerCommon/TriggerMenu/cmt/requirements.
Once all conflicts are resolved, you should compile the package(s) against a recent nightly and check that the code works as expected.
Do not commit your changes yet.
Step 2: Isolate the changes and apply them to the target branch
When happy with the changes, the script should be run again but this time with --stage 2
# All conflicts are fixed, run stage 2 to commit them (to a new branch)
git-package-pseudomerge.py \
--packages Trigger/TrigSteer/TrigSteering Trigger/TriggerCommon/TriggerMenu/ \
--source upstream/21.0 \
--target upstream/main \
--stage 2
This will collate all of the changes and apply them to a fresh branch (which does not have the merge operation in its history - this is very important, the branch from stage one must never be merged into upstream).
This new branch created in --stage 2 can be merged into the official repository. Open a merge request on this branch to complete the procedure.
Any problems? Start again!
As this is procedure is running a lot of commands on a large repository, things might go wrong. Run the command with --reset instead of --stage to start again.
# Something went wrong, reset!
git-package-pseudomerge.py \
--packages Trigger/TrigSteer/TrigSteering Trigger/TriggerCommon/TriggerMenu/ \
--source upstream/21.0 \
--target upstream/main \
--reset