The production Athena jobs are steering using so called Job Transforms.
Examples of those are Reco_tf.py
, Sim_tf.py
and more. They can usually
be recognised by ending with _tf.py
.
Each transform can consist on multiple substeps, also depending on the configuration.
Some arguments are “substep-aware”, which means that one can set the name of the step as a prefix,
e.g. --MyOpt 'Overlay:True'
will only set MyOpt
for the overlay step.
There are a few helpers available:
all
: applies a specific argument to all substeps (--preInclude 'all:Campaigns.MC20e'
)default
: applies a specific argument unless overridden explicitly by another argument using a substep name (--postInclude 'default:PyJobTransforms.UseFrontier.py'
)first
: only apply this argument to the first substepThese options are meant to be combined, e.g.
--postInclude "all:PyJobTransforms.UseFrontier" "RAWtoESD:MyModifter
in this case in all steps Frontier DB will be used, and in addition in RAWtoESD
a custom file will be included to modify the setup.
Note that there is no =
after --postInclude
. This is to allow --postInclude
to consume all arguments (this is standard python options parsing behaviour).
There are many ways on how job transforms can be steered. A --help
argument
can be used to get a list of all available properties (e.g. Reco_tf.py --help
).
In production most of those properties are steered by AMI tags.
Any AMI tag can also be evaluated from the command line e.g. using Reco_tf.py --AMIConfig q442
.
Note that usually inputs and outputs need to be specified explicitly, but in case of this
q-tag they are also present in the definition.
While higher-level job configuration can be steered using different arguments there is also a possibility to affect it lower-level. The following arguments are used for that
preExec
: Can call arbitrary python on flags
(only this is exposed).preInclude
: Calls a function that accepts configuration flags as the only argument. Passed as an “import string” of the form <package>.<module>.<function>
(or <package>.<function>
if aliased in __init__.py
).postExec
: Can call arbitrary python but only flags
and the top-level CA called cfg
are exposed.postInclude
: Calls a function that accepts configuration flags and the main CA instance. If a function with only one mandatory argument (the flags) is passed, it is assumed to be a configuration fragment and is merged with the top-level CA.Any of the described arguments can be applied to all trans
The order of execution can be transform dependent but it is usually like:
preInclude
s.preExec
s.ComponentAccumulator
.postInclude
s.postExec
s.