HLT Control Flow

Last update: 30 Sep 2024 [History] [Edit]

Introduction

The execution of AthenaMT jobs is ruled by the Gaudi scheduler, which executes the algorithms configured for the job in an order respecting inter-algorithm data and control dependencies, defined respectively as Control Flow (CF) and Data Flow (DF). By setting the algorithm state in the Gaudi state machine, the Control Flow allows its execution via the Control-Ready state, while the Data Flow sets the Data-Ready flag when its input data are available. All Athena components (AthAlgorithm, AthSequencer) can return a status flag, indicating a passed or rejected status. The CF can use these flags to apply logical conditions to the execution, that can then change from event to event. The logical unit is represented by the AthSequencer, which groups children algorithms or sequencers. The status of the children are propagated to the sequencer, which combines them with AND/OR logic. Moreover the sequencer rules the children execution order, which can be either concurrent or sequential.

The HLT processing is designed to minimize the number of executions, following two basic principles:

  • regional reconstruction: limiting the reconstruction within a region (through the EventView mechanism);
  • early-rejection: avoiding the execution of algorithms when the required trigger conditions, expressed in the HypoAlgorithms, are not satisfied (through the step-processing).

To achieve these requirements, Control Flow rules are applied by organizing algorithms in CF trees of sequencers.

The Control Flow tree

The root sequencer

To show the full CF tree, can set: flags.Scheduler.ShowControlFlow = True The main root sequencer of the job is called AthMasterSeq that contains the full tree. The HLT execution is implemented in the HLTTop main branch in a sequential OR of these sequencers:

  • HLTPreSeq: containing the CostMonitoring sequence supervisor
  • HLTBeginSeq: containing the L1 seeding and decoders, and the L1 simulation for MC jobs
  • HLTAllSteps: executing the HLT steps in sequential AND, so that when a step is not flagged as ready/passed, the execution stops. The latter represents the main step decision node. For each HLT step, there are two sequencers: StepX_reco and StepX_Filter, where X is the step number (1-N).

The step-reco-sequence (StepX_reco)

It implements the step-wise processing of the HLT, in which a decision is taken after a step. This element reports the step pass/reject decision to the upward main step decision node: a positive result enables the execution of the next step while a negative one terminates the event execution. It’s implemented with a concurrent-OR sequencer whose children are all the signature-steps designed to run at that decision step. This is an example of the Step1_FastCalo_electron step connected to Step1_reco:

Step1_reco [Seq]  [Concurrent]  [OR] 
    Step1_FastCalo_electron [Seq]  [Sequential]  [Prompt] 
        FStep1_FastCalo_electron [Alg]  [n= 0]
        Step1_FastCalo_electron_reco [Seq]  [Concurrent]  [OR] 
            fastCaloSequenceElectron [Seq]  [Sequential]  [Prompt] 
                IM_EMCalo [Alg]  [n= 0]
                ROBPrefetchingAlg_Calo_EMCaloRoIs [Alg]  [n= 0]
                EMCaloInViews [Seq]  [Concurrent]  [OR] 
                    fastCaloVDV [Alg]  [n= 0]
                    FastCaloL2EgammaAlg [Alg]  [n= 0]
        ElectronFastCaloHypo [Alg]  [n= 0]
        ComboHypo_FastCalo_electron [Alg]  [n= 0]

Each signature step (Step1_FastCalo_electron) is a sequential-AND containing:

  • the Filter Algorithm (FStep1_FastCalo_electron): this is a AthFilterAlgorithm/RoRSeqFilter, with two main roles: it is configured to stop/allow the execution of the step, i.e. working as a gate (early-rejection); and it distributes the input TriggerDecisions which defines the regions to be reconstructed, filtering only those from the active chains (regional-reconstruction);
  • the concurrent-OR sequencer (Step1_FastCalo_electron_reco): contains the reconstruction sequence(s), which can be executed in parallel, in order to maximize concurrency, mainly in combined chains. Within this sequence, the execution is governed by the Data Flow. Each sequencer contains few basic components:
    • InputMakerBase::InputMakerAlg algorithm, which prepares the input collections for the initial reconstruction, starting from the filter trigger decisions; if the same reconstruction is included in multiple steps, it merges the trigger decisions from different filters to optimize the reconstruction; if the reconstruction happens in View, it becomes InputMakerBase::EventViewCreatorAlg and launches the regional reconstruction generating the view sequencers (EMCaloInViews);
  • the Hypothesis Algorithm (ElectronFastCaloHypo)
  • the Combo Hypothesis Algorithm (ComboHypo_FastCalo_electron)

This is represented in the figure below:

There is one filter per step, so the name of the filter is inherited from that of the step. There might be more InputMakers per step, for example when the step is merged for a combined chain. There might be more Hypoalg per step, for example when the step is merged for a combined chain. There is one ComboHypo per step, so the name of the ComboHypo is inherited from that of the step.

The step-filter-sequence

The step-reco-sequence described above collects the reconstruction from all the chains in a single decision point, so that is requires a proper synchronization of the execution. For example, if none of the Filters in the step pass the decision with success, the full step reconstruction must be abandoned. For this reason before the step-reco-sequence a step-filter-sequence runs, which prepares the reconstruction ensuring that all the regions of reconstructions (with their collections) can be correctly created. It’s implemented as a concurrent-OR of all the Filters used in that step:

Step1_filter [Seq]  [Concurrent]  [OR] 
    FStep1_FastCalo_electron [Alg]  [n= 0]
    .....

The Control flow generation

The control flow tree is automatically generated in HLTCFConfig.py, when all the Chain objects are passed from the signature specific code. Each chain contains the execution steps, as already described, represented with ChainStep objects, as defined in MenuComponents.py. The CF generation is implemented in decisionTreeFromChains, and it’s made of two main processes: the data flow creation and the proper control flow.

   ( finalDecisions, CFseq_list) = createDataFlow(flags, chains)        
    addChainsToDataFlow(flags, CFseq_list, allDicts) 
    # now connect all algorithms and creates the CAs  
    cfAcc = createControlFlow(flags, HLTNode, CFseq_list)

createDataFlow

def createDataFlow(flags, chains):

This is the first part of the CF Tree generation, implementing these targets:

  • all the steps from all the chains in the input chain list are aligned, and their configuration (in CA) is generated;
  • the full data-flow is defined
  • the filter data-handlers are defined

Since a step can be used in many chains, the step Filter has input decisions from all chains requiring it. All filter inputs and outputs are then collected in this phase. The sequences are generated in CA chainStep.createSequences(), only once. The data-handle connections within the step are created

  • the final decisions of the previous step(s) are set as inputs of the filter;
  • the output of the filter are set as input of the InputMakers;
  • the output of the HypoAlg are set as the input of the ComboHypo;
  • the output of the Combohypo are set as the final decisions of the step;

Each chain implements its own list of ChainSteps, which contain both the reconstruction algorithms, the HypoAlg and the HypoTools proper of the chain. The list of chains, each having a list of steps, is transformed into a list of CFGroups, each containing a list of chains: this allows to group the reconstruction common to multiple chains and perform the sequence creation. This function returns the list of CFGroups and the list of final trigger decisions that will make the HLT final result.

addChainsToDataFlow()

This function simply adds the chain-ID information to the ComboHypo and creates the HypoTools.

createControlFlow

def createControlFlow(flags, HLTNode, CFseqList)

In this second part the CF tree is built, based on the list of CFGroups. The sequencers for the various step nodes are added in a top-bottom manner.

CF components

In HLTCFComponents.py helper classes are defined which allow to build the CF tree:

  • CFSequence: grouping the ChainStep and the Filter
  • CFGroup: grouping chains using the same ChainStep