Component Accumulator

Last update: 27 Nov 2020 [History] [Edit]

Purpose of the Component Accumulator

The ComponentAccumulator is a container for storing configuration of components. It may contain configuration of multiple components that form a consistent set (i.e. an algorithm & tools/services it needs for execution). The job configuration is built by merging several ComponentAccumulators.

...
topCA.merge(alg1CA)
topCA.merge(alg2CA)
...

Deduplication

During the merging it may occur that ComponentAccumulators (here topCA and alg1CA) contain the configuration of the same component. The ComponentAccumulator has functionality to unify their settings. So you don’t need to worry about adding a service/tool twice. On the contrary, ComponentAccumulator instances are supposed to be as much as possible self-contained, so it is recommended to add services/tools needed for an algorithm to work to the ComponentAccumulator and not to rely on the user to add what is necessary.

The settings unification process is called deduplication and is applied to every component in the merged ComponentAccumulators. It works as follows:

  1. Components that have the same type but different name are left intact (i.e. both instances are kept) as this is regular case.
  2. Components that have the same type, name and all properties are silently ignored.
  3. Components that have the same type, the same name and differently set properties are subject to the unification process. For all differently set properties the unification is attempted. It relies on the semantics that can be defined for each configurable parameter separately (see more). By default, though, any differences are considered a mistake in the configuration and results in configuration failure.

Writing Configuration Methods

Methods that configure pieces of the job instantiate a ComponentAccumulator and add to it services, algorithms, etc. as needed. They can call other configuration methods (to obtain the configuration of components they depend on) and merge the result with their own ComponentAccumulator. As parameter they take a configuration flags container and potentially other args and kwargs (positional and keyword arguments). A typical implementation looks like this:

from AthenaConfiguration.ComponentAccumulator import ComponentAccumulator
from AthenaConfiguration.ComponentFactory import CompFactory

def myAlgoCfg(inputFlags):
    acc=ComponentAccumulator()


    #Call the config method of a service that we need:
    from ARequiredSvcPack.ARequiredSvcPackConfig import superServiceCfg

    #We get an accumulator containing possibly other components that our SuperService
    #depends upon and the service itself
    svcAcc=superServiceCfg(inputFlags))

    #SuperService is the primary component that SuperServiceConfig configures
    #Get it, so we can attach it to the ServiceHandle of an algorithm
    mySvc=svcAcc.getPrimary()

    #Merge its accumulator with our accumulator to absorb all required dependencies
    acc.merge(svcAcc)

    # instantiate algorithm configuration object setting its properties
    MyAlgo=CompFactory.MyAlgoMyAlgo("MyAlgo",
                                    isData= not inputFlags.Global.isMC,
                                    svcHandle=mySvc)

    acc.addEventAlgo(myAlgo)
    #Return our accumulator (containing SuperService and its dependencies)
    return acc

ComponentAccumulators with private tools

Private AlgTools are special because they can’t exist without a parent. There is no meaningful way for accumulating them elsewhere. However, configuration methods configuring a private AlgTool with all its dependencies yet without a parent is still a valid use-case. To allow this, the ComponentAccumulator class has methods setPrivateTools and popPrivateTools. A function configuring an AlgTool returns an instance of ComponentAccumulator that has the tool attached via setPrivateTool and contains all the auxiliary components (services, conditions Algorithm etc.) that the tool needs to work. The caller then obtains the private tool via the popPrivateTools method and assigns it to the PrivateToolHandle of the parent and merges the returned ComponentAccumulator with its own ComponentAccumulator. This works for a single AlgTool as well as for lists of AlgTools that are typically assigned to a PrivateToolHandleArray. Merging a ComponentAccumulator that has still private tools attached (e.g. popPrivateTools was never called) will raise an exception complaining about a dangling private tool.

The ComponentAccumulator provides a handy shortcut method popToolsAndMerge that does aforementioned two operations at once. The example below illustrates how this works:

def privToolCfg(flags):
    acc = ComponentAccumulator()
    ...
    # merge/add dependencies
    acc.addService(CompFactory.SomeService(...))

    # it is recommended not to give a custom name to the private tool
    acc.setPrivateTools(CompFactory.ToolA( settingA = True ))
    return acc

def algCfg(flags):
    acc=ComponentAccumulator()
    tool=acc.popToolsAndMerge(privToolCfg(flags))
    # or longer alternative
    toolAcc=privToolCfg(flags)
    tool=toolAcc.popPrivateTools()
    acc.merge(toolAcc)

    alg=CompFactory.Alg("MyAlg",
                        myTool = tool,
                        settingX = 0.5)
    return acc

Designating the primary components

When the ComponentAccumulator is a result of merging of several smaller components it may be useful to designate a component that is a primary concern for a configuration function. This way client code does not need to discover the component by name.

It may also be that the primary component may change depending on the flags. Also in this situation it is convenient to shield the client code by specifying the primary component. Such example is shown below:

def toolCfg(flags):
    acc=ComponentAccumulator()
    # this adds some public tools
    acc.merge(otherTooolsCfg(flags))
    # not a primary component
    acc.addPublicTool( CompFactory.ToolX("X", setting=..))

    # configure different tools depending on the flag value
    # here we will designate the primary component
    if flags.addA:
        acc.addPublicTool( CompFactory.ToolA("ToolA", settingX= ...), primary=True)
    else:
        acc.addPublicTool( CompFactory.ToolA("ToolB", settingY= ...), primary=True)
    return acc

def consumerCfg(flags):
    acc=ComponentAccumulator()

    toolAcc=coolCfg(flags)
    #instead of code like this:
    # toolAcc.getPublicTool( "ToolA" if flags.flagA else "ToolB" ) we can do
    tool=toolAcc.getPrimary() # no need to agree on the name of the tool configured by toolCfg

    alg=CompFactor.MyAlg("MyAlg", toolX=tool )
    ...
    return acc

ComponentAccumulator API

The ComponentAccumulator has the following methods to add components to it:

  • merge(other, sequenceName=None)
    Merge in another instance of ComponentAccumulator. Deduplication is applied. All algorithms form the other top sequence are added to the destination sequence if the second argument is provided. Else the sequence structure is merged.
  • addSequence(sequence, parentSequenceName=None)
    Add a sequence, by default to the top-sequence of the accumulator. If second argument is provided the sequence is added as a subsequence of the sequence with that name. Handy methods to create various types of sequences (parallel/serial with AND/OR logic) are defined in CFElements.
  • addEventAlgo(algo,sequenceName=None,primary=False)
    Add one event-processing algorithm, by default to the top-sequence of the accumulator. If sequenceName argument is provided algorithm is added to this sequence.
  • addCondAlgo(algo,primary=False)
    Add one conditions-processing algorithm. Subject to deduplication.
  • addService(newSvc,primary=False,create=False)
    Add one service. Subject to deduplication. If create is set the service is added to the set of services forcibly created by Athena early in the job even without any client requiring it.
  • addPublicTool(tool,primary=False)
    Add one public tool. Subject to deduplication. Note: Public tools are deprecated for run 3. This feature will be removed.
  • setPrivateTools(tool or list of tools)
    Temporarily attach private AlgTool (or list of private AlgTools) to the accumulator. They need to be removed before merging.

For the explanation of the primary option see above.

Exceptions with ConfigurationError, DeduplicationFailure or plain TypeError are raised in case of misuse of these methods.

The ComponentAccumulator can be queried with these methods:

  • getEventAlgo(name)
    Get an event-processing algorithm by name.
  • getEventAlgos(seqName=None)
    Get all event algorithms (if sequence name is provided all algorithms in this and nested sequences).
  • getCondAlgo(name)
    Get a conditions processing algorithm by name.
  • getService(name)
    Get a service by name.
  • getPublicTool(name)
    Get a public tool by name Note: Public tools are deprecated for run 3. This feature will be removed.
  • getSequence(SequenceName=None)
    Returns a sequence (by searching the tree of sequences). By default returns the top sequence of the accumulator.
  • popPrivateTools()
    Returns the AlgTool or list of AlgTools previously attached to the accumulator.
  • getPrimary()
    Returns the component that is designated to be the primary one (see above for explanation).

Additional methods are available for the use in top level scripts for running the configuration contained in the ComponentAccumulator.

  • run(maxEvents=None,OutputLevel=INFO)
    That starts the athena execution.
  • store(outfile):
    Saves the configuration in the python pickle format to a file (The file needs to be open with open and closed after invocation of store).

Content of the configuration can be printed with: printConfig(withDetails=False, summariseProps=False, onlyComponents = [], printDefaults=False, printComponentsOnly=False) Various flags define level of details that will be emitted from this function. The meaning should be obvious.

Running configuration stored in ComponentAccumulator / self testing

It is advised that in the file defining the function that generates configuration also the self-test is defined. If the configuration contains algorithms a short test athena job can be setup there. Configurations without the algorithms can still be tested in this way to some extent.

# assume this is the content of MyAlgConfig.py
def myAlgCfg(flags):
    acc = ComponentAccumulator()
    ...
    return acc

if __name__ == "__main__":
    from AthenaCommon.Configurable import Configurable
    Configurable.configurableRun3Behavior = 1

    # import the flags and set them
    from AthenaConfiguration.AllConfigFlags import ConfigFlags
    ConfigFlags.Output.ESDFileName = "myESD.pool.root"
    ConfigFlags.Exec.MaxEvents=3
    ...
    # use one of the predefined files
    from AthenaConfiguration.TestDefaults import defaultTestFiles
    ConfigFlags.Input.Files = defaultTestFiles.RAW

    ConfigFlags.lock()

    # create basic infrastructure
    from AthenaConfiguration.MainServicesConfig import MainServicesCfg
    acc = MainServicesCfg(ConfigFlags)

    # add the algorithm to the configuration
    acc.merge(myAlgCfg(ConfigFlags))

    # debug printout
    acc.printConfig(withDetails = True, summariseProps = True)

    # run the job
    status = acc.run()

    # report the execution status (0 ok, else error)
    import sys
    sys.exit(not sc.isSuccess())

See the TileRawChannelMakerConfig example.

This script it then runable with the command:

python -m PackageName.MyAlgConfig

If the test does not take long time (it should not) it can be added as a package unit test in CMakeLists.txt

atlas_add_test( MyAlgConfig_test
                 SCRIPT python -m PackageName.MyAlgConfig
                 PROPERTIES TIMEOUT 300
                 POST_EXEC_SCRIPT nopost.sh)

then this test is always run as part of gitlab CI and can/should be

In an identical way the main driver scripts can be constructed. Useful functionality, to interpret command line options as flags ConfigFlags.fillFromArgs() is used (see more). For examples see: Run3DQTestingDriver.py, runHLT_standalone_newJO.py.