Inspecting Configuration Differences

Last update: 09 Nov 2021 [History] [Edit]

Running the Job Verbosely

There are several options how to dump the configuration and make the job run more verbosely.

Set the whole configuration to DEBUG

To set the whole configuration to debug level do the following at the top:

from AthenaCommon.Constants import DEBUG
from AthenaCommon.Logging import log
log.setLevel(DEBUG)

Dump configuration flags

To dump configuration flags run ConfigFlags.dump(). You should get something like this:

Flag Name                                : Value
Beam.BunchSpacing                        : 25
Beam.Energy                              : [function]
Beam.NumberOfCollisions                  : [function]
Beam.Type                                : [function]
Beam.estimatedLuminosity                 : [function]
Common.MsgSourceLength                   : 50
Common.ProductionStep                    : <ProductionStep.Default: 'Default'>
Common.Project                           : 'Athena'
Common.bunchCrossingSource               : [function]
Common.doExpressProcessing               : False
Common.isOnline                          : False
Common.useOnlineLumi                     : [function]
Concurrency.NumConcurrentEvents          : [function]
Concurrency.NumProcs                     : 0
Concurrency.NumThreads                   : 0
Digitization.AddCaloDigi                 : False
Digitization.AddCaloDigiThinned          : False
Digitization.DataRunNumber               : -1
Digitization.DigiSteeringConf            : 'StandardPileUpToolsAlg'
Digitization.DoCaloNoise                 : True
Digitization.DoDigiTruth                 : False
Digitization.DoInnerDetectorNoise        : True
Digitization.DoPixel3DRadiationDamage    : False
Digitization.DoPixelPlanarRadiationDamage : False
Digitization.DoXingByXingPileUp          : False
Digitization.ExtraInputs                 : [('xAOD::EventInfo', 'EventInfo')]
Digitization.HighGainEMECIW              : True
Digitization.HighGainFCal                : False
Digitization.InputBeamSigmaZ             : 50
Digitization.JobNumber                   : 1
Digitization.PU.BeamGasInputCols         : []
Digitization.PU.BeamHaloInputCols        : []
Digitization.PU.BeamIntensityPattern     : [function]
Digitization.PU.BunchSpacing             : [function]
Digitization.PU.BunchStructureConfig     : ''
Digitization.PU.CavernIgnoresBeamInt     : False
Digitization.PU.CavernInputCols          : []
Digitization.PU.CustomProfile            : ''
Digitization.PU.FinalBunchCrossing       : 6
Digitization.PU.FixedT0BunchCrossing     : 0
Digitization.PU.ForceSequentialEventNumbers : [function]
Digitization.PU.HighPtMinBiasInputColOffset : 0
Digitization.PU.HighPtMinBiasInputCols   : []
Digitization.PU.InitialBunchCrossing     : -32
Digitization.PU.LowPtMinBiasInputCols    : []
Digitization.PU.NumberOfBeamGas          : 0.0
Digitization.PU.NumberOfBeamHalo         : 0.0
Digitization.PU.NumberOfCavern           : 0.0
Digitization.PU.NumberOfCollisions       : 0.0
Digitization.PU.NumberOfHighPtMinBias    : 0.0
Digitization.PU.NumberOfLowPtMinBias     : 0.0
Digitization.PU.ProfileConfig            : ''
Digitization.PU.SignalPatternForSteppingCache : []
Digitization.PileUp                      : False
Digitization.RandomSeedOffset            : 0
Digitization.TRTRangeCut                 : [function]
Digitization.TruthOutput                 : False
Digitization.UseUpdatedTGCConditions     : False
Exec.DebugStage                          : ''
Exec.MaxEvents                           : -1
Exec.OutputLevel                         : 3
Exec.SkipEvents                          : 0
ExecutorSplitting.Step                   : -1
ExecutorSplitting.TotalEvents            : -1
ExecutorSplitting.TotalSteps             : 0
IOVDb.DBConnection                       : 'sqlite://;schema=mycool.db;dbname=OFLP200'
IOVDb.DatabaseInstance                   : 'OFLP200'
IOVDb.GlobalTag                          : 'OFLCOND-MC16-SDR-14'
IOVDb.RunToTimestampDict                 : [function]
Input.Collections                        : ['McEventInfo', 'BCMHits', 'BLMHits', 'PixelHits', 'SCT_Hits', 'LArHitEMB', 'LArHitEMEC', 'LArHitFCAL', 'LArHitHEC', 'LArHitMiniFCAL', 'MBTSHits', 'TileHitVec', 'RPC_Hits', 'TGC_Hits', 'CSC_Hits', 'MDT_Hits', 'TruthEvent', 'EVNTtoHITS_timings', 'StreamHITS', 'TRTUncompressedHits', 'CaloEntryLayer', 'MuonEntryLayer', 'MuonExitLayer', 'LArCalibrationHitDeadMaterial']
Input.Files                              : ['/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/Tier0ChainTests/valid1.410000.PowhegPythiaEvtGen_P2012_ttbar_hdamp172p5_nonallhad.simul.HITS.e4993_s3091/HITS.10504490._000425.pool.root.1']
Input.Format                             : [function]
Input.LumiBlockNumber                    : [function]
Input.OverrideRunNumber                  : False
Input.ProcessingTags                     : [function]
Input.ProjectName                        : [function]
Input.RunAndLumiOverrideList             : []
Input.RunNumber                          : [function]
Input.SecondaryCollections               : [function]
Input.SecondaryFiles                     : []
Input.SpecialConfiguration               : [function]
Input.TimeStamp                          : [function]
Input.TypedCollections                   : [function]
Input.isMC                               : True
Output.AODFileName                       : ''
Output.ESDFileName                       : ''
Output.EVNTFileName                      : ''
Output.EVNT_TRFileName                   : ''
Output.HISTFileName                      : ''
Output.HITSFileName                      : ''
Output.RDOFileName                       : 'myRDO.pool.root'
Output.RDO_SGNLFileName                  : ''
Output.doWriteAOD                        : [function]
Output.doWriteBS                         : False
Output.doWriteESD                        : [function]
Output.doWriteRDO                        : True
Output.doWriteRDO_SGNL                   : [function]
Random.Engine                            : 'dSFMT'
Scheduler.CheckDependencies              : True
Scheduler.EnableVerboseViews             : True
Scheduler.ShowControlFlow                : True
Scheduler.ShowDataDeps                   : True
Scheduler.ShowDataFlow                   : True
TrackingGeometry.MagneticFileMode        : 6
TrackingGeometry.MaterialSource          : 'COOL'

Dump ComponentAccumulator instance

To dump a ComponentAccumulator instance run cfg.printConfig(withDetails=True, summariseProps=True). As you see this is quite verbose (feel free to play with above arguments to reduce the verbosity).

Py:ComponentAccumulator    INFO Event Inputs
Py:ComponentAccumulator    INFO Event Algorithm Sequences
Py:ComponentAccumulator    INFO Top sequence 0
Py:ComponentAccumulator    INFO \__ AthMasterSeq (seq: SEQ AND)
Py:ComponentAccumulator    INFO    \__ AthAlgEvtSeq (seq: SEQ AND)
Py:ComponentAccumulator    INFO       \__ AthBeginSeq (seq: SEQ AND)
Py:ComponentAccumulator    INFO          \__ AthIncFirerAlg/BeginIncFiringAlg (alg)
Py:ComponentAccumulator    INFO              * FireSerial: False set
Py:ComponentAccumulator    INFO              * Incidents: ['BeginEvent'] set
Py:ComponentAccumulator    INFO          \__ IncidentProcAlg/IncidentProcAlg1 (alg)
Py:ComponentAccumulator    INFO       \__ AthAllAlgSeq (seq: PAR AND)
Py:ComponentAccumulator    INFO          \__ AthCondSeq (seq: PAR AND)
Py:ComponentAccumulator    INFO          \__ AthAlgSeq (seq: PAR AND)
Py:ComponentAccumulator    INFO             \__ xAODMaker::EventInfoCnvAlg/EventInfoCnvAlg (alg)
Py:ComponentAccumulator    INFO                 * AODKey: McEventInfo set
Py:ComponentAccumulator    INFO                 * xAODKey: EventInfo set
Py:ComponentAccumulator    INFO             \__ Simulation::BeamSpotReweightingAlg/BeamSpotReweightingAlg (alg)
Py:ComponentAccumulator    INFO                 * BeamSpotKey: BeamSpotData set
Py:ComponentAccumulator    INFO                 * EventInfo: EventInfo set
Py:ComponentAccumulator    INFO                 * InputMcEventCollection: TruthEvent set
Py:ComponentAccumulator    INFO                 * Input_beam_sigma_z: 50.0 set
Py:ComponentAccumulator    INFO                 * OutputLevel: 2 set
Py:ComponentAccumulator    INFO                 * beamSpotWeight: EventInfo.beamSpotWeightForTutorial set
Py:ComponentAccumulator    INFO       \__ AthEndSeq (seq: SEQ AND)
Py:ComponentAccumulator    INFO          \__ AthIncFirerAlg/EndIncFiringAlg (alg)
Py:ComponentAccumulator    INFO              * FireSerial: False set
Py:ComponentAccumulator    INFO              * Incidents: ['EndEvent'] set
Py:ComponentAccumulator    INFO          \__ IncidentProcAlg/IncidentProcAlg2 (alg)
Py:ComponentAccumulator    INFO    \__ AthOutSeq (seq: PAR AND)
Py:ComponentAccumulator    INFO       \__ EventInfoTagBuilder/EventInfoTagBuilder (alg)
Py:ComponentAccumulator    INFO           * AttributeList: SimpleTag set
Py:ComponentAccumulator    INFO           * EventInfoKey: EventInfo set
Py:ComponentAccumulator    INFO           * Tool: EventInfoAttListTool/EventInfoAttListTool
Py:ComponentAccumulator    INFO       \__ AthenaOutputStream/OutputStreamRDO (alg)
Py:ComponentAccumulator    INFO           * ExtraOutputs: [('DataHeader', 'StoreGateSvc+StreamRDO')] set
Py:ComponentAccumulator    INFO           * HelperTools: PrivateToolHandleArray of size 4
Py:ComponentAccumulator    INFO              * MakeEventStreamInfo/StreamRDO_MakeEventStreamInfo
Py:ComponentAccumulator    INFO                 * EventInfoKey: EventInfo set
Py:ComponentAccumulator    INFO                 * Key: StreamRDO set
Py:ComponentAccumulator    INFO              * xAODMaker::EventFormatStreamHelperTool/StreamRDO_MakeEventFormat
Py:ComponentAccumulator    INFO                 * Key: EventFormatStreamRDO set
Py:ComponentAccumulator    INFO              * xAODMaker::FileMetaDataCreatorTool/FileMetaDataCreatorTool
Py:ComponentAccumulator    INFO                 * OutputKey: FileMetaData set
Py:ComponentAccumulator    INFO                 * StreamName: RDO set
Py:ComponentAccumulator    INFO              * Athena::ThinningCacheTool/ThinningCacheTool_StreamRDO
Py:ComponentAccumulator    INFO                 * StreamName: RDO set
Py:ComponentAccumulator    INFO           * ItemList: ['xAOD::EventInfo#EventInfo', 'xAOD::EventAuxInfo#EventInfoAux.', 'xAOD::EventInfo#EventInfo', 'xAOD::EventAuxInfo#EventInfoAux.'] set
Py:ComponentAccumulator    INFO           * MetadataItemList: ['EventStreamInfo#StreamRDO', 'IOVMetaDataContainer#*', 'xAOD::EventFormat#EventFormatStreamRDO', 'xAOD::FileMetaData#FileMetaData', 'xAOD::FileMetaDataAuxInfo#FileMetaDataAux.'] set
Py:ComponentAccumulator    INFO           * MetadataStore: StoreGateSvc/MetaDataStore set
Py:ComponentAccumulator    INFO           * OutputFile: myRDO.pool.root set
Py:ComponentAccumulator    INFO           * WritingTool: AthenaOutputStreamTool/StreamRDOTool
Py:ComponentAccumulator    INFO              * AttributeListKey: SimpleTag set
Py:ComponentAccumulator    INFO    \__ AthIncFirerAlg/EndAlgorithmsFiringAlg (alg)
Py:ComponentAccumulator    INFO        * FireSerial: False set
Py:ComponentAccumulator    INFO        * Incidents: ['EndAlgorithms'] set
Py:ComponentAccumulator    INFO    \__ IncidentProcAlg/IncidentProcAlg3 (alg)
Py:ComponentAccumulator    INFO Condition Algorithms
Py:ComponentAccumulator    INFO  \__ CondInputLoader (cond alg)
Py:ComponentAccumulator    INFO      * Load: [['AthenaAttributeList', '/Indet/Beampos']] set
Py:ComponentAccumulator    INFO  \__ BeamSpotCondAlg (cond alg)
Py:ComponentAccumulator    INFO Services
Py:ComponentAccumulator    INFO ['ClassIDSvc', 'StoreGateSvc', 'DetectorStore', 'HistoryStore', 'ConditionStore', 'CoreDumpSvc', 'MessageSvc', 'PoolSvc', 'AthenaPoolCnvSvc', 'EventPersistencySvc', 'MetaDataStore', 'AthenaPoolAddressProviderSvc', 'ProxyProviderSvc', 'EventSelector', 'IOVDbSvc', 'CondSvc', 'TagInfoMgr', 'InputMetaDataStore', 'MetaDataSvc']
Py:ComponentAccumulator    INFO Public Tools
Py:ComponentAccumulator    INFO [
Py:ComponentAccumulator    INFO   IOVDbMetaDataTool/IOVDbMetaDataTool,
Py:ComponentAccumulator    INFO ]
Py:ComponentAccumulator    INFO Private Tools
Py:ComponentAccumulator    INFO [
Py:ComponentAccumulator    INFO ]
Py:ComponentAccumulator    INFO theApp properties
Py:ComponentAccumulator    INFO   TopAlg : ['AthSequencer/AthMasterSeq']
Py:ComponentAccumulator    INFO   MessageSvcType : MessageSvc
Py:ComponentAccumulator    INFO   EventLoop : AthenaEventLoopMgr
Py:ComponentAccumulator    INFO   ExtSvcCreates : False
Py:ComponentAccumulator    INFO   JobOptionsSvcType : JobOptionsSvc
Py:ComponentAccumulator    INFO   JobOptionsType : NONE
Py:ComponentAccumulator    INFO   JobOptionsPostAction :
Py:ComponentAccumulator    INFO   JobOptionsPreAction :
Py:ComponentAccumulator    INFO   PrintAlgsSequence : True
Py:ComponentAccumulator    INFO   OutStreamType : AthenaOutputStream
Py:ComponentAccumulator    INFO   InitializationLoopCheck : False
Py:ComponentAccumulator    INFO   EvtMax : -1
Py:ComponentAccumulator    INFO   EvtSel : EventSelectorAthenaPool/EventSelector

Using ConfTool.py

The most common use case of confTool.py is inspecting the differences in configuration of two jobs. You already dumped the pickle file earlier in the tutorial using

with open("BeamSpot.pkl", "wb") as f:
    acc.store(f)

Copy BeamSpot.pkl to BeamSpotOld.pkl. Now change again the ConfigFlags.Digitization.InputBeamSigmaZ flag, this time to 60, and run the job again.

To compare the two configurations run

confTool.py --diff BeamSpotOld.pkl BeamSpot.pkl

The tool will report the differences in the configuration.

Run with arguments:
confTool.py --diff BeamSpotOld.pkl BeamSpot.pkl
Step 1: reference file #components: 48
Step 2: file to check  #components: 48
Legend:
Differences in components Settings in 1st file Settings in 2nd file
 Component BeamSpotReweightingAlg differ
     Input_beam_sigma_z =  50.0  vs  60.0   <<

For more details look at Tools and tips section of the guide.