Callgrind

Last update: 24 May 2024 [History] [Edit]

Introduction

Callgrind is a tool that uses the runtime code instrumentation framework of Valgrind for call-graph generation Valgrind is a kind of emulator or virtual machine. It uses JIT (just-in-time) compilation techniques to translate x86 instructions to a simpler form called code on which various tools can be executed. The code processed by the tools is then translated back to the x86 instructions and executed on the host CPU. This way even shared libraries and dynamically loaded plugins can be analyzed but this kind of approach results in a huge slowdown (about 50 times for the callgrind tool) of analyzed applications and big memory consumption.

Simple use case

  • Prepare your development area as usual. You need to run Athena in debug mode only if you want to get a detailed line-by-line profile. Call-graphs can be produced in optimized mode as well.
  • Run Athena with Valgrind:
    valgrind --tool=callgrind --trace-children=yes --enable-debuginfod=no $(which athena.py) your_Job.py`
    
  • the --tool option determines which tool should be executed, callgrind in our case
  • the --trace-children option tells Valgrind to analyze child processes of the main application, otherwise you’ll get only profiles of bash sessions which I think is not what you want wink
  • --enable-debuginfod=no avoid a service providing debug information over an HTTP API. This might be needed or not depending on your system/setup.

Profiling selected algorithms

Profiling an entire Athena job has not only the disadvantage of being very slow but also the resulting profiles can be huge (easily more than 100MB for a few events). This can make it difficult to analyze the results using KCacheGrind. Moreover, the developer might only be interested in his/her algorithm. This is where the ValgrindAuditor (part of Control/Valkyrie) can help out. Once configured with an algorithm name to profile it will turn the callgrind instrumentation on before the algorithm’s execute method and turn it off again afterwards. Once the instrumentation is off the remaining valgrind overhead should only be about a factor of 4 which makes it much easier to run an entire Athena job in a reasonable amount of time.

To enable the ValgrindAuditor add the following lines to your component accumulator job:

flags.PerfMon.Valgrind.ProfiledAlgs=["EMBremCollectionBuilder"] #EMBremCollectionBuilder is an example. Replace as appropriate
from Valkyrie.ValkyrieConfig.ValgrindServiceCfg
acc.merge(ValgrindServiceCfg(flags))

Set flags.PerfMon.Valgrind.ProfiledAlgs to the name of the algorithm you want to profile (you can of course add multiple algorithms). Sometimes you might want to skip a few events before collecting profiling data, e.g. to exclude first-event initializations, dynamic library symbol lookups, etc. This can be done by setting IgnoreFirstNEvents. For a complete documentation of all the ValgrindSvc properties see the [Valkyrie doxygen page] (https://atlas-sw-doxygen.web.cern.ch/atlas-sw-doxygen/atlas_22. 0.X-DOX/docs/html/da/dcc/Valkyrie_page.html).

Before you run your job in Valgrind you might simply want to run Athena with your modified job options. If everything works fine (and you haven’t made a mistake in the algorithm name) you should see the following output

ValgrindAuditor     VERBOSE Starting callgrind: EMBremCollectionBuilder [event 1]
ValgrindAuditor     VERBOSE Stopping callgrind: EMBremCollectionBuilder [event 1]

This tells you that ValgrindAuditor found your algorithm and is enabling/disabling the callgrind instrumentation before/after execution (which won’t have any effect since we are not running in Valgrind yet). Once we have established that the auditor is configured correctly you can run your job in Valgrind:

valgrind --tool=callgrind --trace-children=yes --collect-jumps=yes --instr-atstart=no --enable-debuginfod=no $(which athena.py) --imf your_Job.py`

An example using Reco_tf and some more callgrind options:

InputRDOFile="/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/CampaignInputs/mc20/RDO/mc20_13TeV.410470.PhPy8EG_A14_ttbar_hdamp258p75_nonallhad.recon.AOD.e6337_s3681_r13145/100events.RDO.pool.root"
valgrind --tool=callgrind --trace-children=yes --collect-jumps=yes --instr-atstart=no --cacheuse=yes --cache-sim=yes --branch-sim=yes --enable-debuginfod=no \
$(which Reco_tf.py) \
  --inputRDOFile $InputRDOFile \
  --outputAODFile myAOD.pool.root \
  --preInclude 'egammaConfig.egammaOnlyFromRawFlags.egammaOnlyFromRaw' \
  --preExec 'ConfigFlags.PerfMon.Valgrind.ProfiledAlgs=["EMBremCollectionBuilder"]' \
  --postInclude 'Valkyrie.ValkyrieConfig.ValgrindServiceCfg' \
  --autoConfiguration 'everything' \
  --maxEvents '20' \
  --fileValidation FALSE \
  --perfmon 'none' 

The most important option is --instr-atstart=no. This turns the instrumentation off at the beginning so that it can be turned on by ValgrindAuditor on the first event of the algorithm (s) being profiled.

See the Valgrind manual for other callgrind options. After the job is done you will find several callgrind.out files from which the largest one is again the one you are interested in.

Profiling selected pieces of

The above only works on Algorithms. To profile one method, you need to do a bit more:

  • In the CMakeLists.txt file add: find_package( valgrind ).
  • In the file to be profiled, add #include "valgrind/callgrind.h"
  • At the start of the piece of code to be profiled, add CALLGRIND_START_INSTRUMENTATION; and at the end add CALLGRIND_STOP_INSTRUMENTATION;.

Now run valgrind e.g.

valgrind --tool=callgrind --trace-children=yes --instr-atstart=no $(which athena.py) your_Job.py 

There’s more in the relevant section of the valgrind manual.

KCacheGrind

Data produced by callgrind can be loaded into KCacheGrind tool for browsing the performance results. The actual profile of Athena run is stored in the biggest file produced by callgrind. On lxplus KCacheGrind is already installed so the user doesn’t need to do anything special: