Download Log File From the Grid

Last update: 03 Apr 2023 [History] [Edit]

In the previous step, we saw how to find and open and view a log file in a web browser. But what if we wanted to download it? We can do this using Rucio tools.

After one of your jobs has completed we will now find and download the log file.

tip When using Rucio, it is almost always better to use it in a separate terminal to where you are running your code or submitting grid jobs. This will minimize the potential for conflicts between different python versions.

Setup the Rucio tools if you haven’t done so already.

lsetup rucio

Go back to the BigPanDA web page and find the page with the jediTaskID that we used previously. Search for the Output entry in the Containers table, and note the log file container name, e.g., user.aparker.pruntest.log. Back in your terminal session, try to find this log file in the grid:

$ rucio list-dids user.aparker:*pruntest*log*
| SCOPE:NAME                                       | [DID TYPE]   |
| user.aparker:user.aparker.pruntest.log           | CONTAINER    |
| user.aparker:user.aparker.pruntest.log.340520924 | DATASET      |

We now have two options:

  1. Download the container, and all log files within it (e.g. if the task contained many subjobs)
  2. Download just the dataset specific to the single set of jobs
rucio download user.aparker:user.aparker.pruntest.log.340520924

After it finishes downloading, navigate into the downloaded directory and extract the files from the tarball:

cd user.aparker.pruntest.log.340520924/
tar -xvf user.aparker.pruntest.log.23186476.000001.log.tgz

tip A tarball (a file with the tgz extension) is a set of files packaged together and compressed using gzip. This is an efficient way to transfer large numbers of small files.

This will give you access to the log file (as well as much more related information) from your job. This can be useful for debugging.

There will be a lot of information in here but when you have extracted the logs, the file you are probably looking for is payload.stdout