In the previous step, we saw how to find and open and view a log file in a web browser. But what if we wanted to download it? We can do this using Rucio tools.
After one of your jobs has completed we will now find and download the log file.
When using Rucio, it is almost always better to use it in a separate terminal to where you are running your code or submitting grid jobs. This will minimize the potential for conflicts between different python versions.
Setup the Rucio tools if you haven’t done so already.
lsetup rucio
Go back to the BigPanDA web page and find the
page with the jediTaskID that we used previously. Search for the Output
entry in the Containers table, and note the log file container name, e.g.,
user.aparker.pruntest.log. Back in your terminal session, try to find this
log file in the grid:
$ rucio list-dids user.aparker:*pruntest*log*
+--------------------------------------------------+--------------+
| SCOPE:NAME | [DID TYPE] |
|--------------------------------------------------+--------------|
| user.aparker:user.aparker.pruntest.log | CONTAINER |
| user.aparker:user.aparker.pruntest.log.340520924 | DATASET |
+--------------------------------------------------+--------------+
We now have two options:
- Download the container, and all log files within it (e.g. if the task contained many subjobs)
- Download just the dataset specific to the single set of jobs
rucio download user.aparker:user.aparker.pruntest.log.340520924
After it finishes downloading, navigate into the downloaded directory and extract the files from the tarball:
cd user.aparker.pruntest.log.340520924/
tar -xvf user.aparker.pruntest.log.23186476.000001.log.tgz
tgzextension) is a set of files packaged together and compressed usinggzip. This is an efficient way to transfer large numbers of small files.
This will give you access to the log file (as well as much more related information) from your job. This can be useful for debugging.
There will be a lot of information in here but when you have extracted the
logs, the file you are probably looking for is payload.stdout