Output files

Specifying output files

Output files generated by jobs can be automatically copied to cloud storage. Specify the output filenames when submitting the job, for example:

prominence create --output testfile1.out --output testfile2.out alahiff/tester

The output files are assumed to be in the job’s scratch directory (i.e. the directory specified by HOME, TMP, and TEMP). If the name of the output file cannot be immediately determined, Unix style path name pattern expansion can be used, for example out/*/testfile.out. A current limitation is that this must correspond to a single file only.

Once the job has finished running you can easily obtain the locations of the files as temporary URLs using the describe command, which returns the full JSON representation of the job:

prominence describe --completed 481

output

[
  {
    "id": 481,
    "status": "completed",
    "image": "alahiff/tester",
    "cpus": 1,
    "memory": 1,
    "nodes": 1,
    "disk": 10,
    "runtime": 720,
    "outputFiles": [
      {
        "name": "testfile1.out",
        "url": "https://s3.uk/swift/v1/prominence-jobs/366075e5-35aa-4d80-8d70-15560c4f5ec9/testfile1.out?temp_url_sig=b31edb25c14b43e6e8a54a506bdf06f84861838b&temp_url_expires=1535203453"
      },
      {
        "name": "testfile2.out",
        "url": "https://s3.uk/swift/v1/prominence-jobs/366075e5-35aa-4d80-8d70-15560c4f5ec9/testfile2.out?temp_url_sig=6e9c78982057476ea1320c14e941daedd10cd317&temp_url_expires=1535203453"
      }
    ],
    "events": {
      "creationTime": "2018-08-15T13:23:42Z",    
      "executionStart": "2018-08-15T13:35:49Z",
      "completionTime": "2018-08-15T13:36:20Z"
    }
  }
]

The temporary URLs expire after 10 days, but the data itself is retained.

Note: Even if a task fails, i.e. the exit code of the process is non-zero, stage-out of output files (or directories) will still be attempted.

Specifying output directories

The option --outputdir can be used to specify the name of an output directory. In this case, a tarball of the directory is created and uploaded to object storage.

Downloading output files or directories

The download command enables all output files and/or directories from a specific completed job to be automatically downloaded to the current directory. For example:

prominence download 193

output

Downloading file "frame_001.png"
[==================================================]

In order to download all output files associated with a group of jobs, a constraint can be specified as an alternative to a job id. For example:

prominence download --constraint name=run5

will download the output files from all completed jobs which have a label name=run5. Unless for --force option is specified, output files will not be downloaded if there is an existing file with the same name.

For output directories tarballs (.tgz) are downloaded.