Jobs, tasks and workflows

A task is the smallest unit of compute in PROMINENCE. A task cannot be run on its own but is defined as part of a job. A workflow consists of one or more jobs with optionally dependencies between them.


A job in PROMINENCE consists of the following:

  • Name
  • Labels
  • Input files
  • Output files
  • Storage (e.g. WebDAV or OneData storage mounted in the container)
  • Required resources (e.g. CPU cores, memory, disk)
  • One or more task definitions
  • Policies (e.g. how many times should failing tasks should be retried or where to run the job)
  • Notifications (e.g. to specify email notification upon job completion)


Tasks execute sequentially within a job (except for sidecar tasks which run in parallel to other tasks), and consist of the following:

  • Container image
  • Container runtime (Singularity or udocker)
  • Command to run and optionally any arguments
  • Environment variables
  • Working directory


A workflow consists of:

  • Name
  • Labels
  • One or more job definitions
  • Any dependencies between jobs
  • Factories to generate multiple jobs based on job templates
  • Policies (e.g. how many times should failing jobs should be retried)
  • Notifications (e.g. to specify email notification upon workflow completion)

Jobs within a workflow can be executed sequentially, in parallel or combinations of both. Tasks within a job share the same scratch directory, whereas different jobs within a workflow do not.


An example workflow, including how it is made up of jobs and tasks, is shown below:

Tasks and jobs within a workflow