Use of the Python client makes it straightforward to execute workflows more complex than DAGs or simple parameter sweeps, for example situations where the results of one job (or group of jobs) need to affect the input parameters of the next job(s).
A basic requirement is to be able to pass input parameters or input files to jobs. This can be done programmatically in a variety of ways.
For applications for which input parameters can be provided on the command line, the appropriate values just need to be specified when constructing the task, e.g.
task.command = '/usr/local/blender/blender -b classroom/classroom.blend -o frame_### -f %d' % frame
Here the variable
frame can be adjusted as necessary.
If a small input file needs to be different for each job, the appropriate input files can easily be constructed and passed to the jobs using the Python client. As an example, suppose we need to use a small input file
sample.in with content as follows:
variable nsteps equal 200 # simulation steps
and we want to adjust the value
200 for different jobs. This can done using the
InputFile class, e.g.
contents = 'variable nsteps equal %d # simulation steps' % nsteps
In this case we need to upload the input file to object storage and configure the job to automatically retrieve it.
artifact = Artifact('largefile.h5') # Specify the object name
artifact.upload('/path/to/largefile.h5') # Specify the path to the file and upload it
job.artifacts.append(artifact) # Add the artifact to the job
Another basic requirement is to be able to retrieve output from jobs. Again, this can be done in several ways.
In some cases there may be information we want to extract from the standard output (or error) from a job. This can be obtained easily using the
stderr()) methods of the
Note that if the main application itself doesn’t write the required information into standard output, one option is to add another task to the job which reads an output file from the main application and writes the required information into standard output. An advantage of this is that large output files won’t need to be moved to the machine executing the Python client.
Files generated by jobs which are specified as output files (i.e. uploaded to object storage) can be downloaded as files or loaded into memory. For example, assuming
job here is referring to a completed job:
# Load the output file output1.txt into the variable output1
output1 = job.get_output_file('output1.txt')
# Save the output file output2.txt to a file