Workflow Spec
This describes in more detail the main components of a Gradient Workflow, as seen in the YAML file.

Key Concepts

defaults

At the top of the YAML Workflow file, you can specify default parameters to be used throughout the entire Workflow. This includes environment variables and default machine instance configuration. Instances can also be specified per-job.

inputs

The inputs block allows you to specify named inputs (e.g., a versioned dataset) to be referenced and consumed by your jobs.
Note: you can also collect inputs in a separate YAML and reference this file as an inputPath when creating a Workflow run.
Workflow and job-level inputs can be of type: dataset (a persistent, versioned collection of data), string (e.g., a generated value or ID that may be output from another job) or volume (a temporary workspace mounted onto a job's container).
Note: datasets must be defined in advance of being referenced in a workflow. See Create Datasets for the Workflow for more information.

jobs

Jobs are also sometimes referred to as "steps" within the Gradient Workflow. A job is an individual task that executes code (such as a training a machine learning model) and can consume inputs and produce outputs.

Sample Workflow Spec

To run this Workflow, define datasets named test-one, test-two, and test-three as described in the Create Datasets for the Workflow documentation. Also, to make use of the secret named hello in the inputs section, define a secret as described here.
1
defaults:
2
# Default environment variables for all jobs. Can use any supported
3
# substitution syntax (named secrets, ephemeral secrets, etc.).
4
env:
5
# This environment variable uses a Gradient secret called "hello".
6
HELLO: secret:hello
7
# Default instance type for all jobs
8
resources:
9
instance-type: P4000
10
11
# Workflow takes two inputs, neither of which have defaults. This means that
12
# when the Workflow is run the corresponding input for these values are
13
# required, for example:
14
#
15
# {"inputs": {"data": {"id": "test-one"}, "echo": {"value": "hello world"}}}
16
#
17
inputs:
18
data:
19
type: dataset
20
with:
21
ref: test-one
22
echo:
23
type: string
24
with:
25
value: "hello world"
26
jobs:
27
job-1:
28
# These are inputs for the "job-1" job; they are "aliases" to the
29
# Workflow inputs.
30
#
31
# All inputs are placed in the "/inputs/<name>" path of the run
32
# containers. So for this job we would have the paths "/inputs/data"
33
# and "/inputs/echo".
34
inputs:
35
# The "/inputs/data" directory would contain the contents for the dataset
36
# version. ID here refers to the name of the dataset, not its dataset ID.
37
data: workflow.inputs.data
38
# The "/inputs/echo" file would contain the string of the Workflow input
39
# "echo".
40
echo: workflow.inputs.echo
41
# These are outputs for the "job-1" job.
42
#
43
# All outputs are read from the "/outputs/<name>" path.
44
outputs:
45
# A directory will automatically be created for output datasets and
46
# any content written to that directory will be committed to a newly
47
# created dataset version when the jobs completes.
48
data2:
49
type: dataset
50
with:
51
id: test-two
52
# The container is responsible creating the file "/outputs/<name>" with the
53
# content being a small-ish utf-8 encoded string.
54
echo2:
55
type: string
56
# Set job-specific environment variables
57
env:
58
TSTVAR: test
59
# Set action
61
# Set action arguments
62
with:
63
args:
64
- bash
65
- -c
66
- find /inputs/data > /outputs/data2/list.txt; echo ENV $HELLO $TSTVAR > /outputs/echo2; cat /inputs/echo; echo; cat /outputs/data2/list.txt /outputs/echo2
67
image: bash:5
68
job-2:
69
inputs:
70
# These inputs use job-1 outputs instead of Workflow inputs. You must
71
# specify job-1 in the needs section to reference them here.
72
data2: job-1.outputs.data2
73
echo2: job-1.outputs.echo2
74
outputs:
75
data3:
76
type: dataset
77
with:
78
ref: test-three
79
# List of job IDs that must complete before this job runs
80
needs:
81
- job-1
83
with:
84
args:
85
- bash
86
- -c
87
- wc -l /inputs/data2/list.txt > /outputs/data3/summary.txt; cat /outputs/data3/summary.txt /inputs/echo2
88
image: bash:5
Copied!
Last modified 16d ago