Understanding Inputs & Outputs
A Gradient Workflow is composed of a series of steps. These steps specify how to orchestrate computational tasks. Each step can communicate with other steps through what are known as inputs and outputs.
There are three types of inputs and outputs. Understanding how these function will help you craft concise and elegant Workflows.
    Datasets
    Volumes
    Strings

Datasets

The dataset type leverages the Gradient platform native dataset primitive. Information stored within datasets is not limited to any single type of data. In fact, a generic dataset can include anything from pretrained models to generated images to configuration files. Inherent to datasets is the notion of versions. Workflows can consume and produce new dataset versions as well as tag new versions of existing datasets.
Note: datasets must be defined in advance of being referenced in a workflow. See Create Datasets for the Workflow for more information.
Scenario 1: Consuming a dataset that already exists within Gradient
1
inputs:
2
my-dataset:
3
type: dataset
4
with:
5
ref: my-dataset-id
Copied!
Scenario 2: Generating a new dataset version from a Workflow step
1
my-job:
3
with:
4
args:
5
- bash
6
- '-c'
7
- cp -R /my-trained-model /outputs/my-dataset
8
image: bash:5
9
outputs:
10
my-dataset:
11
type: dataset
12
with:
13
ref: my-dataset-id
Copied!
my-dataset-id can be the actual ID of the dataset, a 15 character string that looks like def123ghi456jkl (or appended with a version ID too), or a name for the dataset.

Volumes

Unlike, e.g., GitHub Actions, it is likely that multiple Gradient Steps/Actions will execute on multiple compute nodes. To facilitate the passing of data between these nodes, Gradient Actions expose the notion of volumes and volume passing.
Volumes enable actions such as the @git-checkout action. Volumes can be defined as input volumes or output volumes or both. When a volume is an output it is mounted in /outputs and is writeable. When a volume is an input it is mounted in /inputs and is read only.
Note: Volumes are limited to 5GB of data currently. If you need more space we recommend using Datasets.
Here is how you would define an output volume:
1
outputs:
2
my-volume:
3
type: volume
Copied!
In this example a volume is first created as an output and then used as an input in a subsequent job step:
1
defaults:
2
resources:
3
instance-type: P4000
4
5
jobs:
6
job1:
8
with:
9
args:
10
- bash
11
- -c
12
- echo hello > /outputs/my-volume/testfile1; echo "wrote testfile1 to volume"
13
image: bash
14
outputs:
15
my-volume:
16
type: volume
17
job2:
18
needs:
19
- job1
21
with:
22
args:
23
- bash
24
- -c
25
- cat /inputs/my-volume/testfile1
26
image: bash
27
inputs:
28
my-volume: job1.outputs.my-volume
Copied!
Volumes cannot currently be used as an output after the job they were created with. This limitation is planned to be removed in the future.

Strings

In some cases, you may need to pass a single value between Workflow steps. The string type makes this possible.
Scenario 1: Passing a string as a Workflow-level input
1
inputs:
2
my-string:
3
type: string
4
with:
5
value: "my string value"
6
7
jobs:
8
job-1:
9
resources:
10
instance-type: P4000
12
with:
13
args:
14
- bash
15
- -c
16
- cat /inputs/my-string
17
image: bash:5
18
inputs:
19
my-string: workflow.inputs.my-string
Copied!
Scenario 2: Passing a string between job steps
1
defaults:
2
resources:
3
instance-type: P4000
4
5
jobs:
6
job-1:
8
with:
9
args:
10
- bash
11
- -c
12
- echo "string output from job-1" > /outputs/my-string; echo job-1 finished
13
image: bash:5
14
outputs:
15
my-string:
16
type: string
17
job-2:
19
with:
20
args:
21
- bash
22
- -c
23
- cat /inputs/my-string
24
image: bash:5
25
needs:
26
- job-1
27
inputs:
28
my-string: job-1.outputs.my-string
Copied!
Scenario 3: Creating a model from a dataset and passing the model ID as a string to a Deployment step
NOTE: There is no native Gradient Actions for Model Deployments today. Instead, you can use the Gradient SDK to create and manage your inference endpoints.
To run this example you will need to a) create a dataset named test-model and upload valid TensorFlow model files to it; b) define a secret named MY_API_KEY with your gradient-cli api-key; c) substitute your clusterId in the deployment create step.
1
defaults:
2
resources:
3
instance-type: P4000
4
5
jobs:
6
UploadModel:
7
uses: create-[email protected]
8
with:
9
name: my-model
10
type: Tensorflow
11
inputs:
12
model:
13
type: dataset
14
with:
15
ref: test-model
16
outputs:
17
model-id:
18
type: string
19
DeployModel:
20
needs:
21
- UploadModel
22
inputs:
23
model-id: UploadModel.outputs.model-id
24
env:
25
PAPERSPACE_API_KEY: secret:MY_API_KEY
27
with:
28
command: bash
29
args:
30
- -c
31
- >-
32
gradient deployments create
33
--clusterId cl1234567
34
--deploymentType TFServing
35
--modelId $(cat inputs/model-id)
36
--name "Sample Deployment"
37
--machineType P4000
38
--imageUrl tensorflow/serving:latest-gpu
39
--instanceCount 1
40
image: paperspace/gradient-sdk
Copied!
Last modified 16d ago
Copy link