Getting Started with Workflows
The following sections cover creating and running Workflows, and authoring new Workflow YAML specs. If you have never run a Workflow before we recommend you step through the Workflows demo in the quick-start guide then return here for more details.

Create a Workflow

Workflows can be created via the GUI / Web UI, or command line (CLI).
Web UI
CLI
    1: Create a Gradient Project
    2: Navigate to its main page by clicking on its entry in the projects list
    3: Click the Workflows tab, which gives this screen
    4: Choose one of the illustrated boxes under Select a template (basic), or click import an existing gradient repository (advanced)
If you did Select a template:
    5: Choose your GitHub username from the dropdown list under Account and Organizations [1]
    6: Give your repository a name
    7: Click Create Workflow. This will create and run the Workflow in your new repository.
If you did import an existing gradient repository:
    5: Choose your GitHub username from the dropdown list under Account and Organizations in the Import existing repository window that pops up [1]
    6: Choose the repository to import
    7: This will take you to the Let's create a Workflow screen, where you can follow its instructions to create one, or run an existing Workflow.
See Run the Workflow below for more details on how Workflows run.
[1] If your username doesn't show up, try clicking the x in the circle on the right of the dropdown, or if it still isn't there, make sure you have the Gradient GitHub app installed from the Quick Start section, and configured so that it can see your repositories.
    1.
    Make sure you have the latest version of the Gradient CLI
    2.
    Create a Gradient Project and grab your project ID. You can create a project that integrates with a Github repo or a create a standalone project. Use a Github project if you already have code you are working with in Github.
    3.
    Create your first Workflow using the Gradient CLI
1
gradient workflows create --name <my-workflow-name> --projectId <project-id>
Copied!
The command will return an ID for the Workflow, for example, 7634c165-5034-4f49-95fa-005fc0e7970b

Create a Workflow Spec

To write your own Workflow, create a Workflow spec in YAML using a text editor. There is one in the Gradient Notebook interface, or you can use your own.
Note: even though YAML and JSON are closely related, Gradient Workflows need to be formatted as YAML and not JSON.
Below is an example of a valid workflow.yaml spec. It clones the repository from https://github.com/NVlabs/stylegan2, generates images from the repo script run_generator.py, and outputs the results to the Gradient-managed dataset demo-dataset.
You will learn more about writing Workflow specs on the following pages.
1
jobs:
2
CloneRepo:
3
resources:
4
instance-type: C5
5
outputs:
6
repo:
7
type: volume
8
9
with:
10
url: https://github.com/NVlabs/stylegan2.git
11
StyleGan2:
12
resources:
13
instance-type: P4000
14
needs:
15
- CloneRepo
16
inputs:
17
repo: CloneRepo.outputs.repo
18
outputs:
19
generatedFaces:
20
type: dataset
21
with:
22
ref: demo-dataset
24
with:
25
script: |-
26
pip install scipy==1.3.3
27
pip install requests==2.22.0
28
pip install Pillow==6.2.1
29
cp -R /inputs/repo /stylegan2
30
cd /stylegan2
31
python run_generator.py generate-images \
32
--network=gdrive:networks/stylegan2-ffhq-config-f.pkl \
33
--seeds=6600-6605 \
34
--truncation-psi=0.5 \
35
--result-dir=/outputs/generatedFaces
36
image: tensorflow/tensorflow:1.14.0-gpu-py3
Copied!

Create Datasets for the Workflow

Datasets referenced in the Workflow spec need to be created before running the Workflow for the first time. On subsequent runs of the Workflow the Datasets will be used again, but different Dataset versions will be created for each output Dataset. For more information about Datasets see Versioned Data.
The above Workflow creates a new output Dataset version in the Dataset named demo-dataset. So before running this Workflow make sure a Dataset with that name already exists. You can run this command to list your Datasets: gradient datasets list.
If you completed the Workflows demo in the quick-start guide you will already have a Dataset with this name. If not, you can create it on the CLI using the following commands.
First, get a list of storage providers that are already part of your account. You should have at least one called Gradient Managed.
1
gradient storageProviders list
2
+------------------+-----------------+------+------------------------------------------+
3
| Name | ID | Type | Config |
4
+------------------+-----------------+------+------------------------------------------+
5
| Gradient Managed | splXXXXXXXXXXXX | s3 | accessKey: XXXXXXXXXXXXXXXXXXXX |
6
| | | | bucket: XXXXXXXXX |
7
| | | | endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
8
| | | | secretAccessKey: ******** |
9
+------------------+-----------------+------+------------------------------------------+
Copied!
Then create a dataset named demo-dataset using the Gradient Managed storage provider ID:
1
gradient datasets create \
2
--name demo-dataset \
3
--storageProviderId splXXXXXXXXXXXX
Copied!
Datasets with other names can be created similarly. The dataset name should match the name referred to in the YAML. Note that Datasets can also be referred to directly by their IDs, but names are usually more convenient unless a specific Dataset version needs to be referenced.

Dataset creation in the GUI

You can also create a Dataset using the GUI. See Creating a Dataset and Dataset Version.

Run the Workflow

Workflows can be run by triggering them to run by making a change to your linked GitHub repository, or by invoking them directly using the command line.
Trigger
CLI
Workflows can be triggered to run from Gradient by placing them in the .gradient/workflows directory in your linked repository. This directory should be created if it does not exist.
Within the Workflow YAML, the on: field is used to indicate that this Workflow is to be triggered to run when the given conditions are met. For the general case of any change to the repo triggering the Workflow to run, the YAML lines are
1
on:
2
github:
3
branches:
4
only: main
Copied!
Currently, this is the condition set that works, so to prevent a Workflow being triggered, comment the lines out. In future, more general cases such as only changes to specific file types being a trigger will be supported.
Run the Workflow with the specified Workflow spec file and the workflow-id from the previously created Workflow. (You can also get a list of Workflows by running gradient workflows list.)
1
gradient workflows run \
2
--id <worklow-id> \
3
--path ./workflow.yaml
Copied!
A Workflow can be run multiple times, each with the same or a different Workflow YAML spec. The Workflow spec is recorded as part of the Workflow run so you can distinguish different runs.
The next sections cover the syntax for authoring new Workflow specs, inputs and outputs to Workflow steps, and various Workflow actions.
Last modified 16d ago