Gradient Workflows
Gradient Workflows provides a simple way to automate machine learning tasks.
Overview​
Gradient Workflows is a new and powerful way to build-out machine learning applications. Workflows utilize GitHub-action style syntax via YAML files to easily create powerful automation.
Workflows is based on the Argo runtime engine, which is a container-native continuous delivery tool for Kubernetes, and makes it easy to build complex and scalable projects with an arbitrary number of discrete steps.
Workflows is used by ML Engineers to build-out deterministic machine learning pipelines.
Where to start​
The best tutorial is the canonical Gradient Workflows Tutorial:
Learn how to create automated pipelines by describing ML tasks in YAML. Best for those interested in MLOps best practices and building production-grade systems.
Key Terminology​
- Workflow: a named or unnamed entity that belongs to a team and project
- Named workflows can be re-run with a default
workflow spec
, or be passed a new spec every time
- Named workflows can be re-run with a default
- Workflow Spec: a YAML list of jobs that is converted into an Argo template and run on the Gradient distributed runtime engine.
- Job: self-contained part of a workflow spec that is similar to an Argo step
- Jobs can define inputs, outputs, and their own environment variables
- Jobs can require other jobs via "needs" and collect/pass info between jobs
- Jobs can be implemented with an action via "use"
- Action: a self-contained, composable set of code building blocks that can perform specific actions within a machine learning project.
- Actions can receive parameters (e.g., args, image) within the job step via the "with" argument
- E.g.,
container@v1
action = run a container, load inputs, and produce outputs
- Workflow Run: the implementation of a workflow
- The most basic run requires a
workflowId
andclusterId
- most will also include a workflowSpec, and the inputs to be passed into the workflow - The workflow run contains everything needed for the workflow to actually be executed, i.e., what (
workflowId
), where (clusterId
), how (workflowSpec
), with (inputs, etc.)
- The most basic run requires a