Gradient Workflows
Workflows are in beta, so try them out and let us know what you think. Hold off on using them for production use cases, as we can't promise bug fixes or new features just yet.
Gradient Workflows
Gradient Workflows provides a simple way to automate machine learning tasks.
Overview​
Gradient Workflows is a new and powerful way to build-out machine learning applications. Workflows utilize GitHub-action style syntax via YAML files to easily create powerful automation.
Workflows is based on the Argo runtime engine, which is a container-native continuous delivery tool for Kubernetes, and makes it easy to build complex and scalable projects with an arbitrary number of discrete steps.
Workflows is used by ML Engineers to build-out deterministic machine learning pipelines.
Where to start​
The best tutorial is the canonical Gradient Workflows Tutorial:
Key Terminology​
- Workflow: a named or unnamed entity that belongs to a team and project
- Named workflows can be re-run with a default
workflow spec
, or be passed a new spec every time
- Named workflows can be re-run with a default
- Workflow Spec: a YAML list of jobs that is converted into an Argo template and run on the Gradient distributed runtime engine.
- Job: self-contained part of a workflow spec that is similar to an Argo step
- Jobs can define inputs, outputs, and their own environment variables
- Jobs can require other jobs via "needs" and collect/pass info between jobs
- Jobs can be implemented with an action via "use"
- Action: a self-contained, composable set of code building blocks that can perform specific actions within a machine learning project.
- Actions can receive parameters (e.g., args, image) within the job step via the "with" argument
- E.g.,
container@v1
action = run a container, load inputs, and produce outputs
- Workflow Run: the implementation of a workflow
- The most basic run requires a
workflowId
andclusterId
- most will also include a workflowSpec, and the inputs to be passed into the workflow - The workflow run contains everything needed for the workflow to actually be executed, i.e., what (
workflowId
), where (clusterId
), how (workflowSpec
), with (inputs, etc.)
- The most basic run requires a