Skip to main content

Gradient Workflows

caution

Workflows are in beta, so try them out and let us know what you think. Hold off on using them for production use cases, as we can't promise bug fixes or new features just yet.

Gradient Workflows

Gradient Workflows provides a simple way to automate machine learning tasks.

Overview​

Gradient Workflows is a new and powerful way to build-out machine learning applications. Workflows utilize GitHub-action style syntax via YAML files to easily create powerful automation.

Workflows is based on the Argo runtime engine, which is a container-native continuous delivery tool for Kubernetes, and makes it easy to build complex and scalable projects with an arbitrary number of discrete steps.

Examining the logs from a simple workflow.

Workflows is used by ML Engineers to build-out deterministic machine learning pipelines.

Where to start​

The best tutorial is the canonical Gradient Workflows Tutorial:

Workflows Tutorial
Learn how to create automated pipelines by describing ML tasks in YAML. Best for those interested in MLOps best practices and building production-grade systems.

Key Terminology​

  • Workflow: a named or unnamed entity that belongs to a team and project
    • Named workflows can be re-run with a default workflow spec, or be passed a new spec every time
  • Workflow Spec: a YAML list of jobs that is converted into an Argo template and run on the Gradient distributed runtime engine.
  • Job: self-contained part of a workflow spec that is similar to an Argo step
    • Jobs can define inputs, outputs, and their own environment variables
    • Jobs can require other jobs via "needs" and collect/pass info between jobs
    • Jobs can be implemented with an action via "use"
  • Action: a self-contained, composable set of code building blocks that can perform specific actions within a machine learning project.
    • Actions can receive parameters (e.g., args, image) within the job step via the "with" argument
    • E.g., container@v1 action = run a container, load inputs, and produce outputs
  • Workflow Run: the implementation of a workflow
    • The most basic run requires a workflowId and clusterId - most will also include a workflowSpec, and the inputs to be passed into the workflow
    • The workflow run contains everything needed for the workflow to actually be executed, i.e., what (workflowId), where (clusterId), how (workflowSpec), with (inputs, etc.)