Skip to main content

Gradient Workflows

Gradient Workflows provides a simple way to automate machine learning tasks.

Overview​

Gradient Workflows is a new and powerful way to build-out machine learning applications. Workflows utilize GitHub-action style syntax via YAML files to easily create powerful automation.

Workflows is based on the Argo runtime engine, which is a container-native continuous delivery tool for Kubernetes, and makes it easy to build complex and scalable projects with an arbitrary number of discrete steps.

Examining the logs from a simple workflow.

Workflows is used by ML Engineers to build-out deterministic machine learning pipelines.

Where to start​

The best tutorial is the canonical Gradient Workflows Tutorial:

Workflows Tutorial
Learn how to create automated pipelines by describing ML tasks in YAML. Best for those interested in MLOps best practices and building production-grade systems.

Key Terminology​

  • Workflow: a named or unnamed entity that belongs to a team and project
    • Named workflows can be re-run with a default workflow spec, or be passed a new spec every time
  • Workflow Spec: a YAML list of jobs that is converted into an Argo template and run on the Gradient distributed runtime engine.
  • Job: self-contained part of a workflow spec that is similar to an Argo step
    • Jobs can define inputs, outputs, and their own environment variables
    • Jobs can require other jobs via "needs" and collect/pass info between jobs
    • Jobs can be implemented with an action via "use"
  • Action: a self-contained, composable set of code building blocks that can perform specific actions within a machine learning project.
    • Actions can receive parameters (e.g., args, image) within the job step via the "with" argument
    • E.g., container@v1 action = run a container, load inputs, and produce outputs
  • Workflow Run: the implementation of a workflow
    • The most basic run requires a workflowId and clusterId - most will also include a workflowSpec, and the inputs to be passed into the workflow
    • The workflow run contains everything needed for the workflow to actually be executed, i.e., what (workflowId), where (clusterId), how (workflowSpec), with (inputs, etc.)