Using Jobs

This feature is currently only available to our Gradient hosted offering. Contact Sales to learn more.

Creating a Job

Web UI
CLI
Web UI

Jobs can be submitted via the Job Builder. There are a variety of optional and required options for your job including the instance type, the command, the container, etc. Note: At the top of the screen, you'll see a few job presets which will give a sense for how these options work.

  • Machine Type. What type of instance to run your Job on. We recommend starting with a GPU+. Many Jobs benefit from a machine with a GPU, but some can run just using a CPU.

  • Container. Jobs are run within a docker container. You can run a public or private container. Learn more here.

  • Workspace. The workspace is the collection of code that is run. It can be a Git repository (public or private), your local working directory (if you are using the CLI) which is uploaded to the docker container during the job running process, or none (default value).

  • Command. The command is the entry point to the container. This is the line of code that will kick off your Job. It could be a bash script ./run.sh or python main.py as just some examples.

  • Ports. You have the option to attach a public IP automatically. Supports opening multiple ports simultaneously, separated by : . Learn more about opening ports here.

  • Custom Metrics. Enter a list of custom metrics to use with Gradient's statd client, such as percent_failure or percent_success.

Once you have examined or specified the parameters, hit "Submit Job" and watch the Job run!

The Job Builder is great for getting started with basic tasks. For more advanced workflows, we recommend using the CLI which offers more flexibility.

CLI

Syntax

gradient jobs create <namespace> <command> [options...]

Job Parameters Basics

  • Machine Type: Such as --P100 or --C7 or --TPU

  • Container: Such as --tensorflow/tensorflow:1.5.1-gpu

  • Command: Such as "./do.sh"

Example Use

gradient jobs create \
--name "my job" \
--container "http://dockerhub.com/mycontainer" \
--machineType "P5000" \
--command "/paperspace/run.sh"

Job Parameters Complete List

Argument

Default

Description

name

[required]

Job name

machineType

K80

An optional machine type to run the job on: either 'GPU+', 'P4000', 'P5000', 'P6000', 'V100', 'K80', 'P100', or 'TPU'.

container

paperspace/tensorflow-python

A reference to a docker image in a public or private docker registry, or a container name provided by Paperspace. Docker image repository references must be in lowercase and may include a tag and a hostname prefix followed by a slash; if committed the hostname defaults to that of the public Docker Hub registry. An example docker image reference: docker.io/mynamespace/myimage:mytag. A container name may be mixed case. (Designated container names are currently only provided as part of various Gradient tutorials and samples.)

command

Job command/entrypoint

ports

An optional list of port mappings to open on the job cluster machine while the job is running. The port mappings are specified as 'XXXX:YYYY' where XXXX is an external port number and YYYY is an internal port number. Multiple port mappings can be provided as a comma separated list. Port numbers must be greater than 1023. Note: only /tcp protocol usage is supported.

workspace

Path to workspace directory. (Soon also will support a path to a workspace archive or git repository URL.)

workspaceArchive

Path to workspace archive. (Currently being deprecated in an upcoming version.)

workspaceUrl

Project git repository URL. (Currently being deprecated in an upcoming version.)

workingDirectory

Working directory for the experiment

experimentId

Experiment Id

jobEnv

Environmental variables in JSON String Format. Example:

{ "HKS_EPOCHS": 1, "HKS_MAX_EVALS": 4, "DATASET_SIZE": 10000 }

project

$CWD

The name of the project for this job. If not provided, this is taken from the .ps_project/config.json file, or the current directory name.

projectID

Project ID

apiKey

API key to use this time only

ignoreFiles

Ignore certain files from uploading

Environment variables are available for use within the context of your job. The following host config options are currently exposed within the container:

$PS_HOST_PUBLIC_IP_ADDRESS - the public IP address of the host machine running the job

$PS_HOST_PRIVATE_IP_ADDRESS - the private IP address of the host machine running the job

$PS_HOSTNAME - the hostname of the host machine running the job

These can be used in conjunction with the ports option to send HTTP traffic to the job while it's in progress for example.