Public Profiles
TensorBoards

Create a Job

About

Create a new Paperspace job

Note: if a project is not defined for the current working directory, and you are running in command line mode, a project configuration settings file will be created. Use--init false or specify--project <your-project-name>to override this behavior.

Syntax

$ gradient jobs create <namespace> <command> [options...]

Job Parameters Basics

  • Machine Type: Such as --P100 or --C7 or --TPU

  • Container: Such as --tensorflow/tensorflow:1.5.1-gpu

  • Command: Such as "./do.sh"

Example Use

$ gradient jobs create \
--name "my job" \
--container "http://dockerhub.com/mycontainer" \
--machineType "P5000" \
--command "/paperspace/run.sh"

Job Parameters Complete List

Argument

Default

Description

name

[required]

Job name

machineType

K80

An optional machine type to run the job on: either 'GPU+', 'P4000', 'P5000', 'P6000', 'V100', 'K80', 'P100', or 'TPU'.

container

paperspace/tensorflow-python

A reference to a docker image in a public or private docker registry, or a container name provided by Paperspace. Docker image repository references must be in lowercase and may include a tag and a hostname prefix followed by a slash; if committed the hostname defaults to that of the public Docker Hub registry. An example docker image reference: docker.io/mynamespace/myimage:mytag. A container name may be mixed case. (Designated container names are currently only provided as part of various Gradient tutorials and samples.)

command

Job command/entrypoint

ports

An optional list of port mappings to open on the job cluster machine while the job is running. The port mappings are specified as 'XXXX:YYYY' where XXXX is an external port number and YYYY is an internal port number. Multiple port mappings can be provided as a comma separated list. Port numbers must be greater than 1023. Note: only /tcp protocol usage is supported.

workspace

Path to workspace directory. (Soon also will support a path to a workspace archive or git repository URL.)

workspaceArchive

Path to workspace archive. (Currently being deprecated in an upcoming version.)

workspaceUrl

Project git repository URL. (Currently being deprecated in an upcoming version.)

workingDirectory

Working directory for the experiment

experimentId

Experiment Id

jobEnv

Environmental variables in JSON String Format. Example:

{ "HKS_EPOCHS": 1, "HKS_MAX_EVALS": 4, "DATASET_SIZE": 10000 }

project

$CWD

The name of the project for this job. If not provided, this is taken from the .ps_project/config.json file, or the current directory name.

projectID

Project ID

apiKey

API key to use this time only

ignoreFiles

Ignore certain files from uploading

Environment variables are available for use within the context of your job. The following host config options are currently exposed within the container:

$PS_HOST_PUBLIC_IP_ADDRESS - the public IP address of the host machine running the job

$PS_HOST_PRIVATE_IP_ADDRESS - the private IP address of the host machine running the job

$PS_HOSTNAME - the hostname of the host machine running the job

These can be used in conjunction with the ports option to send HTTP traffic to the job while it's in progress for example.

Run jobs from Dockerfiles

Note: to run jobs from Dockerfiles, use paperspace-node, or gradient-cli.

Gradient job containers can be created from a Dockerfile. Three options are available:

1) The job can build the container image and push it to a remote registry only. This is useful in cases where you want access to a GPU to build a GPU-enabled container but do not have one on-hand.

2) The job can build the container image and run commands against a running instance of the container without uploading to a remote registry. Useful for experimenting with gradient jobs defined by Dockerfiles and cases where you are only interested in the results of the job.

3) The job can build the container image, upload the image to a remote registry, and then run commands against a running instance of the container . Useful for building images and experimenting with gradient jobs defined by Dockerfiles while retaining the original image used.

The following new job fields are available:

Name

Type

Attributes

Description

useDockerfile

boolean

<optional>

determines whether to build from Dockerfile (default false). Do not include a --container argument when using this flag.

buildOnly

boolean

<optional>

determines whether to only build and not run image (default false)

registryTarget

string

<optional>

registry location to push image to

registryTargetUsername

string

<optional>

registry username

registryTargetPassword

string

<optional>

registry password

relDockerfilePath

string

<optional>

relative location of dockerfile in workspace (default "./Dockerfile")

For example, to run a job that only builds a container image and pushes to a remote registry:

gradient jobs create --apiKey XXXXXXXXXXXX --workspace https://github.com/ianmiell/simple-dockerfile --useDockerfile true --buildOnly true --registryTarget my-registry/image:0.1-test --registryTargetUsername myusername --registryTargetPassword 123456

Note that if you selected buildOnly you should supply always a registry target and credentials.

To run a job that builds a container image, pushes to a remote registry, and then runs a command inside an instance of the running container:

gradient jobs create --apiKey XXXXXXXXXXXX --workspace https://github.com/ianmiell/simple-dockerfile --useDockerfile true --buildOnly false --command "echo hello" --registryTarget my-registry/image:0.1-test --registryTargetUsername myusername --registryTargetPassword 123456

To run a job that builds the container image and then runs an instance of the container, without pushing to a remote registry:

gradient jobs create --apiKey XXXXXXXXXXXX --workspace https://github.com/ianmiell/simple-dockerfile --useDockerfile true --buildOnly false --command "echo hello"

Note: commands run during the build step from the Dockerfile, like CMD ["command", "arg1"...] are run inside the image layers as it's being built. Once the container image is ready to run, the --command argument is used to determine what command to run against the built image.