datasets

Paperspace Gradient CLI reference guide for datasets.

create

Create dataset

Examples

CLI

gradient datasets create \
  --name "new_dataset_1" \
  --storageProviderId "sp65qbxcrsgiv43"

curl

curl -X POST 'https://api.paperspace.io/datasets' \
-H 'x-api-key: d44808a2785d6a...' \
-H 'Content-Type: application/json' \
--data-raw '{
    "storageProviderId": "sp65qbxcrsgiv43",
    "name": "new_dataset_1"
}'

Python

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.create(
    name='new_dataset_1',
    storage_provider_id='sp65qbxcrsgiv43'
))

Options

Name	Type	Attributes	Description
--name	string	required	Dataset name
--storageProviderId	string	required	Storage provider ID
--description	string	optional	Dataset description
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Created dataset: ds7diuwj0av3tqy

curl

{
  "id": "ds7diuwj0av3tqy",
  "name": "new_dataset_1",
  "description": null,
  "isPublic": false,
  "mlObjectTypeId": 2,
  "dtCreated": "2022-03-21T23:26:59.475Z",
  "dtModified": "2022-03-21T23:26:59.475Z",
  "storageProvider": {
    "name": "Gradient Managed",
    "config": {
      "bucket": "tewr3st2z",
      "endpoint": "https://osg-ny2.paperspace.io/",
      "accessKey": "6P0NXXIKSB49ZW7ZXESL",
      "retainData": false,
      "secretAccessKey": "********"
    },
    "dtCreated": "2022-03-14T15:00:11.060Z",
    "dtModified": "2022-03-14T15:00:11.060Z",
    "isTeamDefault": true,
    "isGradientManaged": true,
    "id": "sp65qbxcrsgiv43",
    "type": "s3"
  }
}

Python

ds7diuwj0av3tqy

details

Show dataset details

Examples

CLI

gradient datasets details \
  --id "ds7diuwj0av3tqy"

curl

curl -X GET 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.get(
    dataset_id="ds7diuwj0av3tqy"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset ID
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

+-----------------+------------------------------------+
| Name            | new_dataset_1                      |
+-----------------+------------------------------------+
| ID              | ds7diuwj0av3tqy                    |
| Description     |                                    |
| StorageProvider | Gradient Managed (sp65qbxcrsgiv43) |
+-----------------+------------------------------------+

curl

{
  "id": "ds7diuwj0av3tqy",
  "name": "new_dataset_1",
  "description": "Dataset description",
  "isPublic": false,
  "mlObjectTypeId": 2,
  "dtCreated": "2022-03-21T19:05:04.485Z",
  "dtModified": "2022-03-21T19:21:22.839Z",
  "storageProvider": {
    "name": "Gradient Managed",
    "config": {
      "bucket": "tewr3st2z",
      "endpoint": "https://osg-ny2.paperspace.io/",
      "accessKey": "6P0NXXIKSB49ZW7ZXESL",
      "retainData": false,
      "secretAccessKey": "********"
    },
    "dtCreated": "2022-03-14T15:00:11.060Z",
    "dtModified": "2022-03-14T15:00:11.060Z",
    "isTeamDefault": true,
    "isGradientManaged": true,
    "id": "sp65qbxcrsgiv43",
    "type": "s3"
  }
}

Python

Dataset(
  (id = "ds7diuwj0av3tqy"),
  (name = "new_dataset_1"),
  (description = None),
  (storage_provider_id = None),
  (storage_provider = StorageProvider(
    (id = "sp65qbxcrsgiv43"),
    (type = "s3"),
    (name = "Gradient Managed"),
    (config = {
      bucket: "tewr3st2z",
      endpoint: "https://osg-ny2.paperspace.io/",
      accessKey: "6P0NXXIKSB49ZW7ZXESL",
      retainData: False,
      secretAccessKey: "********",
    })
  ))
);

list

List datasets

Examples

CLI

gradient datasets list

curl

curl -X GET 'https://api.paperspace.io/datasets' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.list())

Options

Name	Type	Attributes	Description
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

+---------------+-----------------+------------------------------------+
| Name          | ID              | Storage Provider                   |
+---------------+-----------------+------------------------------------+
| new_dataset_1 | ds7diuwj0av3tqy | Gradient Managed (sp65qbxcrsgiv43) |
+---------------+-----------------+------------------------------------+

curl

[
  {
    id: "ds7diuwj0av3tqy",
    name: "new_dataset_1",
    description: "Dataset description",
    isPublic: false,
    mlObjectTypeId: 2,
    dtCreated: "2022-03-15T00:23:00.481Z",
    dtModified: "2022-03-15T00:49:20.202Z",
    storageProvider: {
      name: "Gradient Managed",
      config: {
        bucket: "tewr3st2z",
        endpoint: "https://osg-ny2.paperspace.io/",
        accessKey: "6P0NXXIKSB49ZW7ZXESL",
        retainData: false,
        secretAccessKey: "********",
      },
      dtCreated: "2022-03-14T15:00:11.060Z",
      dtModified: "2022-03-14T15:00:11.060Z",
      isTeamDefault: true,
      isGradientManaged: true,
      id: "sp65qbxcrsgiv43",
      type: "s3",
    },
  },
]

Python

[
  Dataset(
    (id = "ds7diuwj0av3tqy"),
    (name = "new_dataset_1"),
    (description = "Updated dataset description"),
    (storage_provider_id = None),
    (storage_provider = StorageProvider(
      (id = "sp65qbxcrsgiv43"),
      (type = "s3"),
      (name = "Gradient Managed"),
      (config = {
        bucket: "tewr3st2z",
        endpoint: "https://osg-ny2.paperspace.io/",
        accessKey: "6P0NXXIKSB49ZW7ZXESL",
        retainData: False,
        secretAccessKey: "********",
      })
    ))
  ),
];

update

Update dataset

Examples

CLI

gradient datasets update \
  --id "ds7diuwj0av3tqy" \
  --description "Updated dataset description"

curl

curl -X POST 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy' \
-H 'x-api-key: d44808a2785d6a...' \
-H 'Content-Type: application/json' \
--data-raw '{
    "description": "Updated dataset description"
}'

Python

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.update(
    dataset_id="ds7diuwj0av3tqy",
    description="Updated dataset description"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset ID
--name	string	optional	Dataset name
--description	string	optional	Dataset description
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Updated dataset: ds7diuwj0av3tqy

curl

{
  "id": "ds7diuwj0av3tqy",
  "name": "new_name",
  "description": "Updated dataset description",
  "isPublic": false,
  "mlObjectTypeId": 2,
  "dtCreated": "2022-03-21T19:05:04.485Z",
  "dtModified": "2022-03-21T23:44:36.440Z",
  "storageProvider": {
    "name": "Gradient Managed",
    "config": {
      "bucket": "tewr3st2z",
      "endpoint": "https://osg-ny2.paperspace.io/",
      "accessKey": "6P0NXXIKSB49ZW7ZXESL",
      "retainData": false,
      "secretAccessKey": "********"
    },
    "dtCreated": "2022-03-14T15:00:11.060Z",
    "dtModified": "2022-03-14T15:00:11.060Z",
    "isTeamDefault": true,
    "isGradientManaged": true,
    "id": "sp65qbxcrsgiv43",
    "type": "s3"
  }
}

Python

None

delete

Delete dataset

Examples

CLI

gradient datasets delete \
  --id "dsltzdafwejmqlq"

curl

curl -X DELETE 'https://api.paperspace.io/datasets/dsltzdafwejmqlq' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.delete(
    dataset_id='ds7diuwj0av3tqy',
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset ID
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Deleted dataset: dsltzdafwejmqlq

curl

Python

None

files get

Get files

Examples

CLI

gradient datasets files get \
  --id "dsj1g16f1z67jai:0z7o247" \
  --target-path "~/downloads"

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--source-path	string	optional	File or directory to get
--target-path	string	required	Target directory path
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

files list

List files

Examples

CLI

gradient datasets files list \
  --id "dsj1g16f1z67jai:xbreynt"

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--path	string	optional	Sub-directory to delete
--recursive	string	optional	Recursive list content
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

+----------+------+
| Name     | Size |
+----------+------+
| data.txt | 4    |
+----------+------+

files put

Put files

Examples

CLI

gradient datasets files put \
  --id "dsj1g16f1z67jai:xbreynt" \
  --source-path "data.txt"

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--source-path	string	required	File or directory to get
--target-path	string	optional	Target directory path
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

files delete

Delete files

Examples

CLI

gradient datasets files delete \
  --id "dsj1g16f1z67jai:0z7o247"

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--path	string	optional	Sub-directory to delete
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

versions create

Create dataset version

Examples

CLI

gradient datasets versions create \
  --id "ds7diuwj0av3tqy"

curl

curl -X POST 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.create(
    dataset_id="ds7diuwj0av3tqy"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--message	string	optional	Dataset version message
--source-path	string	optional	Files to put into dataset versioning (setting this will commit the version)
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Created dataset version: ds7diuwj0av3tqy:dk0c3k1

curl

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": false,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-21T23:51:56.952Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

Python

dk0c3k1

versions commit

Commit dataset version

Examples

CLI

gradient datasets versions commit \
  --id "ds7diuwj0av3tqy:dk0c3k1"

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Committed dataset version: ds7diuwj0av3tqy:dk0c3k1

versions details

Show dataset version details

Examples

CLI

gradient datasets versions details \
  --id "ds7diuwj0av3tqy:dk0c3k1"

curl

curl -X GET 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions/dk0c3k1' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.get(
    dataset_version_id="ds7diuwj0av3tqy:dk0c3k1"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

+-----------+-------------------------+
| ID        | ds7diuwj0av3tqy:dk0c3k1 |
+-----------+-------------------------+
| Message   |                         |
| Committed | true                    |
| Tags      |                         |
+-----------+-------------------------+

curl

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": true,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-22T00:03:38.917Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

Python

DatasetVersion(version='dk0c3k1', message=None, is_committed=True, tags=[], dataset_id=None)

versions list

List dataset versions

Examples

CLI

gradient datasets versions list \
  --id "ds7diuwj0av3tqy"

curl

curl -X GET 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.list(
    dataset_id="ds7diuwj0av3tqy"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset ID
--isCommitted	boolean	optional	Show filter by committed status
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

+-------------------------+---------+------+
| ID                      | Message | Tags |
+-------------------------+---------+------+
| ds7diuwj0av3tqy:dk0c3k1 |         |      |
+-------------------------+---------+------+

curl

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": true,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-22T00:03:38.917Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

Python

[
  DatasetVersion(
    (version = "dk0c3k1"),
    (message = None),
    (is_committed = True),
    (tags = []),
    (dataset_id = None)
  ),
];

versions update

Update dataset version

Examples

CLI

gradient datasets versions update \
  --id "ds7diuwj0av3tqy:dk0c3k1"

curl

curl -X POST 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions/dk0c3k1' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.update(
    dataset_version_id="ds7diuwj0av3tqy:dk0c3k1"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--message	string	optional	Dataset version message
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Updated dataset version: ds7diuwj0av3tqy:dk0c3k1

curl

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": true,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-22T00:03:38.917Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

Python

None

versions delete

Delete dataset version

Examples

CLI

gradient datasets versions delete \
  --id "ds7diuwj0av3tqy:dk0c3k1"

curl

curl -X DELETE 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions/dk0c3k1' \
-H 'x-api-key: d44808a2785d6a...'

Python

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.delete(
    dataset_version_id="ds7diuwj0av3tqy:8vkjy15"
))

Options

Name	Type	Attributes	Description
--id	string	required	Dataset version ID
--apiKey	string	optional	API key to use this time only
--optionsFile	string	optional	Path to YAML with predefined options
--createOptionsFile	string	optional	Generate template options file

Response

CLI

Deleted dataset version: ds7diuwj0av3tqy:dk0c3k1

curl

Python

None