Storage Management

Overview

Storage management will allow you to modify and delete files & directories in Linux based systems. To effectively navigate through a Linux system, we recommend the use of the Terminal, however, through Jupyter Notebooks it is possible to run bash commands to interact with the underlying OS environment via a Code Cell in single line.

To do so, you have to start your line using the ! operator, if you need to add more lines, you can connect them using the && operator.

Determine the amount of storage used

When managing data, it is important to be able to decide which which data should be kept and which should be deleted. First we need to know where that data is.

Determining folder size

Often we manage our files using folders, and often we need to determine folder size as part of our file management, to do so we can use the following commands:

Terminal:

du -sch .[!.]* * | sort -h

Code cell:

!du -sch .[!.]* * | sort -h

Keep in mind that these commands will only list folders (not individual files) located in the folder that you are currently viewing in the terminal. If you want to see the list of folders stored in a different folder, you can use the following commands:

Terminal:

cd <FolderName>
du -sch .[!.]* * | sort -h
## Replace <FolderName> with the path to the folder you want to analyze

Code cell:

!cd <FolderName> && du -sch .[!.]* * | sort -h
## Replace <FolderName> with the path to the folder you want to analyze

Determining file size

Once we know where our data is, we can determine file sizes. This information will help us determine which individual files have had the greatest impact on our storage. To do so you use the following commands:

Terminal:

find . -print0 | du -ch --files0-from=- | sort -h

Code cell:

!find . -print0 | du -ch --files0-from=- | sort -h

Please keep in mind that these commands will only list files (not folders) located in the folder that you are currently viewing in the terminal. If you want to see the information stored in a different folder, you can use the following commands:

Terminal:

cd <FolderName>
find . -print0 | du -ch --files0-from=- | sort -h
## Replace <FolderName> with the path to the folder you want to analyze

Code cell:

!cd <FolderName> && find . -print0 | du -ch --files0-from=- | sort -h
## Replace <FolderName> with the path to the folder you want to analyze

Once you have determined which files you might want to delete, we recommend moving them to a single folder so you can later delete everything in one go. To do so, you can use the following commands:

Terminal

cd <FolderName>
mv <FileName> <DestinationFolder>
## Replace <FolderName> with the address to the folder the file is in
## Replace <FileName> with the name of the file you want to move
## Replace <DestinationFolder> with the path to the folder that will hold the file

Code cell:

!cd <FolderName> && mv <FileName> <DestinationFolder>
## Replace <FolderName> with the address to the folder the file is in
## Replace <FileName> with the name of the file you want to move
## Replace <DestinationFolder> with the path to the folder that will hold the file

Deleting folders and files

Once you have determined which folders and files you want to delete, can use the following command to delete them:

Terminal:

cd <FolderName>
rm -r <DataName>
## Replace <FolderName> with the path of the folder that holds the data/folder you want to delete
## Replace <DataName> with the name of the folder/file you want to delete

Code cell:

!cd <FolderName> && rm -r <DataName>
## Replace <FolderName> with the path of the folder that holds the data/folder you want to delete
## Replace <DataName> with the name of the folder/file you want to delete

Other recommended tools and alternatives

For users that might prefer a visual interface to determine folder and file sizes, free alternative tools have been created that can ease the process, our recommendation is ncdu.

note

Please keep in mind that ncdu will only be available for users with access to the Linux Terminal

To use ncdu, follow these instructions:

1. In the terminal, update your machine’s APT

sudo apt-get update

2. Install ncdu

sudo apt install ncdu

3. Run ncdu by typing it as a command in the terminal

ncdu

This application will scan your machine and show all folders listed by size in the terminal, it will also allow you to move around the file system to open each folder, see it’s content and delete any file or folder.

note

Please keep in mind that all the instructions shown from this point on are executed in the Paperspace Console

Datasets

As we show here, datasets are a completely separate storage source that allows our users to save and/or share large amounts of data, even though datasets are managed separately from the main storage, due to their nature, they are in fact part of the main storage.

Datasets work as read only sources, so, once files have been added to one, they can’t be edited, moved, nor deleted, therefore, those files can only be removed from your storage by deleting the whole dataset itself.

To delete a dataset please follow these instructions:

1. Select any project
2. Click on the Data tab

3. Search for the dataset you want to delete
4. Click on the 3 dot button that corresponds to the dataset you want to delete

5. Select Delete

Models

Depending on their size, some models can take up a significant amount of storage in your account, and thus, require you to delete some of your models to clean up some of that space.

Here you can find all information related to model deletion.

Deployments

Depending on how you have setup your deployment, it might use some of your storage to save data, due to this, disabling a deployment and/or deleting it might be the best option to save up some of that storage.

In the tutorial shown here, you can learn how to stop a deployment.

Now, to delete a deployment, you have to follow these steps:

1. Select the project that holds the deployment you want to delete
2. Click on the Deployments tab

3. Find the deployment you want to delete
4. Click on the 3 dot button

5. Select Delete