Build a custom container and push it to a registry
Launch a Jupyter Notebook based on the custom container
Train and test a model
The jobs submitted to Gradient run in the context of a container. Though the platform comes with mainstream container images, there may be a need to run a job within a custom container.
In this tutorial, we will learn how to build a custom container for training a model based on Apache MXNet. The container encapsulates all the dependencies needed for the job and Jupyter Notebooks for interactive development.
You may want to clone the Github repository that has the Dockerfile and Jupyter Notebook.
Custom containers are Docker images that can be easily built on any machine that has Docker CE installed.
Make sure that Docker is running by executing the following command:
$ docker versionClient: Docker Engine - CommunityVersion: 18.09.2API version: 1.39Go version: go1.10.8Git commit: 6247962Built: Sun Feb 10 04:12:39 2019OS/Arch: darwin/amd64Experimental: falseServer: Docker Engine - CommunityEngine:Version: 18.09.2API version: 1.39 (minimum version 1.12)Go version: go1.10.6Git commit: 6247962Built: Sun Feb 10 04:13:06 2019OS/Arch: linux/amd64Experimental: true
We will now create a Dockerfile that has the commands to install Apache MXNet and Jupyter Notebooks environment.
FROM mxnet/python:latestRUN pip3 install requestsRUN pip3 install jupyterlabRUN pip3 install matplotlib
The first line of Dockerfile instructs the build process to use mxnet/python as the base image. The next three lines simply use pip to install dependencies such as jupyterlab and matplotlib.
If you have cloned the Github Repo, Dockerfile is available in the Container directory.
Let’s go ahead and build the custom container. Replace the USERNAME environment variable with your Docker Hub username.
$ docker build -t $USERNAME/mxnet .Sending build context to Docker daemon 2.048kBStep 1/4 : FROM mxnet/python:latestlatest: Pulling from mxnet/python7e6591854262: Pull complete....6a614067a2c4: Pull complete107776115e90: Pull completeDigest: sha256:628b8e5d6d375d595128e11c8088edae37534524aa1ca2c5086355c89a8d2649Status: Downloaded newer image for mxnet/python:latest---> 121567f56cd1Step 2/4 : RUN pip3 install requests---> Running in 0127fabe8f91Requirement already satisfied: requests in /usr/local/lib/python3.5/dist-packages (2.21.0)....Removing intermediate container 0127fabe8f91---> d839e172a78eStep 3/4 : RUN pip3 install jupyterlab---> Running in 7b9ddfe995a3Collecting jupyterlabDownloading https://files.pythonhosted.org/packages/dd/7f/18b4ecfa055243f1eccdb1d7a1cdc0ae529f3df4c1098cee442ad177511a/jupyterlab-0.35.6-py3-none-any.whl (14.8MB)Collecting jupyterlab-server<0.3.0,>=0.2.0 (from jupyterlab)Downloading https://files.pythonhosted.org/packages/78/77/e8a9c300afbe24aa46abaf1091d9e7b82328559e99cf2d601e858bcb3e1a/jupyterlab_server-0.2.0-py3-none-any.whl...Removing intermediate container 7b9ddfe995a3---> 1bb94afa4112Step 4/4 : RUN pip3 install matplotlib---> Running in a57b4eaa39f6Collecting matplotlibDownloading https://files.pythonhosted.org/packages/89/61/465fb3bfba684b0f53b5c4829c3c89e86e6fe9fdcdfda93e38f1788090f0/matplotlib-3.0.3-cp35-cp35m-manylinux1_x86_64.whl (13.0MB)Collecting cycler>=0.10 (from matplotlib)Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl...Removing intermediate container a57b4eaa39f6---> d34a43f406cfSuccessfully built d34a43f406cfSuccessfully tagged janakiramm/mxnet:latest
We can verify the image creation by running docker images command.
$ docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEjanakiramm/mxnet latest d34a43f406cf 2 minutes ago 742MBmxnet/python latest 121567f56cd1 6 days ago 551MB
Before we can access the custom container image from Gradient, we need to push it a container registry such as Docker Hub.
Login to Docker Hub, and then push the image.
$ docker loginAuthenticating with existing credentials...Login Succeeded
$ docker push $USERNAME/mxnet:latestThe push refers to repository [docker.io/janakiramm/mxnet]8b622d1d381a: Pushed4569983d4e90: Pushed4af26e82c9e0: Pushed5d5ab40659c9: Layer already exists1bd27f114801: Layer already existsf5ea869d942f: Layer already exists26d13f8f529c: Layer already existsd908c11b1a82: Layer already exists7ccfaa7554e3: Layer already exists89ec57aea3bf: Layer already existsa0c1e01578b7: Layer already existslatest: digest: sha256:8e525774d076a84935348539740418bf4726ce349685733d8787e49c990851e2 size: 2623
In the next step, we will launch a Jupyter Notebook based on the container image we just built.
Access the Notebooks from the left navigation bar of Gradient. Choose Custom Container to pull the Apache MXNet image that we built.
Under the container name, type the full container image name including the tag. If you are accessing a private registry, populate the username and password sections.
Provide the command that you would want to run along with the user context. These settings are highly dependent on the way the container is built. Since we didn’t include a default run command within the image, we are instructing Gradient to launch a Jupyter Notebook.
Follow the remaining steps involved in launching a Gradient Notebook. Refer to the Notebook tutorial for a step-by-step guide.
After the Notebook instance is launched successfully, click on the URL to access it.
You can upload the Notebook,
MNIST.ipynb, cloned from the Github repo.
You can now train and test the CNN model based Lenet architecture trained with the popular MNIST dataset.
Gradient is an ML PaaS that removes the friction involved in configuring pipelines for data science and machine learning. Similar to a PaaS, developers, and data scientists can upload data and code to Gradient to train sophisticated models. The scalable infrastructure of Gradient can be used for model serving.
The custom containers feature of Gradient allows developers to add additional runtimes, frameworks, and tools to the platform.