Exercise6.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Exercise 6. Distributed Training"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Distributed training\n\nNow that the setup is working, we can go to the full dataset with 120 classes. We just need to point to a different path on the datastore. "
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "full_dataset = ds.path('breeds')\nprint(full_dataset)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "## AML Compute\nfrom azureml.train.dnn import PyTorch\n\nscript_params = {\n    '--data_dir': full_dataset.as_mount(),\n    '--num_epochs': 25,\n    '--output_dir': './outputs',\n    '--log_dir': './logs',\n    '--mode': 'fine_tune'\n}\n\nestimator120 = PyTorch(source_directory='.', \n                        script_params=script_params,\n                        compute_target=compute_target, \n                        entry_script='pytorch_train.py',\n                        pip_packages=['tensorboardX'],\n                        node_count=1,\n                        use_gpu=True)\n\nrun120 = experiment.submit(estimator120)\n\nfrom azureml.widgets import RunDetails\nRunDetails(run120).show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "But now training takes very long (> 1 hour), so let's see if we can run this job on multiple GPUs to cut down on training time."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "# first let's cancel the above job\nrun120.cancel()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Running the model on multiple nodes is simple (in this case using Horovod MPI-based algorithm running on 4 nodes)"
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "## AML Compute\nfrom azureml.train.dnn import PyTorch\n\nscript_params = {\n    '--data_dir': full_dataset.as_mount(),\n    '--num_epochs': 25,\n    '--output_dir': './outputs',\n    '--log_dir': './logs',\n    '--mode': 'fine_tune'\n}\n\nestimator120 = PyTorch(source_directory='.', \n                        script_params=script_params,\n                        compute_target=compute_target, \n                        pip_packages=['tensorboardX'],\n                        entry_script='pytorch_train_horovod.py',\n                        node_count=4,\n                        distributed_backend='mpi',\n                        use_gpu=True)\n\nrun120 = experiment.submit(estimator120)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "from azureml.widgets import RunDetails\nRunDetails(run120).show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "from azureml.contrib.tensorboard import Tensorboard\n\n# The Tensorboard constructor takes an array of runs, so be sure and pass it in as a single-element array here\ntb = Tensorboard([run120])\n\n# If successful, start() returns a string with the URI of the instance.\ntb.start()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "tb.stop()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Training on 4 nodes completes in about 25 minutes and achieves 81% accuracy, which is similar to accuracy produced by single node training. This is great improvement of training time."
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.4",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}