01_data_collection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "O7WSRPgkFC4O"
      ],
      "authorship_tag": "ABX9TyOPGFR4ds/XJPitz8/fG1kn",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Blignaut24/Braai-Gift-Co-Website/blob/master/01_data_collection.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Collection Notebook"
      ],
      "metadata": {
        "id": "7fXQNSoB8HJi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Objectives\n",
        "* Import bulldozers data from Kaggle and save as raw data\n",
        "* Inspect and save in folder: outputs/datasets/collect/"
      ],
      "metadata": {
        "id": "iE__YwUUAkuq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inputs\n",
        "* JSON file containing Kaggle authentication token"
      ],
      "metadata": {
        "id": "Cfn5twTBAs5S"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Outputs\n",
        "* Generate Dataset: output/dataset/collection/"
      ],
      "metadata": {
        "id": "txAhQ5NEA97j"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Additional Comments\n",
        "\n",
        "The data Kaggle provides for this project is split into three parts:\n",
        "* **Train.csv** is the training set, which contains data through the end of 2011.\n",
        "* **Valid.csv** is the validation set, which contains data from January 1, 2012 - April 30, 2012.\n",
        "* **Test.csv** is the test set. It contains data from May 1, 2012 - November 2012.\n",
        "\n",
        "The key fields are in train.csv are:\n",
        "* **SaleID**: the unique identifier of the sale.\n",
        "* **MachineID**: the unique identifier of a machine. A machine can be sold multiple times.\n",
        "* **saleprice**: what the machine sold for at auction (only provided in train.csv)\n",
        "* **saledate**: the data of the sale\n",
        "\n",
        "Every machine is unique and can be set up in different ways, which we call **\"configurations\"**. Sometimes, certain types of machines don't have all the options, so there might be missing info. Also, the details about what options a machine has and how much it's been used might not always be reliable. There's a file called **\"machine_appendix.csv\"** that gives more details about each machine, like when it was made, what model it is, and what it's used for. Each machine has a special ID number that we can use to find it in all the different data lists for the project.\n"
      ],
      "metadata": {
        "id": "9zj138bUCeU7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---"
      ],
      "metadata": {
        "id": "0syjwa_KHXXj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install Python packages in notebooks"
      ],
      "metadata": {
        "id": "q2p8VN0aHuJY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# %pip install -r /workspace/BulldozerSalePredictor/requirements.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QLFYVPBgH5wl",
        "outputId": "50581a3b-1fff-4328-c6c6-93b8f306a2db"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[31mERROR: Could not open requirements file: [Errno 2] No such file or directory: '/workspace/churnometer/requirements.txt'\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Fetch data from Kaggle"
      ],
      "metadata": {
        "id": "O7WSRPgkFC4O"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://github.com/mrdbourke/zero-to-mastery-ml/raw/master/data/bluebook-for-bulldozers.zip # download files from GitHub as zip\n",
        "\n",
        "import os\n",
        "import zipfile\n",
        "\n",
        "local_zip = 'bluebook-for-bulldozers.zip'\n",
        "zip_ref = zipfile.ZipFile(local_zip, 'r')\n",
        "\n",
        "zip_ref.extractall('.') # extract all data into current working directory\n",
        "zip_ref.close()"
      ],
      "metadata": {
        "id": "a78gH9MjF5zd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f224decf-614f-4761-bb82-7a32b80da2f7"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-05-08 07:26:20--  https://github.com/mrdbourke/zero-to-mastery-ml/raw/master/data/bluebook-for-bulldozers.zip\n",
            "Resolving github.com (github.com)... 140.82.112.4\n",
            "Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/bluebook-for-bulldozers.zip [following]\n",
            "--2024-05-08 07:26:20--  https://raw.githubusercontent.com/mrdbourke/zero-to-mastery-ml/master/data/bluebook-for-bulldozers.zip\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 72077822 (69M) [application/zip]\n",
            "Saving to: ‘bluebook-for-bulldozers.zip’\n",
            "\n",
            "bluebook-for-bulldo 100%[===================>]  68.74M   187MB/s    in 0.4s    \n",
            "\n",
            "2024-05-08 07:26:21 (187 MB/s) - ‘bluebook-for-bulldozers.zip’ saved [72077822/72077822]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Import Packages"
      ],
      "metadata": {
        "id": "D4jzmgfJF-cn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import sklearn"
      ],
      "metadata": {
        "id": "7Gwcp5WSGgrf"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import training and validation sets\n",
        "df= pd.read_csv(\"bluebook-for-bulldozers/TrainAndValid.csv\",\n",
        "               low_memory=False)"
      ],
      "metadata": {
        "id": "KMajGpU5GjW9"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.info()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pfl6bPdkGmj8",
        "outputId": "283b5bd8-31cc-4a52-a63c-df3111ee9b14"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 412698 entries, 0 to 412697\n",
            "Data columns (total 53 columns):\n",
            " #   Column                    Non-Null Count   Dtype  \n",
            "---  ------                    --------------   -----  \n",
            " 0   SalesID                   412698 non-null  int64  \n",
            " 1   SalePrice                 412698 non-null  float64\n",
            " 2   MachineID                 412698 non-null  int64  \n",
            " 3   ModelID                   412698 non-null  int64  \n",
            " 4   datasource                412698 non-null  int64  \n",
            " 5   auctioneerID              392562 non-null  float64\n",
            " 6   YearMade                  412698 non-null  int64  \n",
            " 7   MachineHoursCurrentMeter  147504 non-null  float64\n",
            " 8   UsageBand                 73670 non-null   object \n",
            " 9   saledate                  412698 non-null  object \n",
            " 10  fiModelDesc               412698 non-null  object \n",
            " 11  fiBaseModel               412698 non-null  object \n",
            " 12  fiSecondaryDesc           271971 non-null  object \n",
            " 13  fiModelSeries             58667 non-null   object \n",
            " 14  fiModelDescriptor         74816 non-null   object \n",
            " 15  ProductSize               196093 non-null  object \n",
            " 16  fiProductClassDesc        412698 non-null  object \n",
            " 17  state                     412698 non-null  object \n",
            " 18  ProductGroup              412698 non-null  object \n",
            " 19  ProductGroupDesc          412698 non-null  object \n",
            " 20  Drive_System              107087 non-null  object \n",
            " 21  Enclosure                 412364 non-null  object \n",
            " 22  Forks                     197715 non-null  object \n",
            " 23  Pad_Type                  81096 non-null   object \n",
            " 24  Ride_Control              152728 non-null  object \n",
            " 25  Stick                     81096 non-null   object \n",
            " 26  Transmission              188007 non-null  object \n",
            " 27  Turbocharged              81096 non-null   object \n",
            " 28  Blade_Extension           25983 non-null   object \n",
            " 29  Blade_Width               25983 non-null   object \n",
            " 30  Enclosure_Type            25983 non-null   object \n",
            " 31  Engine_Horsepower         25983 non-null   object \n",
            " 32  Hydraulics                330133 non-null  object \n",
            " 33  Pushblock                 25983 non-null   object \n",
            " 34  Ripper                    106945 non-null  object \n",
            " 35  Scarifier                 25994 non-null   object \n",
            " 36  Tip_Control               25983 non-null   object \n",
            " 37  Tire_Size                 97638 non-null   object \n",
            " 38  Coupler                   220679 non-null  object \n",
            " 39  Coupler_System            44974 non-null   object \n",
            " 40  Grouser_Tracks            44875 non-null   object \n",
            " 41  Hydraulics_Flow           44875 non-null   object \n",
            " 42  Track_Type                102193 non-null  object \n",
            " 43  Undercarriage_Pad_Width   102916 non-null  object \n",
            " 44  Stick_Length              102261 non-null  object \n",
            " 45  Thumb                     102332 non-null  object \n",
            " 46  Pattern_Changer           102261 non-null  object \n",
            " 47  Grouser_Type              102193 non-null  object \n",
            " 48  Backhoe_Mounting          80712 non-null   object \n",
            " 49  Blade_Type                81875 non-null   object \n",
            " 50  Travel_Controls           81877 non-null   object \n",
            " 51  Differential_Type         71564 non-null   object \n",
            " 52  Steering_Controls         71522 non-null   object \n",
            "dtypes: float64(3), int64(5), object(45)\n",
            "memory usage: 166.9+ MB\n"
          ]
        }
      ]
    }
  ]
}