Snowpark_For_Python_ML.ipynb

{
 "metadata": {
  "kernelspec": {
   "display_name": "Streamlit Notebook",
   "name": "streamlit"
  }
 },
 "nbformat_minor": 5,
 "nbformat": 4,
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5976cb51-5bad-44ad-88aa-f8ccbe937f00",
   "metadata": {
    "name": "cell8",
    "collapsed": false
   },
   "source": "# Project Overview\n\nPerform data analysis and data preparation tasks to train a Linear Regression model to predict future ROI (Return On Investment) of variable ad spend budgets across multiple channels including search, video, social media, and email using Snowpark for Python, Snowpark ML and Streamlit. By the end of the session, you will have an interactive web application deployed visualizing the ROI of different allocated advertising spend budgets.\n\n***Prerequisite**: Before proceeding with this Notebook, you must first successfully run through Snowpark_For_Python_DE.ipynb.*\n\n## Machine Learning\n\nIn this Notebook, we will focus on Machine Learning in Snowflake using Snowpark for Python.\n\n- Load features and target from Snowflake table into Snowpark DataFrame\n- Prepare features for model training\n- Train ML model using Snowpark ML in Snowflake and upload the model to Snowflake stage\n- Register ML model and use it for inference from Snowpark ML Model Registry\n\nFor environment setup including loading data into Snowflake tables, and step-by-step instructions, please refer to the [QuickStart Guide](https://quickstarts.snowflake.com/guide/getting_started_with_dataengineering_ml_using_snowpark_python/index.html#0)."
  },
  {
   "cell_type": "markdown",
   "id": "a83ffa8f-faea-4e9e-8fbe-296193a3153d",
   "metadata": {
    "name": "cell9",
    "collapsed": false
   },
   "source": "## Import Libraries"
  },
  {
   "cell_type": "code",
   "id": "3775908f-ca36-4846-8f38-5adca39217f2",
   "metadata": {
    "language": "python",
    "name": "cell1",
    "collapsed": false,
    "codeCollapsed": false
   },
   "source": "# Snowpark for Python\nfrom snowflake.snowpark.context import get_active_session\nfrom snowflake.snowpark.version import VERSION\n\n# Snowpark ML\nfrom snowflake.ml.modeling.compose import ColumnTransformer\nfrom snowflake.ml.modeling.pipeline import Pipeline\nfrom snowflake.ml.modeling.preprocessing import PolynomialFeatures, StandardScaler\nfrom snowflake.ml.modeling.linear_model import LinearRegression\nfrom snowflake.ml.modeling.model_selection import GridSearchCV\nfrom snowflake.ml.registry import Registry\nfrom snowflake.ml.version import VERSION as ml_version\n\n# Misc\n#import pandas as pd\nimport json\nimport logging \nlogger = logging.getLogger(\"snowflake.snowpark.session\")\nlogger.setLevel(logging.ERROR)\n\nsession = get_active_session()\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "4fb8edfb-6c71-40cc-bc23-9aa58b056e63",
   "metadata": {
    "name": "cell10",
    "collapsed": false
   },
   "source": "## Features and Target\n\nAt this point we are ready to perform the following actions to save features and target for model training.\n\n- Delete rows with missing values\n- Exclude columns we don't need for modeling\n- Save features into a Snowflake table called MARKETING_BUDGETS_FEATURES"
  },
  {
   "cell_type": "code",
   "id": "8d50cbf4-0c8d-4950-86cb-114990437ac9",
   "metadata": {
    "language": "python",
    "name": "cell2",
    "collapsed": false
   },
   "source": "# Load data\nsnow_df_spend_and_revenue_per_month = session.table('spend_and_revenue_per_month')\n\n# Delete rows with missing values\nsnow_df_spend_and_revenue_per_month = snow_df_spend_and_revenue_per_month.dropna()\n\n# Exclude columns we don't need for modeling\nsnow_df_spend_and_revenue_per_month = snow_df_spend_and_revenue_per_month.drop(['YEAR','MONTH'])\n\n# Save features into a Snowflake table call MARKETING_BUDGETS_FEATURES\nsnow_df_spend_and_revenue_per_month.write.mode('overwrite').save_as_table('MARKETING_BUDGETS_FEATURES')\nsnow_df_spend_and_revenue_per_month.show()",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "6069de28-2e2c-467d-a20f-df1f1a6bb35b",
   "metadata": {
    "name": "cell11",
    "collapsed": false
   },
   "source": "## Model Training using Snowpark ML in Snowflake\n\nLearn more about [Snowpark ML](https://docs.snowflake.com/developer-guide/snowpark-ml/snowpark-ml-modeling).\n\nNOTE: For workloads that require a large amount of memory and compute resources, consider using [Snowpark-Optimized Warehouses](https://docs.snowflake.com/en/developer-guide/snowpark/python/python-snowpark-training-ml#snowpark-optimized-warehouses)."
  },
  {
   "cell_type": "code",
   "id": "c695373e-ac74-4b62-a1f1-08206cbd5c81",
   "metadata": {
    "language": "python",
    "name": "cell3",
    "collapsed": false,
    "codeCollapsed": false
   },
   "source": "CROSS_VALIDATION_FOLDS = 10\nPOLYNOMIAL_FEATURES_DEGREE = 2\n\n# Create train and test Snowpark DataDrames\ntrain_df, test_df = session.table(\"MARKETING_BUDGETS_FEATURES\").random_split(weights=[0.8, 0.2], seed=0)\n\n# Preprocess the Numeric columns\n# We apply PolynomialFeatures and StandardScaler preprocessing steps to the numeric columns\n# NOTE: High degrees can cause overfitting.\nnumeric_features = ['SEARCH_ENGINE','SOCIAL_MEDIA','VIDEO','EMAIL']\nnumeric_transformer = Pipeline(steps=[('poly',PolynomialFeatures(degree = POLYNOMIAL_FEATURES_DEGREE)),('scaler', StandardScaler())])\n\n# Combine the preprocessed step together using the Column Transformer module\npreprocessor = ColumnTransformer(\n    transformers=[\n        ('num', numeric_transformer, numeric_features)])\n\n# The next step is the integrate the features we just preprocessed with our Machine Learning algorithm to enable us to build a model\npipeline = Pipeline(steps=[('preprocessor', preprocessor),('classifier', LinearRegression())])\nparameteres = {}\n\n# Use GridSearch to find the best fitting model based on number_of_folds folds\nmodel = GridSearchCV(\n    estimator=pipeline,\n    param_grid=parameteres,\n    cv=CROSS_VALIDATION_FOLDS,\n    label_cols=[\"REVENUE\"],\n    output_cols=[\"PREDICTED_REVENUE\"],\n    verbose=2\n)\n\n# Fit and Score\nmodel.fit(train_df)\ntrain_r2_score = model.score(train_df)\ntest_r2_score = model.score(test_df)\n\n# R2 score on train and test datasets\nprint(f\"R2 score on Train : {train_r2_score}\")\nprint(f\"R2 score on Test  : {test_r2_score}\")",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "0dc15f0c-8c57-42b7-b57e-e67cf40a3997",
   "metadata": {
    "name": "cell12",
    "collapsed": false
   },
   "source": "## Log Trained Model to Snowflake Model Registry\n\nThe Model Registry allows to store models as objects in a schema in Snowflake. Note that by default the database and schema of the session is used.\n\nLearn more about [Model Registry](https://docs.snowflake.com/developer-guide/snowpark-ml/model-registry/overview)."
  },
  {
   "cell_type": "code",
   "id": "dc22a0e7-83ca-470c-8430-60b5f9e1e4c1",
   "metadata": {
    "language": "python",
    "name": "cell4",
    "collapsed": false
   },
   "outputs": [],
   "source": "registry = Registry(session)\nMODEL_NAME = \"PREDICT_ROI\"",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "9e9e3849-61b9-4902-bac5-53c3f0131c40",
   "metadata": {
    "language": "python",
    "name": "cell13",
    "collapsed": false
   },
   "outputs": [],
   "source": "# NOTE: If you try to log the model with the same name, you may get \"ValueError: (0000) Model PREDICT_ROI version v1 already existed.\" error. \n# If that's the case, uncomment and run this cell.\n\n# registry.delete_model(MODEL_NAME)",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "813b07ec-cf73-44dd-bbf0-261268f73d8e",
   "metadata": {
    "language": "python",
    "name": "cell5",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "mv = registry.log_model(model,\n                        model_name=MODEL_NAME,\n                        version_name=\"v1\",\n                        metrics={\"R2_train\": train_r2_score, \"R2_test\":test_r2_score},\n                        comment='Model pipeline to predict revenue',\n                        options={\"embed_local_ml_library\": True}\n                    )",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "39a15039-8008-42ae-a639-982585087d2d",
   "metadata": {
    "name": "cell14",
    "collapsed": false
   },
   "source": "## View Logged Model in Snowflake Model Registry"
  },
  {
   "cell_type": "code",
   "id": "bd9caea2-4825-4f86-bec7-80b3d9d07103",
   "metadata": {
    "language": "python",
    "name": "cell6",
    "collapsed": false
   },
   "outputs": [],
   "source": "registry.show_models()",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "0ff01596-ce5d-4efb-aeaf-656972f0b84a",
   "metadata": {
    "name": "cell15",
    "collapsed": false
   },
   "source": "## Inference\n\nOnce the model is logged we can use it for inference on new data.\n\nFirst we will create a Snowpark DataFrame with some sample data and then call the logged model to get new predictions. Note: we will handle negative values in our Streamlit application."
  },
  {
   "cell_type": "code",
   "id": "6f01b5c4-3f5b-4384-9051-e15ded939c48",
   "metadata": {
    "language": "python",
    "name": "cell7",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "test_df = session.create_dataframe([[250000,250000,200000,450000],[500000,500000,500000,500000],[8500,9500,2000,500]], \n                                    schema=['SEARCH_ENGINE','SOCIAL_MEDIA','VIDEO','EMAIL'])\nmv.run(test_df, function_name='predict').show()",
   "execution_count": null
  }
 ]
}