aula1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Curso de Python do Maurão\n",
    "#### Aula 1 - Análise de dados com Python\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMPORTANTE: Quem estiver interessado em uma versão \"turbo\" dessa aula pode ir no [git do chicolucio e pegar](https://github.com/chicolucio-forks/curso_data_science_pro/blob/master/explore/01_exploring_iris.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  **Objetivos:** \n",
    "\n",
    " - Usar os recursos de análise de covariância em uma base de dados conhecida\n",
    " - Explorar as funcionalidades de gráficos de diferentes \n",
    " - Análise de correlação\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Descrição dos dados utilizados\n",
    "\n",
    "Essa base de dados (chamada Iris) foi coletada em 1936 e publicada num paper descrevendo-a, bem como a um modelo linear que foi elaborado para descrevê-la.\n",
    "\n",
    "Mais informações: https://en.wikipedia.org/wiki/Iris_flower_data_set\n",
    "\n",
    "O dado consiste de 150 observações, um terço de cada espécie, e serão feitas várias análises abaixo.\n",
    "\n",
    "Referências:\n",
    "\n",
    "https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Machine+Learning+R/iris-machinelearning.png\n",
    "https://machinelearningmastery.com/\n",
    "\n",
    "https://www.kaggle.com/benhamner\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Carregando as bibliotecas a serem usadas (instaladas pelo Anaconda)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "\n",
    "from pandas.plotting import scatter_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn import model_selection\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.svm import SVC\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2) Configurações do ambiente"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "sns.set(style=\"white\", color_codes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3) Obtendo os dados a serem analisados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "diretorio = r\"..\\data\\iris.csv\"\n",
    "iris = pandas.read_csv(diretorio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4) Espiando ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4) Usando groupby para analisar as classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.groupby('species').size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5) Boxplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6) Histogramas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.hist()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7) Matrix scatter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scatter_matrix(iris)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scatter_matrix(iris, diagonal='kde')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8) Analisando mais de perto uma das correlações"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.plot(kind=\"scatter\", x=\"sepal-length\", y=\"sepal-width\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9) Usando a seaborn, outra lib de gráficos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.jointplot(x=\"sepal-length\", y=\"sepal-width\", data=iris, height=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10) Colorindo os grupos, adicionando legenda..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.FacetGrid(iris, hue=\"species\", size=6) \\\n",
    "   .map(plt.scatter, \"sepal-length\", \"sepal-width\") \\\n",
    "   .add_legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11) Comparando-se os boxplots, dá pra chegar a uma conclusão interessante..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(x=\"species\", y=\"petal-length\", data=iris)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "12) Adicionando os pontos ao gráfico. Como são poucos pontos, isso faz sentido.O parâmetro \"jitter\" tira os pontos da mesma linha, melhorando a visualização."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = sns.boxplot(x=\"species\", y=\"petal-length\", data=iris)\n",
    "ax = sns.stripplot(x=\"species\", y=\"petal-length\", data=iris, jitter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "13) E elas estão de volta!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.violinplot(x=\"species\", y=\"petal-length\", data=iris, size=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "13) Pode-se usar a estimativa de densidade de núcleo (kde) para analisar os dados. A mesma conclusão do item 11 pode ser evidenciada aqui."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.FacetGrid(iris, hue=\"species\", size=6) \\\n",
    "   .map(sns.kdeplot, \"petal-length\") \\\n",
    "   .add_legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "14) Esse é o pairplot, que mostra a relação de bivariância entre cada par de variaveis do dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(iris, hue=\"species\", size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "15) Pode-se adicionar uma regressão linear aos plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(iris, kind=\"reg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(iris, hue=\"species\", kind=\"reg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Machine learning\n",
    "\n",
    "De agora em diante, vamos explorar recursos de machine learning para tentar entender as relações entre as variáveis a classe das espécies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "16) Salvando os valores a serem processados nas analises. A biblioteca scikit-learn usa numpy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array = iris.values\n",
    "print(type(array))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(array.shape)\n",
    "print(array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "16) Isolando Iris-setosa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_target = array[:,4]\n",
    "print(raw_target)\n",
    "y = np.where(raw_target == 'Iris-setosa', 1, 0)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = array[:,0:4]\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "17) Algoritmos de machine learning normalmente exigem que se tenha um conjunto de dados para \"treinar\" o modelo e outro para testar. Isso será feito abaixo. O parâmetro \"Seed\" indica o ponto de partida da \"aleatoridade\" da escolha. Ao fixá-lo garantimos que a escolha será sempre a mesma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_size = 0.20\n",
    "seed = 7\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    model_selection.train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "    \n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)\n",
    "print(X_train[:10])\n",
    "print(X_test[:10])\n",
    "print(y_train[:10])\n",
    "print(y_test[:30])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "18) Criando o modelo, validando o resultado"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "cm = confusion_matrix(y_pred,y_test)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "19) Iris-virinica e Iris-versicolor se sobrepõem parcialmente. Vamos tentar modelar a Iris-virginica."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_target = array[:,4]\n",
    "y = np.where(raw_target == 'Iris-virginica', 1, 0)\n",
    "print(raw_target)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = array[:,0:4]\n",
    "test_size = 0.20\n",
    "seed = 7\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    model_selection.train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "cm = confusion_matrix(y_pred,y_test)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "20) E agora a Iris-versicolor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_target = array[:,4]\n",
    "y = np.where(raw_target == 'Iris-versicolor', 1, 0)\n",
    "print (y)\n",
    "X = array[:,0:4]\n",
    "test_size = 0.20\n",
    "seed = 7\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    model_selection.train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "cm = confusion_matrix(y_pred,y_test)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Talvez a previsão ruim das últimas tentativas deva-se ao fato do modelo tentar separar os grupos através de uma linha reta, o que funcionaria para Iris-setosa mas não para as outras espécies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "21) Vamos tentar um outro modelo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = array[:,0:4]\n",
    "raw_target = array[:,4]\n",
    "y = np.where(raw_target == 'Iris-versicolor', 1, 0)\n",
    "test_size = 0.20\n",
    "seed = 7\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    model_selection.train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "model = KNeighborsClassifier(n_neighbors=15, weights='distance')\n",
    "model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "cm = confusion_matrix(y_pred,y_test)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Muito melhor, né? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "22) Talvez o modelo logístico tenha falhado porque a classe Iris-setosa tenha \"empurrado\" as outras classes uma em direção à outra. Vamos remover a Iris-setosa e tentar criar um novo modelo logístico."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "small_array = array[array[:,4]!='Iris-setosa']\n",
    "print('Shape of small_array: ', small_array.shape)\n",
    "print(small_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "23) Aplicando novamente o modelo linear, agora com uma amostra de treinamento maior, porque são menos dados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = small_array[:,0:4]\n",
    "raw_target = small_array[:,4]\n",
    "y = np.where(raw_target == 'Iris-versicolor', 1, 0)\n",
    "\n",
    "test_size = 0.30\n",
    "\n",
    "seed = 7\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    model_selection.train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "cm = confusion_matrix(y_pred,y_test)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "24) Tentando um outro modelo não linear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "raw_target = array[:,4]\n",
    "y = np.where(raw_target == 'Iris-versicolor', 1, 0)\n",
    "print(y)\n",
    "X = array[:,0:4]\n",
    "test_size = 0.20\n",
    "seed = 7\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    model_selection.train_test_split(X, y, test_size=test_size, random_state=seed)\n",
    "model = SVC()\n",
    "model.fit(X_train,y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "cm = confusion_matrix(y_pred,y_test)\n",
    "print(cm)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}