python-foundation.Rmd

---
title: "Python Foundation for Spatial Analysis (Full Course Material)"
author: "Ujaval Gandhi"
subtitle: A gentle introduction to Python programming with a focus on spatial data.
output:
  # pdf_document:
  #   toc: yes
  #   toc_depth: '3'
  #   latex_engine: xelatex 
  html_document:
    df_print: paged
    highlight: pygments
    toc: yes
    toc_depth: 3
    includes:
      after_body: disqus.html
fontsize: 12pt
header-includes:
- \usepackage{fancyhdr}
- \pagestyle{fancy}
- \renewcommand{\footrulewidth}{0.4pt}
- \fancyhead[LE,RO]{\thepage}
- \geometry{left=1in,top=0.75in,bottom=0.75in}
- \fancyfoot[CE,CO]{{\includegraphics[height=0.5cm]{images/cc-by-nc.png}} Ujaval Gandhi
  http://www.spatialthoughts.com}
classoption: a4paper
---
\newpage

***

```{r echo=FALSE, fig.align='center', out.width='75%', out.width='250pt'}
knitr::include_graphics('images/spatial_thoughts_logo.png')
```

***

\newpage

# Introduction 

This class covers Python from the very basics. Suitable for GIS practitioners with no programming background or python knowledge. The course will introduce participants to basic programming concepts, libraries for spatial analysis, geospatial APIs and techniques for building spatial data processing pipelines.

Learn more about my views on why and how to learn Python in this [Introductory Presentation](https://docs.google.com/presentation/d/125LV0qR47S_OGaZZr5Zv-7lbWfdZ5pNo3iFm6IOX7uI/edit?usp=sharing){target="_blank"}.

# Get the Data Package

The code examples in this class use a variety of datasets. All the required datasets and Jupyter notebooks are supplied to you in the ``python_foundation.zip`` file. Unzip this file to a directory - preferably to the ``<home folder>/Downloads/python_foundation/`` folder. 

*Not enrolled in our instructor-led class but want to work through the material on your own?* [Get free access to the data package](https://docs.google.com/forms/d/e/1FAIpQLSeSrQH38dutYRLLy4kOxw_ltpKG_jKkB4aSZc0dRjHhbma0ow/viewform){target="_blank"}

# Installation and Setting up the Environment 

There are many ways to install Python on your system. Many operating systems comes with a Python distribution built-in too. If you use software such as QGIS - it comes with its own version of Python. Working on Python projects typically require installing third-party packages (i.e. tools). As these packages have different dependencies - they often cause conflicts and errors due to different versions of Python on your system. 

An easy and reliable way to get a Python installation on your system is via [Anaconda](https://www.anaconda.com/). For this course, we will use the Anaconda3 Individual Edition to install Python and required spatial analysis packages.

> Many Python tool names have references to the reptile world. The default Python package manager is called [Pip](https://pypi.org/project/pip/) which references the process of hatching eggs. Interestingly, the naming of the Python language itself had [nothing to do with reptiles](https://en.wikipedia.org/wiki/Guido_van_Rossum#Python).

There are 2 ways to install Anaconda and the required packages. 

1. **Graphical User Interface (GUI)**: This method uses the *Anaconda Navigator* program which provides a user-friendly interface to create environments and install packages. This is the preferred way for beginners.
2. **Command Line Interface (CLI)**: Anaconda also provides the `conda` command that can be run from the *Command Prompt* on Windows or the *Terminal* on Mac/Linux. This method is faster but requires some familiarity with the command-line.

You may follow any of the above methods using the instructions below. We recommend you try the GUI method first and if you face problems, you can try setting up the environment using the CLI method. Both methods will result in an identical setup.

## Graphical User Interface (Anaconda Navigator) 

1. [Download the Anaconda Installer](https://www.anaconda.com/products/individual) for Python 3.8 (or a higher version) for your operating system. Once downloaded, double click the installer and install it into the default suggested directory. Select an install for *Just Me* and **use default settings**. 

> Note: If your username has spaces, or non-English characters, it causes problems. In that case, you can install it to a path such as  `C:\anaconda`.

  ```{r echo=FALSE, fig.align='center', out.width='75%'}
  knitr::include_graphics('images/python_foundation/conda1.png')
  ```

2. Once installed, launch the *Anaconda Navigator* program. It is a good practice to create a new *environment* for each of your Python projects. An environment is a space where you will install required packages. Many packages may contain conflicting requirements which could prevent them all being installed into a single environment. Having a separate environment isolates your project from such problems. Click on the *Environments* tab.

> Linux users can open a terminal and type `anaconda-navigator` to launch the program.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda2.png')
```

3. Click the *+ Create* button and name the environment as ``python_foundation``. Tick python and leave it to default version. Click *Create*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda3.png')
```

4. Once created, click the *Channels* button. A channel is a repository where packages are hosted. The default channel is good for most purposes. But some packages we require for this class are not available in the default channel, so we need to add another channel. In the *Manage channels* dialog, click *Add* and enter ``conda-forge``. Press *Enter*.

> Learn more about [conda-forge](https://conda-forge.org/docs/user/introduction.html) 

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda4.png')
```

5. Click the *Update Channels* button.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda5.png')
```

6. Once the update is done, search for the package ``geopandas``. This is a package that allows you to work with vector geospatial data in Python. Select the check-box next to the first result and click *Apply* to install the package with all its dependencies.

> If you do not see any results, make sure you have selected *All* in the dropdown selector next to *Channels* instead of *Installed*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda6.png')
```

7. Similarly, search and install the following packages

- ``geopy``
- ``rasterio``
- ``matplotlib``

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda7.png')
```

8. Switch to the *Home* tab. Ensure that you have the *python_foundation* environment selected. We will now install some programs which allow us to write and execute Python scripts. From the home screen, install the ``JupyterLab`` application. Windows users may optionally install the ``CMD.exe prompt`` application.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda8.png')
```

9. After the installation completes, click *Launch* button for the *JupyterLab* application. JupyterLab is a web application that allows you to write, document and run Python code. It allow you to interactively run small bits of code and see the results. It also support a variety of output formats such as charts or maps - making it an ideal platform for data science projects. The name Jupyter comes from the 3 primary programming languages it supports - Julia, Python and R. We will be using Jupyter notebooks for all exercises in this course.

> Linux users can open a terminal and first switch to the correct environment using the command `conda activate python_foundation`. After that, JupyterLab can be opened using the `jupyter-lab` command.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda9.png')
```

10. A new browser tab will open with an instance of JupterLab. Click the *Python 3*  button under *Notebook*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda10.png')
```

11. Enter the following statements in the first cell and click the *Run* button. **If nothing happens - it means your installation was successful!**. If you get an *ImportError*, repeat the installation steps carefully again. If you continue getting errors, try the *Command Line Interface* procure outlined in the next section.

```
import geopandas
import geopy
import rasterio
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/conda11.png')
```

## Command Line Interface (Anaconda Prompt / Terminal) 

This section outlines an alternative installation procedure using the command prompt. If you already installed the packages successfully using the GUI method in the previous section, you may skip this part.

1. [Download the Anaconda Installer](https://www.anaconda.com/products/individual#Downloads) for Python 3.8 (or a higher version) for your operating system. Once downloaded, double click the installer and install it into the default suggested directory. Select an install for *Just Me* and **use default settings**. 

> Note: If your username has spaces, or non-English characters, it causes problems. In that case, you can install it to a path such as  `C:\anaconda`.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli1.png')
```
  
2. Once installed, launch the *Anaconda Prompt* program from the Windows Start Menu or the *Terminal* app on Mac/Linux.  It is a good practice to create a new *environment* for each of your Python projects. An environment is a space where you will install required packages. Many packages may contain conflicting requirements, preventing them from all being installed into a single environment. Having a separate environment isolates your project from such problems. Enter the command below and pres *Enter* to create your new environment.

```
conda create --name python_foundation
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli2.png')
```

3. Now you will be asked for confirmation; press *y + Enter* to proceed and complete your environment creation.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli3.png')
```

4. Now that the environment is created, you need to activate it. Type the command below and press *Enter*. Once the environment activates, the ``(base)`` will change to ``(python_foundation)``. 

```
conda activate python_foundation
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli4.png')
```

5. Now we are ready to install the required packages using the `conda install` command. First we will install ``geopandas``. This is a module that allows you to work with vector geospatial data in Python. We will use the `conda-forge` channel to install the package. The `-y` option will skip the confirmation dialog. Enter the command below and press *Enter*. 

```
conda install --channel conda-forge geopandas -y
```

> Learn more about [conda-forge](https://conda-forge.org/docs/user/introduction.html) 


```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli5.png')
```

6. If the installation is successful, you should see a message as shown here. 

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli6.png')
```

7. `geopandas` installation is a bit tricky and can sometimes fail. So it is a good idea to check if it can be imported without errors. We will use the `python` command with the `-c` option which will execute a python statement. Enter the below command and press Enter. 

```
python -c "import geopandas"
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli7.png')
```

8. If your geopandas installation was successful, the command will run without any output. **If nothing happens - it means your installation was successful!**

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli8.png')
```

9. We can now install other required packages for this class. Run the command below to install `geopy`, `rasterio`, `matplotlib` and `jupyterlab` packages.

```
conda install --channel conda-forge geopy rasterio matplotlib jupyterlab -y
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli9.png')
```

10. Once the command finishes, you should see a screen such as below.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli10.png')
```

11. Your Python environment is now ready. Launch the *JupyterLab* application using the command below. This will initiate and run a local server in your system and opens in your default browser. 

> Note: Do not close your anaconda prompt after *JupyterLab* opens up. You need to keep it running as long as you want to use JupyterLab.

```
jupyter-lab
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli11.png')
```

12. A new browser tab will open with an instance of JupyterLab. Click the *Python 3*  button under *Notebook*.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli12.png')
```

13. Enter the following statements in the first cell and click the *Run* button. **If nothing happens - it means your installation was successful!**. If you get an *ImportError*, repeat the installation steps carefully again.

```
import geopandas
import geopy
import rasterio
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/condacli13.png')
```


# Using Jupyter Notebooks

Your class data package contain multiple Jupyter notebooks containing code and exercises for this class. 

1. Launch the *JupyterLab* application. It will open your Web Browser and load the application in a new tab. From the left-hand panel, navigate to the directory where you extracted the data package.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/setup1.png')
```

2. Jupyter notebooks have a `.ipynb` extensions. Double-click on a notebook file to open it. Code in the notebook is executed cell-by-cell. You can select a cell and click the *Run* button to execute the code and see the output.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/setup2.png')
```

3. At the end of each notebook, you will find an exercise. Before adding a new cell and attempting to complete the exercise, make sure you go to **Run &rarr; Run All Cells** to execute all the code in the notebook. Doing this will ensure all the required variables are avalable to you to use in the exervise.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/setup3.png')
```
\newpage

----

Open the notebook named ``00_hello_world.ipynb``.

----

```{r child='00_hello_world.md'}
```

\newpage


Open the notebook named ``01_variables.ipynb``.

----

```{r child='01_variables.md'}
```

\newpage

Open the notebook named ``02_data_structures.ipynb``.

----

```{r child='02_data_structures.md'}
```

\newpage

Open the notebook named ``03_string_operations.ipynb``.

----

```{r child='03_string_operations.md'}
```

\newpage
Open the notebook named ``04_loops_and_conditionals.ipynb``.

----

```{r child='04_loops_and_conditionals.md'}
```

\newpage

Open the notebook named ``05_functions.ipynb``.

----

```{r child='05_functions.md'}
```

\newpage

Open the notebook named ``06_standard_library.ipynb``.

----

```{r child='06_standard_library.md'}
```

\newpage

Open the notebook named ``07_third_party_modules.ipynb``.

----

```{r child='07_third_party_modules.md'}
```

\newpage

Open the notebook named ``08_using_web_apis.ipynb``.

----

```{r child='08_using_web_apis.md'}
```

\newpage

Open the notebook named ``09_reading_files.ipynb``.

----

```{r child='09_reading_files.md'}
```

\newpage

Open the notebook named ``10_reading_csv_files.ipynb``.

----

```{r child='10_reading_csv_files.md'}
```

\newpage

Open the notebook named ``11_working_with_pandas.ipynb``.

----

```{r child='11_working_with_pandas.md'}
```

\newpage

```{r child='12_working_with_geopandas.md'}
```

\newpage

Open the notebook named ``13_creating_spatial_data.ipynb``.

----

```{r child='13_creating_spatial_data.md'}
```

\newpage

Open the notebook named ``14_introduction_to_numpy.ipynb``.

----

```{r child='14_introduction_to_numpy.md'}
```

\newpage

Open the notebook named ``15_working_with_rasterio.ipynb``.

----

```{r child='15_working_with_rasterio.md'}
```

\newpage

# Writing Standalone Python Scripts

So far we have used Jupyter Notebooks to write and execute Python code. A notebook is a great choice to interactively explore, visualize and document workflows. But they are not suited for writing scripts for automation. If you have tasks that are long running or want to execute certain tasks on a schedule, you have to write scripts in a standalone `.py` file and run it from a Terminal or Console.

## Get a Text Editor

Any kind of software development requires a good text editor. If you already have a favorite text editor or an IDE (Integrated Development Environment), you may use it for this course. Otherwise, each platform offers a wide variety of free or paid options for text editors. Choose the one that fits your needs.

Below are my recommendations editors that are simple to use for beginners.

- Windows: [Notepad++](https://notepad-plus-plus.org/downloads/) is a good free editor for windows. Download and install the Notepad++ editor. Tip: Before writing Python code in Notepad+++, make sure to go to Settings &rarr; Preferences &rarr; Tab Settings and enable `Replace by space`. Python is very sensitive about whitespace and this setting will ensure tabs and spaces are treated properly.
- Mac: [TextMate](https://macromates.com/) is an open-source editor for Mac that is currently available for free.

## Writing a Script

Copy the following code and paste it in your text editor. Browse to the data package directory and save the file as `get_distance.py`. Make sure that there is no `.txt` extension at the end. 

```{python eval=FALSE, code=readLines('code/python_foundation/get_distance.py')}
```

## Executing a Script

### Windows

1. Launch the *Anaconda Navigator* program. Select the `python_foundation` environment and launch the `CMD.exe Prompt` program.

```{r echo=FALSE, fig.align='center', out.width='75%', out.width='80%'}
knitr::include_graphics('images/python_foundation/win_script1.png')
```

2. Navigate to the directory containing the script using the `cd` command.

```
cd Downloads\python_foundation
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/win_script2.png')
```

3. Run the script using the `python` command. The script will run and print the distance.

```
python get_distance.py
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/win_script3.png')
```

### Mac and Linux

1. Open a Terminal Window.

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/mac_script1.png')
```

2. Switch to the correct conda environment. 

```
conda activate python_foundation
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/mac_script2.png')
```

3. Navigate to the directory containing the script using the `cd` command.

```
cd Downloads/python_foundation
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/mac_script3.png')
```

4. Run the script using the `python` command. The script will run and print the distance.

> If you have multiple python installations on your system, you will have to pick the right Python binary. If the command fails, try `python3.7` instead of just `python` in the command below. The script will run and print the distance.

```
python get_distance.py
```

```{r echo=FALSE, fig.align='center', out.width='75%'}
knitr::include_graphics('images/python_foundation/mac_script4.png')
```

\newpage

# What next?

To achieve mastery in programming, you need to practice writing code and solve many diverse type of problems. I strongly recommend picking up a project and implementing it in Python to improve your skills. You can also take other courses to learn aspects of Python and spatial data processing.

## Do a Project

I have some recommendation for projects that are a good fit for beginners, but are still challenging to help you become proficient. You can try to work on as many of the following problems as you like.

- **Geocoding**: Find list of addresses and geocode them to create a point layer. Here's a good overview of [geocoding using geopy](https://towardsdatascience.com/geocode-with-python-161ec1e62b89)
- **Network Analysis**: Use the [osmnx package](https://github.com/gboeing/osmnx) to do network analysis with OpenStreetMap data and Python. Start by looking at [examples](https://github.com/gboeing/osmnx-examples/tree/master/notebooks)
- **Visualization and Interactive Mapping**: Use the plotting library [geoplot](https://residentmario.github.io/geoplot/index.html) or mapping library [folium](https://python-visualization.github.io/folium/) to create an interactive map. See this [example](https://medium.com/analytics-vidhya/interactive-choropleth-map-in-python-using-folium-4e1479d9e568)
- **Geo-processing**: I have several intermediate level geoprocessing tutorials that use QGIS. You can take any of the following tutorials and implement it using libraries such as *geopandas* and *rasterio*.
    - [Performing Table Joins](https://www.qgistutorials.com/en/docs/3/performing_table_joins.html) 
    - [Performing Spatial Joins](https://www.qgistutorials.com/en/docs/3/performing_spatial_joins.html) 
    - [Performing Spatial Queries](https://www.qgistutorials.com/en/docs/3/performing_spatial_queries.html)
    - [Nearest Neighbor Analysis](https://www.qgistutorials.com/en/docs/3/nearest_neighbor_analysis.html)
    - [Sampling Raster Data using Points or Polygons](https://www.qgistutorials.com/en/docs/3/sampling_raster_data.html)

You may want to check out this [list of useful python geo-spatial libraries](https://github.com/sacridini/Awesome-Geospatial/blob/master/README.md#python) for your project.


## Keep Learning

Here are some recommendations for courses you could take after you have completed this course.

* University of Helsinki's Free [Geo-Python Course](https://geo-python.github.io/site/)
* University of Helsinki's Free [Automating GIS Processes Course](https://automating-gis-processes.github.io/site/)
* Kaggle's Free [Python Courses](https://www.kaggle.com/learn), including [Geospatial Analysis](https://www.kaggle.com/learn/geospatial-analysis)
* Allen Downey's Free [Elements of Data Science Course](https://allendowney.github.io/ElementsOfDataScience/)
* [Doing Geospatial in Python](https://geopython.github.io/geopython-workshop/) online workshop. Covers some more geospatial libraries, with a focus on OGC services and metadata.

\newpage

# Supplement

----

```{r child='supplement1_plotting.md'}
```
[View the Notebook &#8599;](https://nbviewer.jupyter.org/github/spatialthoughts/courses/blob/master/code/python_foundation/supplement1_plotting.ipynb)


```{r child='supplement2_working_with_xarray.md'}
```
[View the Notebook &#8599;](https://nbviewer.jupyter.org/github/spatialthoughts/courses/blob/master/code/python_foundation/supplement2_working_with_xarray.ipynb)


\newpage

# Data Credits
* [World Cities Database](https://simplemaps.com/data/world-cities): Basic database. Copyright SimpleMaps.com 2010-2020
* [Geonames](https://www.geonames.org) places database for USA, Canada and Mexico
* Karnataka Admin Boundaries: Downloaded from [Datameet Spatial Data repository](https://github.com/datameet/maps/tree/master/States).
* OpenStreetMap (osm) data layers: Data/Maps Copyright 2019 Geofabrik GmbH and OpenStreetMap Contributors. [OSM India free extract](https://download.geofabrik.de/asia/india.html) downloaded from Geofabrik.
* Bangalore Sentinel-2 Imagery. Downloaded from [Copernicus Open Access Hub](https://scihub.copernicus.eu/). Copyright European Space Agency - ESA.
* NASA Shuttle Radar Topography Mission (SRTM) Elevation Dataset. Downloaded from [30m SRTM Tile Downloader](https://dwtkns.com/srtm30m/]).


# License


This course material is licensed under a [Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/). You are free to use the material for any non-commercial purpose. Kindly give appropriate credit to the original author.

If you would like to utilize these materials as part of a commercial offering, you can purchase a *Trainer License* for a small fee. 

Please [contact us](https://spatialthoughts.com/contact/) for pricing and terms.
<!--
This material is part of the *Trainer's Package* and the buyer gets a non-exclusive, non-transferable, perpetual license to the material. You can fully customise and brand the materials to your requirements.
-->

&copy; 2020 Ujaval Gandhi [www.spatialthoughts.com](https://spatialthoughts.com)

***

**This course is offered as an instructor-led online class. Visit [Spatial Thoughts](https://spatialthoughts.com/events/) to know details of upcoming sessions.**