pyspark-project-example

A simple example for PySpark based project.

Prerequisites

JDK (>=7) (as the local test downloads Spark 1.6.0, which dropped JDK 6 support)
Python PEP 8 (for checking code-style in local test)
```
pip install pep8
```
Python Py4J == 0.9.0 (for initiating Spark in local test)
```
pip install -r requirements.txt
```

Test

For test in your local, just run below:
```
./dev/run_tests
```
This contains some codes to download Spark 1.6.0 to run tests. So, it is okay to run the script without some specific modifications.

Warnings

As currently PySpark cannot be installed via pip install due to this issue, Spark has to be installed and SPARK_HOME="your Spark path" and PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH should be set in order to use this library outside after building.