Skip to content
This repository was archived by the owner on Dec 2, 2021. It is now read-only.

Commit ad51891

Browse files
bkinmanbkinman
committed
Initial commit
0 parents  commit ad51891

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+5171
-0
lines changed

‎.gitignore

+105
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
.idea/
6+
7+
# C extensions
8+
*.so
9+
10+
# Distribution / packaging
11+
.Python
12+
env/
13+
build/
14+
develop-eggs/
15+
dist/
16+
downloads/
17+
eggs/
18+
.eggs/
19+
lib/
20+
lib64/
21+
parts/
22+
sdist/
23+
var/
24+
wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
29+
# PyInstaller
30+
# Usually these files are written by a python script from a template
31+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
32+
*.manifest
33+
*.spec
34+
35+
# Installer logs
36+
pip-log.txt
37+
pip-delete-this-directory.txt
38+
39+
# Unit test / coverage reports
40+
htmlcov/
41+
.tox/
42+
.coverage
43+
.coverage.*
44+
.cache
45+
nosetests.xml
46+
coverage.xml
47+
*.cover
48+
.hypothesis/
49+
50+
# Translations
51+
*.mo
52+
*.pot
53+
54+
# Django stuff:
55+
*.log
56+
local_settings.py
57+
58+
# Flask stuff:
59+
instance/
60+
.webassets-cache
61+
62+
# Scrapy stuff:
63+
.scrapy
64+
65+
# Sphinx documentation
66+
docs/_build/
67+
68+
# PyBuilder
69+
target/
70+
71+
# Jupyter Notebook
72+
.ipynb_checkpoints
73+
74+
# pyenv
75+
.python-version
76+
77+
# celery beat schedule file
78+
celerybeat-schedule
79+
80+
# SageMath parsed files
81+
*.sage.py
82+
83+
# dotenv
84+
.env
85+
86+
# virtualenv
87+
.venv
88+
venv/
89+
ENV/
90+
91+
# Spyder project settings
92+
.spyderproject
93+
.spyproject
94+
95+
# Rope project settings
96+
.ropeproject
97+
98+
# mkdocs documentation
99+
/site
100+
101+
# mypy
102+
.mypy_cache/
103+
104+
RUNS/
105+

‎README.md

+158
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
[![Udacity - Robotics NanoDegree Program](https://s3-us-west-1.amazonaws.com/udacity-robotics/Extra+Images/RoboND_flag.png)](https://www.udacity.com/robotics)
2+
3+
## Deep Learning Project ##
4+
5+
In this project, you will train a deep neural network to identify and track a target in simulation and then issue commands to a drone to follow that target. So-called “follow me” applications like this are key to many fields of robotics and the very same techniques you apply here could be extended to scenarios like advanced cruise control in autonomous vehicles or human-robot collaboration in industry.
6+
7+
[image_0]: ./docs/misc/sim_screenshot.png
8+
![alt text][image_0]
9+
10+
## Setup Instructions
11+
**Clone the repository**
12+
```
13+
$ git clone https://github.com/udacity/RoboND-DeepLearning-Private.git
14+
```
15+
16+
**Download the QuadSim binary**
17+
18+
To interface your neural net with the QuadSim simulator, you must use a version QuadSim that has been custom tailored for this project. The previous version that you might have used for the Controls lab will not work.
19+
20+
The simulator binary can be downloaded [here](https://github.com/udacity/RoboND-DeepLearning-Private/releases)
21+
22+
**Install Dependencies**
23+
24+
You'll need Python 3 and Jupyter Notebooks installed to do this project. The best way to get setup with these if you are not already is to use Anaconda following along with the [RoboND-Python-Starterkit](https://github.com/ryan-keenan/RoboND-Python-Starterkit).
25+
26+
If for some reason you choose not to use Anaconda, you must install the following framworks and packages on your system:
27+
* Python 2.7
28+
* Tensorflow 1.2.1
29+
* NumPy 1.11
30+
* OpenCV 2
31+
* SciPy 0.17.0
32+
* eventlet
33+
* Flask
34+
* h5py
35+
* PIL
36+
* python-socketio
37+
* scikit-image
38+
* socketIO-client
39+
40+
## Implement the Segmentation Network
41+
1. Download the training dataset from [here](https://github.com/udacity/RoboND-DeepLearning-Private/releases), and extract to the project `data` directory.
42+
2. Complete `make_model.py`by following the TODOs in `make_model_template.py`
43+
3. Complete `data_iterator.py` by following the TODOs in `data_iterator_template.py`
44+
4. Complete `train.py` by following the TODOs in `train_template.py`
45+
5. Train the network locally, or on [AWS](docs/aws_setup.md).
46+
6. Continue to experiement with the training data and network until you attain the score you desire.
47+
7. Once you are comfortable with performance on the training dataset, see how it performs in live simulation!
48+
49+
## Collecting Training Data ##
50+
A simple training dataset has been provided in the [releases](https://github.com/udacity/RoboND-DeepLearning-Private/releases) section of this repository. This dataset will allow you to verify that you're segmentation network is semi-functional. However, if you're interested in improving your score, you may be interested in collecting additional training data. To do, please see the following steps.
51+
52+
The data directory is organized as follows:
53+
```
54+
data/runs - contains the results of prediction runs
55+
data/train/images - contains images for the training set
56+
data/train/masks - contains masked (labeled) images for the training set
57+
data/validation/images - contains images for the validation set
58+
data/validation/masks - contains masked (labeled) images for the validation set
59+
data/weights - contains trained TensorFlow models
60+
```
61+
62+
### Training Set: with Hero Present ###
63+
1. Run QuadSim
64+
2. Select `Use Hero Target`
65+
3. Select `With Other Poeple`
66+
4. Click the `DL Training` button
67+
5. With the simulator running, press "r" to begin recording.
68+
6. In the file slection menu navigate to the `data/train/target/hero_train1` directory
69+
7. **optional** to speed up data collection, press "9" (1-9 will slow down collection speed)
70+
8. When you have finished collecting data, hit "r" to stop recording.
71+
9. To exit the simulator, hit "`<esc>`"
72+
73+
### Training Set: without Hero Present ###
74+
1. Run QuadSim
75+
2. Make sure `Use Hero Target` is **NOT** selected
76+
3. Select `With Other Poeple`
77+
4. Click the `DL Training` button
78+
5. With the simulator running, press "r" to begin recording.
79+
6. In the file slection menu navigate to the `data/train/non_target/run_train1` directory.
80+
7. **optional** to speed up data collection, press "9" (1-9 will slow down collection speed)
81+
8. When you have finished collecting data, hit "r" to stop recording.
82+
9. To exit the simulator, hit "`<esc>`"
83+
84+
### Validation Set ###
85+
To collect the validation set, repeat both sets of steps above, except using the directory `data/validation` instead rather than `data/train`.
86+
87+
### Image Preprocessing ###
88+
Before the network is trained, the images first need to be undergo a preprocessing step.
89+
**TODO**: Explain what preprocessing does, approximately.
90+
To run preprocesing:
91+
```
92+
$ python preprocess_ims.py
93+
```
94+
**Note**: If your data is stored as suggested in the steps above, this script should run without error.
95+
96+
## Training, Predicting and Scoring ##
97+
With your training and validation data having been generated (or downloaded from the [releases](https://github.com/udacity/RoboND-DeepLearning-Private/releases) section of this repository, you are free to begin working with the neural net.
98+
99+
**Note**: Training CNNs is a very compute-intensive process. If your system does not have a recent Nvidia graphics card, with [cuDNN](https://developer.nvidia.com/cudnn) installed , you may need to perform the training step in the cloud. Instructions for using AWS to train your network in the cloud may be found [here](docs/aws_setup.md)
100+
101+
### Training your Model ###
102+
**Prerequisites**
103+
- Net has been implemented as per these instructions
104+
- Training data is in `data` directory
105+
106+
To train, simply run the training script, `train.py`, giving it the name of the model weights file as a parameter:
107+
```
108+
$ python train.py my_amazing_model.h5
109+
```
110+
After the trianing run has completed, your model will be stored in the `data/weights` directory as an [HDF5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) file.
111+
112+
### Predicting on the Validation Set and Evaluating the Results ###
113+
**Prerequisites**
114+
-Model has been trained
115+
-Validation set has been collected
116+
117+
Once the network has been trained, you can run inference on the validation set using `predict.py`. This script requires two parameters, the name of the model file you wish to perform prediction with, and the output directory where you would like to store your prediction results.
118+
119+
```
120+
$ python predict.py my_amazing_model.h5 my_prediction_run
121+
```
122+
123+
For the prediction run above, the results will be stored in `data/runs/my_prediction_run`.
124+
125+
To get a sense of the overall performance of the your net on the prediction set, you can use `evaluation.py` as follows:
126+
127+
```
128+
$ python evaluate.py validation my_prediction_run
129+
average intersection over union 0.34498680536
130+
number of validation samples evaluated on 1000
131+
number of images with target detected: 541
132+
number of images false positives is: 4
133+
average squared pixel distance error 11.0021170157
134+
average squared log pixel distance error 1.4663195103
135+
```
136+
137+
## Scoring ##
138+
**TODO**
139+
140+
**How the Final score is Calculated**
141+
142+
**TODO**
143+
144+
**Ideas for Improving your Score**
145+
146+
**TODO**
147+
148+
**Obtaining a leaderboard score**
149+
150+
**TODO**
151+
152+
## Experimentation: Testing in Simulation
153+
1. Copy your saved model to the weights directory `data/weights`.
154+
2. Launch the simulator, select "Spawn People", and then click the "Follow Me" button.
155+
3. Run `server.py` to launch the socketio server.
156+
4. Run the relatime follower script `$ realtime_follower.py my_awesome_model`
157+
158+
**Note:** If you'd like to see an overlay of the detected region on each camera frame from the drone, simply pass the `--overlay_viz` parameter to `realtime_follower.py`

‎code/build_model.py

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
import model_builder
2+
3+
# run this file with 'python model_builder.py' to generate plots of network defined in network config
4+
if __name__ == '__main__':
5+
net = model_builder.make_example_network()

‎code/evaluate.py

+62
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
import glob
2+
import numpy as np
3+
import os
4+
import sys
5+
from scipy import misc
6+
7+
from utils import scoring_utils
8+
9+
10+
def score_run(gt_dir, pred_dir):
11+
gt_files = sorted(glob.glob(os.path.join(gt_dir, 'masks', '*.png')))
12+
pred_files = sorted(glob.glob(os.path.join(pred_dir, '*.png')))
13+
14+
iou = 0
15+
error1 = 0
16+
error2 = 0
17+
n_valid_detections = 0
18+
n_invalid_detections = 0
19+
20+
for e, gt_file in enumerate(gt_files):
21+
gt_mask = misc.imread(gt_file)[:, :, 1].clip(0, 1)
22+
pred_mask = (misc.imread(pred_files[e])[:, :, 1] > 127).astype(np.int)
23+
24+
iou += scoring_utils.intersection_over_union(pred_mask, gt_mask)
25+
# only compute centroid distance if it found some object
26+
if pred_mask.sum() > 3:
27+
if gt_mask.sum() > 3:
28+
gt_centroid = scoring_utils.get_centroid_largest_blob(gt_mask)
29+
pred_centroid = scoring_utils.get_centroid_largest_blob(pred_mask)
30+
error1 += scoring_utils.average_squared_distance(pred_centroid, gt_centroid)
31+
error2 += scoring_utils.average_squared_log_distance(pred_centroid, gt_centroid)
32+
n_valid_detections += 1
33+
else:
34+
error1 += 1
35+
error2 += 1
36+
n_invalid_detections += 1
37+
38+
return iou, error1, error2, len(gt_files), n_valid_detections, n_invalid_detections
39+
40+
41+
if __name__ == '__main__':
42+
43+
if len(sys.argv) != 3:
44+
raise ValueError('evaluate.py the ground truth folder name and prediction folder name as commandline input')
45+
46+
gt_folder = sys.argv[1]
47+
pred_folder = sys.argv[2]
48+
49+
gt_path = os.path.join('..', 'data', gt_folder)
50+
pred_path = os.path.join('..', 'data', 'runs', pred_folder)
51+
52+
(iou, err1, err2, n_preds, n_valid_detections, n_invalid_detections
53+
) = score_run(gt_path, pred_path)
54+
55+
print('average intersection over union {}'.format(iou / n_preds))
56+
print('number of validation samples evaluated on {}'.format(n_preds))
57+
print('number of images with target detected: {}'.format(n_valid_detections))
58+
print('number of images false positives is: {}'.format(n_invalid_detections))
59+
if n_valid_detections > 0 or n_invalid_detections > 0:
60+
n_detections = n_valid_detections + n_invalid_detections
61+
print('average squared pixel distance error {}'.format(err1 / n_detections))
62+
print('average squared log pixel distance error {}'.format(err2 / n_detections))

‎code/make_model_template.py

+53
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
from tensorflow.contrib.keras.python import keras
2+
from tensorflow.contrib.keras.python.keras import layers as l
3+
4+
from network_definitions.separable_conv2d import SeparableConv2DKeras
5+
6+
7+
# TODO implement a encode block, it should reduce spatial edge width by half
8+
def encode_block(x, filts, n_blocks=2):
9+
pass
10+
11+
# TODO implement a decode block, it should increase spatial edge_width by half
12+
def decode_block(x_small, x_large, filts, n_blocks=2):
13+
pass
14+
15+
def make_encode_decode_net(inputs, num_classes, n_blocks, base=32, filter_list=None):
16+
"""Flexible semantic segmentation architecture using seperable convolutions"""
17+
18+
if filter_list is None:
19+
filter_list = list()
20+
for i in range(n_blocks):
21+
filter_list.append(base * (2 ** i))
22+
23+
x0 = l.Conv2D(base, 3, activation='relu', padding='same')(inputs)
24+
x0 = l.BatchNormalization()(x0)
25+
26+
encode_layers = list()
27+
for i in range(n_blocks):
28+
if i == 0:
29+
encode_layers.append(encode_block(x0, filter_list[i]))
30+
else:
31+
encode_layers.append(encode_block(encode_layers[i - 1], filter_list[i]))
32+
33+
encode_layers = [x0] + encode_layers
34+
encode_layers = list(reversed(encode_layers))
35+
filter_list = list(reversed(filter_list))
36+
37+
for e, layer in enumerate(encode_layers[1:]):
38+
if e == 0:
39+
x = decode_block(encode_layers[0], layer, filter_list[e])
40+
else:
41+
x = decode_block(x, layer, filter_list[e])
42+
43+
return l.Conv2D(num_classes, 3, activation='sigmoid', padding='same')(x)
44+
45+
46+
def make_example_model():
47+
im_shape = (256, 256, 3)
48+
inputs = keras.layers.Input(im_shape)
49+
out_layer = make_encode_decode_net(inputs, num_classes=3, n_blocks=3, base=16)
50+
51+
model = keras.models.Model(inputs=inputs, outputs=out_layer)
52+
model.compile(optimizer=keras.optimizers.Adam(0.001), loss='binary_crossentropy')
53+
return model

0 commit comments

Comments
 (0)