Skip to content

Commit

Permalink
Merge pull request openvinotoolkit#605 from DmitriySidnev/ds/update_mcmt
Browse files Browse the repository at this point in the history 
Update multi camera multi person tracker
  • Loading branch information
Roman Donchenko authored Mar 18, 2020
2 parents 55a7696 + 5fcf598 commit 0687878
Show file tree
Hide file tree
Showing 18 changed files with 1,922 additions and 299 deletions.
58 changes: 55 additions & 3 deletions ci/requirements-demos.txt
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,58 @@
absl-py==0.9.0 # via tensorboard
attrs==19.3.0 # via pytest
cachetools==4.0.0 # via google-auth
certifi==2019.11.28 # via requests
chardet==3.0.4 # via requests
cycler==0.10.0 # via matplotlib
entrypoints==0.3 # via flake8
flake8-import-order==0.18.1 # via motmetrics
flake8==3.7.9 # via motmetrics
google-auth-oauthlib==0.4.1 # via tensorboard
google-auth==1.11.3 # via google-auth-oauthlib, tensorboard
grpcio==1.27.2 # via tensorboard
idna==2.9 # via requests
importlib-metadata==1.5.0 # via pluggy, pytest
joblib==0.14.1 # via scikit-learn
nibabel==3.0.1
kiwisolver==1.1.0 # via matplotlib
markdown==3.2.1 # via tensorboard
matplotlib==3.0.3
mccabe==0.6.1 # via flake8
more-itertools==8.2.0 # via pytest
motmetrics==1.2.0
nibabel==3.0.2
numpy==1.18.1 ; python_version >= "3.4"
scikit-learn==0.22.1
oauthlib==3.1.0 # via requests-oauthlib
packaging==20.3 # via pytest
pandas==0.24.2 # via motmetrics
pathlib2==2.3.5 # via pytest
pluggy==0.13.1 # via pytest
protobuf==3.11.3 # via tensorboard, tensorboardx
py-cpuinfo==5.0.0 # via pytest-benchmark
py==1.8.1 # via pytest
pyasn1-modules==0.2.8 # via google-auth
pyasn1==0.4.8 # via pyasn1-modules, rsa
pycodestyle==2.5.0 # via flake8, flake8-import-order
pyflakes==2.1.1 # via flake8
pyparsing==2.4.6 # via matplotlib, packaging
pytest-benchmark==3.2.3 # via motmetrics
pytest==5.4.1 # via motmetrics, pytest-benchmark
python-dateutil==2.8.1 # via matplotlib, pandas
pytz==2019.3 # via pandas
requests-oauthlib==1.3.0 # via google-auth-oauthlib
requests==2.23.0 # via requests-oauthlib, tensorboard
rsa==4.0 # via google-auth
scikit-learn==0.22.2.post1
scipy==1.4.1
tqdm==4.42.0
six==1.14.0 # via absl-py, cycler, google-auth, grpcio, packaging, pathlib2, protobuf, python-dateutil, tensorboard, tensorboardx
tensorboard==2.1.1
tensorboardx==2.0
tqdm==4.43.0
urllib3==1.25.8 # via requests
wcwidth==0.1.8 # via pytest
werkzeug==1.0.0 # via tensorboard
wheel==0.34.2 # via tensorboard
xmltodict==0.12.0 # via motmetrics
zipp==1.2.0 # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools
184 changes: 170 additions & 14 deletions demos/python_demos/multi_camera_multi_person_tracking/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This demo demonstrates how to run Multi Camera Multi Person demo using OpenVINO<

The demo expects the next models in the Intermediate Representation (IR) format:

* Person detection model
* Person detection model (or person instance segmentation model)
* Person re-identification model

It can be your own models or pre-trained model from OpenVINO Open Model Zoo.
Expand All @@ -20,7 +20,8 @@ As input, the demo application takes:

The demo workflow is the following:

1. The demo application reads tuples of frames from web cameras/videos one by one. For each frame in tuple it runs person detector
1. The demo application reads tuples of frames from web cameras/videos one by one.
For each frame in tuple it runs person detector
and then for each detected object it extracts embeddings using re-identification model.
2. All embeddings are passed to tracker which assigns an ID to each object.
3. The demo visualizes the resulting bounding boxes and unique object IDs assigned during tracking.
Expand All @@ -40,55 +41,210 @@ pip3 install -r requirements.txt
Run the application with the `-h` option to see the following usage message:

```
usage: multi_camera_multi_person_tracking.py [-h] -i I [I ...] -m M_DETECTOR
usage: multi_camera_multi_person_tracking.py [-h] -i I [I ...]
[--config CONFIG]
[--detections DETECTIONS [DETECTIONS ...]]
[-m M_DETECTOR]
[--t_detector T_DETECTOR]
[--m_segmentation M_SEGMENTATION]
[--t_segmentation T_SEGMENTATION]
--m_reid M_REID
[--output_video OUTPUT_VIDEO]
[--config CONFIG]
[--history_file HISTORY_FILE]
[-d DEVICE] [-l CPU_EXTENSION]
[--save_detections SAVE_DETECTIONS]
[--no_show] [-d DEVICE]
[-l CPU_EXTENSION]
Multi camera multi person tracking live demo script
optional arguments:
-h, --help show this help message and exit
-i I [I ...] Input sources (indexes of cameras or paths to video
files)
--config CONFIG Configuration file
--detections DETECTIONS [DETECTIONS ...]
JSON file with bounding boxes
-m M_DETECTOR, --m_detector M_DETECTOR
Path to the person detection model
--t_detector T_DETECTOR
Threshold for the person detection model
--m_reid M_REID Path to the person reidentification model
--m_segmentation M_SEGMENTATION
Path to the person instance segmentation model
--t_segmentation T_SEGMENTATION
Threshold for person instance segmentation model
--m_reid M_REID Path to the person re-identification model
--output_video OUTPUT_VIDEO
--config CONFIG
Optional. Path to output video
--history_file HISTORY_FILE
Optional. Path to file in JSON format to save results
of the demo
--save_detections SAVE_DETECTIONS
Optional. Path to file in JSON format to save
bounding boxes
--no_show Optional. Don't show output
-d DEVICE, --device DEVICE
-l CPU_EXTENSION, --cpu_extension CPU_EXTENSION
MKLDNN (CPU)-targeted custom layers.Absolute path to a
shared library with the kernels impl.
--no_show Optional. Don't show output
```
Minimum command examples to run the demo:

```
# videos
python multi_camera_multi_person_tracking.py \
-i path/to/video_1.avi path/to/video_2.avi \
-d path/to/person-detection-retail-0013.xml \
-r path/to/person-reidentification-retail-0076.xml \
--m_detector path/to/person-detection-retail-0013.xml \
--m_reid path/to/person-reidentification-retail-0103.xml \
--config config.py
# videos with instance segmentation model
python multi_camera_multi_person_tracking.py \
-i path/to/video_1.avi path/to/video_2.avi \
--m_segmentation path/to/instance-segmentation-security-0050.xml \
--m_reid path/to/person-reidentification-retail-0107.xml \
--config config.py
# web-cameras
python multi_camera_multi_person_tracking.py \
-i 0 1 \
-d path/to/person-detection-retail-0013.xml \
-r path/to/person-reidentification-retail-0076.xml \
--m_detector path/to/person-detection-retail-0013.xml \
--m_reid path/to/person-reidentification-retail-0103.xml \
--config config.py
```

The demo can use a JSON file with detections instead of a person detector.
The structure of this file should be as follows:
```json
[
[ # Source#0
{
"frame_id": 0,
"boxes": [[x0, y0, x1, y1], [x0, y0, x1, y1], ...], # N bounding boxes
"scores": [score0, score1, ...], # N scores
},
{
"frame_id": 1,
"boxes": [[x0, y0, x1, y1], [x0, y0, x1, y1], ...],
"scores": [score0, score1, ...],
},
...
],
[ # Source#1
{
"frame_id": 0,
"boxes": [[x0, y0, x1, y1], [x0, y0, x1, y1], ...], # N bounding boxes
"scores": [score0, score1, ...], # N scores
},
{
"frame_id": 1,
"boxes": [[x0, y0, x1, y1], [x0, y0, x1, y1], ...],
"scores": [score0, score1, ...],
},
...
],
...
]
```
Such file with detections can be saved from the demo. Specify the argument
`--save_detections` with path to an output file.

## Demo Output

The demo displays bounding boxes of tracked objects and unique IDs of those objects.
To save output video with the result please use the option `--output_video`, to change configuration parameters please open the `config.py` file and edit it.
To save output video with the result please use the option `--output_video`,
to change configuration parameters please open the `config.py` file and edit it.

Visualization can be controlled using the following keys:
* `space` - pause or next frame
* `enter` - resume video
* `esc` - exit

Also demo can dump resulting tracks to a json file. To specify the file use the
`--history_file` argument.

## Quality measuring

The demo provides tools for measure quality of the multi camera multi person tracker:
* Evaluation MOT metrics
* Visualize the demo results from a history file

For MOT metrics evaluation we use [py-motmetrics](https://github.com/cheind/py-motmetrics) module.
It is necessary to have ground truth annotation file for the evaluation. Supported format
of the ground truth annotation can be obtained via the annotation tool [CVAT](https://github.com/opencv/cvat).
The annotation must includes the following labels and attributes:
```json
[
{
"name": "person",
"id": 0,
"attributes": [
{
"id": 0,
"name": "id",
"type": "text",
"mutable": false,
"values": [
" "
]
}
]
}
]
```

To run evaluation MOT metrics use the following command:
```bash
python run_evaluate.py \
--history_file path/to/history/file.json \
--gt_files \
path/to/ground/truth/annotation/for/source_0.xml \
path/to/ground/truth/annotation/for/source_1.xml \
```
Number of ground truth files depends on the number of used video sources.

For the visualization of the demo results please use the next command:
```
python run_history_visualize.py \
-i path/to/video_1.avi path/to/video_2.avi \
--history_file path/to/history/file.json \
```
This a minimum arguments set for the script. To show all available arguments
run the command:
```
python3 run_history_visualize.py -h
usage: run_history_visualize.py [-h] [-i I [I ...]] --history_file
HISTORY_FILE [--output_video OUTPUT_VIDEO]
[--gt_files GT_FILES [GT_FILES ...]]
[--timeline TIMELINE] [--match_gt_ids]
[--merge_outputs]
Multi camera multi person tracking visualization demo script
optional arguments:
-h, --help show this help message and exit
-i I [I ...] Input videos
--history_file HISTORY_FILE
File with tracker history
--output_video OUTPUT_VIDEO
Output video file
--gt_files GT_FILES [GT_FILES ...]
Files with ground truth annotation
--timeline TIMELINE Plot and save timeline
--match_gt_ids Match GT ids to ids from history
--merge_outputs Merge GT and history tracks into one frame
```
Ground truth files have the same format was described in the MOT metrics evaluation part.

## Process analysis

During the demo execution are available two options for analysis the process:
1. Visualize distances between embeddings that are criterion for matching tracks.
2. Save and visualize embeddings (via `tensorboard`).

Also demo can dump resulting tracks to a json file. To specify the file use the `--history_file` argument.
By default these options are disabled.
To enable the first one please set in configuration file for `analyzer` parameter
`enable` to `True`, for the second one for `embeddings` specify parameter `path`
that is a directory where data related to embeddings will be saved
(if it is an empty string the option is disabled). In `embeddings` is a parameter
`use_images`. If it is `True` for every embedding will be drawn an image with a person
instead a point.
49 changes: 41 additions & 8 deletions demos/python_demos/multi_camera_multi_person_tracking/config.py
View file
Original file line number Diff line number Diff line change
@@ -1,18 +1,51 @@
# This config is tuned for the bundle of person-detection-retail-xxx and person-reidentification-retail-xxx
# models, but should be suitable for other well-trained detector and reid models
# Alse all tracking update intervals are set assumin input frequency about 30FPS
random_seed = 100

time_window = 20
global_match_thresh = 0.35
mct_config = dict(
time_window=20,
global_match_thresh=0.2,
bbox_min_aspect_ratio=1.2
)

sct_config = dict(
time_window=10,
continue_time_thresh=2,
track_clear_thresh=3000,
match_threshold=0.475,
merge_thresh=0.3,
match_threshold=0.25,
merge_thresh=0.15,
n_clusters=4,
max_bbox_velocity=0.2,
detection_occlusion_thresh=0.7,
track_detection_iou_thresh=0.5
track_detection_iou_thresh=0.5,
process_curr_features_number=0,
interpolate_time_thresh=10,
detection_filter_speed=0.6,
rectify_thresh=0.1
)

normalizer_config = dict(
enabled=False,
clip_limit=.5,
tile_size=8
)

visualization_config = dict(
show_all_detections=True,
max_window_size=(1920, 1080),
stack_frames='vertical'
)

analyzer = dict(
enable=False,
show_distances=True,
save_distances='',
concatenate_imgs_with_distances=True,
plot_timeline_freq=0,
save_timeline='',
crop_size=(32, 64)
)

embeddings = dict(
save_path='',
use_images=True, # Use it with `analyzer['enable'] = True` to save crops of objects
step=0 # Equal to subdirectory for `save_path`
)
Loading

0 comments on commit 0687878

Please sign in to comment.