A benchmark framework for measuring different deep learning tools. Please refer to http://dlbench.comp.hkbu.edu.hk/ for our testing results and more details.
| Dirctory | Description |
|---|---|
| configs/ | Configuration files for running benchmark |
| network-configs/ | Description of our tested models |
| synthetic/ | Our benchmark tests with fake data |
| tools/ | Contains running scripts and network configurations of each deep learning tool |
| logs/ | Will be generated by running benchmark.py. Running logs should be put in here |
Prepare data for the tools you want to run and put them under $HOME/data. Note that the name of each data directory should be the same as the name of the tool for convenience.
You can download data we used for our benchmark through following links:
- Caffe: http://dlbench.comp.hkbu.edu.hk/s/data/caffe.zip
- CNTK: http://dlbench.comp.hkbu.edu.hk/s/data/cntk.zip
- MXNet: http://dlbench.comp.hkbu.edu.hk/s/data/mxnet.zip
- TensorFlow: http://dlbench.comp.hkbu.edu.hk/s/data/tensorflow.zip
- Torch: http://dlbench.comp.hkbu.edu.hk/s/data/torch.zip
For the synthetic data generation, please refer to scripts in the link: http://dlbench.comp.hkbu.edu.hk/s/html/v5/index.html.
There are some sample configuration files in configs/, you can choose one of them as example and change values of each item according to your needs and environment.
To run benchmark test just execute
python benchmark.py -config configs/<your config file>.config
Follow the instructions in tools/Readsme.md preparing the running scripts and netowrk configurations. Note that training data should be put in $HOME/data/ so that we can test new tools in our machines and update benchmarking results to our website.