diff --git a/.dockerignore b/.dockerignore index e8a37bf..3cbfe6f 100644 --- a/.dockerignore +++ b/.dockerignore @@ -5,4 +5,5 @@ research tensorboard agents data/tensorboard -data/agents \ No newline at end of file +data/agents +data/postgres \ No newline at end of file diff --git a/.gitignore b/.gitignore index 3cdcf7d..6b49044 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ **/__pycache__ data/tensorboard/* data/agents/* +data/postgres/* data/log/* *.pkl *.db diff --git a/README.md b/README.md index a9e2d4c..8409a2e 100644 --- a/README.md +++ b/README.md @@ -21,28 +21,123 @@ https://towardsdatascience.com/using-reinforcement-learning-to-trade-bitcoin-for # Getting Started -The first thing you will need to do to get started is install the requirements in `requirements.txt`. +### How to find out if you have nVIDIA GPU? +Linux: ```bash +sudo lspci | grep -i --color 'vga\|3d\|2d' | grep -i nvidia +``` +If this returns anything, then you should have an nVIDIA card. + +### Basic usage + +The first thing you will need to do to get started is install the requirements. If your system has an nVIDIA GPU that you should start by using: + +```bash +cd "path-of-your-cloned-rl-trader-dir" pip install -r requirements.txt ``` +More information regarding how you can take advantage of your GPU while using docker: https://github.com/NVIDIA/nvidia-docker + + +If you have another type of GPU or you simply want to use your CPU, use: + +```bash +pip install -r requirements.no-gpu.txt +``` + +Update your current static files, that are used by default: +```bash + python ./cli.py update-static + ``` + +Afterwards you can simply see the currently available options: + +```bash +python ./cli.py --help +``` + +or simply run the project with default options: + +```bash +python ./cli.py opt-train-test +``` + +If you have a standard set of configs you want to run the trader against, you can specify a config file to load configuration from. Rename config/config.ini.dist to config/config.ini and run + +```bash +python ./cli.py --from-config config/config.ini opt-train-test +``` + +```bash +python ./cli.py opt-train-test +``` + +### Testing with vagrant + +Start the vagrant box using: +```bash +vagrant up +``` + +Code will be located at /vagrant. Play and/or test with whatever package you wish. +Note: With vagrant you cannot take full advantage of your GPU, so is mainly for testing purposes + + +### Testing with docker + +If you want to run everything within a docker container, then just use: +```bash +./run-with-docker (cpu|gpu) (yes|no) opt-train-test +``` +- cpu - start the container using CPU requirements +- gpu - start the container using GPU requirements +- yes | no - start or not a local postgres container +Note: in case using yes as second argument, use + +```bash +python ./ cli.py --params-db-path "postgres://rl_trader:rl_trader@localhost" opt-train-test +``` + +The database and it's data are pesisted under `data/postgres` locally. + +If you want to spin a docker test environment: +```bash +./run-with-docker (cpu|gpu) (yes|no) +``` + +If you want to run existing tests, then just use: +```bash +./run-tests-with-docker +``` + +# Fire up a local docker dev environment +```bash +./dev-with-docker +``` -The requirements include the `tensorflow-gpu` library, though if you do not have access to a GPU, you should replace this requirement with `tensorflow`. # Optimizing, Training, and Testing While you could just let the agent train and run with the default PPO2 hyper-parameters, your agent would likely not be very profitable. The `stable-baselines` library provides a great set of default parameters that work for most problem domains, but we need to better. -To do this, you will need to run `optimize.py`. +To do this, you will need to run `cli.py`. ```bash -python ./optimize.py +python ./cli.py opt-train-test ``` This can take a while (hours to days depending on your hardware setup), but over time it will print to the console as trials are completed. Once a trial is completed, it will be stored in `./data/params.db`, an SQLite database, from which we can pull hyper-parameters to train our agent. From there, agents will be trained using the best set of hyper-parameters, and later tested on completely new data to verify the generalization of the algorithm. +# Common troubleshooting + +##### The specified module could not be found. +Normally this is caused by missing mpi module. You should install it according to your platorm. +- Windows: https://docs.microsoft.com/en-us/message-passing-interface/microsoft-mpi +- Linux/MacOS: https://www.mpich.org/downloads/ + # Project Roadmap If you would like to contribute, here is the roadmap for the future of this project. To assign yourself to an item, please create an Issue/PR titled with the item from below and I will add your name to the list. @@ -59,14 +154,10 @@ If you would like to contribute, here is the roadmap for the future of this proj - ~Optionally replace SQLite db with Postgres to enable multi-processed Optuna training~ - This is enabled through Docker, though support for Postgres still needs to be improved - ~Replace `DummyVecEnv` with `SubProcVecEnv` everywhere throughout the code~ **[@archenroot, @arunavo4, @notadamking]** -- Find sources of CPU bottlenecks to improve GPU utilization - - Replace pandas or improve speed of pandas methods by taking advantage of GPU -- Find source of possible memory leak (in `RLTrader.optimize`) and squash it - + - Allow features to be added/removed at runtime + - Create simple API for turning off default features (e.g. prediction, indicators, etc.) + - Create simple API for adding new features to observation space ## Stage 1: -- Allow features to be added/removed at runtime - - Create simple API for turning off default features (e.g. prediction, indicators, etc.) - - Create simple API for adding new features to observation space - Add more optional features to the feature space - Other exchange pair data (e.g. LTC/USD, ETH/USD, EOS/BTC, etc.) - Twitter sentiment analysis @@ -90,6 +181,9 @@ If you would like to contribute, here is the roadmap for the future of this proj - Experiment with Auto-decoders to remove noise from the observation space - Implement self-play in a multi-process environment to improve model exploration - Experiment with dueling actors vs tournament of dueling agents +- Find sources of CPU bottlenecks to improve GPU utilization + - Replace pandas or improve speed of pandas methods by taking advantage of GPU +- Find source of possible memory leak (in `RLTrader.optimize`) and squash it # Contributing diff --git a/Vagrantfile b/Vagrantfile index 1fe70b4..5404f8e 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -52,7 +52,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| vm_config.vm.synced_folder '.', '/vagrant', disabled: false vm_config.vm.provision "default setup", type: "shell", inline: <