streamlining notebook release

Dockerfile

@@ -11,9 +11,6 @@ RUN apt-get update && \
 RUN mkdir -p /root/.config/matplotlib && \
 	echo backend:Agg > /root/.config/matplotlib/matplotlibrc
 
-RUN git clone https://github.com/Jamonek/Robinhood && \
-	pip3 install Robinhood/
-
 RUN VERSION=${VERSION} git clone https://github.com/omdv/robinhood-portfolio && \
 	pip3 install --upgrade --force-reinstall -r robinhood-portfolio/requirements.txt
 

README.md

@@ -1,7 +1,7 @@
 # Note
 This app is heavily relying on [pandas-datareader](https://pydata.github.io/pandas-datareader/stable/remote_data.html#) for financial quotes. Over the last couple years multiple APIs were obsoleted by their providers (Google, Morningstar) and as I am no longer a RH client I have no time to keep up with those changes. If you encounter the "Bad Gateway" error or similar it is likely that the current market data source is no longer valid. You are welcome to fork and try different sources of quotes - I will try to fix it, when/if I have time.
 
-**Current APIs are [TIINGO](https://api.tiingo.com/account/token) for stock and [STOOQ](https://stooq.com) for market index.**
+**Current API is [TIINGO](https://api.tiingo.com/account/token) for stock and market index.**
 
 # Robinhood Portfolio
 Python client to access and analyze the Robinhood portfolio.
@@ -10,39 +10,37 @@ Based on unofficial [robinhood-api](https://github.com/Jamonek/Robinhood) and se
 - [portfolioopt](https://github.com/czielinski/portfolioopt)
 
 ## Current Features 
-- Creates a Flask web server with lightweight page
+- ~~Creates a Flask web server with lightweight page~~ Replaced with Jupyter notebook
 - Downloads orders and dividends from Robinhood account.
 - Downloads market data from google API and market index from open source. Supports incremental download for additional dates to reduce a number of requests to open APIs.
 - Calculates the total return of the portfolio, including dividend payouts and risk metric for the portfolio
 - Calculates the risk metric for individual securities and correlations between securities
 - Calculates Markowitz portfolios
 
-## Screenshots
+## Screenshots (OLD FLASK APP)
 ![Image1](https://github.com/omdv/robinhood-portfolio/blob/master/docs/image_1.png)
 ![Image2](https://github.com/omdv/robinhood-portfolio/blob/master/docs/image_2.png)
 ![Image3](https://github.com/omdv/robinhood-portfolio/blob/master/docs/image_3.png)
 ![Image4](https://github.com/omdv/robinhood-portfolio/blob/master/docs/image_4.png)
 ![Image5](https://github.com/omdv/robinhood-portfolio/blob/master/docs/image_5.png)
 
 
-## Future Possible Features
-- Backtesting using one of existing libraries. Enabled, but not implemented because none of the existing libraries support dividend payouts and do not provide any significant advantages vs what is implemented already.
-- Automatic trading for simple portfolio allocations
-
-### How To Install:
-It is recommended to use virtualenv:
+### Non-Docker way
+Install dependencies, it is recommended to use virtualenv:
 ```
 git clone https://github.com/omdv/robinhood-portfolio && cd robinhood-portfolio
 virtualenv robinhood && source robinhood/bin/activate && pip3 install -r requirements.txt
 ```
 
-### How to Use
-Make sure you have set the `TIINGO_API_KEY` environment variable.
+To run:
 ```
-python3 app.py
+jupyter notebook
 ```
 
-### Docker container
+Open `main.ipynb`, enter TIINGO API KEY, Robinhood credentials, set DEMO_RUN variable to `False` to run with your data, execute all cells.
+
+
+<!-- ### Docker way
 Docker container based on Ubuntu is [available](https://hub.docker.com/r/omdv/robinhood-portfolio/). To launch it in a background mode you need to get TIINGO API key and provide it to docker.
 ```
 docker run -e TIINGO_API_KEY=<API-KEY> -d -p 8080:8080 --name robinhood omdv/robinhood-portfolio:ubuntu
@@ -53,11 +51,7 @@ Once up and running connect to [http://localhost:8080](http://localhost:8080). I
 To specify a different port run:
 ```
 docker run -e TIINGO_API_KEY=<API-KEY> -d -e PORT=$PORT -p $PORT:$PORT --name robinhood omdv/robinhood-portfolio:ubuntu
-```
-
-
-### Jupyter notebook (draft)
-You can find the Jupyter notebook using the backtrader library with pyfolio in "notebooks" folder.
+``` -->
 
 
 ### Disclaimer

backend/backend.py

@@ -11,10 +11,10 @@ class BackendClass(object):
     Backend wrapper class, provides wrappers to donwload robinhood and market
     data, provides access to portfolio models. Mostly deals with UI logic.
     --------
-    datafile: path to hdf datafile
+    datadir: path to data directory
     """
-    def __init__(self, datafile, userfile):
-        self.datafile = datafile
+    def __init__(self, datadir, userfile):
+        self.datadir = datadir
         self.userfile = userfile
         self.df_returns = None
         self.daily_returns = None
@@ -26,9 +26,9 @@ def __init__(self, datafile, userfile):
         self._date_fmt = '{:%d-%m-%Y}'
 
         # read dataframe
-        self._df_ord = pd.read_hdf(datafile, 'orders')
-        self._df_div = pd.read_hdf(datafile, 'dividends')
-        self._market = pd.read_hdf(datafile, 'market')
+        self._df_ord = pd.read_pickle(self.datadir+'orders.pkl')
+        self._df_div = pd.read_pickle(self.datadir+'dividends.pkl')
+        self._market = pd.read_pickle(self.datadir+'market.pkl')
 
         # handle user dictionary
         self._init_user_dict()
@@ -89,7 +89,7 @@ def update_market_data(self, **keyword_parameters):
         user dict to download the entire history from scratch, otherwise
         download only new dates in addition to the existing set
         """
-        md = MarketData(self.datafile)
+        md = MarketData(self.datadir)
 
         # check if symbols match
         s1 = list(self._df_ord.symbol.unique())
@@ -116,7 +116,7 @@ def update_market_data(self, **keyword_parameters):
         return self
 
     def update_robinhood_data(self, user, password):
-        rd = RobinhoodData(self.datafile)
+        rd = RobinhoodData(self.datadir)
         self._df_div, self._df_ord, _, _ =\
             rd.download_robinhood_data(user, password)
         self.user['rb_dates'] = [
@@ -137,7 +137,7 @@ def _get_daily_portfolio_panel(self):
         """
         Generate a panelframe with daily portfolio changes
         """
-        self._ptfm = PortfolioModels(self.datafile)
+        self._ptfm = PortfolioModels(self.datadir)
         self._panel = self._ptfm.daily_portfolio_changes().panelframe
         return self
 
@@ -189,7 +189,7 @@ def _get_sell_orders(self):
         """
         Get three best/worst closed positions by realized gains
         """
-        df = pd.read_hdf(self.datafile, 'closed')
+        df = pd.read_pickle(self.datadir, 'closed')
         df1 = df.nlargest(min(3, df.shape[0]), 'realized_gains')
         df2 = df.nsmallest(min(3, df.shape[0]), 'realized_gains')
         df = pd.concat([df1, df2]).sort_values(by='realized_gains')
@@ -237,7 +237,7 @@ def _get_buy_orders(self):
         """
         market_prices = self._panel['close'].iloc[-1]
 
-        df = pd.read_hdf(self.datafile, 'open')
+        df = pd.read_pickle(self.datadir, 'open')
         df['current_price'] =\
             df.apply(lambda x: market_prices[x.symbol], axis=1)
         df['unrealized_gains'] =\
@@ -284,8 +284,8 @@ def _get_buy_orders(self):
         return self
 
     def _get_all_orders(self):
-        cl = pd.read_hdf(self.datafile, 'closed')
-        op = pd.read_hdf(self.datafile, 'open')
+        cl = pd.read_pickle(self.datadir, 'closed')
+        op = pd.read_pickle(self.datadir, 'open')
         mkt = self._panel['close'].iloc[-1]
 
         cl['average_buy_price'] = cl['current_cost_basis'] / cl['signed_size']
@@ -483,5 +483,5 @@ def calculate_all(self):
 
 
 if __name__ == '__main__':
-    bc = BackendClass('../data/data.h5')
+    bc = BackendClass('data/data.h5', 'user.pkl')
     bc = bc.calculate_all()

backend/market_data.py

@@ -3,6 +3,7 @@
 """
 import pandas_datareader.data as web
 
+
 def download_save_market_data(api_key, symbols, start_date, end_date):
     """
     Return market data

backend/portfolio_model.py → backend/portfolio_models.py

@@ -146,7 +146,6 @@ def _calculate_daily(self):
         # merge orders with market
         pf = self._merge_market_with_orders(df, market)
 
-
         df = pd.read_pickle(self.datafolder + "/dividends.pkl")
 
         # calculate cumulative dividends
@@ -155,9 +154,8 @@ def _calculate_daily(self):
         # merge orders with market
         pf = self._merge_market_with_dividends(df, pf)
 
-
-        #replace null stock prices using backfill to avoid issues with
-        #daily_change and beta calculations
+        # replace null stock prices using backfill to avoid issues with
+        # daily_change and beta calculations
         close_price = pf['close']
         close_price.values[close_price.values == 0] = np.nan
         close_price.fillna(method='bfill', inplace=True)
@@ -696,4 +694,4 @@ def markowitz_portfolios(self):
             'Tangency portfolio')
         mrk.append(case)
 
-        return mrk
+        return mrk

backend/robinhood_data.py

@@ -244,5 +244,6 @@ def demo_dividends(self):
         dividends['payable_date'] = pd.Timestamp('2019-01-02', tz='UTC')
         return dividends
 
+
 if __name__ == "__main__":
     pass