Data Source

[bitnom]

Just got here and read the readme. Data sourcing is no problem with BTC or any other crypto. We can get it straight from the exchanges for free. Check out the library CCXT.

[methenol]

I think CCXT would be a great option as it would standardize the data format from several exchanges and gets around writing individual API calls on a per exchange basis.

I started tinkering with the CCXT library some today out of curiosity and noticed some initial issues that would need to be ironed out. The names of the asset vary between some exchanges (for example Kraken which uses XBT instead of BTC for ISO 3316 naming convention), and the granularity that you can pull is based on the granularity that the exchange supports (for example you can't pull 1m data on poloniex, but 5m works fine which matches the API references). Fortunately it looks like CCXT has some good methods for checking the capabilities of the exchanges so we may be able to set some requirements like minimum granularity or maximum history and populate the SQL tables with data from exchanges that meet the requirements.

A risk is if an exchange updates their APIs we would be stuck until CCXT is updated to comply with the new API format.

Going to explore this some more after we iron out some of the issues with the agent converging on the data in the current v0.2 branch. For now there is probably a benefit to having everyone working off the same dataset, which the kaggle dump ensures.