Static typing after data download

[jaredbroad]

Hey Dominic, this is a nice API for the request strings but once the data is downloaded its loosely typed if I understand correctly?

Perhaps we could merge in some of QuantConnect's work to get static typing on the datasets? https://github.com/QuantConnect/Quandl.NET

Given how varied the Quandl sets are its tricky but we've written a stock prices model, and perhaps with time we could build up models for a large variety of data-types.

We're looking to build this into QuantConnect so I'd be happy to help.

[HubertJ]

Hi Jared,

You are correct. 😄 In it's current state the QuandlCS stuff will just download the string data from Quandl and it is up to the user to convert that to a typed object for use... which is far from ideal.

The main reason for this is that I wasn't sure of the "best" approach for doing it. I wanted to avoid creating a library of datasets because, as you say, Quandl contains such a wide variety of datasets which is constantly growing it would be an impossible task.

Whilst I was looking into the upload functionality on Quandl I started to write something that would allow users of QuandlCS to specify a collection of objects which would then be converted to a format that could be uploaded to Quandl. I didn't bother going much further with it as the public upload API isn't available for Quandl at the moment (as far I am aware, that may have changed since I last checked) and so I didn't know what format it needed to end up in.

I have just created a branch where I have added some of that stuff mentioned above so that it can be used for downloading datasets and returning you a type-safe collection. The QuandlCSConsole project has been re-purposed to contain some example downloads of Gold, Bananas and USD\NZD FX data.

It's far from finished but if you get a chance, take a look at it and let me know what you think, I'd be happy to get some advice on a better approach. I've been a little bit busy after work the last week but will try and take a look through the Quandl.NET stuff you linked to.

Hopefully over the next month or so I will get a chance to pick up where I left the QuandlCS stuff. Still a fair bit of tidying up to do in the implementation.

[jaredbroad]

Summary of Quandl.NET is that it has a common base factory interface - which defines an Initializer to accept a line from the CSV to parse into the class. We pondered the best way as well, but this seemed like the only way to go, but I'm open to suggestions.

[HubertJ]

For the upload I considered an interface approach with two methods similar to:

using System;

namespace QuandlCS
{
  public interface IQuandlDataset
  {
    string ToRawString(); 
    void FromRawString(string data);
  }
}

and this was one of my dilemmas. Using an interface such as the one above allows for each dataset class to have custom behaviour that is unique to that type alone which is very powerful for both upload and download.

But there were a few downsides which turned me away from that approach. Mainly, whilst it is powerful, it does place much greater responsibility on the dataset class for getting the parsing and initialization correct, formatting the data correctly for upload, handling any backwards compatibility for datasets that might change. etc. etc. which I felt would lead to classes that do much more than they should.

Due to those concerns, and for a rather large desire to want to play around with the .NET reflection stuff more I decided to investigate the attribute approach. The main benefit I saw with this approach is that it takes most of the responsibility away from the designer of the dataset class so that they do not need to worry about the heavy lifting.

The way the branch I committed yesterday works is through a couple of attributes. The main attribute gets put onto a property in a class and specifies the column in the Quandl data that the property refers to. In the column extractor we use the column name set in the attribute and the type of the property to take the string data from the xml and convert it to the correct data type. Which we then set on an instance of the dataset class. For each data point it loops through all the attributes that were specified and try to set the data.

It's far from perfect (for example I don't think the logic in the Quandl.NET/QuandlDemo.cs to set Open,High,Low,Close to Close when there are no trades can be done with the QuandlCS reflection approach without some changes) but I think it makes it a lot easier to create the classes and makes them a lot simpler.

Have you faced any problems with the approach Quandl.NET has taken where the class is responsible for its own initialization? I'm starting to wonder if the attributes make things a bit complicated.