Establish Concrete Metrics for Pipeline Tasks

[Micky774]

It may be worth codifying exactly what the goal of the pipeline is. Roughly speaking, it can be described as creating a potent representation for downstream tasks, however classification can be considered one manifestation of this task (e.g. a simple ML model using the representation as features). What other tasks can we use to act as a test of representation power? And what qualitative features are desirable? What baseline classification scores can be achieved with "null models" e.g. random forest or simple conv-net?

TL;DR

• How can we directly calculate "representation power"?
• What downstream tasks can demonstrate "representation power"?
• What baseline performance can we generate on such downstream tasks?

[rmattson1008]

We should check robustness to noise. I don’t think it would be too hard to compare this on “downstream” tasks like the ones in the last scipy paper.

[Micky774]

Agreed, that would be a great addition