Token filtering with a measure of association?

[mfra123]

Would it be possible (and useful) to add a functionality to filter token lists by a measure of association with the outcome e.g. a chi squared / log odds?

When working with heavily imbalanced data selecting top n tokens (or similar) with step_tokenfilter is unlikely to include discriminative features, unless the number of terms included is very large or the training data is artificially balanced before token filtering. Similarly if you wish to include a mixture of unigrams and bigrams etc less frequent terms which are discriminative are unlikely to be included unless large numbers of tokens are included, many of which will be common words often with little predictive value.