RFNNRegressor should support both classifier and regressor RFs when building forests

[grovduck]

In #85, we created the RFNNRegressor estimator that uses a suite of random forests (one for each attribute in y (or y_fit)) to support a kNN approach to match targets to reference data based on similarity in the node IDs across forests and their trees. In that PR, we relied exclusively on sklearn.ensemble.RandomForestRegressor to build the individual forests, assuming that all y features were continuous rather than categorical. We decided to defer the implementation of using both sklearn.ensemble.RandomForestRegressor and sklearn.ensemble.RandomForestClassifier to build the forests for a later time. Per this comment:

The other obvious thing I need to do to make this estimator more useful is to have the set of random forests to be both RandomForestClassifier and RandomForestRegressor objects. The RandomForestClassifier forests could either be built from classifying continuous y attributes into classes or from y features that already represent classes, either numeric or strings. In my mind, this seems like we need to both: 1) tell the estimator how to build each forest (some kind of mapper between y attribute and random forest type); and 2) have utility functions to bin continuous data into categorical data. I can definitely see the argument that 2. should be outside the scope of the estimator such that the user does this on their own, but it might be nice to have example utility functions to show how this is done.

Design considerations

• We need to decide what feature dtypes will we support with RFNNRegressor. Ideally, we would be able to support np.object (string) and pd.Categorical features and well as integer and floating-point features (I suppose we could also support np.bool, but building a forest with just two classes seems pretty limited). Both pandas dataframes and numpy arrays can handle mixed data types, so passing a y frame or array should be possible. (Note that, at present, our test datasets only support floating-point features - this isn't a show-stopper, but we'll need to generate mixed-type dataframes if we want to support other data types).

• Automatically inferring the type of RF classifier to associate with each y attribute will be probably be the trickiest part of this implementation. I propose that we associated certain data types with specific RF estimators (with allowable user overrides). np.object and pd.Categorical features would naturally use RandomForestClassifier estimators, whereas floating-point features would naturally use RandomForestRegressor estimators (although see below point about data binning). Integer types are a bit trickier, in that they can be nominal or ordinal. ChatGPT suggests that we could use heuristics to handle integer features (e.g. if the number of unique values is >= 10% of the number of total samples, assume regression). Of course, user specification/override will be a valuable tool to have for integer features.

• In terms of user specification of which forest type to use, I had originally thought that we could provide a mapping of column name to estimator type that would be passed to the initializer, something like:

  est = RFNNRegressor(rf_mapper={"var_0": "classification", "var_1": "regression"}).fit(X, y).

  But because we need to support both numpy arrays and pandas dataframes for fit, we can't consistently rely on column names being available. We would likely have to support index-based specification as well to support numpy arrays. (It does feel a bit strange specifying information the y data before the model is actually fit (i.e. in the RFNNRegressor initialization), although ColumnTransformer does use this pattern).

• In yaImpute, the default behavior for building forests with floating-point features is to bin the feature into discrete classes rather than running random forest in regression mode. Do we want to provide the same functionality with a hyperparmeter keyword (e.g. build_classes=True) that would implement this same behavior or should we just provide examples for users to pre-bin their data before setting up the estimator.

@aazuspan, lots of unresolved questions at this point and I would value your help thinking through the knotty data type questions before we start implementing this. I think working with dataframes will be fairly straightforward by setting up the forests based on column names, but type discovery and variable mapping with numpy arrays seems trickier to me.

[aazuspan]

Thanks for the detailed writeup, @grovduck!

I think working with dataframes will be fairly straightforward by setting up the forests based on column names, but type discovery and variable mapping with numpy arrays seems trickier to me.

Agreed! I'll defer to however you'd prefer to work on this, but I think there's probably enough features here that it could be split into a few different PRs, e.g. manually specifying RF types, automatically inferring RF types, and automatically converting continuous columns to categorical.

But because we need to support both numpy arrays and pandas dataframes for fit, we can't consistently rely on column names being available. We would likely have to support index-based specification as well to support numpy arrays. (It does feel a bit strange specifying information the y data before the model is actually fit (i.e. in the RFNNRegressor initialization), although ColumnTransformer does use this pattern)

The ColumnTransformer is a great find, since that gives an example of specifying columns at instantiation time and a precedent for selecting columns by name or index. I do prefer your suggestion of using a dict to map columns to RF type rather than the association list used by ColumnTransformer. Another possibility since this is a binary choice would be to choose a default RF type and take a column list to override to the other type, e.g. use_classifier=[1, 3, "foo", "bar"], but that's less explicit than the alternative, so I don't necessarily prefer that.

Integer types are a bit trickier, in that they can be nominal or ordinal. ChatGPT suggests that we could use heuristics to handle integer features (e.g. if the number of unique values is >= 10% of the number of total samples, assume regression).

Given that users have a few different ways to mark an int column for classification or regression (via the mapping or by changing data type), I'd be worried that a heuristic approach adds more complexity and potential surprise than it's worth. If we do use heuristics, I'd suggest we make it clear when that happens, e.g. by emitting a warning or storing the selected estimator type in an attribute.

In yaImpute, the default behavior for building forests with floating-point features is to bin the feature into discrete classes rather than running random forest in regression mode. Do we want to provide the same functionality with a hyperparmeter keyword (e.g. build_classes=True) that would implement this same behavior or should we just provide examples for users to pre-bin their data before setting up the estimator.

I can see the value in this feature to achieve closer parity with yaImpute, although it certainly adds some complexity. I assume you'd need some additional parameters like the number of bins and binning method, but would you also need to be able to specify those per-column? Also, I'm not quite sure how this would interact with the rf_mapper, e.g. if you specify a float column for regression and use build_classes=True, would it still run regression on the binned column, or would one of the parameters override the other?

Another (unrelated) consideration is how do we handle random forest parameters that differ between regression and classification, e.g. criterion which accepts different values based on the RF type? When you originally implemented RFNNRegressor we discussed separating the RF and kNN params using dicts or by prefixing/suffixing the parameter names, and either of those could potentially work here too, but they aren't without issues...

[grovduck]

Agreed! I'll defer to however you'd prefer to work on this, but I think there's probably enough features here that it could be split into a few different PRs, e.g. manually specifying RF types, automatically inferring RF types, and automatically converting continuous columns to categorical.

@aazuspan, I totally agree with this approach. I've been ruminating over the order that makes most sense and I think I've settled on:

1. automatically inferring RF types based on feature data types
2. handling RF parameters that differ between regression and classification
3. adding RF type overrides via user-defined parameterizations
4. (possibly) adding binning capacity for continuous features

I'll be splitting these into separate PRs and hoping to make some progress on these.

I do prefer your suggestion of using a dict to map columns to RF type rather than the association list used by ColumnTransformer. Another possibility since this is a binary choice would be to choose a default RF type and take a column list to override to the other type, e.g. use_classifier=[1, 3, "foo", "bar"], but that's less explicit than the alternative, so I don't necessarily prefer that.

For now, I think I'll focus on the dict approach unless that raises issues. That's why I wanted the automatic inferring to come before user specification, so that the default option is in place.

Given that users have a few different ways to mark an int column for classification or regression (via the mapping or by changing data type), I'd be worried that a heuristic approach adds more complexity and potential surprise than it's worth. If we do use heuristics, I'd suggest we make it clear when that happens, e.g. by emitting a warning or storing the selected estimator type in an attribute.

Yep, I agree with you here. This can be communicated to users via clear examples rather than through code.

I can see the value in this feature to achieve closer parity with yaImpute, although it certainly adds some complexity. I assume you'd need some additional parameters like the number of bins and binning method, but would you also need to be able to specify those per-column? Also, I'm not quite sure how this would interact with the rf_mapper, e.g. if you specify a float column for regression and use build_classes=True, would it still run regression on the binned column, or would one of the parameters override the other?

All great questions and no revelations from me yet 😄! Again, I think you've influenced me enough to not to try to handle everything through code and that having users massage certain features into categorical bins (perhaps via ColumnTransformer) will definitely be a path of less resistance. That also slyly avoids your second question ...

[grovduck]

Another (unrelated) consideration is how do we handle random forest parameters that differ between regression and classification, e.g. criterion which accepts different values based on the RF type? When you originally implemented RFNNRegressor we discussed separating the RF and kNN params using dicts or by prefixing/suffixing the parameter names, and either of those could potentially work here too, but they aren't without issues...

@aazuspan, I started looking into RandomForestClassifier and RandomForestRegressor to compare parameters. Just based on parameters, here are the differences:

Parameters in RandomForestClassifier, not in RandomForestRegressor

• class_weight

Parameters in RandomForestRegressor, not in RandomForestClassifier

• none

Parameters with different default values

• criterion
• max_features

Parameters passed through to BaseForest

• n_estimators
• bootstrap
• oob_score
• n_jobs
• random_state
• verbose
• warm_start
• max_samples

Remainder of shared parameters

• max_depth
• min_samples_split
• min_samples_leaf
• min_weight_fraction_leaf
• max_leaf_nodes
• min_impurity_decrease
• ccp_alpha
• monotonic_cst

My assumption (although I'm not certain about it) is that the parameters that are passed through to BaseForest are agnostic to the type of RF being built, but the remaining parameters are being sent through to instantiate different classes (DecisionTreeRegressor and DecisionTreeClassifier). If this holds true, we would only need to differentiate the parameters that are not passed through to BaseForest (still a healthy number).

Looking more carefully at DecisionTreeRegressor and DecisionTreeClassifier (the individual "tree" parameters), they both inherit from BaseDecisionTree and all of the parameters in the "Remainder of shared parameters" section seem to be passed directly to this class. Again, my working assumption is that those parameters would be treated the same, so I think only class_weights, criterion, and max_features would need to be specialized keywords. For RandomForestRegressor forests, we could even override class_weights to None.

Reading back through our discussion on this, I still think we want to avoid using parameter dictionaries, even though that would be more concise and organized. But now that we might need to only specialize a couple of keywords, this may not be so bad.

This is just a marker for now (no response needed).

[aazuspan]

That all sounds like a good plan, @grovduck! Thanks for the thorough research! Distinguishing those two shared but incompatible parameters by name seems totally reasonable.