[Question] How to best use AutoSklearn in a nested cross validation setting?

[verakye]

Short Question Description

What is the suggested way of using AutoSklearn in a nested CV setting, i.e. for example in combination with scikitlearn's cross_validate?

Further details

The general setup/idea/goal:

AutoSklearn in a nested CV setup (inner CV), using Dask on a Slurm cluster for parallelisation and scikitlearn's cross_validate for the outer CV.

The problem:

Combining these 3 aspects leads to a TypeError: cannot pickle '_asyncio.Task' object error which doesn't appear when using Autosklearn just with fit() on a test data set instead of embedding it in the nested CV scenario.

A code snippet that should lead to the problem:

from dask_jobqueue import SLURMCluster
from dask.distributed import Client
import time
import sys
import logging
from sklearn import model_selection
from sklearn.datasets import make_regression
from sklearn.model_selection import ShuffleSplit, cross_validate
from autosklearn.regression import AutoSklearnRegressor
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

# input arguments
cluster_jobs = 1
n_samples = 10000
t_ASL = 5  # in minutes
n_repeats_outer = 2
n_jobs_outer = 1

# Dask cluster and client
cluster = SLURMCluster(nanny=True)  # other specifications in config file
cluster.scale(jobs=cluster_jobs)
client = Client(cluster)

# make fake regression data
X, y, true_weights = make_regression(
    n_samples= n_samples,
    n_features=1300,
    n_informative=400,
    noise=8,
    coef=True,
    random_state=0,
)

# further definitions
random_state = 43

# inner CV
cv_inner_folds = 2

# initialise ASL regressor
auto_model = AutoSklearnRegressor(
        time_left_for_this_task=t_ASL*60,  # multiplies by no. outer folds
        memory_limit=400000,
        resampling_strategy="cv",
        resampling_strategy_arguments={"folds": cv_inner_folds},
        metric=None,  # None: Use default of each algorithm (not mean_squared_error)
        n_jobs=1,  # n_jobs is ignore when passing a dask client
        initial_configurations_via_metalearning=0,  # avoid config. warnings
        dask_client=client,
        tmp_folder=(
            '/p/project/comanukb/vkomeyer/motorpred/ukb/code/Dask_ASL_minexpl/'
            'logs_SlurmCluster/auto_sklearn_log1')
    )

# outer CV
outer_cv = ShuffleSplit(
    n_splits=n_repeats_outer, random_state=random_state,
    train_size=0.7)
scoring_outer = ['r2']

# Model fitting
if __name__ == "__main__":
    score = cross_validate(
        estimator=model,
        X=X, y=y,
        cv=outer_cv,
        scoring=scoring_outer,
        return_estimator=True,
        n_jobs=n_jobs_outer)

The resulting error:

TypeError: cannot pickle '_asyncio.Task' objectoriginating from cross_validate() (I can provide you with the entire traceback if that was helpful)

Further info

The error doesn't occur neither when the nested cross validate setting is replace by a simple train/test split and auto_model.fit(X_train, y_train) nor when using the cross validate setting without AutoSklearn but e.g. scikitlearn's GridSearchCV (comparison code can be provided in case that would be helpful).

Thank you very much for any hints to solve this!

[eddiebergman]

Hi @verakye,

Thank you for the detailed issue :) So at a high level, I'm not sure autosklearn satisfies the criterion of being a scikit-learn model fully. To be more explicit, most scikit-learn cross validation steps use scikit learns clone(estimator), get_params(estimator, params), set_params(estimator, params) to essentially reproduce the estimator in n different objects that can then all be run in parallel.

https://github.com/scikit-learn/scikit-learn/blob/36958fb24/sklearn/model_selection/_validation.py#L268

# sklearn.model_selection._validation
def cross_validate(estimator, ...)
    ...
    clone(estimator)
    ...

And then in clone(estimator) the call to get_params(estimator) is called.

We don't satisfy the criterion of being able to clone an instance of AutoSklearnEstimator because copying parameters like the tmp_dir mean they are technically not a perfect clone and they share some state.

This is why we don't satisfy the API required.

Now how this manifests as the error TypeError: cannot pickle '_asyncio.Task' object, I'm not entirely sure but I am fairly certain it is something to do with not satisfying their required API.

As for what to do about that, you would have to roll your own cross-validate loop unfortunatly, with a different path for each auto-sklearn.

Hope this helps!

Best,
Eddie

[verakye]

Hi @eddiebergman,

thanks a lot for your quick reply and the useful information. From what I see in the error's traceback your assumption seems to be very likely.

Could you maybe elaborate a bit on what you mean with the "different path for each auto-sklearn"? Are you referring to the tmp_dir here, i.e. giving AutoSklearn a new tmp_dir in every loop iteration to avoid the aforementioned copying?

Thanks a lot!
Best,
Vera

[eddiebergman]

Exactly! Autosklearn uses the filesystem as part of its state so they need to have a different tmp_dir so they don't have issues with state.