Skip to content
This repository was archived by the owner on Nov 14, 2023. It is now read-only.

Resuming from checkpoint? #260

@jonathanmiller2

Description

@jonathanmiller2

There doesn't seem to be any documentation for how to resume the train from the local_dir checkpoints.
Using the following provided example:

"""Example using an sklearn Pipeline with TuneGridSearchCV.
Example taken and modified from
https://scikit-learn.org/stable/auto_examples/compose/
plot_compare_reduction.html
"""

from tune_sklearn import TuneSearchCV
from tune_sklearn import TuneGridSearchCV
from sklearn.datasets import load_digits
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
from sklearn.decomposition import PCA, NMF
from sklearn.feature_selection import SelectKBest, chi2

pipe = Pipeline([
    # the reduce_dim stage is populated by the param_grid
    ("reduce_dim", "passthrough"),
    ("classify", LinearSVC(dual=False, max_iter=10000))
])

N_FEATURES_OPTIONS = [2, 4, 8]
C_OPTIONS = [1, 10]
param_grid = [
    {
        "reduce_dim": [PCA(iterated_power=7), NMF()],
        "reduce_dim__n_components": N_FEATURES_OPTIONS,
        "classify__C": C_OPTIONS
    },
    {
        "reduce_dim": [SelectKBest(chi2)],
        "reduce_dim__k": N_FEATURES_OPTIONS,
        "classify__C": C_OPTIONS
    },
]

random = TuneSearchCV(pipe, param_grid, search_optimization="random", local_dir='checkpoints')
X, y = load_digits(return_X_y=True)
random.fit(X[:100], y[:100])
print(random.cv_results_)

Trying to load the generated checkpoint using normal Ray methods such as:

from ray.tune import Tuner
Tuner.restore('checkpoints/_Trainable_2022-12-28_11-47-41')

Yields:

RuntimeError: Could not find Tuner state in restore directory. Did you passthe correct path (including experiment directory?) Got: checkpoints/_Trainable_2022-12-28_11-47-41

What is the intended way of loading checkpoints?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions