[Question] Extending AutoSklearn for ML model constraints

[FelixNeutatz]

Hi everybody,

I am currently trying to extend AutoSklearn to support ML model constraints, such as model size, inference time, and training time. My idea is to adjust the loss function in this way:

The current scorer function only allows access to the model predictions.
However, to compute e.g. the model size, I need access to the model.
I would appreciate any help or hints on what you think is the best approach to tackle this within AutoSklearn.

Best regards,
Felix

[eddiebergman]

Hey @FelixNeutatz, that's not currently possible. We currently only allow access to the predictions, y, X_data and sample_weight to the metrics.

auto-sklearn/autosklearn/metrics/__init__.py

Lines 47 to 54 in 0ae2463

	def __call__(
	self,
	y_true: np.ndarray,
	y_pred: np.ndarray,
	X_data: Optional[SUPPORTED_XDATA_TYPES] = None,
	sample_weight: Optional[List[float]] = None,
	) -> float:
	pass

I can see why you might like to do this and it makes sense. In practice during evaluation, this is possible but during ensemble building, this might end up making things extremely slow to load/reload models constantly so any thought towards this has to be quite careful. We also don't want to make each of these seperate parameters, i.e. inference_time and model_size, that are passed to the metrc. If you have any ideas towards this then we'd be happy to hear :)

Unfortunately I don't see an easy hacky way to do this at the moment :/

If you're willing to make a PR towards this, that may help but I imagine we would aim for this as part of Usability v0.16 if we have a good idea o how to implement this in an extensible way.

Best,
Eddie

[mlindauer]

Hi Eddie,
Could we provide these kind of additional information (inference time, model size, etc) simply as optional dictionary argument to the metrics? We also don't have to recompute this all the time, but only have to do the bookkeeping once. I believe that these kind of meta-information about the model is useful in many ways, for the user to get more insights, for hardware-specific or application-specific metrics; maybe even for DeepCAVE to provide more detailed overviews.