[GENERAL SUPPORT]: Question about scaling using thresholds in Ax for Bayesian Optimization

[Amanichab]

Question

I’m working on a Bayesian Optimization competition, and I’m trying to understand the best way to apply scaling based on thresholds .

Could you please share any advice or recommendations on how to properly implement scaling (like winsorization or normalization) when thresholds are involved? I want to make sure the scaling is done correctly, especially in the context of Bayesian Optimization, and for cases like multi-objective .

Here is the link to our main issue for more context .

Please provide any relevant code snippet if applicable.

Code of Conduct

• I agree to follow this Ax's Code of Conduct

[mgarrard]

Hi @Amanichab - thanks for reaching out. Would be curious to understand a bit more specifics of what you are asking, and if you are currently using Ax and/or BoTorch for this work?

If you are, could you provide any relevant code snippets? I looked through the linked issue, but it is quite long so any help narrowing down the context to what is relevant for this question would be awesome :)

[Amanichab]

Thanks a lot for your interest @mgarrard . To keep it short, we're organizing a BO benchmarking competition.

For the evaluation metric, we plan to use the hypervolume indicator, with the reference point defined by the objective thresholds provided for the problem .

We compute the hypervolume indicator using pymoo . That said, to make the hypervolume meaningful, we also want to scale the objectives appropriately. We're currently considering winsorization - we’re aiming to follow the approach used in Ax as a reference. But we're still exploring how to implement scaling based on thresholds and what the best practices are .

[Balandat]

Is there observation noise in your setting? Winsorization is typically used as a way to handle outliers in the BO algorithm and less so in terms of performance evaluation (esp. if you have deterministic observations).

As far as scaling the objectives within a particular problem is concerned, I'm not sure there is general advice - it will really depend on what the decision maker cares about in the particular problem.

If you're more concerned about making the HV across multiple problems comparable so you can aggregate it across problems, my advice would be to compute the HV relative to some problem baseline (could be the HV of the PF found by using MOO with some evolutionary algorithm and massive budget, or the HV of the PF found by generating some number of random exploration points that characterizes some notion of "average performance").

[Balandat]

cc @sdaulton

[sgbaird]

A few clarifications:

• we have known objective thresholds, which we plan to supply to the participants
• We're using a "Turing test" optimization benchmark - a while back, I brute forced ~200k hyperparameter tuning runs for a property prediction task which included repeats. This was purely for the purpose of making a reusable optimization benchmark. The objectives are RMSE, MAE, training time, and model size
• there is observation noise, at least from the participant perspective, but "behind the scenes" we're randomly sampling a number from 0 to 1 that controls heteroskedastic noise (possible because I ran repeats in the dataset described above). Setting this to 0.5 makes it deterministic and is what will define the "true" Pareto front from a leaderboard perspective
• we can approximate the true Pareto front easily, behind-the-scenes (i.e., what you talked about with large scale evaluations). For example, we could run Sobol on 1B combinations or use a generic algorithm, etc.
• The following is a hosted version of that benchmark and describes the task in optimization language: https://huggingface.co/spaces/AccelerationConsortium/crabnet-hyperparameter

This could possibly be rephrased another way: what scaling would be applied by default to objectives to compute expected hypervolume improvement for a multi-objective task, assuming that a user has supplied custom objective thresholds? I thought this was winsorization (mostly based on warning logs when custom thresholds aren't supplied), but I might be missing something. For example, a chain of transforms.

[Balandat]

I'd have to dig into the details what exactly is happening in Ax (@sdaulton would know better), but I believe it would be winsorization and then a standardization making both objectives zero mean and unit variance.

If this is for evaluation purposes, then I'm not sure winsorization or standardization matters if you have access to the "noiseless ground truth". Basically, if you feel as the person solving the problem it makes sense to scale the objectives you should do that, and then simply provide the thresholds to the users. Whatever their algorithm does is up to them, but you'd have a consistent way of computing the HV based on the ground truth results you can compute from the candidate trace they'll generate.