Question: Best practice for early stopping with custom evaluation metric?

[lit40190]

Hi XGBoost team,

I’m using XGBoost with a custom evaluation metric and early stopping. I’m a bit unsure about the recommended setup to ensure early stopping is triggered on the intended metric.

Environment

OS: [Windows 10]

xgboost version: [ 2.1.4]

Python: [3.11.9]

What I tried

I pass eval_metric and a custom feval / callback metric.

I set early_stopping_rounds=[N] with an evaluation set.

Question

When both built-in and custom metrics are present, which metric does early stopping use by default?

Is the recommended approach to explicitly set the metric ordering / direction (higher is better vs lower is better) for custom metrics?

Any pointer to the best-practice docs or examples would be appreciated. Thanks!

[trivialfis]

You can use the EarlyStopping callback to explicitly control all these options.

[memeplex]

Hi, slightly OT, but I need to use the callback for another reason and am perplexed by the following situation:

• I get "best_iteration is only defined when early stopping is used" when I'm not passing early_stopping_rounds to the initializer, even though early stopping is actually being done (at least eval_set is passed down to fit and the metric is evaluated).

• The callback itself has the attribute rounds that may conflict with early_stopping_rounds.

So how is this solved? Should I set the same value in both places? Because of the second point, early_stopping_rounds is in practice mandatory, but I also need to use the callback.

[trivialfis]

The callback is made to be extensible, whenever in doubt, you can refer to the source code

xgboost/python-package/xgboost/callback.py

Line 448 in 25e6278

model.set_attr(best_score=str(get_s(record)), best_iteration=str(epoch))

Basically, the built-in Early stop callback sets the attribute whenever the metric stops improving for a specific number of rounds.

[memeplex]

I understand that, or at least I believe so. What I'm pointing out is that it seems to be two overlapping ways of achieving the same and both are required in certain situations.

If you want to use the callback then you have to pass the mandatory rounds callback initialization argument.

It you want to enable early stopping at all, you also have to pass early_stopping_rounds to the regressor initializer, alongside the callback with a redundant rounds attribute.

Perhaps I'm missing something obvious here, but I'm confused about this.

[memeplex]

Perhaps this statement in the documentation:

early_stopping_rounds: Activates early stopping

is what's confusing. Coupled with what I now think was a side effect of refitting the same instance and getting "best_iteration is only defined when early stopping is used" the second time (maybe a bug, maybe expected, but definitely a different issue) which I attributed to not having set early_stopping_rounds since I was using the callback anyway.

[trivialfis]

The early_stopping_rounds is just a convenient parameter for XGBoost to add the early stop callback automatically. If you explicitly pass the call back, then there's no need to specify that parameter.

Feel free to make suggestions for improving the document. ;-)