Reproducibility issue of `XGBoostSampler` results for Windows and Linux

[marcofavoritobi]

Subject of the issue

There is a reproducibility issue for the XGBoostSampler across platforms. In particular, when I use XGBoostSampler on Linux and on Windows, I get different results even if random_state is set correctly.

Your environment

• OS: Windows 11, Ubuntu 22.04
• Python version: Python 3.10.9
• Package Version 0.2.1, commit cffd29e
• Anything else you consider helpful.

Steps to reproduce

Run the following script on both Windows and Ubuntu:

import numpy as np

from black_it.calibrator import Calibrator
from black_it.loss_functions.msm import MethodOfMomentsLoss
from black_it.samplers.halton import HaltonSampler
from black_it.samplers.xgboost import XGBoostSampler
from black_it.search_space import SearchSpace

xs = np.linspace(0, 1, 6)
ys = np.linspace(0, 1, 6)
xys_list = []
losses_list = []

for x in xs:
    for y in ys:
        xys_list.append([x, y])
        losses_list.append(x**2 + y**2)

xys = np.array(xys_list)
losses = np.array(losses_list)

sampler = XGBoostSampler(batch_size=4, random_state=0)
param_grid = SearchSpace(
    parameters_bounds=np.array([[0, 1], [0, 1]]).T,
    parameters_precision=np.array([0.01, 0.01]),
    verbose=False,
)
new_params = sampler.sample(param_grid, xys, losses)

On Ubuntu, the value of new_params is:

[[0.24 0.26]
 [0.37 0.21]
 [0.43 0.14]
 [0.11 0.04]]

On Windows, the value of new_params is:

[[0.24 0.26]
 [0.19 0.11]
 [0.13 0.22]
 [0.11 0.05]]

Expected behaviour

I expected to get the same result.

Actual behaviour

Different result, see above.

Furhter comments

I understand that the provided details might not be enough for the exact reproduction of the bug. This issue is more a way to point out the reproducibility issue when using the XGBoostSampler.

On the other hand, I suspect this does not depends on black-it, but rather on the underlying implementation of XGBoost, provided by the package xgboost. The following links already document reproducibility issues for that package:

• Reproducibility: different result on first run with gpu_hist on single GPU  dmlc/xgboost#8820
• gblinear non-determinism dmlc/xgboost#4919
• boosters that should be training identically are not dmlc/xgboost#6711
• Non deterministic results with dask when using version 1.6.2 dmlc/xgboost#8701
• Cannot reproduce optimization results for XGBClassifier dmlc/xgboost#7057

[jblaese]

is this still being worked on? Would be important to have XGBoost be deterministic across all hardware.