Same model giving different predictions in Python and C++

[sidrgupte]

I trained an XGB Regressor in Python and saved it as following:

trained_model = model.train_model(best_params)
json_path = os.path.join(manual_save_path, f"{target}.json")
booster = trained_model.get_booster()
booster.save_model(json_path)

Predictions:

1. Just to be sure, I do the predictions in a separate file:

booster = xgb.Booster()
booster.load_model(json_path)
point = [-3.17492, 17.9437, 34.5295, 3.3425]
dtest_point = xgb.DMatrix([point])
pred = booster.predict(dtest_point)
print(f"Input vector: {point}\nPrediction: {pred[0]}")

I get the following:

Input vector: [-3.17492, 17.9437, 34.5295, 3.3425]
Prediction: 1.0478190183639526

2. Now, I do exactly the same in C++:

// test_xgb.cpp
#include <xgboost/c_api.h>
#include <iostream>
#include <vector>
#include <cmath> // for NAN

int main()
{
    BoosterHandle booster;
    if (XGBoosterCreate(nullptr, 0, &booster) != 0)
    {
        std::cerr << "Error creating booster!" << std::endl;
        return -1;
    }
    const char *model_path = "P.json";

    if (XGBoosterLoadModel(booster, model_path) != 0)
    {
        std::cerr << "Error loading model from " << model_path << std::endl;
        return -1;
    }

    std::vector<float> input = {-3.17492f, 17.9437f, 34.5295f, 3.3425f};
    std::cout << "Input vector: ";
    for (float f : input)
        std::cout << f << " ";
    std::cout << std::endl;
    DMatrixHandle dmat;

    if (XGDMatrixCreateFromMat(input.data(), 1, input.size(), NAN, &dmat) != 0)
    {
        std::cerr << "Error creating DMatrix!" << std::endl;
        return -1;
    }
    const char *json_config = R"({
    "type": 0,
    "training": false,
    "iteration_begin": 0,
    "iteration_end": 0,
    "strict_shape": true
})";

    const bst_ulong *out_shape;
    bst_ulong out_dim;
    const float *out_result;

    if (XGBoosterPredictFromDMatrix(
            booster,
            dmat,
            json_config,
            &out_shape,
            &out_dim,
            &out_result) != 0)
    {
        std::cerr << "Error predicting!" << std::endl;
        return -1;
    }
    bst_ulong out_len = 1;
    for (bst_ulong i = 0; i < out_dim; ++i)
    {
        out_len *= out_shape[i];
    }

    std::cout << "Prediction: ";
    for (bst_ulong i = 0; i < out_len; ++i)
    {
        std::cout << out_result[i] << " ";
    }
    std::cout << std::endl;

    XGDMatrixFree(dmat);
    XGBoosterFree(booster);

    return 0;
}

This gives me the following output:

Input vector: -3.17492 17.9437 34.5295 3.3425 
Prediction: 0.562144 

Python XGBoost version: 3.1.0
C++ XGBoost version: 3.1.0

I also tried saving the models as .bin file but did not work either!
Have I committed a silly mistake somewhere? Been stuck at this for the last 2 hours!

[trivialfis]

If the difference is significant, then the tree is taking different branches. I can't provide a definite answer without reproducing it myself. My guess is that the Python code uses f64 by default, and is converted to f32 inside XGBoost. During the text -> binary or during the f64 -> f32, the data is slightly changed due to floating-point parsing or conversion.

[trivialfis]

To illustrate, using the first number in your input:

>>> from decimal import Decimal
>>> a = -3.17492
>>> Decimal(a)
Decimal('-3.1749200000000001864464138634502887725830078125')

[sidrgupte]

That is something I have already explored. Still, here is a snippet where I tried both - f32 and f64.

point64 = np.array([-3.17492, 17.9437, 34.5295, 3.3425], dtype=np.float64)
point32 = np.array([-3.17492, 17.9437, 34.5295, 3.3425], dtype=np.float32)

d64 = xgb.DMatrix(point64.reshape(1, -1))
d32 = xgb.DMatrix(point32.reshape(1, -1))

pred64 = e_p.predict(d64)[0]
pred32 = e_p.predict(d32)[0]

print(f"f64 input prediction: {pred64:.6f}")
print(f"f32 input prediction: {pred32:.6f}")

It gives:

f64 input prediction: 1.047819
f32 input prediction: 1.047819

I am also attaching the model if you want to check it out. P.json

[trivialfis]

I couldn't reproduce the issue using the "P.json" you provided, tested with both 3.1 branch and the development branch.

#include <xgboost/c_api.h>

#include <cmath>
#include <iostream>
#include <vector>

int main() {
    BoosterHandle booster;
    XGBoosterCreate(nullptr, 0, &booster);
    if (XGBoosterLoadModel(booster, "P.json") != 0) {
        std::cerr << "Error loading model: " << XGBGetLastError() << std::endl;
        return 1;
    }

    std::vector<float> input = {-3.17492f, 17.9437f, 34.5295f, 3.3425f};

    DMatrixHandle dmat;
    XGDMatrixCreateFromMat(input.data(), 1, input.size(), NAN, &dmat);

    const char *json_config = R"({
        "type": 0,
        "training": false,
        "iteration_begin": 0,
        "iteration_end": 0,
        "strict_shape": true
    })";

    const bst_ulong *out_shape;
    bst_ulong out_dim;
    const float *out_result;

    if (XGBoosterPredictFromDMatrix(booster, dmat, json_config,
                                    &out_shape, &out_dim, &out_result) != 0) {
        std::cerr << "Error predicting: " << XGBGetLastError() << std::endl;
        return 1;
    }

    std::cout << "Prediction: " << out_result[0] << std::endl;

    XGDMatrixFree(dmat);
    XGBoosterFree(booster);
    return 0;
}

import numpy as np
import xgboost as xgb

booster = xgb.Booster()
booster.load_model("P.json")

point = [-3.17492, 17.9437, 34.5295, 3.3425]
dtest = xgb.DMatrix(np.array([point]))
pred = booster.predict(dtest, strict_shape=True)
print(f"Prediction: {pred[0]}")

[trivialfis]

You referenced an issue with the evaluation metric. That's irrelevant to the base score

[sidrgupte]

I referenced an issue which is literally what I showed you above i.e. when I use XGBoosterSaveJsonConfig to print the base_score which is literally the first key in learner_model_param dictionary, it prints 0.5 which is not the correct value.

[trivialfis]

I used 3.1 to print the config as in your code, the model config looks right:

./repro 

Model config:
{"learner":{"generic_param":{"device":"cpu","fail_on_invalid_gpu_id":"0","n_jobs":"0","nthread":"0","random_state":"0","seed":"0","seed_per_iteration":"0","validate_parameters":"0"},"gradient_booster":{"gbtree_model_param":{"num_parallel_tree":"1","num_trees":"422"},"gbtree_train_param":{"process_type":"default","tree_method":"auto","updater":"grow_quantile_histmaker","updater_seq":"grow_quantile_histmaker"},"name":"gbtree","specified_updater":false,"tree_train_param":{"alpha":"0","cache_opt":"1","colsample_bylevel":"1","colsample_bynode":"1","colsample_bytree":"1","eta":"0.300000012","gamma":"0","grow_policy":"depthwise","interaction_constraints":"","lambda":"1","learning_rate":"0.300000012","max_bin":"256","max_cat_threshold":"64","max_cat_to_onehot":"4","max_delta_step":"0","max_depth":"6","max_leaves":"0","min_child_weight":"1","min_split_loss":"0","monotone_constraints":"()","refresh_leaf":"1","reg_alpha":"0","reg_lambda":"1","sampling_method":"uniform","sketch_ratio":"2","sparse_threshold":"0.20000000000000001","subsample":"1"},"updater":[{"hist_train_param":{"debug_synchronize":"0","extmem_single_page":"0","max_cached_hist_node":"18446744073709551615"},"name":"grow_quantile_histmaker"}]},"learner_model_param":{"base_score":"[9.856754E-1]","boost_from_average":"1","num_class":"0","num_feature":"4","num_target":"1"},"learner_train_param":{"booster":"gbtree","disable_default_eval_metric":"0","multi_strategy":"one_output_per_tree","objective":"reg:squarederror"},"metrics":[],"objective":{"name":"reg:squarederror","reg_loss_param":{"scale_pos_weight":"1"}}},"version":[3,1,3]}
Prediction: 1.04782

/**
g++ -std=c++17 -o repro repro.cpp -I../xgboost/include -L../xgboost/lib -L$CONDA_PREFIX/lib -lxgboost -lgomp -Wl,-rpath,$(realpath ../xgboost/lib) -Wl,-rpath,$CONDA_PREFIX/lib
 */

#include <xgboost/c_api.h>

#include <cmath>
#include <iostream>
#include <vector>

int main() {
    BoosterHandle booster;
    XGBoosterCreate(nullptr, 0, &booster);
    if (XGBoosterLoadModel(booster, "P.json") != 0) {
        std::cerr << "Error loading model: " << XGBGetLastError() << std::endl;
        return 1;
    }

    // the model config:
    bst_ulong config_len;
    const char *config;
    if (XGBoosterSaveJsonConfig(booster, &config_len, &config) == 0)
    {
      std::cout << "\nModel config:\n"
                << config << std::endl;
    }

    std::vector<float> input = {-3.17492f, 17.9437f, 34.5295f, 3.3425f};

    DMatrixHandle dmat;
    XGDMatrixCreateFromMat(input.data(), 1, input.size(), NAN, &dmat);


    const char *json_config = R"({
        "type": 0,
        "training": false,
        "iteration_begin": 0,
        "iteration_end": 0,
        "strict_shape": true
    })";

    const bst_ulong *out_shape;
    bst_ulong out_dim;
    const float *out_result;

    if (XGBoosterPredictFromDMatrix(booster, dmat, json_config,
                                    &out_shape, &out_dim, &out_result) != 0) {
        std::cerr << "Error predicting: " << XGBGetLastError() << std::endl;
        return 1;
    }

    std::cout << "Prediction: " << out_result[0] << std::endl;

    XGDMatrixFree(dmat);
    XGBoosterFree(booster);
    return 0;
}

[sidrgupte]

Alright, It definitely had to do something with the older version. I downloaded the latest (XGBoost version: 3.3.0), did not touch a single thing in the old code and the problem is resolved! I am getting the correct predictions now. Thank you for your time though.

[trivialfis]

If you can't reproduce it using the scripts above, then we can narrow the issue down to something more specific:

• Are you copying the prediction output before the next call?
• Are you re-creating the DMatrix for new inputs?

[sidrgupte]

• I tried your code exactly as it is. Same old result.
• No. For now, I am just doing it for a single input.