Is eval_num_neg implemented?

[ghost]

According to the documentation:

--eval_num_neg (default 3): number of random negatives per positive used to generate the fixed evaluation sets mentioned above

However, I see it only once in the code, with a DEFINE
./qmf/bpr.cpp:42:DEFINE_uint64(eval_num_neg, 3, "number of negatives generated per positive in evaluation");

What is the purpose of this flag?

[albietz]

Hi.

The flag is passed to the BPREngine constructor (https://github.com/quora/qmf/blob/master/qmf/bpr.cpp#L101). Since the training objective consists of expectations over all pairs of items, computing these at each epoch is expensive, so instead we evaluate on a (fixed) monte-carlo approximation of the expectations, where eval_num_neg is the number of negative items sampled for each positive item.

[ghost]

Agreed. However, what is the purpose of --eval_num_neg?

When evaluating the test data, the number of negative examples should not matter. Or is it used for something else? I'm not sure why there are two parameters about negative examples.

--eval_num_neg
--num_negative_samples

[albietz]

num_negative_samples is used at training time when sampling negative examples in each SGD iteration. eval_num_neg is simply used once in the beginning to generate training and test evaluation sets for computing approximate training and test losses (note that it isn't used in the computation of other average test metrics like auc/ap/etc).
These loss values are mainly useful to check convergence (training loss) and overfitting (test loss), since they approximately correspond to what the algorithm is optimizing, while the other average metrics are more of a final measure of how the algorithm is doing.
Leaving eval_num_neg = 3 is a reasonable choice in most settings. (in contrast, changing num_negative_samples can have some impact on the final results or the convergence speed)

[ghost]

Thank you for clarifying. I'm using precision and recall, so eval_num_neg should have no effect.

Thank you,

[albietz]

Indeed. Thanks for raising the issue - I've added some clarifications in the readme.