Evaluation output

[sbourke]

Hi Guys - we're playing around with QMF. Quick question - when running the following command

/qmf/bin$ ./wals --train_dataset=space.train --user_factors=user_factors --item_factors=item_factors --regularization_lambda=0.05 --confidence_weight=40 --nepochs=10 --nfactors=30 --nthreads=4 --test_avg_metrics=p@5,r@5 --test_always --num_test_users=30

We're expecting some form of output or information regarding the evaluation based on the documentation. The output from the binary is here.

./wals --train_dataset=space.train --user_factors=user_factors --item_factors=item_factors --regularization_lambda=0.05 --confidence_weight=40 --nepochs=10 --nfactors=30 --nthreads=4 --test_avg_metrics=p@5,r@5 --test_always --num_test_users=30
I0926 13:56:18.471247 18817 wals.cpp:85] loading training data
I0926 13:56:22.179965 18817 wals.cpp:95] training
I0926 13:56:33.102653 18817 WALSEngine.cpp:80] epoch 1: train loss = 54.118
I0926 13:56:43.987740 18817 WALSEngine.cpp:80] epoch 2: train loss = 53.8029
I0926 13:56:54.861482 18817 WALSEngine.cpp:80] epoch 3: train loss = 53.79
I0926 13:57:05.761577 18817 WALSEngine.cpp:80] epoch 4: train loss = 53.7868
I0926 13:57:16.658241 18817 WALSEngine.cpp:80] epoch 5: train loss = 53.7855
I0926 13:57:27.557302 18817 WALSEngine.cpp:80] epoch 6: train loss = 53.7849
I0926 13:57:38.434825 18817 WALSEngine.cpp:80] epoch 7: train loss = 53.7845
I0926 13:57:49.277307 18817 WALSEngine.cpp:80] epoch 8: train loss = 53.7843
I0926 13:58:00.133127 18817 WALSEngine.cpp:80] epoch 9: train loss = 53.7841
I0926 13:58:11.047634 18817 WALSEngine.cpp:80] epoch 10: train loss = 53.784
I0926 13:58:11.047685 18817 wals.cpp:99] saving model output

and no files are created in the bin dir. Can you give any suggestions as to what we could be doing wrong?

[JelleZijlstra]

Since you passed in --user_factors=user_factors, it should have created a file called user_factors. Did that not happen?

[sbourke]

The user and item factors output the feature space correctly. My assumption was that if I'm asking the binary to perform an evaluation of p@n that precision would be reported some place. If I have to compute p myself thats ok. But as a user of the binary I was assuming thats what the flag did 🍔

So my assumption was wrong?

[albietz]

@sbourke looks like you didn't specify your test dataset (--test_dataset). The computed test metrics will be reported in the logs.