test.join aggregation does not join probabilities

[berndbischl]

task = sonar.task
lrn = makeLearner("classif.rpart", predict.type = "prob")

rin = makeResampleDesc("Holdout")
mm = setAggregation(auc, test.join)
r = resample(lrn, task, rin, measures = mm)

This leads to
Error in getPredictionProbabilities(pred, cl = levs) :
Trying to get probabilities for nonexistant classes: M,R

It is also obvious from the source code.
NB: we have potentially more columns we need to join like se and the stuff from survival...

@mllg as you pushed this some time ago

[larskotthoff]

Did the PR solve this completely or should we leave this open?

[giuseppec]

The part

NB: we have potentially more columns we need to join like se and the stuff from survival...

was not addressed in the PR, therefore we could leave this open.

[jakob-r]

This error still prevails

task = wpbc.task
lrn = makeLearner("surv.rpart")
rin = makeResampleDesc("Holdout")
mm = setAggregation(cindex, test.join)
r = resample(lrn, task, rin, measures = mm)
#[Resample] holdout iter: 1
#Error in FUN(X[[i]], ...) : 
#  You need to have 'truth.time' and 'truth.event' columns in your pred object for measure cindex!

[jakob-r]

And

task = yeast.task
lrn = makeLearner("multilabel.cforest")
rin = makeResampleDesc("Holdout")
mm = setAggregation(multilabel.acc, test.join)
r = resample(lrn, task, rin, measures = mm)
#[Resample] holdout iter: 1
#Error in `row.names<-.data.frame`(`*tmp*`, value = value) : 
#  invalid 'row.names' length