METEOR bug when input has newline character

[skasai5296]

Hi,

I've been using this library for evaluation, but I get this error every now and then...
ValueError: could not convert string to float: '29.0 8.0 0.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0'

When I print my eval_line, it shows this (partly omitted):
eval_line: EVAL ||| 29.0 9.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 12.0 12.0 4.0 6.0 2.0 2.0 2.0 2.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 1.0 4.0 6.0 7.0 ||| Error: specify SCORE or EVAL or SING ||| 29.0 12.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 10.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 13.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 13.0 0.0 7.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 12.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 8.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 13.0 0.0 7.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 8.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 11.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 10.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 11.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 10.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 11.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 10.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 11.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 11.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 9.0 0.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 13.0 0.0 7.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 11.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 12.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ||| 29.0 16.0 0.0 9.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

So self._stat in meteor.py seemed to be returning an error string.
I don't know a good workaround, do you have an idea of why this is occurring?

[juharris]

Thanks for letting us know. If you're able to consistently reproduce the issue and you know the exact string that causes it, you can get more details by uncommenting this line: https://github.com/Maluuba/nlg-eval/blob/master/nlgeval/pycocoevalcap/meteor/meteor.py#L91
There's a comment above the line to explain more. Sorry we couldn't find a good way to consistently debug Meteor so we have to resort to this.

[juharris]

Actually I'm not sure that will help much. Do you have a stacktrace?
Perhaps some debugging in the _stat method would help.

[skasai5296]

Thank you for the advice! I'll try debugging the _stat method.

Actually, when I uncommented the error line, I retrieved this from meteor:

Error from Meteor:
b'Exception in thread "main" java.util.InputMismatchException\n\tat java.base/java.util.Scanner.throwFor(Scanner.java:939)\n\tat java.base/java.util.Scanner.next(Scanner.java:1594)\n\tat java.base/java.util.Scanner.nextDouble(Scanner.java:2564)\n\tat edu.cmu.meteor.scorer.MeteorStats.<init>(Unknown Source)\n\tat Meteor.scoreStdio(Unknown Source)\n\tat Meteor.main(Unknown Source)\n'

not sure of what this implies though...

I guess the input to self._stat is wrong???

[skasai5296]

I wasn't able to reproduce this error consistently, so I'm closing the issue.

[skasai5296]

I prepared a minimal sample! It seems that a newline character is doing the bad stuff.

from nlgeval import NLGEval
evaluator = NLGEval()
ref = ["hello\nworld"]
hyp = "hello world"
evaluator.compute_metrics(hyp, ref)

Error handling value: b'Error: specify SCORE or EVAL or SING\n'
Decoded value: Error: specify SCORE or EVAL or SING
eval_line: EVAL ||| 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aab10821no/anaconda3/envs/anet2/lib/python3.7/site-packages/nlgeval/__init__.py", line 297, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/home/aab10821no/anaconda3/envs/anet2/lib/python3.7/site-packages/nlgeval/pycocoevalcap/meteor/meteor.py", line 84, in compute_score
    scores.append(float(dec(v.strip())))
ValueError: could not convert string to float: 'Error: specify SCORE or EVAL or SING'

[skasai5296]

It seems to be referenced in the original pycocoevalcap repository, but isn't merged.
So I'm guessing we need to work it around in our codes?
tylin/coco-caption#35

[kracwarlock]

@temporaer The fix in tylin/coco-caption#35 just removes the \r, \n characters which I don't like. Should we replace them with a space? Do you have a better strategy?

[temporaer]

generally replacing with a space makes more sense, how about something like

re.sub(r'\s+', ' ', score_line)

? Or should we aim for maximum comparability here?

[kracwarlock]

@skasai5296 Does #94 resolve your errors?

[skasai5296]

@kracwarlock
Yes! Thank you for the fix.