How is the answer sentences split?

[DingQiang2018]

Hi,

Thanks for sharing your dataset. However, I had a problem processing it. In the English part, it is unclear how each annotation is mapped to the answer sentences. For example, the following answer has three sentences but has annotations for five segments. Although this example may indicate the sentences are split by ". ", other more complicated examples violate this rule. May I ask how you split the sentences?

answer:

When the consistory of Eastern Ave. CRC was approached by three men with a protest against Rev. Hoeksema, they initially disagreed that it was sin. However, they agreed that it was a public matter. The consistory then put the three men under discipline.

annotation:

["\n None\n None", " None\n On January 19, 1924, three men of the Eastern Ave.congregation (Hoeksema's congregation), approached his door.They brought against him a protest against some of his views", " None\n The consistory agreed that this was a public matter, but disagreed that it was sin.They then put the three men under discipline", " None\n The consistory agreed that this was a public matter, but disagreed that it was sin", " None\n They then put the three men under discipline"]

[Liqu1d-G]

You can use the sentence_tokenize function of eval/anah_v2/utils.py.

We have included the example in eval/anah_v2/eval.py you can refer to.

    for line in lines:
        response = line["response"]
        question = line["question"]
        language = ref_lines[question]["language"]
        document = ref_lines[question]["document"]
        sentences = sentence_tokenize(response, language, keep_end=False)
        for sent in sentences:
            annotation = process_question(model, tokenizer, question, sent, document, language)
            new_lines.append({"question": question, "response": response, "sentence": sent, "annotation": annotation, "language": language})