MedReason

[warner-benjamin]

Implement verifiers env for MedReason

Implement a verifiers environment for the MedReason dataset. Use the authors’ question, system, and LLM-as-judge prompts if provided. Only create a training dataset if a training split exists.

Paper: https://arxiv.org/abs/2504.00993
Dataset / Project: https://huggingface.co/datasets/UCSC-VLAA/MedReason

Format & Env

• Mixed (some multiple-choice, some open-ended):
  • MC tasks: accuracy-based evaluation
  • Open-ended reasoning: vf.JudgeRubric()

[tmabraham]

Correct me if I am wrong, but my understanding is that MedReason is mainly a combination of existing datasets like PubMedQA, MedQA, etc. except that they are using some specialized knowledge graph-based approach for generating CoT traces which they can use for SFT. But this is not relevant for RL and evals I think, so for that it's just the same as existing datasets no?

[warner-benjamin]

I was thinking the knowledge graph-based reasoning could be used for a LLM as a Judge reward, potentially making this dataset more useful than just MCQ datasets.

[Manishram-ai]

I can try to take on this task :)

[AdepojuJeremy]

Can you reassign this task to me