-
Notifications
You must be signed in to change notification settings - Fork 437
Open
Description
Describe the bug
The lsat_qa tasks were configured with evaluation_splits = ["validation", "test"], but lighteval/lsat_qa only has test split
To Reproduce
task = "lsat_qa|5"
pipeline = Pipeline(
tasks=task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
pipeline.evaluate()
pipeline.save_and_push_results()
pipeline.show_results()```
## Expected behavior
The configuration should only reference the `test` split to ensure successful loading and evaluation.
## Version info
- OS: mac
- Lighteval version: main (local development)Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels