Ability to choose columns from dataset

[etemiz]

Describe the feature

I want to be able to choose specific column (gt_response) because response column is for thinking process.

Dataset: https://huggingface.co/datasets/yingyingzhang/metamath-qwen2-math

Paste any useful information

When I add this to dataset-info.json:

"metamath": {
"hf_dataset_id": "yingyingzhang/metamath-qwen2-math",
"split": ["validation"],
"columns": {
"gt_response": "response"
}
},

I get the following error.

ValueError: New column name response already in the dataset. Please choose a column name which is not already in the dataset. Current columns in the dataset: ['prompt', 'response', 'gt_response', 'gold_ans', 'pred_ans', 'type', 'score', 'query']

Is there a way to overwrite existing columns? Or specifying what to download from hf or what to include in training?

Thanks!

[Jintao-Huang]

You might be able to do it this way:

"columns": {
"gt_response": "response", "response": "_"
}

[etemiz]

this worked

"columns": {
"response": "-",
"gt_response": "response"
}