[Feature] 请问使用API评测如何支持自定义数据集？

[Jimmy-L99]

Describe the feature

command：

(opencompass) root@node1:~/user/OpenCompass/opencompass# python run.py configs/vllm-glm4-9b-chat-custom.py

vllm-glm4-9b-chat-custom.py

from opencompass.models import OpenAISDK
from mmengine.config import read_base

with read_base():
    from .datasets.cmmlu.cmmlu_gen import cmmlu_datasets

datasets = []
datasets = cmmlu_datasets

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
    reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')],
)

models = [
    dict(
        abbr='glm-4-9b-chat-vllm-API',
        type=OpenAISDK,
        key='EMPTY',
        openai_api_base='http://localhost:port/v1', 
        path='glm-4-9b-chat',
        tokenizer_path='/root/user/models/glm-4-9b-chat',
        rpm_verbose=True, 
        meta_template=api_meta_template, 
        query_per_second=10, 
        max_out_len=1024,
        max_seq_len=4096,
        temperature=0.01,
        batch_size=8,
        retry=3,
    )
]

想在不更改OpenCompass代码的情况下实现，但是文档示例只有以下用法：

python run.py \
    --models hf_llama2_7b \
    --custom-dataset-path xxx/test_qa.jsonl \
    --custom-dataset-data-type qa \
    --custom-dataset-infer-method gen

Will you implement it?

• I would like to implement this feature and create a PR!

[tonysy]

You can change the models with the API models.