compare型数据集主观测评

[MHHL-Frank]

想用alpacaeval进行主观评测，参考了eval_subjective_alpacaeval_official.py，
有两个问题：
1、eval_subjective_alpacaeval_official.py文件里只有一个模型chatglm3-6b，还有一个gpt4-turbo作为judgemodel，这指的是chatglm3-6b和got4-turbo进行比较吗，如果不是的话怎么写入两个模型，并用gpt-turbo进行judge；

2、gpt4_judge = dict(
abbr='GPT4-Turbo',
path='gpt-4-1106-preview',
key='', # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
config='weighted_alpaca_eval_gpt4_turbo'
)
这个里面config是什么意思，如果想用别的API模型如qwen-max该怎么写

[bittersweet1999]

首先这里会有三个模型，一个是你需要测试他性能的模型，也就是config里的models变量，在demo的config里是chatglm3-6b。第二个是需要和它比较的模型，在alpacaeval的官方榜单里是某个版本的gpt4，这个是固定的不能更改的，所以在OC里隐藏起来了，不管哪个模型进来都是和它进行比较，如果需要改的话需要更改另一个地方的设置，但是你如果要和官方榜单比较的话不需要管这一点。第三个就是judge model了，利用judge model对刚刚前两个模型的结果进行比较，选出一个好的来，也就是config里的judgemodels。 你的第二个问题，如果你想要替换judge model，那么注意你的结果也是不能和榜单比较的（因为judge model也替换了，官方榜单的judge model是gpt4-turbo，只能变成相当于自己内部建立一个榜单进行比较了），如果要换judge model，直接和写一个model的config是一样的，这样即可

from opencompass.models import Qwen
qwen_max_0428 = dict(
    abbr="qwen-max-0428",
    retry=10,
    type=Qwen,
    # path="qwen-max-0428",
    path="qwen-max",  # identical to qwen-max-0428, for now
    key="",
    generation_kwargs={
        "enable_search": False,
    },
    meta_template=api_meta_template,
    query_per_second=1,
    max_out_len=2048,
    max_seq_len=2048,
    batch_size=8,
)```

[MHHL-Frank]

好的，谢谢，不过qwen_max_0428=这里应该还有个中括号吧[dict(...)]

[bittersweet1999]

你写成judge_models = [qwen_max_0428]就行了

[MHHL-Frank]

我需要调用qwen的API，如下
judgemodel = [dict(
abbr="qwen-max-latest",
retry=10,
type=Qwen,

path="qwen-max-latest",

path="qwen-max-latest", # identical to qwen-max-0428, for now
key="xxx",
generation_kwargs={
"enable_search": False,
},
meta_template=api_meta_template,
query_per_second=1,
max_out_len=2048,
max_seq_len=2048,
batch_size=8,
)]
后面
eval = dict(
partitioner=dict(
type=NaivePartitioner
),
runner=dict(
type=LocalRunner,
max_num_workers=256,
task=dict(type=AlpacaEvalTask, judge_cfg=judgemodel),
)
)
judge_cfg这里是怎么写的，我改成task=dict(type=AlpacaEvalTask, judge_cfg=judgemodel)后会出现问题：
File "/home/jovyan/opencompass/opencompass/tasks/outer_eval/alpacaeval.py", line 95, in init
assert type(judge_cfg) == ConfigDict
AssertionError

[MHHL-Frank]

删掉括号后还是同样的问题

[bittersweet1999]

opencompass/configs/eval_subjective.py

Line 70 in 85a2887

judge_models = [dict(

这个关键字段应该是judge_models，你这里是judgemodel。
还有一个问题忘了跟你说了，你这边用的是qwen做judgemodel的话，走不了这个official的config，因为那个只能用GPT做judge model，alpacaeval那边是这样的，alpacaeval的lc win rate用了logit prob，其他模型不确保能输出这个。所以我推荐你如果要换judge model的话，参考https://github.com/open-compass/opencompass/blob/main/configs/eval_subjective.py 这个config即可。
也就是写成下面这样就能跑了

from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.subjective.alpaca_eval.alpacav2_judgeby_gpt4 import alpacav2_datasets
from opencompass.models import HuggingFaceCausalLM, HuggingFace, HuggingFaceChatGLM3, OpenAI
from opencompass.partitioners import NaivePartitioner, SizePartitioner
from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
from opencompass.partitioners.sub_size import SubjectiveSizePartitioner
from opencompass.partitioners.sub_num_worker import SubjectiveNumWorkerPartitioner
from opencompass.runners import LocalRunner
from opencompass.runners import SlurmSequentialRunner
from opencompass.tasks import OpenICLInferTask
from opencompass.tasks.subjective_eval import SubjectiveEvalTask
from opencompass.summarizers import SubjectiveSummarizer

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ]
)

# -------------Inference Stage ----------------------------------------
# For subjective evaluation, we often set do sample for models
models = [your models
]


datasets = [*alpacav2_datasets] # add datasets you want


infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=OpenICLInferTask)),
)
# -------------Evalation Stage ----------------------------------------

## ------------- JudgeLLM Configuration
judge_models = [your qwen]

## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=SubjectiveSummarizer, function='subjective')
work_dir = 'outputs/subjective/'

把models和judge_models换成你需要的即可

[bittersweet1999]

可能和多个因素有关系，你的模型的prediction是否采样，以及你是使用的哪个evaluate的config跑的呢

[chanchimin]

from mmengine.config import read_base
from opencompass.partitioners import NaivePartitioner, NumWorkerPartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask, OpenICLEvalTask
# from opencompass.models import VLLM, HuggingFaceBaseModel
from opencompass.models import HuggingFaceBaseModel
# from opencompass.models import TurboMindModel
from opencompass.models import OpenAI
from opencompass.tasks.subjective_eval import SubjectiveEvalTask
from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
from opencompass.partitioners.sub_num_worker import SubjectiveNumWorkerPartitioner
from opencompass.models import VLLM
from opencompass.summarizers import SubjectiveSummarizer

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ]
)


with read_base():

    from opencompass.configs.datasets.subjective.alpaca_eval.alpacav2_judgeby_gpt4 import alpacav2_datasets
    #from opencompass.configs.datasets.subjective.multiround.mtbench_single_judge_diff_temp import mtbench_datasets
    #from opencompass.configs.datasets.subjective.arena_hard.arena_hard_compare import arenahard_datasets
    # from opencompass.configs.datasets.subjective.multiround.mtbench101_judge import mtbench101_datasets


work_dir = ###

datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])

paths = [
    ###
]
#
abbrs = [
    "llama3_8b_instruct",
]

all_models = [
    dict(
            type=VLLM,
            abbr=abbr,
            path=path,
            max_out_len=2048,
            max_seq_len=2048,
            batch_size=16,
            generation_kwargs=dict(temperature=0),
            run_cfg=dict(num_gpus=1),
            # model_kwargs=dict(gpu_memory_utilization=0.6)
        ) for abbr, path in zip(abbrs, paths)
]


models = [
*all_models
]

infer = dict(
    partitioner=dict(
        type=NumWorkerPartitioner,
        num_worker=16,
    ),
    runner=dict(
        type=LocalRunner,
        max_num_workers=16,
        max_workers_per_gpu=1,
        retry=1, # Modify if needed
        task=dict(type=OpenICLInferTask)
    ),
)



judge_models = [dict(
    abbr='GPT4-Turbo',
    # abbr='GPT4O-mini',
    type=OpenAI,
    # path='gpt-4o-mini',
    path='gpt-4-1106-preview',
    openai_api_base='###',
    key='###',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
    meta_template=api_meta_template,
    query_per_second=32,
    max_out_len=2048,
    max_seq_len=2048,
    batch_size=8,
    temperature=0,
)]

## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNumWorkerPartitioner, num_worker=8, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)


summarizer = dict(type=SubjectiveSummarizer, function='subjective')

我使用的是这个脚本主要是参考configs/eval_subjective.py 改的，主要改动是1) type改成了vllm，以及2）load datasets的时候源代码有一个错误我修正成了“from opencompass.configs.datasets.subjective.alpaca_eval.alpacav2_judgeby_gpt4 import alpacav2_datasets”， 可以辛苦看看这个config这么写有问题吗？

[bittersweet1999]

看看是不是因为greedy造成的呢，greedy测主观能力的时候一般会比sample要低的

[chanchimin]

感谢您的及时回复，我debug发现是因为在最终送入vllm进行推理的时候并没有加上template, 请问这个template应该是在哪个地方通过什么方式设置呢？

[bittersweet1999]

opencompass/configs/models/hf_llama/lmdeploy_llama3_1_8b_instruct.py

Line 14 in bcb707d

stop_words=['<|end_of_text|>', '<|eot_id|>'],

你说的是这个stopwords?

[ftgreat]

@chanchimin 请问你有跑成功的AlpacaEval评测的例子吗，谢谢

[bittersweet1999]

如果要跑AlpacaEvalv2，得到win rate的话，用这个config可以跑的https://github.com/open-compass/opencompass/blob/aeded4c4db6e070a8ee9e8126358477c6d08dcee/configs/eval_subjective.py#L5，其他主观数据集也可以一起跑。如果要得到length control win rate，由于涉及到需要模型的logprob，需要使用这个config跑，https://github.com/open-compass/opencompass/blob/main/configs/eval_subjective_alpacaeval_official.py

[ftgreat]

谢谢回复。我这边是 AzureOpenAI 的key。想问下怎么适配

[bittersweet1999]

建议用这个config
https://github.com/open-compass/opencompass/blob/aeded4c4db6e070a8ee9e8126358477c6d08dcee/configs/eval_subjective.py#L5%EF%BC%8C%E5%85%B6%E4%BB%96%E4%B8%BB%E8%A7%82%E6%95%B0%E6%8D%AE%E9%9B%86%E4%B9%9F%E5%8F%AF%E4%BB%A5%E4%B8%80%E8%B5%B7%E8%B7%91%E3%80%82%E5%A6%82%E6%9E%9C%E8%A6%81%E5%BE%97%E5%88%B0length

[ftgreat]

多谢。看judge model是这个type=OpenAI。想问下怎么适配AzureOpenAI，直接适配了就可以使用吗？

[bittersweet1999]

Azure没有特殊支持过，你可以参考

opencompass/opencompass/models/openai_api.py

Line 36 in aeded4c

class OpenAI(BaseAPIModel):

OpenAI这个类，改一改

opencompass/opencompass/models/openai_api.py

Line 152 in aeded4c

def generate(self,

generate函数里的一些传参，就可以使用了