The results of the GA and NPO tests on the TOFU dataset showed almost no change.

[zh133767710]

Tasks

• Benchmark
• Unlearning method
• Evaluation
• Dataset
• None of the above

Feature request

We executed the tofu_unlearn.sh. But it was found that when evaluating the models saved in saves/unlearn, the TOFU_EVAL.json and TOFU_SUMMARY.json for different tasks were the same. For example, for forget01's GA and NPO. In additional, the DPO indicates that an idk.jsonl file is required in the /data directory, we have not found this file. Therefore, we hope for your assistance. Here is our experimental setup:

Motivation

..

Implementation

No response

[Duguce]

Hi @zh133767710, I encountered the same issue: when using GA and NPO on the TOFU dataset, the evaluation results for different forget tasks show almost no difference.

Have you been able to identify or resolve this problem later on?