CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation

• Dataset: LeetCode-Contest

Download the .jsonl file and put it to data/leetcode_contest.jsonl

Run

• Run {run_type}

# run sampling
python {run_type}.py \
--config 'config/config-leetcode-contest-qwen2.5-coder-32b.json' \
--run_type {run_type} \
--api_key 'xxx' \
--base_url 'xxx'

The required order of method execution, the methods following the arrow need to rely on the results generated by the previous methods for execution, including codes or test cases.

Sampling / Gen_tests -> Sampling+Filtering / CodeT / MBR_Exec / Self_repair, Gen_tests -> Reflexion

• run_type

Method	run_type	Require LLM (api_key and base_url)
Sampling	sampling	y
Sampling+Filtering	sampling_filtering	n
Gen_tests	gen_tests	y
CodeT	codet	n
MBR_Exec	mbr_exec	n
Self_repair	self_repair	y
Reflexion	reflexion	y
CoCoEvo	coevod	y
Evolution (CoCoEvo w/o test evolution)	evolutiond	y

• Gen_tests

# evaluate generated tests
python gen_tests_eval.py \
--config 'config/config-leetcode-contest-qwen2.5-coder-32b.json' \
--run_type 'gen_tests_eval'

# show evaluation result
python show_tests.py --result_dir='result/leetcode_contest/qwen2.5-coder-32b' --run_type='gen_tests_eval'

• For other methods, submit {run_type} to private tests

# submit to private tests
python submit.py \
--config 'config/config-leetcode-contest-qwen2.5-coder-32b.json' \
--run_type {run_type}

Other Baselines

baselines (AgentCoder, CodeCOT, INTERVENOR)

Citation

@article{li2025cocoevo,
  title={CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation},
  author={Li, Kefan and Yu, Hongyue and Guo, Tingyu and Cao, Shijie and Yuan, Yuan},
  journal={arXiv preprint arXiv:2502.10802},
  year={2025}
}