Skip to content

Commit 3133993

Browse files
committed
instructions on how to run a refactory benchmark
1 parent 136428e commit 3133993

File tree

1 file changed

+33
-1
lines changed

1 file changed

+33
-1
lines changed

README.md

+33-1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ as an [_edit format_](https://aider.chat/docs/benchmarks.html#edit-formats).
1212
## Table of Contents
1313
- [What is CEDARScript?](#what-is-cedarscript)
1414
- [Installation](#installation)
15+
- [Running a Benchmark](#running-a-benchmark)
1516
- [Why Use CEDARScript?](#why-use-cedarscript)
1617
- [Performance Comparison](#performance-comparison)
1718
- [Notable Achievements](#notable-achievements)
@@ -30,7 +31,7 @@ AI-assisted development tools to understand and execute these tasks.
3031

3132
## Installation
3233

33-
1. Install _**Aider with CEDARScript**_ via this command below:
34+
1. Install **Aider with _CEDARScript and CedarTL support_** via this command below:
3435
```shell
3536
python -m ensurepip --upgrade
3637
pip install --upgrade --force-reinstall \
@@ -42,6 +43,37 @@ aider-chat
4243
aider --edit-format cedarscript
4344
```
4445

46+
## Running a Benchmark
47+
48+
### One-Time Actions
49+
First, [install Aider with CEDARScript](#installation);
50+
51+
Then, follow the [**benchmark setup instructions**](https://github.com/Aider-AI/aider/blob/main/benchmark/README.md#setup-for-benchmarking) once.
52+
53+
Following that, install the [`refactor-benchmark`](https://github.com/Aider-AI/refactor-benchmark/tree/main#benchmark-details),
54+
which will perform refactorings on a _non-trivial_ amount of code found in fairly **large** files:
55+
```shell
56+
( cd tmp.benchmarks && git clone https://github.com/Aider-AI/refactor-benchmark.git )
57+
```
58+
59+
## For Every Benchmark Run
60+
Finally, for every new benchmark you want to run:
61+
```shell
62+
# Launch the docker container
63+
./benchmark/docker.sh
64+
65+
# Inside the container, install aider as a development build.
66+
# This way you're running the code that you cloned above, including any local changes.
67+
pip install -e .
68+
69+
###
70+
./benchmark/benchmark.py gemini-flash-cedarscript-version-refactor \
71+
--model gemini/gemini-1.5-flash-latest \
72+
--edit-format cedarscript \
73+
--exercises-dir refactor-benchmark \
74+
--threads 1 #### Must be only 1 ####
75+
```
76+
4577
## Why use CEDARScript?
4678

4779
`TL;DR`: You can get higher success rates when refactoring large files, comparing to other edit formats.

0 commit comments

Comments
 (0)