@@ -12,6 +12,7 @@ as an [_edit format_](https://aider.chat/docs/benchmarks.html#edit-formats).
12
12
## Table of Contents
13
13
- [ What is CEDARScript?] ( #what-is-cedarscript )
14
14
- [ Installation] ( #installation )
15
+ - [ Running a Benchmark] ( #running-a-benchmark )
15
16
- [ Why Use CEDARScript?] ( #why-use-cedarscript )
16
17
- [ Performance Comparison] ( #performance-comparison )
17
18
- [ Notable Achievements] ( #notable-achievements )
@@ -30,7 +31,7 @@ AI-assisted development tools to understand and execute these tasks.
30
31
31
32
## Installation
32
33
33
- 1 . Install _ ** Aider with CEDARScript ** _ via this command below:
34
+ 1 . Install ** Aider with _ CEDARScript and CedarTL support _ ** via this command below:
34
35
``` shell
35
36
python -m ensurepip --upgrade
36
37
pip install --upgrade --force-reinstall \
@@ -42,6 +43,37 @@ aider-chat
42
43
aider --edit-format cedarscript
43
44
```
44
45
46
+ ## Running a Benchmark
47
+
48
+ ### One-Time Actions
49
+ First, [ install Aider with CEDARScript] ( #installation ) ;
50
+
51
+ Then, follow the [ ** benchmark setup instructions** ] ( https://github.com/Aider-AI/aider/blob/main/benchmark/README.md#setup-for-benchmarking ) once.
52
+
53
+ Following that, install the [ ` refactor-benchmark ` ] ( https://github.com/Aider-AI/refactor-benchmark/tree/main#benchmark-details ) ,
54
+ which will perform refactorings on a _ non-trivial_ amount of code found in fairly ** large** files:
55
+ ``` shell
56
+ ( cd tmp.benchmarks && git clone https://github.com/Aider-AI/refactor-benchmark.git )
57
+ ```
58
+
59
+ ## For Every Benchmark Run
60
+ Finally, for every new benchmark you want to run:
61
+ ``` shell
62
+ # Launch the docker container
63
+ ./benchmark/docker.sh
64
+
65
+ # Inside the container, install aider as a development build.
66
+ # This way you're running the code that you cloned above, including any local changes.
67
+ pip install -e .
68
+
69
+ # ##
70
+ ./benchmark/benchmark.py gemini-flash-cedarscript-version-refactor \
71
+ --model gemini/gemini-1.5-flash-latest \
72
+ --edit-format cedarscript \
73
+ --exercises-dir refactor-benchmark \
74
+ --threads 1 # ### Must be only 1 ####
75
+ ```
76
+
45
77
## Why use CEDARScript?
46
78
47
79
` TL;DR ` : You can get higher success rates when refactoring large files, comparing to other edit formats.
0 commit comments