forked from gkamradt/SnakeBench
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathllms.txt
More file actions
4653 lines (4046 loc) · 152 KB
/
Copy pathllms.txt
File metadata and controls
4653 lines (4046 loc) · 152 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file is a merged representation of a subset of the codebase, containing files not matching ignore patterns, combined into a single document by Repomix.
================================================================
File Summary
================================================================
Purpose:
--------
This file contains a packed representation of the entire repository's contents.
It is designed to be easily consumable by AI systems for analysis, code review,
or other automated processes.
File Format:
------------
The content is organized as follows:
1. This summary section
2. Repository information
3. Directory structure
4. Multiple file entries, each consisting of:
a. A separator line (================)
b. The file path (File: path/to/file)
c. Another separator line
d. The full contents of the file
e. A blank line
Usage Guidelines:
-----------------
- This file should be treated as read-only. Any changes should be made to the
original repository files, not this packed version.
- When processing this file, use the file path to distinguish
between different files in the repository.
- Be aware that this file may contain sensitive information. Handle it with
the same level of security as you would the original repository.
Notes:
------
- Some files may have been excluded based on .gitignore rules and Repomix's configuration
- Binary files are not included in this packed representation. Please refer to the Repository Structure section for a complete list of file paths, including binary files
- Files matching these patterns are excluded: backend/completed_games, backend/model_lists, frontend/.next
- Files matching patterns in .gitignore are excluded
- Files matching default ignore patterns are excluded
Additional Info:
----------------
================================================================
Directory Structure
================================================================
backend/
cli/
generate_matchups.py
.env.local
.gitignore
app.py
backend.railway.json
elo_tracker.py
llm_providers.py
main.py
requirements.txt
video.py
frontend/
public/
file.svg
globe.svg
next.svg
vercel.svg
window.svg
src/
app/
about/
page.tsx
match/
[id]/
page.tsx
models/
[id]/
page.tsx
error.tsx
globals.css
layout.tsx
page.tsx
PostHogPageView.tsx
providers.tsx
components/
home/
HeroSection.tsx
LeaderboardSection.tsx
StatsSection.tsx
layout/
Footer.tsx
Navbar.tsx
match/
GameCanvas.tsx
GameControls.tsx
GameViewer.tsx
MatchInfo.tsx
Modal.tsx
PlayerThoughts.tsx
ScorePanel.tsx
ui/
button.tsx
table.tsx
lib/
utils.ts
.gitignore
components.json
eslint.config.mjs
frontend.railway.json
next.config.ts
nixpacks.toml
package.json
postcss.config.mjs
tailwind.config.js
tailwind.config.ts
tsconfig.json
.gitignore
.startsession
README.md
run_repomix.sh
================================================================
Files
================================================================
================
File: backend/cli/generate_matchups.py
================
#!/usr/bin/env python3
"""
This script generates battle matchups for LLM Snake Arena.
Two modes are supported:
1. all: Generate all unique combinations from the model list, each repeated as specified.
2. single: Generate matchups with a fixed model (provided via --model) against all other models from the input file.
Usage Examples:
-------------
Generate all matchups from model_lists.txt for 3 rounds and output to matchups.txt:
python cli/generate_matchups.py --mode all --rounds 3
Generate matchups for a single fixed model against all other models:
python cli/generate_matchups.py --mode single --model my_fixed_model --rounds 3
"""
import argparse
import itertools
import sys
def read_models(filename):
"""Reads models from a file, ignoring blank lines."""
try:
with open(filename, 'r') as f:
models = [line.strip() for line in f if line.strip()]
return models
except Exception as e:
print(f"Error reading file {filename}: {e}")
sys.exit(1)
def generate_all_combinations(models, rounds):
"""
Generates all unique matchup combinations from the list of models.
Each matchup is repeated 'rounds' times.
"""
matchups = []
# itertools.combinations returns unique pairs (order doesn't matter).
for model_a, model_b in itertools.combinations(models, 2):
for _ in range(rounds):
matchups.append(f"{model_a} {model_b}")
return matchups
def generate_single_matchups(fixed_model, models, rounds):
"""
Generates matchups where the fixed_model battles every other model in the list.
Each matchup is repeated 'rounds' times.
"""
matchups = []
for model in models:
if model == fixed_model:
continue # Skip the fixed model if it appears in the list.
for _ in range(rounds):
matchups.append(f"{fixed_model} {model}")
return matchups
def main():
parser = argparse.ArgumentParser(
description="Generate model battle selection matchups for the LLM Snake Arena."
)
parser.add_argument(
'--mode',
choices=['all', 'single'],
default='all',
help="Mode to generate matchups. 'all' for all combinations; 'single' for a fixed model vs all others."
)
parser.add_argument(
'--model',
type=str,
help="Fixed model name (required for 'single' mode)."
)
parser.add_argument(
'--rounds',
type=int,
default=1,
help="Number of rounds to generate for each matchup combination (default: 1)."
)
parser.add_argument(
'--input',
type=str,
default='model_lists/models_all.txt',
help="Input file containing the list of models (default: model_lists.txt)."
)
parser.add_argument(
'--output',
type=str,
default='model_lists/matchups.txt', # Updated default path
help="Output file to write matchups (default: model_lists/matchups.txt)."
)
args = parser.parse_args()
# Update output path after args are parsed if no explicit output was provided
if args.output == 'model_lists/matchups.txt' and args.mode == 'single':
args.output = f'model_lists/{args.model}_matchups.txt'
models = read_models(args.input)
if not models:
print("No models found in the input file.")
sys.exit(1)
if args.mode == 'single':
if not args.model:
print("Error: --model argument must be specified in 'single' mode.")
sys.exit(1)
if args.model not in models:
print(f"Warning: Fixed model '{args.model}' not found in the input list. It will still be used as the fixed model.")
matchups = generate_single_matchups(args.model, models, args.rounds)
else:
matchups = generate_all_combinations(models, args.rounds)
try:
with open(args.output, 'w') as f:
for matchup in matchups:
f.write(matchup + "\n")
print(f"Matchups generated and written to {args.output}")
except Exception as e:
print(f"Error writing to output file {args.output}: {e}")
sys.exit(1)
if __name__ == '__main__':
main()
================
File: backend/.env.local
================
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
================
File: backend/.gitignore
================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
#uv.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# Ruff stuff:
.ruff_cache/
# PyPI configuration file
.pypirc
.DS_Store
================
File: backend/app.py
================
import os
import json
import random
import logging
from flask import Flask, jsonify, request
from dotenv import load_dotenv
load_dotenv()
app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
# Endpoint to mimic GET requests for a list of games.
# Mimics functionality in frontend/src/app/api/games/route.ts
@app.route("/api/games", methods=["GET"])
def get_games():
try:
print("Getting games")
# Get the number of games to return from query parameters, default to 10
limit = request.args.get("limit", default=10, type=int)
sort_by = request.args.get("sort_by", default="start_time", type=str)
# Load the game index
game_index_path = os.path.join(os.getcwd(), "completed_games", "game_index.json")
with open(game_index_path, "r", encoding="utf-8") as f:
game_index = json.load(f)
# Sort the index based on the sort_by parameter
if sort_by == "start_time":
sorted_index = sorted(game_index, key=lambda x: x["start_time"], reverse=True)
elif sort_by == "total_score":
sorted_index = sorted(game_index, key=lambda x: x["total_score"], reverse=True)
elif sort_by == "actual_rounds":
sorted_index = sorted(game_index, key=lambda x: x["actual_rounds"], reverse=True)
else:
# For random order, just take random sample directly from index
selected_games_index = random.sample(game_index, min(limit, len(game_index)))
sorted_index = None
# For sorted queries, take only the top N records we need
if sorted_index is not None:
selected_games_index = sorted_index[:min(limit, len(sorted_index))]
# Only load the specific games we need
valid_games = []
games_dir = os.path.join(os.getcwd(), "completed_games")
for record in selected_games_index:
file_path = os.path.join(games_dir, record["filename"])
try:
with open(file_path, "r", encoding="utf-8") as f:
game_data = json.load(f)
valid_games.append(game_data)
except Exception as e:
logging.error(f"Error reading or parsing file {file_path}: {e}")
continue
print(f"Returning {len(valid_games)} games")
return jsonify({"games": valid_games})
except Exception as error:
logging.error(f"Error reading game index or files: {error}")
return jsonify({"error": "Failed to load game list"}), 500
# Endpoint to mimic the stats API.
# Mimics functionality in frontend/src/app/api/stats/route.ts
@app.route("/api/stats", methods=["GET"])
def get_stats():
# Get the query parameters: simple for summary stats,
# model for full stats for a single model
simple = request.args.get("simple", default=False, type=bool)
model = request.args.get("model", default=None, type=str)
if simple:
# This branch returns the simple version
stats_path = os.path.join(os.getcwd(), "completed_games", "stats_simple.json")
try:
with open(stats_path, "r", encoding="utf-8") as f:
stats_data = json.load(f)
except Exception as e:
logging.error(f"Error loading simple stats data: {e}")
stats_data = {}
return jsonify({
"totalGames": 0, # You could update this if available in stats_data
"aggregatedData": stats_data
})
# For full stats, we require a model parameter.
if model is None:
return jsonify({"error": "Please provide a model parameter for full stats."}), 400
stats_path = os.path.join(os.getcwd(), "completed_games", "stats.json")
try:
with open(stats_path, "r", encoding="utf-8") as f:
stats_data = json.load(f)
except Exception as e:
logging.error(f"Error loading full stats data: {e}")
return jsonify({"error": "Failed to load stats data."}), 500
model_stats = stats_data.get(model)
if model_stats is None:
return jsonify({"error": f"Stats for model '{model}' not found."}), 404
# Since the full stats already include wins/losses, simply return the model's stats.
total_games = model_stats.get("wins", 0) + model_stats.get("losses", 0) + model_stats.get("ties", 0)
return jsonify({
"totalGames": total_games,
"aggregatedData": {model: model_stats}
})
# Endpoint to get details for a single game by id.
# Mimics functionality in frontend/src/app/api/games/[gameId]/route.ts
@app.route("/api/matches/<match_id>", methods=["GET"])
def get_game_by_id(match_id):
try:
# Construct the file path using the game_id.
match_filename = f"snake_game_{match_id}.json"
match_file_path = os.path.join(os.getcwd(), "completed_games", match_filename)
with open(match_file_path, "r", encoding="utf-8") as f:
match_data = json.load(f)
return jsonify(match_data)
except Exception as error:
logging.error(f"Error reading match data for match id {match_id}: {error}")
return jsonify({"error": "Failed to load match data"}), 500
if __name__ == "__main__":
# Run the Flask app in debug mode.
app.run(debug=os.getenv("FLASK_DEBUG"))
================
File: backend/backend.railway.json
================
{
"$schema": "https://railway.app/railway.schema.json",
"build": {
"builder": "NIXPACKS",
"buildCommand": "pip3 install -r requirements.txt",
"watchPatterns": [
"backend/*"
]
},
"deploy": {
"startCommand": "gunicorn app:app",
"restartPolicyType": "ON_FAILURE",
"restartPolicyMaxRetries": 3
}
}
================
File: backend/elo_tracker.py
================
#!/usr/bin/env python3
import os
import json
import glob
from datetime import datetime
import argparse
import math
# Elo parameters
K = 32
INITIAL_RATING = 1500
# Define a ranking for game result strings.
RESULT_RANK = {"won": 2, "tied": 1, "lost": 0}
def get_pair_result(result_i, result_j):
"""
Given the result strings (e.g., "won", "lost", "tie") for two players,
return a tuple (S_i, S_j) representing the head-to-head score:
- S = 1 means win, 0 means loss, 0.5 means tie.
If both players have the same result (e.g., both "won"), treat it as a tie.
"""
rank_i = RESULT_RANK.get(result_i, 1)
rank_j = RESULT_RANK.get(result_j, 1)
if rank_i > rank_j:
return 1, 0
elif rank_i < rank_j:
return 0, 1
else:
return 0.5, 0.5
def expected_score(rating_i, rating_j):
"""Compute the expected score for player i vs. player j."""
return 1 / (1 + 10 ** ((rating_j - rating_i) / 400))
def process_game(game_data, ratings):
"""
Process one game and update the 'ratings' dictionary (for Elo).
Returns updated ratings.
"""
metadata = game_data.get("metadata", {})
models = metadata.get("models", {}) # {player_id: model_name}
game_result = metadata.get("game_result", {}) # {player_id: "won"/"lost"/"tie"}
player_ids = list(models.keys())
n = len(player_ids)
# Ensure all models exist in our ratings dictionary
for pid in player_ids:
model = models[pid]
if model not in ratings:
ratings[model] = INITIAL_RATING
# For each model (player) in this game, accumulate actual/expected scores
score_sum = { models[pid]: 0 for pid in player_ids }
expected_sum = { models[pid]: 0 for pid in player_ids }
# Loop over all unordered pairs of players
for i in range(n):
for j in range(i+1, n):
pid_i = player_ids[i]
pid_j = player_ids[j]
model_i = models[pid_i]
model_j = models[pid_j]
res_i = game_result.get(pid_i, "tie")
res_j = game_result.get(pid_j, "tie")
# Determine the head-to-head result
S_i, S_j = get_pair_result(res_i, res_j)
# Compute expected scores from the current ratings
R_i = ratings[model_i]
R_j = ratings[model_j]
E_i = expected_score(R_i, R_j)
E_j = expected_score(R_j, R_i)
# Accumulate results
score_sum[model_i] += S_i
score_sum[model_j] += S_j
expected_sum[model_i] += E_i
expected_sum[model_j] += E_j
# Update each player's rating
for pid in player_ids:
model = models[pid]
delta = (K / (n - 1)) * (score_sum[model] - expected_sum[model]) if (n > 1) else 0
ratings[model] += delta
return ratings
### ADDED FOR STATS ###
def update_model_stats(game_data, stats, ratings):
"""
Updates stats for each model after one game:
- Increments wins/losses/ties
- Adds to total apples eaten
- Syncs the Elo rating from 'ratings'
- Appends a game history record with details including:
game_id, my_score, opponent_score, opponent_model, result,
and, if applicable, death_info.
"""
metadata = game_data.get("metadata", {})
game_id = metadata.get("game_id")
models = metadata.get("models", {}) # {player_id: model_name}
game_result = metadata.get("game_result", {}) # {player_id: "won"/"lost"/"tie"}
final_scores = metadata.get("final_scores", {}) # {player_id: score}
death_info = metadata.get("death_info", {}) # {player_id: death info dictionary}
# For each player in the game...
for pid, model_name in models.items():
# Ensure this model is in stats
if model_name not in stats:
stats[model_name] = {
"wins": 0,
"losses": 0,
"ties": 0,
"apples_eaten": 0,
"elo": INITIAL_RATING,
"games": [] # New field for game history tracking
}
# Update W/L/T counts
result = game_result.get(pid, "tie")
if result == "won":
stats[model_name]["wins"] += 1
elif result == "lost":
stats[model_name]["losses"] += 1
else:
stats[model_name]["ties"] += 1
# Update apples eaten
apples = final_scores.get(pid, 0)
stats[model_name]["apples_eaten"] += apples
# Update current Elo rating
stats[model_name]["elo"] = ratings[model_name]
# Determine opponent's score and opponent's model.
if len(models) == 2:
opponent_pid = [other for other in models if other != pid][0]
opponent_score = final_scores.get(opponent_pid, 0)
opponent_model = models.get(opponent_pid)
else:
opponent_score = None
opponent_model = None
# Build game history record for this model
game_record = {
"game_id": game_id,
"my_score": final_scores.get(pid, 0),
"opponent_score": opponent_score,
"opponent_model": opponent_model,
"opponent_elo": ratings.get(opponent_model, INITIAL_RATING),
"result": result,
"start_time": metadata.get("start_time"),
"end_time": metadata.get("end_time")
}
# Include death_info if this model died in this game.
if pid in death_info:
game_record["death_info"] = death_info[pid]
# Append the record to the model's game history list.
stats[model_name]["games"].append(game_record)
def summarize_game_results(models, game_result):
"""
Summarizes both the overall game result and the pairwise matchups.
Returns a tuple of strings for (overall_summary, matchup_summary)
"""
# Sort by rank (won>tie>lost) just for display
results = []
for pid, model in models.items():
result = game_result.get(pid, "tie")
rank = RESULT_RANK.get(result, 1)
results.append((rank, result, model))
results.sort(reverse=True)
overall = "Overall result:\n"
for _, result, model in results:
overall += f" {model}: {result}\n"
# Pairwise
matchups = "Pairwise matchups:\n"
player_ids = list(models.keys())
for i in range(len(player_ids)):
for j in range(i+1, len(player_ids)):
pid_i = player_ids[i]
pid_j = player_ids[j]
model_i = models[pid_i]
model_j = models[pid_j]
res_i = game_result.get(pid_i, "tie")
res_j = game_result.get(pid_j, "tie")
score_i, score_j = get_pair_result(res_i, res_j)
if score_i == 0.5:
result_str = "ties"
elif score_i == 1:
result_str = "wins against"
else:
result_str = "loses to"
matchups += f" {model_i} {result_str} {model_j}\n"
return overall, matchups
def main():
parser = argparse.ArgumentParser(
description="Calculate Elo ratings and gather stats from a folder of game result JSON files."
)
parser.add_argument("folder", help="Path to folder containing game result JSON files.")
parser.add_argument("--output", required=True, help="Path to output folder for stats.json")
args = parser.parse_args()
# Find all JSON files
files = [f for f in glob.glob(os.path.join(args.folder, "*.json")) if not f.endswith("game_index.json")]
games = []
for filename in files:
try:
with open(filename, "r") as f:
data = json.load(f)
# We sort by end_time so we process in chronological order
end_time_str = data.get("metadata", {}).get("end_time")
if end_time_str:
end_time = datetime.fromisoformat(end_time_str)
else:
end_time = datetime.min
games.append((end_time, data))
except Exception as e:
print(f"Error reading {filename}: {e}")
# Sort the games by their end_time
games.sort(key=lambda tup: tup[0])
# Elo ratings dict: model -> rating
ratings = {}
# Stats dict: model -> {"wins", "losses", "ties", "apples_eaten", "elo"}
stats = {}
# print("Initial Elo ratings:")
# print(f" (New models start at {INITIAL_RATING})")
# print("-" * 40)
# print("\nProcessing games in chronological order...\n")
for end_time, game_data in games:
metadata = game_data.get("metadata", {})
models = metadata.get("models", {})
game_result = metadata.get("game_result", {})
overall_summary, matchup_summary = summarize_game_results(models, game_result)
# print(overall_summary)
# print(matchup_summary)
# Update ratings
ratings = process_game(game_data, ratings)
# Update stats (wins, losses, ties, apples, Elo)
update_model_stats(game_data, stats, ratings)
print("Updated Elo ratings:")
for model, rating in sorted(ratings.items(), key=lambda x: x[1], reverse=True):
print(f" {model}: {rating:.2f}")
print("-" * 40)
### ADDED FOR STATS: SAVE stats.json ###
# Write out stats aggregated across all games
output_path = os.path.join(args.output, "stats.json")
with open(output_path, "w") as f:
json.dump(stats, f, indent=2)
print(f"\nAggregated stats saved to {output_path}")
# Additionally, write out a version of stats that excludes game history.
# This mirrors stats.json but without the "games" list for each model.
model_stats = {
model: {k: v for k, v in data.items() if k != "games"}
for model, data in stats.items()
}
# Add first and last game timestamps and top score for each model
for model, data in stats.items():
if "games" in data and data["games"]:
# Sort games by start_time to ensure correct ordering
sorted_games = sorted(data["games"], key=lambda g: g.get("start_time", ""))
# Get first and last game timestamps
first_game = sorted_games[0].get("start_time", "")
last_game = sorted_games[-1].get("start_time", "")
# Find the highest score across all games
top_score = max([game.get("my_score", 0) for game in data["games"]])
# Add to model_stats
model_stats[model]["first_game_time"] = first_game
model_stats[model]["last_game_time"] = last_game
model_stats[model]["top_score"] = top_score
simple_output_path = os.path.join(args.output, "stats_simple.json")
with open(simple_output_path, "w") as f:
json.dump(model_stats, f, indent=2)
print(f"\nModel-only stats saved to {simple_output_path}")
### NEW FUNCTIONALITY: BUILD AND SAVE A GAME INDEX ###
# Build a lightweight index for game metadata to speed up future queries.
game_index = []
for end_time, game_data in games:
metadata = game_data.get("metadata", {})
game_id = metadata.get("game_id")
if not game_id:
continue
# Use known keys; here we compute total_score from final_scores.
final_scores = metadata.get("final_scores", {})
total_score = sum(final_scores.values()) if final_scores else 0
start_time = metadata.get("start_time", "")
actual_rounds = metadata.get("actual_rounds", 0)
# Construct the filename from the game_id using your naming convention.
filename = f"snake_game_{game_id}.json"
game_index.append({
"game_id": game_id,
"filename": filename,
"start_time": start_time,
"total_score": total_score,
"actual_rounds": actual_rounds
})
# Save the game index into the same folder as the game files.
index_path = os.path.join(args.folder, "game_index.json")
with open(index_path, "w") as f:
json.dump(game_index, f, indent=2)
print(f"\nGame index saved to {index_path}")
if __name__ == "__main__":
main()
================
File: backend/llm_providers.py
================
import os
from openai import OpenAI
import anthropic
import google.generativeai as genai # Add this import
from together import Together
from ollama import chat
from ollama import ChatResponse
class LLMProviderInterface:
"""
A common interface for LLM calls.
"""
def get_response(self, model: str, prompt: str) -> str:
raise NotImplementedError("Subclasses should implement this method.")
class OpenAIProvider(LLMProviderInterface):
def __init__(self, api_key: str):
self.client = OpenAI(api_key=api_key)
def get_response(self, model: str, prompt: str) -> str:
response = self.client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_completion_tokens=4096,
)
return response.choices[0].message.content.strip()
class AnthropicProvider(LLMProviderInterface):
def __init__(self, api_key: str):
self.client = anthropic.Anthropic(api_key=api_key)
def get_response(self, model: str, prompt: str) -> str:
# According to Anthropic docs, this is one way to call the API.
response = self.client.messages.create(
model=model,
max_tokens=4096,
messages=[{"role": "user", "content": prompt}],
)
return response.content[0].text.strip()
class GeminiProvider(LLMProviderInterface):
def __init__(self, api_key: str):
genai.configure(api_key=api_key)
def get_response(self, model: str, prompt: str) -> str:
model = genai.GenerativeModel(model)
response = model.generate_content(
contents=prompt,
generation_config={
"max_output_tokens": 4096,
},
stream=False
)
return response.text.strip()
class TogetherProvider(LLMProviderInterface):
def __init__(self, api_key: str):
self.client = Together(api_key=api_key)
def get_response(self, model: str, prompt: str) -> str:
response = self.client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=20000,
)