LLM Python Raytracer - Benchmark Experiment

This repository contains various Python raytracer implementations generated by different Large Language Models (LLMs) using a single prompt and one attempt.

Prompt Used

The following prompt was used for all LLMs:

Write a raytracer that renders an interesting scene with many colourful lightsources in python. Output a 800x600 image as a png

Implementations

The following LLMs did not return properly functioning code: Grok3-thinking, DeepSeek-R1, Le Chat (Mistral Large 2), 8b models, llama3-405b/70b, llama3-hermes3-405b/70b, Hunyuan-T1,Qwen3-235, Qwen3.5-397B-A17B. Generally, the R1-adjacent reasoning models tend to overthink code fragments and then come up with dysfunctional code.

Results

Performance Notes

Most implementations take several minutes to render, which is not unexpected for a Python raytracer. (Try C next? :D)
Performance varies between implementations, with some taking significantly longer to complete
DeepSeek R1 spent 566 thinking and then outputted a broken file. I was not able to restart due to a busy server.
4o's first attempt required corrections, but the second attempt was successful

Output Images for different LLMs

Claude Sonnet 3.5 (new)	Claude Sonnet 3.7	Claude Sonnet 3.7 Thinking	GPT-4o
DeepSeek v3	Gemini Flash Thinking	Gemini 2 Flash	Grok 2
Grok 3	O1	O3-Mini	Qwen Max
GPT-4.5	QwQ32B	Codestral	Deepseek V3 0324
Gemini 2.5 pro exp	LLama 3.3 70b	Quasar Alpha	Optimus Alpha

Variability test

Since the tests above are based on one-shot prompts, they are not necessarily representative of the LLM's average capabilities. I ran the experiment 4 times with Sonnet-3.5, Sonnet-3.7, GPT4.5 and grok3 to test for consistency.

We can clearly see a significant change in behavior between the two models. Sonnet-3.5 produces slight variations of a basic scene with red, green and blue spheres, while Sonnet-3.7 uses more objects, colors and more variations in general. GPT 4.5 and Grok3 are also more of a three-sphere llm.