OpenDino toy ESP32 push-to-talk with GPT-4o mini Realtime over raw WebSockets

[RASPIAUDIOadmin]

Summary

This PR introduces a new example that demonstrates a practical implementation of the GPT-4o mini Realtime API on an ESP32.
Unlike existing tutorials, it streams audio via raw WebSockets only—no local bridge, Raspberry Pi, or PC required. Because an ESP32 has very limited RAM and CPU, the code is carefully optimised to keep the entire dialogue stack on the microcontroller. To the best of my knowledge, this is the first open-source example on GitHub that connects an ESP32 directly to OpenAI’s Realtime endpoint.

Motivation

Today there is no reference project showing how to run OpenAI Realtime over WebSockets on a bare ESP32. This example fills that gap by proving:

Push-to-talk conversation with round-trip latency under one second

JSON-Schema function calling handled entirely on the microcontroller

A minimal, reproducible hardware setup that others can copy or adapt

By publishing this, I hope to unblock—and inspire—developers who want to bring real-time LLM interaction to low-cost, resource-constrained devices.

---

For new content

When contributing new content, read through our contribution guidelines, and mark the following action items as completed:

• I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
• I have conducted a self-review of my content based on the contribution guidelines:
  • Relevance: This content is related to building with OpenAI technologies and is useful to others.
  • Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
  • Spelling and Grammar: I have checked for spelling or grammatical mistakes.
  • Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
  • Correctness: The information I include is correct and all of my code executes successfully.
  • Completeness: I have explained everything fully, including all necessary references and citations.

We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our contribution guidelines for more details.