ten-moss

Voice Assistant with Moss (TEN Framework)

A real-time voice agent built on the TEN Framework that grounds every answer in a Moss session. On each final ASR transcript, the control extension queries Moss for session-scoped context (single-digit milliseconds, in-process) and injects it into the LLM prompt before the model responds, so answers reflect your knowledge base with no perceptible added latency.

The integration is powered by the ten-moss package (MossSessionManager) and lives entirely in the main_python control extension.

Role	Component
Transport	Agora RTC
Speech-to-text	Deepgram
LLM	OpenAI
Text-to-speech	ElevenLabs
Grounding	Moss (in-process session)

Architecture

flowchart LR
    mic(["Mic"]) --> rtc_in["agora_rtc"]
    rtc_in --> adapter["streamid_adapter"]
    adapter --> stt["stt<br/>Deepgram"]
    stt -- "asr_result (final)" --> ctl["main_control<br/>main_python"]
    ctl <-. "query_context<br/>in-process, single-digit ms" .-> idx[("Moss<br/>index")]
    ctl -- "grounding + user question" --> llm["llm<br/>OpenAI"]
    llm --> tts["tts<br/>ElevenLabs"]
    tts --> rtc_out["agora_rtc"]
    rtc_out --> spk(["Speaker"])

    classDef moss fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
    class ctl,idx moss

Loading

Everything on the retrieval path runs inside the agent process. There is no network hop between the transcript arriving and the grounded prompt reaching the LLM.

What's in this directory

This example ships the TEN app plus a small index builder; the run harness (playground, server, Taskfile, Dockerfile) comes from the TEN Framework, so tenapp/ drops into any TEN checkout.

• tenapp/: the TEN app, i.e. the graph (property.json) and the main_python control extension that carries the Moss delta.
• create_index.py + data/knowledge.jsonl: build the demo Moss index.
• .env.example: every credential the agent needs.

Prerequisites

• A TEN Framework checkout. This example runs with TEN's tooling and references shared TEN extensions by relative path (../../../ten_packages/extension/...), so it must live inside a TEN Framework repo.
• A Moss project (MOSS_PROJECT_ID / MOSS_PROJECT_KEY) from moss.dev.
• Provider keys: Agora (transport), Deepgram (STT), OpenAI (LLM), ElevenLabs (TTS).

Quick start

1. Build the demo knowledge index (from this directory; needs only the Moss SDK):

   pip install moss python-dotenv   # the Moss SDK (+ python-dotenv to load .env)
   cp .env.example .env             # fill in MOSS_PROJECT_ID / MOSS_PROJECT_KEY / MOSS_INDEX_NAME
   python create_index.py           # reads data/knowledge.jsonl, creates MOSS_INDEX_NAME

2. Drop the app into a TEN checkout:

   ./setup.sh /path/to/ten-framework

   The script reuses the sibling voice-assistant example's harness (Taskfile, scripts/, Dockerfile), swaps in this tenapp/, and seeds ai_agents/.env from this directory's .env if you created one in step 1. To do it by hand instead:

   cd ten-framework/ai_agents/agents/examples
   cp -r voice-assistant voice-assistant-with-moss        # reuse its Taskfile, scripts/, Dockerfile
   rm -rf voice-assistant-with-moss/tenapp
   cp -r /path/to/moss/apps/ten-moss/tenapp voice-assistant-with-moss/tenapp

   main_python depends on ten-moss (listed in main_python/requirements.txt), so task install pulls it from PyPI automatically. task install also pre-downloads the moss-minilm embedding model (when the MOSS_* env vars are set) so the first agent session does not have to.

3. Run with TEN's tooling from that example directory (task install && task run, per the TEN docs), with the MOSS_* vars from step 1 plus the provider keys from Prerequisites (Agora, Deepgram, OpenAI, ElevenLabs). Open the TEN playground at http://localhost:3000, select the voice_assistant graph (a predefined_graph in tenapp/property.json, or open ?graph=voice_assistant), and ask something covered by data/knowledge.jsonl, for example "how long do refunds take?", to hear grounded answers.

   Apple Silicon note: TEN's ten_agent_build dev image is amd64-only. On colima, start the VM with Rosetta (colima start --vz-rosetta); under plain qemu emulation the Go toolchain segfaults during task install. Docker Desktop and OrbStack enable Rosetta by default.

Under the hood

The difference from the stock TEN voice assistant is small and lives in three places in main_python:

Location	Change
config.py	MainControlConfig inherits MossSessionConfig (the moss_* properties).
extension.py (on_init)	Opens the Moss session via MossSessionManager.from_config(...).open(), best-effort.
extension.py (_on_asr_result)	Calls query_context(text) and prepends the grounding to the user's turn.

Anatomy of a turn:

sequenceDiagram
    autonumber
    participant User
    participant STT as Deepgram STT
    participant Ctl as main_control
    participant Moss as Moss session (in-process)
    participant LLM as OpenAI LLM
    participant TTS as ElevenLabs TTS

    User->>STT: speech (via agora_rtc + streamid_adapter)
    STT->>Ctl: asr_result (final)
    Ctl->>Moss: query_context(text)
    Moss-->>Ctl: grounding (single-digit ms)
    Ctl->>LLM: context + [Current User Question] + text
    LLM-->>TTS: streamed response
    TTS-->>User: audio (via agora_rtc)

Loading

Measure the latency

Every turn, the control extension logs the retrieval cost using the SDK's own SearchResult.time_taken_ms (surfaced by ten-moss as last_time_taken_ms), with the wall clock alongside for reference:

[retrieval-latency] backend=moss(in-process) time_taken_ms=2 (wall_clock=64ms)

In the playground transcript you see, per turn, what Moss retrieved plus the SDK time_taken_ms, followed by the LLM's answer:

🔎 Moss · retrieved in 2 ms (SDK time_taken_ms)
   Relevant knowledge from Moss: [1] Refunds are processed within 3-5 business days…
<the assistant's spoken answer>

The extension also emits a per-turn latency breakdown, both as a grep-able log line and as a note in the transcript, so you can see where each turn's time goes:

[latency-breakdown] turn=3 moss_retrieval_ms=2 llm_ttft_ms=480 llm_total_ms=1150 turn_total_ms=1160

Field	Meaning
moss_retrieval_ms	The SDK's SearchResult.time_taken_ms (in-process retrieval engine time).
llm_ttft_ms	Time to the LLM's first token after dispatch.
llm_total_ms	Full LLM generation for the turn.
turn_total_ms	ASR-final to LLM-final (the whole control-side turn).

ASR timing appears in the Deepgram STT extension logs and TTS audio-out in the ElevenLabs TTS logs (both per turn in the worker log), so between those and the lines above you get the full component-by-component breakdown.

Benchmark against TEN's default retrieval

TEN's shipped memory/RAG backends (memU, OceanBase PowerRAG, EverMemOS) are remote services that pay a network round trip every turn, whereas Moss retrieves in-process, so the same grounding is a local call of single-digit milliseconds.

Configuration

Moss is configured on the main_control node in tenapp/property.json (env-substituted):

Property	Default	Description
moss_project_id	${env:MOSS_PROJECT_ID}	Moss project ID.
moss_project_key	${env:MOSS_PROJECT_KEY}	Moss project key (kept masked in logs).
moss_index_name	${env:MOSS_INDEX_NAME}	Index to query.
moss_model_id	moss-minilm	Embedding model; empty string adopts the stored index's model.
moss_top_k	3	Results retrieved per query.
moss_alpha	0.8	Hybrid search weighting (0.0 to 1.0).
moss_context_header	Relevant knowledge from Moss:	Header prepended to the injected grounding.
moss_max_context_chars	2000	Cap on the injected grounding block; 0 means unlimited.
enable_moss	true	Set to false to run the plain voice assistant with no grounding.

Provenance

The tenapp/ baseline (graph, main_python control extension, agent runtime, scripts) is vendored from the TEN Framework voice-assistant example at commit c385d27, licensed under Apache-2.0 (headers preserved). Only the Moss delta described above is Moss-authored.

Three small patches were applied on top of the vendored baseline: agent/decorators.py fixes the agent_event_handler annotation to type[AgentEvent]; extension.py parses session_id defensively so a non-numeric value cannot crash the ASR handler; and scripts/install_python_deps.sh fails fast (set -euo pipefail) and pre-warms the moss-minilm model cache, because two workers racing the first download corrupt the cache and grounding then silently degrades to an ungrounded assistant.

Testing status

The ten-moss package is covered by offline unit tests (packages/ten-moss/tests/). This end-to-end app is not run in CI; it requires the TEN toolchain plus paid Agora, Deepgram, OpenAI, and ElevenLabs credentials, so it is validated manually via the steps above.