MotionEngine — data-driven motion plugin for TalkingHead (proof of concept)

[lhupyn]

Hi! First of all, thanks for TalkingHead — it's an amazing project.

I'm working on a real-time 3D avatar SDK where an LLM controls a TalkingHead avatar via voice. I needed richer animations than the built-in emojis/gestures, so I extracted a small plugin called MotionEngine.

What it does

MotionEngine plays multi-layered motions defined in pure JSON — synchronizing facial expressions, hand gestures, and bone oscillation overlays (via poseDelta) in a single call:

const engine = new MotionEngine(head);
engine.registerMotions(motions);
head.opt.update = (dt) => engine.update(dt);

await engine.play('wave_right'); // smile + hand raise + wrist oscillation

Each motion combines up to 3 layers:

• Visemes/blendshapes — facial expressions (smile, eyebrow raise, etc.)
• Gesture commands — TalkingHead hand poses (handup, thumbup, etc.)
• Bone overlays — sinusoidal oscillation on skeleton bones (e.g., waving hand, hip bounce)

It falls back gracefully to native TalkingHead gestures, emojis, and poses when a custom motion isn't found.

Live demo

https://lhupyn.github.io/motion-engine/

The demo loads brunette.glb from your GitHub Pages and shows a side-by-side comparison: TalkingHead native animations vs MotionEngine custom motions.

Repo

https://github.com/lhupyn/motion-engine

• src/MotionEngine.js — ~250 lines, no DOM dependencies, works as a pure plugin
• src/motions.json — 20 built-in motions (wave, thumbs up, nod, shrug, celebrate, jump, etc.)

Why this matters for LLM-driven avatars

When an LLM controls an avatar, it needs a semantic vocabulary — not low-level blendshape values. MotionEngine lets an AI agent call engine.play("thinking") instead of orchestrating 8 individual parameters. And since motions are pure JSON, they can be authored or even generated by an LLM at runtime.

This is still a proof of concept, but I wanted to share it in case it's useful for the TalkingHead community. Would love to hear your thoughts!

[met4citizen]

Thank you for sharing. I think your approach is great. Your plugin introduces new gestures, and I like the "bone overlays" feature, which adds some movement to otherwise static gestures.

From the beginning, I have been waiting for text-to-3D-animation AI models to emerge and fill this need, but we are not quite there yet. Even commercial products such as DeepMotion's SayMotion, Cartwheel, and others take too long to generate for real-time use and the resulting animations aren't yet reliable enough. At this point, the best options are to use FBX animations and/or extend the existing algorithmic approach, as you have done here.

I will leave this issue open for a while, and the next time I update the README, I will add a link to your repo in the Add-on Modules section. Thanks again!

[lhupyn]

Thanks for the feedback! I've kept exploring this path and wanted to share what I've found.

The problem I'm trying to solve:
An LLM in a real-time conversation needs two distinct things:

• Speed — during a conversation, the avatar must react instantly. It can't wait 3-15 seconds to generate an animation.
• Variety — a fixed dictionary of 40 motions falls short. You need to create new animations without being a 3D animator.

I've concluded that the solution is two complementary layers:

Layer 1: Semantic dictionary (runtime, fast)

The LLM agent simply says play("thinking") or play("shy") and the motion executes instantly from a pre-registered JSON. The dictionary is organized by semantic tags (greeting, emotion, body...) so the LLM can navigate it efficiently with few tokens.

For this, MotionEngine exposes getLLMContext() — a compact summary of all available motions grouped by category, designed to be injected into the agent's system prompt.

Layer 2: LLM motion creator (authoring, slow)

I've built a "lab" (https://lhupyn.github.io/motion-engine/playground.html) where you describe a movement in natural language and an LLM (Gemini, OpenAI or Claude) generates the motion JSON. This is NOT for real-time — it's an authoring tool to expand the dictionary.

The trick is in the prompt. MotionEngine scans the loaded avatar at runtime:

• Discovers the available morph targets (ARKit blendshapes from the GLB model)
• Discovers the skeleton bones (via armature traverse)
• Lists existing motions as reference

All of this is injected into the system prompt along with 6 real examples from the dictionary (few-shot). The LLM receives exactly what it can move and what well-crafted motions look like, and generates valid, playable JSON.

https://github.com/user-attachments/assets/6f8e7c94-46b7-46cb-9271-d0c670665af5
https://github.com/user-attachments/assets/6efd572f-2a5e-4b3a-a058-e45dba8974ac
https://github.com/user-attachments/assets/b1c26423-effd-4d01-b162-ac36941bcf71

On FBX

I agree that text-to-3D-animation models aren't ready for real-time yet. But for my use case (conversational avatar), I don't need full FBX animations either. What I need are expressive micro-animations: a 2-3 second gesture that accompanies what the avatar is saying. The pure JSON format (morph keyframes + bone oscillations) is sufficient and has advantages:

• Weighs bytes, not megabytes
• Can be generated, edited, and mixed programmatically
• No retargeting needed — works directly with TalkingHead's skeleton

Next step: multiple dictionaries

I'm exploring the idea of swappable dictionaries by context — one motion set for casual conversation, another for presentations, another for gaming. The LLM agent would select the active dictionary based on the situation, keeping each context's vocabulary compact and token-efficient.

[lhupyn]

Quick update — I've been iterating on the plugin and wanted to share a couple of things that might be relevant for TH:

Persistent multi-track moods: MotionEngine now injects custom moods directly into TH's animMoods during registration. This means setMood() handles them natively — the avatar keeps breathing, blinking, and all idle animations while the mood persists. Temporal gestures play on top without resetting the emotional state.

Expanded catalog: The motion dictionary grew to 137+ entries, mostly generated using LLM batch processing (Gemini) with a few-shot prompt that includes the avatar's actual morph targets and bones.

LLM-generated motions: I've been testing motion generation with different models. Gemini 3.1 Pro produces usable results for creating new animations from natural language descriptions. One direction I was exploring is a "bone atlas" — a reference file that maps bone rotations to semantic descriptions — which could help LLMs reason about body movement more accurately. I wasn't able to pull it off since my 3D animation skills are limited.

On the approach: The problem I'm trying to solve is: how does an LLM agent pick the right animation during a conversation without hallucinating motions that don't exist, and without burning tokens on long catalogs? The current approach is autodiscovery, packages the available motions into a compact context, and exposes it as a single tool for the LLM. It reduces hallucinations because the LLM only sees what actually exists, and it's token-efficient because the catalog is grouped by semantic tags. Not sure if this is the right direction.

Thanks for TalkingHead, it's been a great platform to experiment with.