feat: add multimodal engagement estimation and adaptive tutoring#272
Open
cldanass wants to merge 2 commits into
Open
feat: add multimodal engagement estimation and adaptive tutoring#272cldanass wants to merge 2 commits into
cldanass wants to merge 2 commits into
Conversation
…gagement # Conflicts: # CLAUDE.md # gateway/http/app.py # gateway/http/routers/providers.py
There was a problem hiding this comment.
Pull request overview
Adds a new multimodal “engagement” domain (text/audio/video scoring + fusion) with a dedicated SQLite DB and API endpoints, and wires the live engagement signal into chat completions via adaptive system-prompt injection. It also updates the UI to capture webcam frames and attach voice + webcam context to engagement events, plus adds evaluation tooling and tests.
Changes:
- Introduce
ai/engagement/(scoring cores, fusion, cache, persistence, prompt injection) backed by an isolated engagement SQLite DB. - Add
/api/v1/engagement/*routes, initialize engagement DB on app startup, and inject engagement directives into provider proxy chat bodies when enabled. - Update UI to capture webcam engagement, send engagement events for text/voice, and add offline evaluation scripts + new tests.
Reviewed changes
Copilot reviewed 36 out of 37 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/vite.config.ts | Makes dev proxy target configurable via VITE_BACKEND_URL. |
| ui/src/lib/stores/index.ts | Adds a store for the live webcam engagement score. |
| ui/src/lib/components/chat/WebcamCapture.svelte | New webcam capture + periodic scoring component publishing a live score. |
| ui/src/lib/components/chat/MessageInput/VoiceRecording.svelte | Changes voice confirm flow to include base64 audio payload for engagement. |
| ui/src/lib/components/chat/MessageInput.svelte | Records engagement on text/voice send; adds engagement camera toggle and headless capture. |
| ui/src/lib/apis/engagement/index.ts | New frontend API client for engagement endpoints. |
| tests/test_media.py | Adds test coverage for local Whisper STT fallback. |
| tests/test_engagement.py | Adds fusion/service/router/prompt-injection tests for engagement. |
| tests/conftest.py | Adds isolated in-memory engagement DB wiring for tests. |
| run.py | Adds a minimal app entrypoint for local dev (uvicorn run:app). |
| requirements.txt | Adds engagement-related dependencies (e.g., librosa, mediapipe). |
| manual_check/sortie_benchmark.md | Adds benchmark output artifact for engagement latency. |
| manual_check/RESULTATS_ANALYSE.md | Adds written analysis/results summary for engagement evaluation. |
| manual_check/README.md | Documents manual evaluation tooling and dataset format. |
| manual_check/labels.csv | Adds sample ground-truth labels file used by evaluation scripts. |
| manual_check/grid_search.py | Adds fusion-weight grid search tooling. |
| manual_check/calibrate.py | Adds threshold calibration tooling for engagement levels. |
| manual_check/analyze.py | Adds core evaluation script (ablation, correlations, separability, benchmark). |
| manual_check/ab_analyze.py | Adds A/B analysis tooling (adaptive vs control). |
| learning/sessions/service.py | Notes engagement is recorded client-side (no backend hook here). |
| gateway/http/routers/providers.py | Injects engagement directive into proxied OpenAI/Ollama chat requests (guarded by setting). |
| gateway/http/routers/engagement.py | Adds engagement API router (video/audio/chat + session score/summary). |
| gateway/http/routers/audio.py | Adds local faster-whisper STT fallback when no external STT URL is configured. |
| gateway/http/app.py | Initializes engagement DB on startup and registers engagement router. |
| config/settings.py | Adds ENGAGEMENT_ADAPTIVE_PROMPT setting to gate adaptive prompt injection. |
| ai/media/whisper_stt.py | Implements local STT via faster-whisper with lazy model loading. |
| ai/engagement/video_core.py | Implements video engagement scoring (MediaPipe FaceMesh + optional DeepFace). |
| ai/engagement/text_core.py | Implements text engagement metrics/scoring with session-scoped cache. |
| ai/engagement/service.py | Orchestrates scoring, fusion, caching, and persistence. |
| ai/engagement/repository.py | Adds repository methods for recent rows and averages. |
| ai/engagement/prompt.py | Builds/injects engagement directive into system prompt based on live cached signals. |
| ai/engagement/models.py | Adds ORM model for engagement metrics table (dedicated DB). |
| ai/engagement/fusion.py | Implements normalized weighted fusion over present modalities. |
| ai/engagement/database.py | Adds dedicated DB engine/session + init logic for engagement DB. |
| ai/engagement/cache.py | Adds runtime caches (EMA smoothing, TTL eviction, freshness window). |
| ai/engagement/audio_core.py | Implements audio engagement scoring with soundfile + PyAV fallback decode. |
| ai/engagement/init.py | Exposes engagement service/repository as a domain package. |
Comment on lines
+617
to
+618
| // Voice messages send instantly. | ||
| dispatch('submit', prompt); |
Comment on lines
+71
to
+74
| cameraStatus = 'starting'; | ||
| captureInterval = window.setInterval(capture, captureMs); | ||
| // Grab a first frame quickly so a video score is available right away. | ||
| window.setTimeout(capture, 700); |
Comment on lines
+140
to
+146
| console.debug( | ||
| '[engagement] video frame sent', | ||
| canvasEl.width + 'x' + canvasEl.height, | ||
| imageCapture ? '(grabFrame)' : '(drawImage)', | ||
| '-> score', | ||
| data?.video_score | ||
| ); |
Comment on lines
+56
to
+63
| if not {"user_id", "text_score", "created_at"}.issubset(columns): | ||
| print( | ||
| "[Engagement DB] Dropping incompatible legacy engagement_metrics " | ||
| "table and recreating with the current schema.", | ||
| flush=True, | ||
| ) | ||
| with engagement_engine.begin() as conn: | ||
| conn.execute(text("DROP TABLE engagement_metrics")) |
| user_id=user_id, | ||
| session_id=session_id, | ||
| modality=modality, | ||
| message=message, |
Comment on lines
+188
to
+200
| const bytes = new Uint8Array(await audioBlob.arrayBuffer()); | ||
| let binary = ''; | ||
| const CHUNK = 0x8000; | ||
| for (let i = 0; i < bytes.length; i += CHUNK) { | ||
| binary += String.fromCharCode(...bytes.subarray(i, i + CHUNK)); | ||
| } | ||
| const base64String = btoa(binary); | ||
| const res = await transcribeHandler(audioBlob); | ||
| dispatch('confirm', { | ||
| text: res?.text ?? '', | ||
| audio_base64: base64String, | ||
| duration_seconds: durationSeconds | ||
| }); |
Comment on lines
+78
to
+82
| if not WHISPER_AVAILABLE: | ||
| raise HTTPException( | ||
| status_code=status.HTTP_400_BAD_REQUEST, | ||
| detail="Audio STT URL not configured", | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multimodal Engagement Estimation & Adaptive Tutoring
Project: Multimodal LLM-based estimation and adaptation of learner engagement
Platform: Open TutorAI CE (standalone FastAPI build)
Branch:
feature/multimodal-engagementThis document explains the full feature: what it does, how it is built, how to run
it, how it was evaluated, and the results.
1. What this feature does
LLM tutors are "blind": they only see the text of a message. They cannot tell
whether a learner is engaged, confused, or disengaged. This feature adds an
external module that estimates the learner's engagement in real time from
three complementary modalities — text, video (webcam), and audio (voice) — fuses
them into a single engagement score, and injects a pedagogical directive into the
LLM's system prompt so the tutor adapts its behavior (simplify, check
understanding, deepen) — without any fine-tuning.
2. Architecture overview
A real-time pipeline in four stages, plus persistence:
Layered design (keeps computation isolated from infrastructure):
Code lives in
ai/engagement/:text_core.pyvideo_core.pyaudio_core.pyfusion.pycache.pyservice.pyprompt.pyrepository.py/models.py/database.pyAPI routes are in
gateway/http/routers/engagement.py.3. The three modalities
3.1 Text (
text_core.py)Seven signals extracted from each message:
Combined score (content 60% / rhythm 40%):
3.2 Video (
video_core.py)MediaPipe FaceMesh (478 landmarks) + DeepFace emotion. Six descriptors:
Penalty when gaze is strongly averted:
gaze < 0.3 -> video ×= 0.6.When no face is detected, the cached score decays instead of freezing.
3.3 Audio (
audio_core.py)librosa prosody. Four descriptors:
Penalty when mostly silent:
silence < 0.3 -> audio ×= 0.5.Browser audio (WebM/Opus) is decoded via PyAV when soundfile cannot read it; a
local
faster-whispertranscription is available as an STT fallback.4. Fusion and engagement levels
Normalized weighted mean over present modalities (missing ones are dropped and
the weights renormalized — "graceful degradation"):
A text-only event therefore always produces a valid score. Weights are
configurable per request.
Engagement levels (data-driven calibrated thresholds):
These thresholds were calibrated from the real score distribution (tertiles)
instead of arbitrary 0.40/0.70 cut-points (see §7).
5. Adaptive tutoring (
prompt.py)At each completion, a short directive is injected into the LLM message (compatible
with OpenAI/Ollama and streaming), preceded by a marker telling the model not to
reveal it:
re-engagement question, warm tone.
If no signal is available (no webcam, audio, or history), no directive is
injected and the tutor behaves normally.
6. API endpoints (
/engagement)/engagement/video/engagement/audio/engagement/chat/engagement/session/{id}/summary/engagement/session/{id}/scoreMetrics are persisted in a dedicated, isolated SQLite database (
engagement.db),separate from the application DB.
7. Privacy & ethics
is cached and persisted when a message is actually sent.
8. Evaluation (
manual_check/)Evaluation is offline: the scripts read
engagement.db+ a ground-truth file(
labels.csv, learner self-rating on a 1–5 Likert scale) and produce the analysis.See
manual_check/README.mdfor full usage andmanual_check/RESULTATS_ANALYSE.mdfor the written results.
9. Results summary
Protocol: 8 balanced sessions (4 engaged / 4 disengaged, 2 per scenario
S1 visual / S2 textual / S3 vocal / S4 multimodal), 41 labeled measurements.
Mann-Whitney U = 3.0, p < 0.001 (statistically significant).
levels.
Perspective: a larger, multi-user dataset will further quantify each modality's
marginal contribution and consolidate the calibration; evaluating the pedagogical
impact of the adaptation is the natural next step.
10. How to run the system
11. Task checklist (what was done)
/engagement/*) integrated into the gateway12. Additional informations :
(Mann-Whitney p < 0.001); ordinal validity Spearman rho = 0.873; real-time latency
validated (text/audio instantaneous, video ~4-5 fps).
librosa==0.11.0,mediapipe==0.10.14,deepface(optional;pulls in TensorFlow — video scoring degrades gracefully when absent).