Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/agents/mirror.prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ If you'd find prior reflections useful — for example, you suspect the student

## Hard constraints

- **English only.** Write every field in English unless the product explicitly passes a different language instruction.
- **No diagnostic language.** Do not label the student's personality, ability, or identity. Describe what they did and what they said, never who they are.
- **No advice.** Do not suggest what to do. That is not your job.
- **No careers, no pathways.** That is Pathfinder's job.
Expand Down
31 changes: 31 additions & 0 deletions src/agents/openai-realtime/mirror-payloads.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,43 @@ import LIVE_PROMPT_RAW from './mirror-realtime-live.prompt.md?raw'

const MIRROR_JSON_SHAPE = '{"validation":"","inferred_meaning":"","story_reframe":""}'
export const OPENAI_REALTIME_MIRROR_VOICE = 'marin'
export const OPENAI_REALTIME_MIRROR_TRANSCRIPTION_LANGUAGE = 'en'
export const OPENAI_REALTIME_MIRROR_TRANSCRIPTION_MODEL = 'gpt-4o-mini-transcribe'

const LIVE_INSTRUCTIONS = LIVE_PROMPT_RAW.trim()

export function buildRealtimeMirrorLiveInstructions(): string {
return LIVE_INSTRUCTIONS
}

export function buildRealtimeMirrorLiveAudioInputConfig() {
return {
transcription: {
model: OPENAI_REALTIME_MIRROR_TRANSCRIPTION_MODEL,
language: OPENAI_REALTIME_MIRROR_TRANSCRIPTION_LANGUAGE,
},
noise_reduction: { type: 'far_field' },
turn_detection: {
type: 'server_vad',
create_response: false,
interrupt_response: true,
threshold: 0.5,
prefix_padding_ms: 700,
silence_duration_ms: 800,
},
} as const
}

export function buildRealtimeMirrorLiveResponseInstructions(): string {
return [
buildRealtimeMirrorLiveInstructions(),
'',
'The student has just finished one English voice turn.',
'Reply in English only.',
'Keep this spoken reply short and natural.',
].join('\n')
}

export function buildRealtimeMirrorUserInput(transcript: string): string {
return [
'The student had this live voice session with the Companion while looking into the mirror scene.',
Expand Down Expand Up @@ -42,6 +72,7 @@ export function buildRealtimeMirrorRepairInput(previousText: string): string {
export function buildRealtimeMirrorResponseInstructions(): string {
return [
'Use the latest student transcript item in this conversation.',
'Write every field in English.',
'Return ONLY a JSON object with validation, inferred_meaning, and story_reframe.',
`The object must match this shape: ${MIRROR_JSON_SHAPE}.`,
'Do not ask a question. Do not give advice. Do not include Markdown.',
Expand Down
34 changes: 18 additions & 16 deletions src/agents/openai-realtime/mirror-prompt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,21 @@ import { fileURLToPath } from 'node:url'
import type { RealtimeSessionCreateRequest } from 'openai/resources/realtime/realtime'
import { OPENAI_REALTIME_MIRROR_DEFAULT_MODEL } from './config'
import {
buildRealtimeMirrorLiveAudioInputConfig,
buildRealtimeMirrorLiveInstructions,
OPENAI_REALTIME_MIRROR_TRANSCRIPTION_LANGUAGE,
OPENAI_REALTIME_MIRROR_TRANSCRIPTION_MODEL,
OPENAI_REALTIME_MIRROR_VOICE,
} from './mirror-payloads'

export {
buildRealtimeMirrorLiveAudioInputConfig,
buildRealtimeMirrorLiveInstructions,
buildRealtimeMirrorRepairInput,
buildRealtimeMirrorResponseInstructions,
buildRealtimeMirrorUserInput,
OPENAI_REALTIME_MIRROR_TRANSCRIPTION_LANGUAGE,
OPENAI_REALTIME_MIRROR_TRANSCRIPTION_MODEL,
OPENAI_REALTIME_MIRROR_VOICE,
} from './mirror-payloads'

Expand All @@ -32,6 +38,7 @@ export function buildRealtimeMirrorInstructions(): string {
getMirrorSystemPrompt(),
'',
'## Realtime session rules',
'- Always write the final Mirror JSON fields in English.',
'- The student is not in an interview. Do not ask questions.',
'- For voice input, listen until the app sends the explicit stop/commit event.',
'- Return text only.',
Expand Down Expand Up @@ -61,22 +68,17 @@ export function buildRealtimeMirrorSessionConfig({
output_modalities: [mode === 'live_audio' ? 'audio' : 'text'],
max_output_tokens: 1000,
audio: {
input: {
transcription: {
model: 'gpt-4o-mini-transcribe',
language: 'en',
},
noise_reduction: { type: 'near_field' },
turn_detection:
mode === 'live_audio'
? {
type: 'semantic_vad',
create_response: true,
interrupt_response: true,
eagerness: 'auto',
}
: null,
},
input:
mode === 'live_audio'
? buildRealtimeMirrorLiveAudioInputConfig()
: {
transcription: {
model: OPENAI_REALTIME_MIRROR_TRANSCRIPTION_MODEL,
language: OPENAI_REALTIME_MIRROR_TRANSCRIPTION_LANGUAGE,
},
noise_reduction: { type: 'far_field' },
turn_detection: null,
},
...(mode === 'live_audio' ? { output: { voice } } : {}),
},
tool_choice: 'none',
Expand Down
10 changes: 10 additions & 0 deletions src/agents/openai-realtime/mirror-realtime-live.prompt.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
You are a reflective journaling companion. Students talk with you in live conversation. You are a good listener who prompts when needed but uses very short and simple acknowledgments (hmm, nodding, short ack) and prompts to go deeper when needed.

Always respond in English. If the student speaks in English, keep every spoken reply and follow-up in natural English. Do not switch to Indonesian, Malay, Singlish, or any other language unless the student explicitly asks to practice that language.

## Two modes

You are always in one of two modes. Read the conversation to know
Expand Down Expand Up @@ -157,6 +159,14 @@ A student who slows down and goes quiet gets more space.

If the student is checking whether the mic works: say only
"I can hear you."
If the student only says a very short or vague thing ("hi",
"hello", "I don't know", "nothing", "not sure", or one
unclear fragment), do not assume an event has already
happened. Do not ask "who were you with?", "where were you?",
or any detail that implies a story exists. Instead, invite
them gently toward school or after-school life:
"Anything interesting happen during school or after school today?"
Or: "Anything from school or after school still on your mind?"
If the student asks what to talk about: give one simple
invitation about something that happened recently,
then leave space.
Expand Down
26 changes: 24 additions & 2 deletions src/components/student-space/EngineHost.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,17 @@ const SURFACES_REQUIRING_HYDRATION = new Set(['trajectory'])
* is unsafe under SSR: some engine modules still expect a browser-owned
* `window` / `document` during evaluation.
*/
export function EngineHost({ className, children }: { className?: string; children?: ReactNode }) {
export function EngineHost({
className,
children,
showOnboardingFlow = true,
hideCompanion = false,
}: {
className?: string
children?: ReactNode
showOnboardingFlow?: boolean
hideCompanion?: boolean
}) {
const containerRef = useRef<HTMLDivElement | null>(null)
const [error, setError] = useState<Error | null>(null)
const backend = useMemo(() => createStudentSpaceBackendBridge(), [])
Expand Down Expand Up @@ -95,6 +105,18 @@ export function EngineHost({ className, children }: { className?: string; childr
game.setRenderActive(isWorldRoute)
}, [game, isWorldRoute])

useEffect(() => {
if (!game || !hideCompanion) return
const group = (game as unknown as { view?: { kira?: { group?: { visible: boolean } } } }).view
?.kira?.group
if (!group) return
const previousVisible = group.visible
group.visible = false
return () => {
group.visible = previousVisible
}
}, [game, hideCompanion])

useEffect(() => {
document.body.classList.toggle('student-space-page-route', !isWorldRoute)
return () => document.body.classList.remove('student-space-page-route')
Expand Down Expand Up @@ -255,7 +277,7 @@ export function EngineHost({ className, children }: { className?: string; childr
<CaptureChooser />
<AskSheet />
<MoodSheet />
<OnboardingFlow />
{showOnboardingFlow ? <OnboardingFlow /> : null}
{import.meta.env.DEV && game ? <CameraTuneBridge game={game} /> : null}
{children}
</EngineOverlayProvider>
Expand Down
Loading
Loading