Add speech-to-phrase Home Assistant app - #21
Merged
Merged
Conversation
- overrides.py: per-language voice-targeting doc. Entities/areas/floors can
be switched off globally or per-command, and given spoken aliases. Applied
uniformly to the grammar (training), the matcher (hassil from_tuples so an
alias resolves back to the HA name) and the UI.
- training: combo_cost() estimates grammar size per intent/combo/domain, so
the UI can show what enabling a command costs.
- presets: example "shapes" -- sentences rendered with slots as placeholders
(domain noun for {name}), keeping the UI about command shape rather than a
wall of device names.
- app: expose cost, shapes, targets and overrides to the web UI; keep the
lists tab showing unfiltered devices/areas/floors so a switched-off one is
greyed rather than gone.
- tools/audio_test.py + tools/subset_check.py, tests/test_intent_server.py
and WAV fixtures (noise, OOV, room impulse responses).
- docs/MODEL_COVERAGE.md: per-language model coverage report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The add-on now does one thing: turn audio into a transcript and hand it to
Home Assistant, whose own conversation agent decides what it means.
- app: the Wyoming intent service is no longer started. intent_server.py and
the matcher stay in the tree; `--intent` brings the service up for
development. Drops /api/test and /api/validate_sentence (the Test tab and
the phrasing validator were their only callers) and the now-unused
intent/scripts_scenes catalogs from /api/state.
- Custom commands are speech-to-text only: sentences, plus the device types
{name} may match. No intent/action/response editing, and a blank card is
no longer saved as an empty command.
- Command Details is read-only -- the ways a command can be said, and what it
costs the grammar. Per-command target narrowing and user-added phrasings
are gone; extra_sentences.json is still honoured if present, so /api/save
only rewrites it when the client sends one.
- Devices & Lists is a plain on/off switch per entity/area/floor: exclusion is
global. Aliases are gone from the UI; anything already in targets.json
(aliases, per-command exclusions) round-trips untouched.
- Test tab removed.
Also fixes the add-on run script, which passed a --intents-yaml flag app.py
does not accept -- argparse aborted before the server ever started.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither was consumed: the run script never passed them and app.py has no matching argument. Removed from config.yaml (options + schema) and the DOCS.md options table, which now documents max_score on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A row on the devices page is a spoken *name*, not an entity, and a name is not
unique -- two devices can be called "Media Player", and every alias is a name
of its own. Nothing said where a row came from.
Entity records now carry `entity_id` and `alias_of` (an alias record remembers
the registry name it stands in for), /api/state groups them by name into
`{name, sources}`, and a row whose name covers more than one entity or stands
in for another lists them underneath:
Media Player [on] 2 entities
• media_player.living_room
• media_player.office
Speaker [on]
• media_player.office — alias of Media Player
The switch stays on the name, since that is what the grammar holds: switching
"Media Player" off silences every entity called that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both in _entity_records, both silent:
1. An entity whose name comes from its device has no registry name of its own
-- name and original_name are null and friendly_name is the only place the
name exists. friendly_name was consulted only when the entity had no
aliases either, so an aliased entity of this kind was reachable *only* by
its alias: "Home Assistant Light" never entered the grammar, just "cool
light". Fall back to friendly_name whenever the registry has nothing.
2. Capabilities were read as `sorted(caps(attrs.get(eid, {})))`, so an entity
missing from /api/states (unavailable, restarting) recorded `features: []`.
gating.py distinguishes None (unknown -> permissive) from empty (known to
support nothing -> gate it out), and [] took the second path: the entity
silently vanished from every brightness/position/speed/volume command.
gating.py's own docstring promises the opposite. Record None when there is
no state to read; [] still means a genuinely incapable entity.
The second one is what makes a dimmable light stop accepting "set <light>
brightness to 10 percent": with the brightness templates gone from the
grammar, the decoder can only reach the nearest path it does have, which for
HassTurnOn/name_only is "[the] {name} on".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
speech-to-phrase-lib takes a word-insertion reward per emitted token,
documented there as countering "the CTC length bias that otherwise lets a
short parse win over a longer, better-fitting one purely on token count".
The add-on never passed one, so it sat at the library default of 0.
That bias is visible: the FST decode picks the lowest-cost path, and audio a
path doesn't account for is absorbed by CTC blanks almost for free, so a
shorter in-grammar phrase can beat the longer one actually spoken -- an
optional trailing "percent" gets dropped, and in the worst case a whole
command collapses onto a short phrase like "[the] {name} on".
Plumbed through wyoming_server (GrammarHolder/serve/start_background/CLI),
app.py (--token-bonus) and the add-on config, plus tools/audio_test.py so the
value can be swept against the TTS + RIR + noise harness. Default 0 -- no
behaviour change until it is set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two bugs kept the harness from running at all:
* load_templates walked a sentences/<lang>/*.yaml tree that no longer exists
(dbc9a91 moved templates to the home-assistant-intents package), so it found
zero templates and train() raised "No sentence templates provided". It now
goes through training.assemble -- the same path production compiles -- so the
sweep measures the real grammar and picks up domain/capability gating instead
of maintaining a second, drifting copy of the scoping rules.
* _spellout_words is memoized and returns a tuple, which the library documents
callers must copy. enumerate_realizations concatenated it onto a list and
raised TypeError on the first template with a {0..100} range -- i.e. exactly
the brightness/position/speed commands worth testing. A --limit under 127
never reached one, which is why this stayed hidden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he env token_bonus: fit with tools/audio_test.py against live HA TTS. On 12 long brightness/speed commands exact decodes went 3/12 at 0 -> 5/12 at 1.0 -> 10/12 at 2.0, saturating there; 4.0 gained nothing and drove OOV false-accepts from 1 to 18. At 0 the failures were the reported ones -- a trailing "percent" dropped, and "set overhead light brightness to four" heard as "overhead light off". Both decode exactly at 2.0. Defaulted per backend like max_score, since the cost scales differ: citrinet 2.0, coqui 0.0 (unmeasured, so left off). max_score needed no code change -- DEFAULT_MAX_SCORE is already 5.0 for citrinet. The 7.0 in use was a per-language override in the dev data dir's settings.json; removed, so it follows the backend default again. audio_test.py: the Home Assistant token was hardcoded in the source. It now comes from HA_TOKEN (and HA_URL), with a clear error when a clip isn't cached and there is no token to synthesize it with. Cached clips need no token at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
slot_lists = _current_slot_lists(cfg, lang)
...
for lang in langs:
`lang` is only bound by the loop underneath, so the first pass raised
UnboundLocalError and every scheduled retrain was skipped. Later passes saw the
value left over from the previous iteration's last language, so the failure was
intermittent rather than permanent, and the except-log kept it quiet.
Hoisting the language set above the fetch exposed a second bug: `entities` and
`slot_lists` were computed once per pass, but override filtering is
per-language, so every language was trained with cfg.language's exclusions --
an entity switched off for voice in one language vanished from all of them.
Now the registry is fetched once per pass (the Home Assistant round-trip should
not be per-language) and each language's own overrides are applied to it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… mapping * The Wyoming service advertised a hardcoded version "0.1.0" while the manifest said 0.2.0. It now reads config.yaml (copied into the image) so there is one source of truth and it cannot drift from what Supervisor installed. * wyoming_server.py --backend auto hardcoded citrinet with a TODO, even though models.resolve_backend already does per-language selection. A coqui-only language (sl/nl/cs) resolved to citrinet and found no model. Dev entry point only -- app.py was already correct -- but it is now the same rule. * settings.set_max_score persisted MIN_MAX_SCORE (0.1) when the value would not parse. 0.1 accepts nothing, so a bad save silently killed recognition until someone thought to look at the gate. It now writes nothing and returns None. * Dropped `map: share:rw` from config.yaml: nothing reads /share, so the add-on was asking for a permission it never used. * /api/save said "skipped retrain (matcher still updates)" -- the matcher is not loaded in a speech-to-text-only build. It now names the language whose model is missing. * custom_commands.py documents that stt is the only mode with any effect while the intent service is not started. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A nested worktree checkout appeared there during development; nothing under it belongs in the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Purge the build toolchain. cmake/g++/git/libfst-dev exist only to compile speech-to-phrase-lib's native _fst module; they are now removed in the same layer that installs them, dropping ~200 MB of gcc/cmake/git from the shipped image. libfst-tools went too -- nothing shells out to the OpenFST CLI. Keeping the right runtime library needed care. _fst links libfst.so.<N> dynamically, but only libfst-dev was named on the command line, so apt treats the runtime package as auto-installed and --auto-remove would delete it. The package name tracks the soname (libfst26 today), so instead of hardcoding it, each shared library the built module links is resolved back to its package and marked manual. `readlink -f` is required: ldd reports /lib/... while dpkg records /usr/lib/... under usrmerge, so querying ldd's path matches nothing -- verified, this silently kept zero packages before the fix. A counter turns that case into a build failure, and a post-purge `import speech_to_phrase` (which loads _fst) is the backstop. Bundle the model. The default en Citrinet model (~140 MB) is fetched at build time into <addon_root>/models, and ensure_model prefers it over the network, so a fresh install boots offline instead of downloading on first run. Anything already in --models-dir still wins, so an existing /data/models copy is kept across updates. Only src/models.py is copied before the fetch, so editing the app doesn't invalidate that layer. --build-arg BUNDLE_MODEL= opts out. Verified with a simulated image layout: a fresh data dir and an empty --models-dir trains and serves with zero download attempts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Self-contained build. The recognition library now lives in lib/ and is built
from source via requirements.txt's `./lib` entry, so the add-on no longer
depends on a git checkout of speech-to-phrase-lib -- a dependency that could
never have installed anyway: it was requested as "speech-to-phrase-lib" while
the project declares itself "speech-to-phrase", and pip rejects a direct
reference whose metadata name differs from the requested one. `git` is gone
from the image with it.
The vendored copy is taken from speech-to-phrase-2, which is ahead of
speech-to-phrase-lib on templates/tokenizer/grammar. Most notably it adds a
subword-segmentation lattice, so the grammar accepts any valid tokenization of
a word instead of one hard-coded segmentation -- its own docstring cites
"percent" (heard as "per cent") as the motivating case.
Speakability, backported from the same repo. Package templates and list values
are authored for hassil's *text* matcher and carry written-only forms; a
grammar path through one asks the acoustic model to emit a token for something
with no sound, so it sits there matching nothing. Values and templates are now
screened by the same rule: written separators (-/–—) become a space rather
than disqualifying the form, so Bulgarian "по-силно" enters as "по силно"
instead of demanding an unpronounceable token; format characters are silent;
combining marks count as spoken, without which Malayalam and Devanagari words
would be discarded wholesale.
Measured on en: 667 -> 539 templates, of which 126 were hyphen spellings
collapsing onto a spoken form already present and only 2 were genuinely
removed -- "set [the] brightness to {0..100}%", which no one can say. The
`1/2` dead path in timer_half is gone. A list left with nothing speakable, or
a template with no spoken phrasing, now logs rather than silently emptying.
Also includes tools/lang_check.py: a per-language round trip (package
templates -> grammar -> TTS -> decode) for validating a new language.
Verified against the vendored library end to end: wheel builds with the native
_fst module, intent-server test passes, the app trains and serves, and the
grammar it writes loads and decodes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nl_NL-coqui is the one shipped model with a demonstrated accuracy bug: it decodes below the score gate, so a misrecognition is acted on rather than deferred to the cloud. A Dutch Citrinet exists (citrinet_nl_256_v2), so point MODEL_NAMES["nl"] at it and keep Coqui as the fallback. Packaged as stt_nl_citrinet_256.tar.gz (model.onnx + tokens.json, matching the layout of stt_de_citrinet_1024.tar.gz); ensure_model downloads and extracts it, and the recognizer loads and trains from the extracted copy. Measured over the Dutch Speech-to-Phrase examples, TTS in, transcript out: Citrinet resolves 52/53 to the right command against Coqui's 46/53. tools/lang_check.py is what produced that number. It runs the production path for one language -- package JSON, training.assemble, FST, recognizer -- then synthesizes each tagged block's example with that language's HA Cloud TTS voice and decodes it. It scores by whether hassil resolves the transcript to the command that was spoken, not by string equality: the decoder legitimately picks a different in-grammar realization (Dutch drops the article, the grammar spells numbers out), and that is the same command, not an error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ways the check reported failures that were not there. It trained the grammar on two entities per domain while the examples are written against the whole fixture set, so an example could name an entity no template could emit; the utterance was then out-of-grammar and the miss looked like a template problem. Dutch lost three cases to this. Taking every entity of an addressable domain also builds a more realistic grammar -- a real install has more than two lights. And a tagged block can be dropped from the Speech-to-Phrase grammar while still belonging in home-assistant-intents: capability gating removes the valve branch of HassSetPosition when no valve supports set_position. Decoding an example built on such a block measures nothing about the templates, so report those as NOT_IN_GRAMMAR instead of scoring them. With both fixed, and against the vendored library, every language's examples resolve to the right command except three cases with identified causes: a French lock/unlock minimal pair, and Czech "2" (the number spellout has no gender agreement, so it compiles "dva" where feminine "hodiny" needs "dvě"). de 59/59, es 55/55, it 50/50, nl 53/53, ca 56/56, fr 55/56, cs 58/59. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither needed a better acoustic model in the sense of "train one".
French: stt_fr_citrinet_1024_gamma_0_25 cannot resolve the "dé-" prefix and
decodes "déverrouille la porte" as "verrouille la porte" at 0.73 -- confidently,
so the score gate does not catch it, and the user gets a locked door when they
asked to unlock. Choosing a different lock verb only moves the error to the more
dangerous direction ("verrouille" then decodes as "déverrouille" at 2.72). Of
the four French models in the dataset, two resolve the pair; the Conformer does
it best and takes French from 55/56 to 56/56 commands resolved, so use it.
Czech: not an acoustic problem at all. The model hears "dvě hodiny" perfectly --
the grammar simply had no path for it. ICU has gender-specific rulesets for
numerals, but the icu_rbnf binding exposes no ruleset argument and always
returns the masculine reading, so "2" compiled only as "dva" while the feminine
"hodiny"/"minuty" require "dvě". _expand_ref already models a number as a list
of alternatives, so offering every reading is a one-line change plus a table of
the low numerals that inflect. Czech goes 58/59 -> 59/59; the same table covers
sk/ru/uk/pl/hr/sl, which have the same agreement and would have hit the same
wall.
Every supported language now resolves every example to the right command:
de 59/59, fr 56/56, es 55/55, it 50/50, nl 53/53, ca 56/56, cs 59/59.
NOTE: templates.py is vendored from speech-to-phrase-2. The numeral-variant
change belongs upstream -- it is carried here only so the fix is not lost when
the vendored copy is refreshed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A `conversation:` sentence trigger, or an `assist_satellite.ask_question`
answer, can only work if the recognizer can produce it: HA matches the
transcript, so a phrase absent from the grammar is a trigger that never fires
and a question that can't be answered. The original add-on fetched both; this
one had stopped.
Both are fetched over the websocket API and added to the grammar as plain
sentences, each behind its own gate -- the add-on options `sentence_triggers`
and `question_answers` (both on, for parity with the original), overridable
per-language in the web UI. Every added phrase widens the FST search space, and
the answer crawl costs a round-trip per automation/script, so an installation
may prefer to pay for neither.
Sentences carrying Jinja2 templates are skipped (no fixed spoken form), as are
answers belonging to disabled automations. HA writes hassil, which the FST
trainer cannot parse, so they go through the same expansion as the packaged
templates; a `{ref}` that survives it has nothing to bind to and takes its
sentence with it.
The web UI shows both sources under Settings, listing the phrases each
contributes with their cost and flagging any that cannot be recognized, and the
Commands meter now counts them -- it named the whole grammar while describing
only the built-in commands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Diagnosing a misrecognition currently means reading DEBUG logs, and tuning max_score means guessing: the score that decided an utterance is never shown. Settings -> Debug mode lists each utterance with the phrasing it matched, the source that phrasing came from, its score against the gate, and the verdict. Rejections are the useful rows -- they show what a phrase was misheard as and by how much it missed, which is the difference between raising the gate and adding a custom command. While it is on, Home Assistant is handed an empty transcript for every utterance. Tuning the gate means deliberately saying things that should be rejected, and the ones that pass must not run the user's lights while they do it. The toggle is per-language in settings.json, re-read by the STT server every utterance, and applies without a save -- it changes only runtime reporting, not the grammar, so making the user retrain for it would misstate the cost. Attribution is reconstructed rather than guessed: the FST returns tokens, not the template it walked, so training.assemble_sources now reports which source contributed each template and sources.py turns those back into patterns over the same list values and -- via the library's own spellout -- the same number spellings the grammar was compiled from. So "set the brightness to fifty percent" is attributed instead of missed for having guessed a different spelling of 50. Built only while the debug view is open, and discarded when the grammar is retrained. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Debug mode reported "not in the grammar" for a custom command using
`[optional]` or `(a|b)`. Built-in templates are pre-expanded into flat phrasings
by hassil before they reach the grammar, but a custom command is not: the
trainer expands the construct inside the FST, so one template stays one
template while the decoder can emit any of its phrasings. Attribution built its
pattern from the template as written, where `[please]` is five literal
characters, and matched nothing the decoder could actually say.
Expand each template through s2p_intents.phrasings -- the same hassil expansion
the built-ins go through, now that it is no longer named for list values only --
and report the hit against the template as authored, which is what the author
will recognize.
Also `{0..100:slot}`, the range form DOCS.md tells users to write: the `:slot`
suffix binds the match to a slot name and says nothing about the reference's
contents, but both the attributor and phrase_count tested the whole reference
against a range pattern anchored at the end. So it read as a list named
"0..100", which nothing defines -- every numeric custom command was
unattributable, and the grammar-size meter priced a 101-value range at one
phrase.
Verified with real audio end to end: a custom command `what time is it
[please]` now attributes to it, having previously come back unattributed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Debug mode lived in <lang>/settings.json, so it survived a restart -- and while it is on the add-on answers Home Assistant with an empty transcript. Leaving it on and restarting therefore came back up with voice doing nothing, and the one thing a person would try to fix that is the very thing that preserved it. It is now held in memory beside the recognition log it fills, so every start is a start with it off. The switch and the log are the same feature and the same lifetime: turning it off discards the log, which described a session that has ended. Nothing about it is written to disk any more, so a settings file from the previous build cannot switch it back on -- and settings.read_bool_file, added only to read it, goes with it. It also stops being per-language, which it never really was: one recognizer runs, and debug mode observes it. The UI said nothing would appear while another language was selected; it now says whose utterances these are instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
home-assistant-intents 2026.8.25 ships the speech_to_phrase blocks for Catalan, Czech, Dutch, French, German, Italian and Spanish. 2026.7.21 had en.json and nothing else, so those seven languages had a model mapped but no grammar to put in front of it. English is byte-identical between the two releases; this is purely the new languages. Every example command in each language round-trips through the real path (package templates -> grammar -> that language's HA Cloud TTS voice -> decode) to the command that was spoken, all within the score gate: de 59/59, ca 56/56, cs 59/59, es 55/55, fr 56/56, it 50/50, nl 53/53. Getting there needed a fix to the harness. lang_check did a plain "import speech_to_phrase", and a development machine usually has the upstream library installed editable -- scikit-build's editable install hooks sys.meta_path, which Python consults before sys.path, so it shadows the vendored lib/ however the path is ordered. Upstream has no subword-segmentation lattice, so the grammar pins one tokenization and the model's natural output cannot match it: clearly-spoken commands come back empty or as a different command. The first run of this bump read ca 37/56 and it 13/50 that way, which is a language looking broken when only the harness was. lang_check now puts lib/ on sys.path and drops any finder that claims the name, leaving the ordinary path-based import as the only one that can answer, and loads the native _fst from lib/build/<tag>/ since the .so is a build artifact rather than a source file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five findings from a review pass over the add-on. Three of them stop it
working entirely; all three were reproduced before being fixed.
A grammar.fst is compiled against one model's vocabulary -- its arc labels
*are* that model's token ids -- but the staleness fingerprint covered only
templates, slot values and backend. So swapping the model for a language left
the old grammar in place, and the recognizer returned an empty transcript for
every utterance, silently, until something else happened to change the
grammar. The French Citrinet -> Conformer change (1024 tokens -> 128) is
exactly that shape: loading the old grammar into the new model decodes '' at
score inf where a retrain decodes correctly at 1.67. The model is in the
fingerprint now, so it retrains once on the first start after an update.
models.py maps a model for thirteen languages; the package ships templates for
eight. On the other five, assemble returned nothing, the library rejected the
empty grammar, and the ValueError escaped create_app -- so `language: ru` was
a container that died on every start with only a traceback to go on. The
configured language is checked up front and reported as a setting to change,
with the supported ones listed; an empty grammar anywhere else now leaves the
previous one alone, and /api/save says so instead of claiming a retrain.
model_name_for fell back to "whatever model this language has" when the
requested backend had none, so `language: cs` with `backend: citrinet`
downloaded the Coqui model and then died on its missing tokens.txt. A missing
model reads as missing: the add-on serves the web UI and logs which backends
that language does have. es_ES-coqui goes with it -- that model cannot load at
all (alphabet of 36 where the decode path expects 30), so listing it only
offered a download followed by a crash.
<data>/<lang>/ is a path join and lang came from the request, so
POST /api/save {"lang": "../../pwned"} wrote enabled.json, settings.json and
custom_commands.json outside the data directory. The set of languages is
small, closed and known, so every route that takes one checks it first.
Also: model archives extract with tarfile's "data" filter (the Python 3.14
default, so behaviour stops depending on the interpreter), and the comment
claiming all commands are on by default now says what actually ships --
config.yaml passes default_importance: usable, which is about half the
catalogue, and DOCS.md lists what that leaves off and how to change it.
Every language still resolves every example to the right command:
de 59/59, ca 56/56, cs 59/59, es 55/55, fr 56/56, it 50/50, nl 53/53.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four leftovers from the pre-release review, none of them load-bearing on their own. The audio buffer grew on every AudioChunk and only AudioStop emptied it, so a satellite that stopped sending one -- crashed, wedged, or streaming an open microphone -- grew it until the add-on was killed for using too much memory. There is now a 30-second budget derived from the announced format, under a 16 MiB absolute ceiling because rate/width/channels are whatever the client says they are. Past the cap the tail is dropped and the head kept: the command follows the wake word, so the start is the part worth decoding, and an over-long capture decodes to something the score gate rejects anyway. Verified by streaming 120s at a 30s cap -- the buffer stops at 960000 bytes and the warning names the likely cause once, not once per chunk. Every value on the Devices & Lists page was escaped except the "Used by:" labels, and while a built-in's label is "Intent/combo", a custom command's is its own first sentence -- so text the user wrote reached the page as markup. A -inf score would have serialized as invalid JSON and broken the debug feed. The old check enumerated +inf and nan by hand; math.isfinite covers all three. docs/MODEL_COVERAGE.md was written when a model was the only thing a language needed, so it counted 13 supported languages. Sentence templates are the other half, and five of those 13 have none -- which reframes the report's central recommendation: the cheapest coverage win is no longer training acoustic models, it is authoring ~50 sentence blocks each for zh/ru/hr/hi/sl, five languages whose acoustic side is already done and validated. Also folds in what has since been fixed (Dutch on Citrinet, the es_ES-coqui mapping dropped, fr/it resolving every command), replaces the hand-built 6-command test with the per-language round trip that superseded it, and reconciles the arithmetic: 9 HA codes served + 7 model-but-no-sentences + 50 no model = 66. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One recognizer runs, for one language, but the web UI offered a picker over all eight and then let you edit whichever you chose. Tuning commands for a language speech-to-text was not serving looks exactly like the add-on ignoring you. So the header states the active language instead of offering it: every request is about the configured one, /api/languages is gone, and the debug view loses its "speech-to-text is actually running for X" warning because that can no longer happen. A page loaded before the option changed is refused with a 409 rather than having its edits applied to the current language -- those choices were made against a different set of devices and commands -- and the reason is shown in the status line instead of a bare "Failed." Debug mode gains a Took column: the time from audio-stop to transcript, which is the pause a user actually experiences. It covers format conversion, level normalization, VAD trimming and the decode as one number, because that is what is perceived; hovering gives the length of speech and the ratio between them. Sub-second values render in milliseconds, since "0.05 s" and "0.52 s" both read as "fast" while 50 ms and 520 ms do not. Measured over the real Wyoming path: 524 ms on the first utterance after a start, then 51 and 46 ms -- the lazy model and VAD load, which is worth being able to see rather than guess at. Verified in a browser against a running add-on: the header renders the language as text with no picker, the feed table takes the extra column without overflowing, and a save carrying a stale language surfaces "this add-on is set to 'de', not 'en'. Reload the page." de 59/59, nl 53/53, cs 59/59 unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The options page had seven entries, and five of them were the wrong place for what they configured. Which commands are on, how confident a decode must be, whether to pull in the phrases Home Assistant already listens for -- these belong to a language, and their cost is only legible next to the grammar-size meter in the web UI. An add-on option cannot show that, and applies to every language at once. So the page is down to the language and debug_logging, and the run script passes nothing else: app.py's defaults are now the shipped configuration rather than something config.yaml overrode on the way past. Behaviour is deliberately unchanged. Verified by running the exact argv the s6 script now execs: backend auto (citrinet for en/de, coqui for cs -- the old argparse default of "citrinet" would have left Czech with no model), the same "usable" 24-of-46 starting set, both sentence sources on, and the per-backend gate and word-insertion reward. A real boot seeds 24 combos, trains 112 sentences and comes up with max_score=5.0, token_bonus=2.0, as before. default_importance is the one whose removal is a small mercy: it seeded a language's *first* run and was never read again, so changing it later did nothing whatsoever. token_bonus is the one real loss -- it is now fixed at the fitted per-backend value, tunable only from the command line. It was never in the web UI and re-fitting it needs tools/audio_test.py, so an options entry was not how anyone was going to arrive at a better number. language becomes a list defaulting to English. Only the eight languages that ship Speech-to-Phrase templates can be recognized, and free text let you name one that could not -- an add-on that then refused to start with a message nobody had asked to see. DOCS.md keeps a table of what each removed option became, since anyone upgrading will look for them. Also: cd || exit in the run script, so a failed cd blames the cd rather than a missing app.py (shellcheck SC2164), and the CHANGELOG's top section is renamed to match the 2.0.0 in config.yaml. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The docs had accumulated a running commentary on their own history: an options
table with a companion table of what each removed option used to be, a coverage
report opening with what changed since its first version, resolved issues kept
under strikethrough, and rationale phrased as the bug it once prevented. All
true, none of it what a reader needs, and each line another thing to keep
correct.
DOCS.md loses the "things that used to be options" table -- what the add-on
chooses for you is worth stating, but as a short list of current behaviour
rather than a migration guide -- and the justifications that only make sense if
you know what came before ("was a way to wonder why nothing changed").
MODEL_COVERAGE.md keeps every fact and drops the narrative. The
model-plus-sentences distinction that motivated its rewrite now simply opens
the report, because it is the thing to understand first. The Spanish and French
model choices stay, restated as live constraints -- don't re-add es_ES-coqui,
don't swap the French Conformer -- since that is why they are worth documenting
at all. The resolved-issues list is replaced by what the test cannot tell you,
which is the part that stays true. Struck-through dataset rows for Dutch and
Slovenian are gone, along with the priority-shift note explaining why they were
struck.
Every number left in both files was re-checked against the code: the coverage
table against MODEL_NAMES and the package's language list, the 13/8/9 and
9+7+50 splits, Gap A and Gap B membership, the cited gates, and the "about half
the catalogue" claim (24 of 46 on German) including each command named as
starting off.
Not touched: CHANGELOG.md, which is a history on purpose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The add-on did not install. The vendored library declared requires-python = ">=3.12" and the Home Assistant Debian base image ships 3.11.2, so pip refused it before compiling anything: "Package 'speech-to-phrase' requires a different Python: 3.11.2 not in '>=3.12'". That floor was never a real requirement. Nothing in src/, lib/ or tools/ needs anything newer than 3.9 (checked with vermin against a 3.11 target, and with ast feature_version for syntax); the only constraint is the native module's limited-API level, and what sets that is the buffer protocol -- Py_buffer and PyObject_GetBuffer, used to read the log-prob array without copying -- which entered the limited API in 3.11, not 3.12. Compiling fstmodule.cc at 0x030A0000 fails on exactly those three symbols and at 0x030B0000 is clean, so 3.11 is the true floor. The wheel is cp311-abi3 now; all 17 CPython symbols the built module leaves undefined are stable-ABI ones. Two more breaks were hiding behind that one, both found by building the real image rather than reasoning about it: python3-dev was never installed, so CMake reported "Could NOT find Python (missing: Interpreter Development.SABIModule)" -- which names the component, not the missing headers. It joins the toolchain that is purged in the same layer, and libfst22/libstdc++6 are still correctly kept. Model archives were extracted with tarfile's filter="data", which does not exist on 3.11.2: it arrived in 3.12 and was backported only as far as 3.11.4, and the base image is *older* than that (verified: hasattr(tarfile, "data_filter") is False in the image). Every model download would have raised TypeError. Making the hardening depend on the interpreter's patch level is not acceptable either way, so where the filter is absent the same restrictions are applied by hand -- traversal, absolute paths, and anything that is not a plain file or directory. Both paths were tested against ../escape, /etc/pwned and a symlink archive, on the host and inside the image on real 3.11.2. Verified by building the image and running it: the grammar trains (byte-identical to the host's), the Wyoming server answers describe, the HEALTHCHECK passes, and three cached TTS clips decode correctly through the real Wyoming protocol on 3.11.2. Image is 863 MB with the toolchain purged. Also added a .dockerignore: the context was ~750 MB, nearly all of it downloaded models under local/ and a host-built CMake cache in lib/build/ that the in-image build had to detect and throw away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The build fails on any Debian release that ships python3-wheel from apt: "Cannot uninstall wheel 0.46.1 ... no RECORD file was found", because pip cannot remove a distribution it did not install. Trixie has it (pulled in by python3-dev); bookworm does not, which is why this only showed up on trying a newer base. The upgrade was never doing anything. The vendored library is a PEP 517 build, so pip creates an isolated environment and installs the backend named in lib/pyproject.toml into it -- the system setuptools and wheel are not used to build anything here. Both bases build after this: bookworm 857 MB, trixie 820 MB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
librosa was used for three calls -- filters.mel, stft and resample -- and hard
-depends on numba and scikit-learn, hence llvmlite, whose single shared object
is 170 MB. That is about a fifth of the image for code the recognizer never
runs: importing librosa loads numba even for a plain decode, but nothing on any
path we exercise ever reaches a jitted function.
_dsp.py now builds the Slaney filterbank and the centred, reflect-padded STFT
directly, and audio.resample calls soxr -- which is what librosa was delegating
to. The image drops from 857 to 582 MB on bookworm (820 to 549 on trixie);
site-packages goes 631 -> 362 MB.
The models were trained on librosa's features, so "close" would not do. Getting
to bit-identical needed two details that are easy to get wrong and invisible
when wrong:
- librosa rounds the filterbank triangles to float32 *before* applying the
Slaney normalization. Computing in float64 and casting at the end differs in
the last bit.
- the transform has to come from scipy.fft, librosa's backend. numpy.fft is
equally correct but differs by ~1e-7 relative, and scipy stays a dependency
for exactly this reason -- 40 MB is a fair price for the front end being the
one the acoustic model was trained against rather than merely close to it.
tests/test_features_parity.py is the gate, and compares four things: the
filterbank, the STFT over 25 real clips, the featurizer's whole output, and the
acoustic log-probabilities -- the last by swapping only the featurizer inside a
loaded model, so the ONNX graph and its inputs are identical and the front end
is the sole variable. All four are exactly equal; resample is too. It skips
where librosa is absent, which is everywhere except a development machine.
Beyond the unit gate: all 408 example utterances across the eight supported
languages produce byte-identical transcripts and identical scores to the
pre-change baseline, and both images decode real audio and train a
byte-identical grammar.
Also: tools/vendored_lib.py now holds the "bind speech_to_phrase to lib/" logic
that lang_check.py had inlined, and the tests call it too. test_debug_mode.py
had been importing the *upstream* library all along without anyone noticing --
harmless while the two agreed, an ImportError the moment lib/ grew a function
upstream did not have.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dev launcher carries a Home Assistant long-lived access token inline, and nothing but its untracked status was keeping it out of the repository -- which a single `git add -A` defeats. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both arches move to the trixie base image: Python 3.13.5 instead of 3.11.2, OpenFST 1.8.4 instead of 1.8.2, and 33 MB less image (amd64 582 -> 549 MB, aarch64 579 MB). Nothing in the tree had to change for it. The soname-agnostic "keep whatever _fst links" logic in the Dockerfile picked up libfst26 on both architectures by itself, which is what it was written for. The abi3 floor stays at 3.11 rather than rising to meet the image. 3.11 is the real floor -- the buffer protocol, which is what the module needs from the limited API -- and targeting it means one wheel that also installs on a Debian 12 base and on whatever a contributor's machine runs. For the same reason the hand-rolled tar extraction check stays: the image now has tarfile's data filter, but the library's floor is below where that was backported, so the hardening still must not depend on the interpreter. Three comments justified the 3.11 floor by pointing at the base image shipping 3.11, which this makes false; they now say what is actually true. Verified on both: amd64 builds from build.yaml, trains a byte-identical grammar (d6555e3b…, unchanged across every base and Python version tried), answers describe, passes its own HEALTHCHECK and decodes three cached clips correctly. aarch64 was cross-built under qemu and decodes correctly too, which exercises the native FST module and onnxruntime on that architecture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Preserve externally bounded audio, disable destructive VPE processing, improve Citrinet candidate selection, and add human-audio/OOV regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Require Citrinet's unbiased grammar decode to pass the production confidence envelope before using the token bonus to rerank longer candidates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use agreement with Citrinet's unconstrained CTC transcript to bound a stronger length-corrected retry, preserving OOV rejection while recognizing the workout command. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat optional timer wording as equivalent only when Home Assistant resolves the same intent and duration, and require a shared acoustic content word before rescuing rejected long phrases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add an eight-language semantic audio corpus, refresh the VPE and noise regression fixtures, and align tokenizer/matcher behavior with home-assistant-intents 2026.8.28.\n\nAdd recommended configurable numeric ranges that narrow both the acoustic grammar and intent matcher while preserving explicit full-range choices. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation