fix: name a representative error in the deep-search non-404 failed fragment

[whoamiTM]

What does this PR do?

Follow-up to my comment on #1784. That fix split the timeout class out of the generic "per-id fetch raised a non-404 error" bucket, but every other non-404 exception still collapses into the same opaque string. On my box that bucket was hiding a fast HTTP 500 on every per-id script fetch: HA core's script config view file-loads scripts.yaml on each request and dies on a !secret reference (annotatedyaml.YAMLException: Secrets not supported in this YAML file) before it ever gets to the id lookup. Nothing timed out, but the warning read identically to the pre-#1810 timeout case, and the actual error was only visible after a debug-log dive. I know ha_search is reworked in the custom component, but the container/addon/uvx server path in src/ still carries this exact bucket, so filing the fix there.

This PR makes the failed fragment name one representative error inline:

27 script(s) not scanned (per-id fetch raised a non-404 error; e.g. HTTP 500: 500 Internal Server Error) — their match status is unknown; this result is not exhaustive. A per-id config fetch returning HTTP 500 is most often a `!secret` reference in the config file HA loads for this entity, which the per-id config endpoint rejects; the specific cause is in the Home Assistant log (the 500 body is a generic placeholder), not in this response.

Correction from the first revision (thanks @kingpanther13 for catching it): the rendered sample is HTTP 500: 500 Internal Server Error, not HTTP 500: Secrets not supported in this YAML file. That YAMLException message never reaches the HTTP body — core's BaseEditConfigView.get() file-loads the YAML with no try/except and the request handler only catches vol.Invalid/ServiceNotFound/Unauthorized, so the exception escapes to aiohttp, whose production 500 body is the fixed 500 Internal Server Error text. The !secret detail goes to the HA log only. So the sample can name the status but not the cause — which is why the fragment now also appends a static hint pointing at the HA log and the single most common cause (a !secret the per-id config endpoint rejects). The hint is deliberately endpoint-agnostic — it names no specific YAML file, since the same string rides the automation, script, and scene fragments and a scene 500 has nothing to do with scripts.yaml. That delivers the five-minute diagnosis the report asked for without the response claiming a cause it can't actually confirm. My test mocks now use the realistic 500 Internal Server Error body throughout.

Implementation: a new summarize_fetch_error() in _fetch.py renders an exception as HTTP <code>: <first line of the message> for HomeAssistantAPIError (stripping the client's own API error: <code> - prefix so the code isn't stated twice) or <ExceptionType>: <first line> otherwise, capped at 160 chars. A companion http_500_diagnosis_hint() returns the static HA-log hint for an HTTP 500: sample and "" for anything else (so non-500 fragments stay byte-identical). The Attempt-C fetch closures for automations, scripts, and scenes capture a summary on the generic failed class only, and only the first one — the append is now guarded (if not failed_errors:), so the motivating "every per-id fetch 500s" case does N−1 fewer summarize_fetch_error calls; every failure still counts. The per-type helpers return the sample as a new trailing tuple slot, and _apply_per_type_partial_flag / _apply_scene_partial_flag append it as an e.g. (plus the 500 hint) when present. With no sample the fragment wording is byte-identical to today, so anything matching on the current string is unaffected.

One asymmetry worth flagging: automations and scripts have a dedicated yaml_skipped branch that classifies a per-id 404 out of the generic bucket, so their 404s never produce a sample. Scenes have no such branch (their integration-managed scenes are pre-filtered upstream via the registry walk instead), so a scene per-id 404 does fall through to the generic failed class and can surface an e.g. HTTP 404: … sample. I left that as-is rather than adding a scene 404 branch — it's a faithful "this scene fetch failed with a 404" signal, not a misclassification — but calling it out so the "404s never produce a sample" statement is scoped to automations/scripts, not scenes.

Tests were written first per the TDD guideline: unit tests for the summarizer and the 500-hint scoping, fragment-assembly tests for both flag helpers with and without a sample (including that the hint rides only HTTP-500 samples), component tests pinning the new tuple slot (including that 404s and timeouts leave it None on the automation/script path, and a new test through the script closure's generic except Exception branch), a test proving the first-error guard summarizes exactly once for N failures, and seam tests driving my exact scenario end to end through public deep_search for scripts and scenes. One existing scene test (test_partial_reason_distinguishes_integration_from_failure) pinned the exact old fragment while its fixture drives a real RuntimeError; updated it to pin the new sample-carrying wording. Full unit suite is green locally apart from 3 failures in test_helper_response_shape.py that fail identically on a clean master checkout here, so I'm reading those as environment-specific; I didn't run the Docker e2e suite locally and am counting on CI for that.

Type of change

• 🐛 Bug fix

Testing

• I have tested these changes with a LLM agent
• All automated tests pass (uv run pytest)
• Code follows style guidelines (uv run ruff check)

Checklist

• I have updated documentation if needed

[ghhamcp]

@codex review — apply the review criteria in .gemini/styleguide.md in addition to AGENTS.md guidance

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 43a5309dfc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kingpanther13]

Thanks — the mechanism is right, and I verified the plumbing end to end (classification isolation, byte-identical no-sample wording, all callers updated). Requesting changes on three items, all within files the PR already touches:

1. The headline example can't actually happen — re-mock the automation/script tests with a realistic 500 body. HA core's BaseEditConfigView.get() file-loads the YAML with no try/except before the id lookup, and the request handler only catches vol.Invalid/ServiceNotFound/Unauthorized — so the YAMLException escapes to aiohttp, whose production 500 body is the fixed 500 Internal Server Error text. "Secrets not supported in this YAML file" goes to the HA log, never the HTTP body. The real rendered sample for your scenario is HTTP 500: 500 Internal Server Error, not HTTP 500: Secrets not supported in this YAML file. Your own scene test already mocks the realistic form — but the automation/script tests in test_deep_search_tier3_parallel.py / test_ha_search_merge.py mock API error: 500 - Secrets not supported in this YAML file, pinning output real HA can't produce. Please re-mock those realistically, update the asserted samples, and correct the description's example. (The fix is still worthwhile — the sample names the 500 and sends the user to the logs.)

2. Capture only the first error instead of summarizing all N. Each closure runs summarize_fetch_error() and appends on every failure, but only failed_errors[0] is read — in exactly the scenario motivating this PR, that's N summarizations with N−1 discarded. Guard the append (if not failed_errors:) or use a scalar; three sites (_deep.py automation + script closures, _scenes.py). The "List membership tracks failed_count" comments describe an invariant nothing relies on — drop them with the change. While in there: the script generic except Exception branch (_deep.py:593) has no test through it; the guard change is a natural moment to add one.

3. Two test docstrings went stale. The 5→6-tuple widening makes test_deep_search_tier3_parallel.py:389 and :721 ("…returned 5-tuple") wrong — one-word fixes (5→6).

Minor, take or leave:

• Scene 404s can produce an e.g. HTTP 404: … sample on the registry-failed fallback (the scene closure has no yaml_skipped branch), contradicting the description's "404s never produce a sample" — either exclude 404s from the scene sample or scope the claim to automations/scripts.
• Since the real sample can't name the cause (see 1), consider a static hint when the sample is an HTTP 500: "check HA logs; a common cause is a !secret in scripts.yaml/automations.yaml, which the per-id config endpoint rejects." That would deliver the five-minute diagnosis your report asked for.

[whoamiTM]

@kingpanther13 thanks for the thorough pass, all three are fixed and pushed. Walking through them:

The headline example. You're right, and this was the important one. I traced it back through the client and the sample my own PR text showed can't actually render. _raw_request builds the message from response.json()'s message field, falling back to response.text, so when the YAMLException escapes BaseEditConfigView.get() into aiohttp's production 500 handler the body is the fixed 500 Internal Server Error string, not the secrets text (that goes to the HA log only). So the real rendered sample is HTTP 500: 500 Internal Server Error. I re-mocked the automation/script tests in test_deep_search_tier3_parallel.py / test_ha_search_merge.py to that realistic body, updated the asserted samples, and corrected the PR description. Since the sample can now only name the status and not the cause, I added a static hint on HTTP-500 fragments pointing at the HA log plus the common !secret cause, which is the five minute diagnosis I was after without the response claiming something it can't confirm.

Capture only the first error. Good catch on the N summarizations. Pulled the guarded append into a record_first_failure() helper in _fetch.py (if not failed_errors:), so all three sites now capture only the first and everything else is counted but not summarized. Dropped the "List membership tracks failed_count" comments like you said. The guard also happened to be the natural moment for the untested script except Exception branch, so I added a test through it, plus a spy test that pins summarize_fetch_error at exactly one call for N failures.

Stale docstrings. Fixed both, 5-tuple to 6-tuple at :389 and :721.

On the two minor ones: I took the scene-404 point by scoping the "404s never produce a sample" claim to automations/scripts in the description rather than adding a scene 404 branch, since a scene per-id 404 falling into the generic failed class is a faithful "this fetch 404'd" signal, not a misclassification (scenes pre-filter integration-managed ones upstream via the registry walk instead). And I took the static-500-hint suggestion. Worth flagging: my first cut of that hint named scripts.yaml/automations.yaml literally, which is wrong on the shared scene path since a scene 500 has nothing to do with those files, so the hint is now endpoint-agnostic ("the config file HA loads for this entity") and rides all three fragments correctly. There's a regression test asserting the scene fragment never names those files.

Full unit suite is green here apart from the same 3 test_helper_response_shape.py failures that fail identically on a clean master checkout, so still reading those as environment-specific on my end. Still no local Docker e2e or live-HA run, counting on CI for that.

[kingpanther13]

Round 2 verified — all three requested items addressed cleanly (realistic 500 mocks, guarded first-capture with the spy test, docstring fixes), plus both minors taken with the endpoint-agnostic hint done right. Remaining minor polish (a scene-404 regression pin and hardening the 500-hint selection) will land in a follow-up. Thanks for the thorough response — great first contribution.

[github-actions]

🧪 Your changes are now in the dev channel!

Your PR has been merged to master and is available for testing in the dev channel.

Test your changes before the next stable release (biweekly Wednesday):
📖 Dev Channel Documentation

Quick start

# Run dev version
uvx ha-mcp-dev

# Check version
uvx ha-mcp-dev --version

Docker:

docker pull ghcr.io/homeassistant-ai/ha-mcp:dev
docker run --rm -i \
  -e HOMEASSISTANT_URL=http://your-ha:8123 \
  -e HOMEASSISTANT_TOKEN=your_token \
  ghcr.io/homeassistant-ai/ha-mcp:dev

---

Found an issue? Please open a new bug report and mention this PR for context.