Skip to content

llm_factory: cache parsed llm_config per profile#671

Merged
neoneye merged 1 commit intomainfrom
perf/cache-llm-config
May 5, 2026
Merged

llm_factory: cache parsed llm_config per profile#671
neoneye merged 1 commit intomainfrom
perf/cache-llm-config

Conversation

@neoneye
Copy link
Copy Markdown
Member

@neoneye neoneye commented May 5, 2026

Summary

Production logs were full of repeated INFO/WARNING noise at every Luigi task start:

INFO  planexe_config        - Optional file '.env' not found …
WARN  planexe_llmconfig     - Environment variable 'OPENAI_API_KEY' not found.
WARN  planexe_llmconfig     - Environment variable 'DASHSCOPE_API_KEY' not found.
WARN  planexe_llmconfig     - Environment variable 'DEEPSEEK_API_KEY' not found.
INFO  planexe_llmconfig     - Applied PLANEXE_LLM_CONFIG_WHITELISTED_CLASSES=…

…because every LLM lookup was re-reading the JSON, re-substituting env vars, and re-applying the class whitelist.

Now _load_llm_config caches the parsed PlanExeLLMConfig in a module-level dict keyed by resolved ModelProfileEnum. Subsequent calls hit the cache. Pricing and Anthropic usage-hook side effects move into the cache-miss branch — same end state, just no longer redone per call.

Latent mutation bug fixed in passing

get_llm was mutating config["arguments"] in place when merging caller kwargs. Pre-cache that was harmless (the dict was rebuilt every call). With caching it would leak state across calls. Shallow-copy the dict before update().

Not in this PR

The OPENAI_API_KEY / DASHSCOPE_API_KEY / DEEPSEEK_API_KEY warnings still fire on the first load even though those entries are dropped by the OpenRouter class whitelist. Fixing that needs filter_by_whitelisted_classes to run before substitute_env_vars in PlanExeLLMConfig.load. Saved for a separate PR.

Test plan

  • Boot worker_plan, run a plan, confirm the Optional file '.env' not found / Environment variable …_API_KEY not found lines now appear at most once per profile per worker (instead of per task).
  • get_llm("foo", max_tokens=8192) followed by get_llm("foo") — the second call must NOT see max_tokens=8192 leaked in.

🤖 Generated with Claude Code

Production logs were full of repeated INFO/WARNING noise on
every Luigi task start:

    INFO  planexe_config        - Optional file '.env' not found …
    WARN  planexe_llmconfig     - Environment variable 'OPENAI_API_KEY' not found.
    WARN  planexe_llmconfig     - Environment variable 'DASHSCOPE_API_KEY' not found.
    WARN  planexe_llmconfig     - Environment variable 'DEEPSEEK_API_KEY' not found.
    INFO  planexe_llmconfig     - Applied PLANEXE_LLM_CONFIG_WHITELISTED_CLASSES=…

…each LLM lookup was re-reading the JSON, re-substituting env
vars, and re-applying the class whitelist.

Cache the parsed `PlanExeLLMConfig` in a module-level dict
keyed by resolved `ModelProfileEnum`. The pricing and
Anthropic-usage-hook side effects only need to run on first
load, so they move into the cache-miss branch — same end state.

Side effect: `get_llm` was mutating `config["arguments"]` in
place when merging caller kwargs. Pre-cache that was harmless
(the dict was rebuilt every call); with caching it would leak
state across calls. Shallow-copy the dict before `update()`.
@neoneye neoneye merged commit 676196b into main May 5, 2026
3 checks passed
@neoneye neoneye deleted the perf/cache-llm-config branch May 5, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant