LLM auto update a system prompt to summarize relationships and glossaries

[windfox1243]

Instead of getting the context by chunks - which is quite ineffective and inconsistent, i think a great idea is that we keep summarizing genders, the way the characters address each other – in short, their relationship, and we can also store glossaries too. All in a txt file like system prompt

[NhutEco2]

I'm translating into Vietnamese, a language with an extremely complex system of personal pronouns and forms of address. Unlike English, Vietnamese does not have a fixed, small set of pronouns; depending on how they are counted, there are well over 100 commonly used personal pronouns and kinship-based forms of address, and the actual number is effectively open-ended because many nouns and titles can function as pronouns.

To make the translations reasonably accurate, I currently do the following: I convert the entire EPUB(s) into plain text, then feed it to an AI chatbot of my choice (I'm currently using Gemini) and ask it to generate a custom instruction that clearly describes the relationships and forms of address between the characters. I then attach this .txt file to the custom instruction.

In addition, I use relatively large chunks (currently 5,000 tokens per chunk when using Gemini 3.1 Flash Lite). My hope is that providing larger chunks gives the LLM broader context and helps it make better decisions, although I'm not actually sure whether this is true. As far as I know, this approach is largely unverified and based more on intuition than evidence.

Sometimes the model can be a bit dumb and still mix things up if it fails to understand the context—especially in dialogue, where it may lose track of who is speaking—but overall the results are fairly good.

[windfox1243]

Hi, i'm currently adding this feature myself on my fork, and surprisingly it works really well. I divide context into 2 type: global context and chunk context, and new context will be created / delete / summarize over time depends on past context. After all, the refine translate will do it job lol (there're many bugs right now btw, mostly UI). Hope to see the official implementation here, as i believe it might be a better approach than current chunk system:)

[hydropix]

Hey @windfox1243, this is exactly the kind of context handling I'd love to bring into the main repo. Could you share a bit more about what you built?

Most useful of all: is your fork public? A link to the branch (or even a rough draft PR) would let me read the exact approach and help get it merged.

If a code share isn't handy yet, a few specific questions:

1. Data shape — what does the "global context" hold versus the "chunk context"? Is the global context one evolving document (free text? JSON?) covering character gender, relationships, forms of address, and glossary, or separate structures per category?

2. Update loop — when do you refresh it? An extra LLM call after each chunk to re-summarize, or is the summary piggybacked on the translation response itself?

3. Prompt injection — do you put the global context in the system prompt or the user prompt? The repo currently keeps the glossary in the user prompt on purpose, so the system prompt stays stable and cacheable across chunks. Curious how you handle caching with an evolving context.

4. Growth control — you mentioned automatic creation, deletion, and summarization "based on historical patterns." What triggers a deletion, and how do you keep the global context from growing past the token budget? A hard cap, LRU-style eviction, periodic re-summarization?

5. Relation to the existing glossary — the repo already has an NER-based glossary (SQLite store + per-chunk filtering injected into the prompt). Does your global context replace that, or sit alongside it?

6. Cost/latency — roughly how much slower is a full book versus the current pipeline, and how many extra tokens/calls per chunk?

7. Testing — which models and language pairs have you tried it on?

Happy to help turn this into a PR. Thanks for prototyping it.

[windfox1243]

Hi! Thanks for your interest. I should clarify that this was a experiment rather than a clean implementation. The code is currently quite messy and contains a lot of abandoned or unnecessary experiments.
My fork is public, so you’re welcome to inspect the implementation directly here: https://github.com/windfox1243/TranslateBooksWithLLMs---RAG-context/tree/global-context-experiment.
The exe is also available, but the source code in the fork will be much easier to understand than trying to extract it.
I haven’t prepared reliable benchmarks or documented all the design decisions yet, so I don’t want to give you inaccurate answers to the detailed questions. Please feel free to reuse or adapt anything useful you find. If you point me to specific parts of the code that need clarification, I’ll be happy to explain what I was trying to do.
P/s: There's still a bug about align chunks before / after resume job which might affect context snapshot. I'm trying to fix it right now:)
Edit 2: Fixed

[windfox1243]

Hey @hydropix, roughly the idea is:

1. The “global context” is currently one evolving text file, not SQLite/JSON. It has a stable lore section and a dynamic state section.
   The stable part stores things like:

• characters and gender
• aliases / identity links
• glossary-style terminology

The dynamic part stores things that can change across the story:

• current addressing forms
• relationship evolution
• dialogue speaker/addressee state
  It is one document with sections, instead of separate stores per category

2. When auto-update is enabled, it does an extra LLM call before translating each chunk. That call analyzes the source chunk and updates the context first, then the updated context is injected into the translation prompt for that chunk. So it is not piggybacked on the translation response. I chose this because I wanted the translator to already know the new character/gender/addressing info before translating the current chunk.

3. At the moment, the novel context is injected into the translation system prompt. That part is probably not ideal for your current caching design. The existing glossary still stays in the user prompt. If this were cleaned up for the main repo, I think the novel context should probably be moved to the user prompt too, similar to the glossary, so the system prompt can stay stable/cacheable.

4. This part is still rough. There is no clever LRU/token-budget system yet.
   Right now growth is controlled mostly by:

• asking the model to output replacements/updates instead of appending repeated facts
• canonicalizing characters/aliases so titles or forms do not become duplicate characters
• allowing explicit DELETE outputs for obsolete characters, aliases, glossary entries, addressing forms, or relationships
• normalizing/merging sections after each update

So deletion is model-driven plus merge/dedupe logic. There is no hard token cap yet, and no real periodic compression pass beyond the incremental summarizing behavior.

5. It sits alongside the existing glossary. It does not replace the SQLite/NER glossary. The existing glossary is still useful for fixed known terms and per-chunk filtering. The novel context glossary is more like “terms discovered while reading the book,” mixed with characters, aliases, relationships, and forms of address. There is definitely overlap, and this part would need design cleanup before merging. But if glossary is in user prompt, it will override the context (i guesss :v)

6. With auto-update enabled, it adds about one extra LLM call per chunk. Also, because the context evolves chunk by chunk, the pipeline forces sequential translation instead of parallel workers -> it is noticeably slower than the current pipeline. I have not benchmarked it properly, but the simple mental model is: translation call + smaller context-analysis call per chunk, and no parallelism while auto-update is active. Cos i haven't tested it properly, idk what to say.

7. Just English / Chinese / Korean to Vietnamese (most complicated language to translate btw). I have not tested enough models/language pairs to make strong claims yet but from the results, there are substantial improvements, especially for small models.

[hydropix]

That’s interesting. I’m also working on a role-playing system with a two-level summary. I’m not sure it’s necessary to anticipate the content. Summarizing progressively can be incredibly effective for maintaining narrative coherence; I’m going to implement several solutions and test them.

And yes, then there are the fundamental limitations of LLMs, which often perform poorly with languages that are less represented in the corpora. We can improve some of these limitations to a certain extent, but there’s a limit that’s impossible to overcome.

[windfox1243]

Yeah, but more work for LLM, less work for human is a W