Working-memory PUT/GET latency grows with session length and concurrency

[tylerhutcherson]

Problem

In Story 1 load testing, working-memory latency grew almost linearly with session size, and degraded further as concurrent
active sessions increased.

This was reproduced with:

• long-term memory disabled
• summarization effectively disabled
• one long-running session that kept growing
• a Locust run ramping up to 25 active sessions doing the same thing

That isolates the core working-memory read/write path. The current behavior does not meet the expected “recent context
stays fast as conversations grow” requirement.

Observed behavior

As sessions grew:

• PUT /v1/working-memory/{session_id} latency increased steadily with transcript size
• GET /v1/working-memory/{session_id} latency also increased steadily
• the effect became much worse under concurrency

This suggests the current implementation is doing work proportional to total session size on each update/read.

Likely cause in the current code

The hot path appears to be full-session based:

• put_working_memory_core() replaces the entire session on every write
• _summarize_working_memory() token-counts the full message list before deciding whether to summarize
• set_working_memory() serializes the full messages array and writes the whole JSON document back to Redis
• get_working_memory() loads the full working-memory document, instantiates all messages, then sorts/slices in memory for
  recent_messages_limit

Relevant files:

• agent_memory_server/api.py
• agent_memory_server/working_memory.py

Why this matters

Working-memory operations appear to scale with total conversation length instead of recent context size. Under multi-
session load, that cost compounds quickly.

Proposed fix

Refactor the working-memory path so reads and writes do not require full-session work on every request.

Acceptance criteria

• PUT latency should remain approximately flat as a session grows
• GET with recent_messages_limit should not depend on total transcript length
• concurrent long sessions should degrade gracefully rather than linearly with total stored history

[rbs333]

My testing agrees with this finding. See recreate: https://github.com/rbs333/ams-sim-proj

[tylerhutcherson]

Same, sharing my results from a locust test run:

Far left is single client/session long running.
For right is multiple clients/sessions long running.

[tylerhutcherson]

@abrookins the approach we were considering for this is to use JSON path api on Redis to read/write specific entries from the JSON array instead of pulling the entire object. No changes to schema or data model. WDYT?

[abrookins]

PUT to completely replace the object still seems useful, while we should also have a way to read and write smaller batches of messages from the session. Now that it's JSON, I agree, we can do it without any schema changes.

Just a note though about this:

The current behavior does not meet the expected “recent context stays fast as conversations grow” requirement.

The current setup for this test basically says: If we do nothing, then latency grows as the size of the data grows. That's true, of course. The latency involved in reading and writing subsets of the JSON array will also grow as the size of the data grows, so you'll move the problem farther out for sure, but it'll still be there.

The solution that actually provides "fast recent context as conversations grow" is automatic summarization, which you turned off. That feature trims the working memory size, replacing messages with continual summaries of the conversation. So while it remains an important feature to be able to (IMO) add a small number of messages to an existing session without accessing the entire session, the promise of the server is not that adding and removing messages to a JSON object will be especially fast compared to native storage. Instead, it's the intelligence of automatically maintaining a summary of an ongoing long conversation, so the data does not explode in size.

[tylerhutcherson]

PUT to completely replace the object still seems useful, while we should also have a way to read and write smaller batches of messages from the session. Now that it's JSON, I agree, we can do it without any schema changes.

Just a note though about this:

The current behavior does not meet the expected “recent context stays fast as conversations grow” requirement.

The current setup for this test basically says: If we do nothing, then latency grows as the size of the data grows. That's true, of course. The latency involved in reading and writing subsets of the JSON array will also grow as the size of the data grows, so you'll move the problem farther out for sure, but it'll still be there.

The solution that actually provides "fast recent context as conversations grow" is automatic summarization, which you turned off. That feature trims the working memory size, replacing messages with continual summaries of the conversation. So while it remains an important feature to be able to (IMO) add a small number of messages to an existing session without accessing the entire session, the promise of the server is not that adding and removing messages to a JSON object will be especially fast compared to native storage. Instead, it's the intelligence of automatically maintaining a summary of an ongoing long conversation, so the data does not explode in size.

Interesting. I actually did not realize summarization replaces working memory (if enabled). And noted on PUT.

The usefulness of summarization for long running sessions is definitely the "best of both worlds" sort of solution. We also want to prevent memory issues in AMS and increasing latency as the session grows so probably best to still not read the entire session object (regardless of length) in general.

Regarding this:

The latency involved in reading and writing subsets of the JSON array will also grow as the size of the data grows

I need to research the perf implications of the RedisJSON api for these operations if we're appending and reading from the end of an array up to some count.

Lastly, my other assumption is that the pending product will use a different data model for all of this. So I am also sort of thinking of ways to hold off on longer term changes for now.

[abrookins]

Giving clients the ability to avoid the memory and deserialization cost from moving around full session objects will be great either way. 👍