Should Rig support composable multi-layer agent memory?

[Parikalp-Bhardwaj]

Rig currently has ConversationMemory for managing ordered conversation history, along with memory policies such as compaction and demotion.

There is also ongoing work around durable conversation-memory backends and session/event persistence.

I wanted to discuss a slightly different problem:

Should Rig eventually provide a small abstraction for composing different kinds of agent memory?

For production agents, memory often serves different purposes:

• Working / short-term memory — recent context required for the current interaction
• Conversation memory — chronological conversation history
• Semantic memory — facts retrieved based on relevance
• Episodic memory — summaries or observations from previous interactions
• Profile/entity memory — relatively stable information about users, projects, or organizations

These have different retrieval and storage semantics.

A possible flow could look like:

User Prompt
    |
    v
+----------------------+
| Conversation Memory  |
+----------------------+
    |
    +-------------------------+
    |                         |
    v                         v
Semantic Recall         Profile / Entity Memory
(vector search)         (structured storage)
    |                         |
    +------------+------------+
                 |
                 v
          Context Assembly
                 |
                 v
               Agent

I don't think ConversationMemory itself should necessarily grow to handle all of these cases.

Instead, could a higher-level composable abstraction make sense?

For example, conceptually:

trait MemorySource {
    async fn recall(
        &self,
        context: &MemoryContext,
    ) -> Result<Vec<MemoryItem>, MemoryError>;
}

With implementations such as:

ConversationMemorySource
SemanticMemorySource
EpisodicMemorySource
ProfileMemorySource

Then an application could compose them:

MemoryPipeline
    |
    +-- Recent conversation
    |
    +-- Semantic retrieval
    |
    +-- Profile facts
    |
    +-- Episodic recall
    |
    v
Ranking / token budget
    |
    v
Agent context

The distinction I have in mind is:

ConversationMemory
    = persistence and management of ordered conversation state

MemorySource / MemoryPipeline
    = retrieval and composition of relevant context

This could potentially reuse Rig's existing vector-store abstractions for semantic memory rather than introducing another storage system.

Some questions I'd be interested in hearing opinions on:

1. Should this remain entirely application-level, with Rig exposing only low-level primitives?
2. Would a common MemorySource abstraction be useful, or would it make Rig too opinionated?
3. Should semantic memory directly reuse Rig's existing vector-store abstractions?
4. Should memory retrieval happen before every model turn or once per agent run?
5. Where should ranking and token-budget management live?
6. Should recalled memories carry metadata such as source, timestamp, confidence, or conversation ID?
7. How should this interact with ConversationMemory, compaction, and future session/event persistence?

I'm not proposing replacing or expanding ConversationMemory into a catch-all memory abstraction.

I'm mainly interested in whether a small composition primitive could be useful for applications that need multiple memory strategies while keeping storage and retrieval implementations independent.

Would this fit Rig's design philosophy, or is this better kept entirely outside the core library?