Better Memory Management

[bmc0923]

We need better memory management! I'm constantly hitting rate limits and just doing standard queries. I've reduced the context window down to 5000, and still happening. Not sure the best way to manage this. I'm going to create a custom memory class based off neuron and simply make it based of session ID, and remembers the last 8 messages. Thoughts?

[ilvalerione]

Rate limit has nothing to do with context window. Rate limit depends by your tier on the AI provider you choose.

[bmc0923]

I don't believe this is correct. When using the API directly, you don't need to send instructions again as part of the tool call. However, that seems to be what is happening here. I looked back at my previous API direct logs, and it was always executing correctly with system instructions only listed once, suggesting there is a mistake in the code with Neuron AI. I'm going to explore this further and report back.

[ilvalerione]

I have to say I never tested providing the tool call result to the LLM without sending back all the context. It should be tested or even asked to the providers because it will improve the efficiency and speed of the tool orchestration process. If you have any update let us know! 👍

[ilvalerione]

This is what Claude from topic says about itself.

[bmc0923]

interesting - maybe it differs between llm providers then. I've mainly only used OpenAI. Good to know.

[bmc0923]

sorry you're right - OpenAI suggests the same thing. Nevertheless, something is wrong still in that it's recording the system message twice. I'll come back on this. It was a while ago I used the API direct.