[Feature] Allow developer to save and restore chat history more cleanly

[JohnDuncanScott]

Feature Request

Loading a model is expensive. Therefore, being able to use the same model instance for use with all users is desirable. The current API is seems to be designed around 1 user interacting with the model and so the API is a little clunky for swapping message history in and out.

At the moment, for each user session you need to do something like:

    def generate(
        self, session_id: str, system_prompt: str, prompt_template: str, user_input: str
    ) -> str:
        output: Final[str]

        with self.model.chat_session(system_prompt, prompt_template):
            if session_id in self.id_to_chat_messages:
                # Load chat messages
                chat_messages = self.id_to_chat_messages[session_id]
                self.model.current_chat_session = chat_messages

            # Generate output
            output = self.__generate(user_input)

            # Save chat messages
            self.id_to_chat_messages[session_id] = self.model.current_chat_session

        return output

The above works and history is successfully swapped in and out (the model seems to behave correctly at least). However, it feels wrong to have to do this and I'm not sure what consequences it has on the backend given I imagine there's some optimisations it does with a long message history.

Having a model instance per user does not scale due the resources required, so having a cleaner API to swap message histories in and out would be super useful.

[JohnDuncanScott]

Actually I've just been trying much longer system prompts with the above technique. It looks like whatever optimisation is being done on the backend causes the chat sessions to bleed into one another, despite using the "with" statement and replacing the chat messages with those from a different session

[JohnDuncanScott]

I think I've spotted the latter issue:
# check if there is only one message, i.e. system prompt: generate_kwargs["reset_context"] = len(self.current_chat_session) == 1

So it looks like because I'm replacing the chat session with a different one, the context doesn't get reset, so although the chat history that gets sent to the model is different, the context is old and confuses things on the backend with the kv caching that happens. I can't see any way around this at the moment

[JohnDuncanScott]

I've tried sending an empty message before loading the history to trigger a context clear, but this causes context to stop working completely, it doesn't rebuild it. It looks like there are no easy workarounds. Any suggestions would be appreciated and it would be good to know if you think this feature is useful / possible.

[ebronasco]

I think this feature is absolutely necessary. Last time I checked, it should be implemented on the CPP side, as was attemped in #1161.

[cebtenzzre]

#2061 is relevant here. Right now fakeReply is only used by the node.js bindings, but it could be used by the Python bindings to restore a previous chat session, similar to how the chat UI does it.

[adbenitez]

I am creating a public chatbot for some chatting platform and I am blocked from using gpt4all exactly for this reason, I need each user to have their own chat history / session, that should be loaded before processing each new request/prompt from user

[adbenitez]

#2061 is relevant here. Right now fakeReply is only used by the node.js bindings, but it could be used by the Python bindings to restore a previous chat session, similar to how the chat UI does it.

@cebtenzzre great! as a temporary solution patched LLModel.prompt_model and GPT4All.generate to add/use fake_reply param and it works like a charm! looking forward to see it as part of the bindings

(NOTE: but it is rather slow to load all Q&A one by one this way)

[stanleybenjamin]

@adbenitez please any sample code I am currently in this situation

[stanleybenjamin]

Although quite slow, I was able to use #2935 from @adbenitez to modify my files. I did this, and it works perfectly, although quite slow.

with model.chat_session(system_prompt="You're an helpful AI assistant"):
    #the fake reply is used to provide the model with it own response as I understand it, 
    #and i/o is a bit faster since it does not respond here
    print(model.generate("Tell me about france", fake_reply="Yes France is a country")) #load previous chat
    print(model.generate("What is the capital City?"))
    print("\n")

Note: You may need to do this to reset memory between for example different users. so they don't share memory

    # Reset the model's internal history completely
    model._history = []
    model.history.append({"role": "system", "content": "You're a helpful assistant."})