How to process result lists for question answering

[reuschling]

I know this is a nontrivial task, and maybe there is no clear answer. But I am wondering if there is some best practice implementation or support to process result lists in order to answer user questions, like RAG.
Inserting result lists into a prompt comes always with the risk of exceeding the llm context size. There would be potential that Embabel knows the context size and consider it somehow. Methods like context compression are not trivial to realize. Also pagination and incremental processing of a result list comes into my mind, and for sure there exists a plenty other possibilities.
The need of processing result lists is common. I for example have scored Opensearch results of a hybrid search, giving chunks of documents.

Embabel must have some processing of result lists yet, e.g. in the context of the web search tool group. I am not sure if or how much support Embabel gives or can give for result list processing.
Is something automatic like this realistic?
String answer = llm.withResultList(list).withQuestion(query).createAnswer();
Or with some control
String answer = llm.withResultList(list).compressListForQuery().resultListPaginationProcessing().withQuestion(query).createAnswer();

I want to ask for this and looking forward to interesting and fruitful thoughts.

[johnsonr]

Interesting idea. Our RAG API is finally stable and I'm happy with it. Still work to do on result quality, but I think it's a solid basis.

One approach here would be to do this via RAG, adding the results and creating a RagReference. The notion of an ephemeral RagService definitely makes sense. Lucene is an option here.

Another approach might be pagination functions, also exposed as an LlmReference. Where you have the content (which fits in memory) but a standard pagination mechanism surfacing LLM functions. Maybe this is closer to what you have in mind?

How large are the results you have? How many items, and how large is each item? Are they all in memory at once? Is there any way you'd like to search within them? You could also persist them to a temporary file and use FileTools to access them. Please see the code module for the implementation of GitHubRepository.

[reuschling]

Thank you @johnsonr for your answer. In my scenario, the size of the result list necessary to answer questions depends on the question and the structure of the items inside the result list. Technically, I have influence how to index my data, from chunk sizes over appended metadata to the used embedding model, but this mustn't be the case in other use cases in the future.

At the moment, I have thousands of meeting minutes. These minutes include metadata listing the participants and the organization that held the meeting. The room and time of the meeting is also there. The meeting minutes are chunked to a fixed max size. I have a sophisticated hybrid search with OpenSearch, so the result lists are finally scored, the search scales up to any degree, the data is incrementally updated to the current state.

When I search for "What was the last meeting where the term X was mentioned" it is straight forward, the agent creates an according OpenSearch query with search term X, sorted by time. Same for similar questions like "I search for a specific meeting last week where we discussed on topic Y" or "specific meeting at time Z inside room R", etc. The top search result documents should contain the information to answer the question.

Other questions are different, for example if someone asks for 'how many meetings'. There is the chance to create an (OpenSearch) query out of it that counts result documents. This can be the case if the question relies on available metadata: "how many meetings with participant X last month" or "with specific search term Y". For this, a preceding query classification in the agent workflow seems to be feasible, creating the according query and interpreting the result (count) accordingly.

But there are also questions where this doesn't help, where it is necessary to process all result documents/meetings in order to generate the answer : "Create a list of all people that leads a meeting last month", "Show me all topics discussed regarding my home village the last half year". The information is not necessarily inside the metadata, so there are at least two different cases:

1. The information is inside the metadata, no according (e.g OpenSearch/SQL) query can be generated: "What was the average participant count in the room X last year". Preceding query classification to this situation => result list loop (with possible pagination) that calculates the average, without LLM.
2. The information is inside the unstructured text of the meeting minutes "Show me all topics last week". Loop with possible pagination, LLM processing necessary

Of course, for answering specific questions, there is also the possibility to precalculate data or enhancing index/database representations for performance reasons. But let this not be our current topic.

So it is all about getting information out of half-structured data (fulltext+metadata), depending on the user question. My goal is to implement the according agent workflow(s). These "query with necessary steps to answer" couples comes currently into my mind. I have the feeling that there must be a critical mass in order to support sophisticated question answering out of half or even full structured data(bases).

Thus, query classification with according query creation and succeeding result list processing could lead to a generalized approach, at least with some parametrization.

For this, Embabel support for different result list processing techniques would be really helpful, in order not to fall into some basic stuff like exceeding LLM context window sizes.