feat!: expose refresh parameter in `OpenSearchDocumentStore` #2623

GunaPalanivel · 2025-12-18T16:33:19Z

Related Issues

Part of #2605

Description

This PR exposes the refresh parameter to relevant methods in OpenSearchDocumentStore, allowing users to control when index changes become visible to search operations.

Motivation

OpenSearch is a "near real-time" search engine where changes are not immediately visible after write/update/delete operations. Previously, users had to use time.sleep() workarounds to ensure consistency. This change exposes the underlying refresh parameter, giving users explicit control over this behavior.

Changes

New RefreshType parameter added to the following methods:

Method	Default Value
`write_documents` / `write_documents_async`	`"wait_for"`
`delete_documents` / `delete_documents_async`	`"wait_for"`
`delete_all_documents` / `delete_all_documents_async`	`True`
`delete_by_filter` / `delete_by_filter_async`	`"wait_for"`
`update_by_filter` / `update_by_filter_async`	`"wait_for"`

Parameter values:

True: Force refresh immediately after the operation
False: Do not refresh (better performance for bulk operations)
"wait_for": Wait for the next refresh cycle (default, ensures read-your-writes consistency)

Test Updates

Updated integration tests to use refresh=True instead of time.sleep(), making tests more reliable and faster.

Additional Fixes

Fixed delete_all_documents_async to set wait_for_completion=True, ensuring the async delete operation completes before returning

How was this tested?

Existing integration tests updated to use the new refresh parameter
All tests pass with the new implementation

Add configurable refresh parameter to write, delete, and update methods in OpenSearchDocumentStore. This allows users to control when index changes become visible to search operations, enabling read-your-writes consistency without relying on time.sleep() workarounds. Methods updated: - write_documents / write_documents_async - delete_documents / delete_documents_async - delete_all_documents / delete_all_documents_async - delete_by_filter / delete_by_filter_async - update_by_filter / update_by_filter_async The refresh parameter accepts: - True: Force immediate refresh - False: No refresh (best for bulk performance) - 'wait_for': Wait for next refresh cycle (default) Additional fix: - Fixed delete_all_documents_async to set wait_for_completion=True, ensuring the async delete operation completes before returning Closes deepset-ai#2065

anakin87 · 2025-12-18T16:40:10Z

Thanks for opening a separate PR. Let's put this on hold and first agree on #2622

anakin87 · 2025-12-19T15:23:25Z

@GunaPalanivel when you have time, feel free to adapt this PR based on #2622 🙂

GunaPalanivel · 2025-12-19T15:25:11Z

@anakin87

I'm working on it 🙂

…ings to match ES pattern (deepset-ai#2623)

GunaPalanivel · 2025-12-19T18:34:13Z

@anakin87 Based on #2622 , I've adapted this PR for OpenSearch:

Main changes:

Split refresh parameter types: Literal[True, False, "wait_for"] for write/delete methods, bool for delete_by_query/update_by_query operations (since they don't support "wait_for")
Set refresh="wait_for" as default for write_documents and delete_documents
Kept refresh=True for delete_all_documents and filter-based operations
Added proper OpenSearch API doc links to all refresh parameters
Updated tests to use refresh=True instead of time.sleep() hacks

Let me know if I missed anything or got something wrong, happy to fix! 🙂

GunaPalanivel · 2026-01-05T04:48:54Z

Hi @anakin87, just a gentle follow-up on this PR when you get a moment.

All checks are passing, and I’m happy to make any changes if needed.

A review would really help me move forward and pick up the next issues.

vblagoje · 2026-01-05T14:17:54Z

@GunaPalanivel coordinate with @davidsbatista as he's doing work on OpenSearchDocumentStore, there is a merge conflict most likely due to that

davidsbatista · 2026-01-05T14:34:25Z

@GunaPalanivel, thanks for the contribution! Please sync with the latest main and fix the conflicts; they should be easy. In any case, let me know if you need help.

anakin87 · 2026-01-05T14:37:13Z

I'll review it soon (once conflicts are fixed).

…ents_async

integrations/opensearch/src/haystack_integrations/document_stores/opensearch/document_store.py

- Sync calls: always wait_for_completion=True (to get result dict with deleted count) - Async calls: wait_for_completion=True only if refresh=True - Logic: wait_for_completion = (not is_async) or refresh

@davidsbatista

Align with ElasticSearch behavior as suggested by @davidsbatista: - Set wait_for_completion = not is_async (True only for sync calls) - This ensures sync calls get the result dict for logging deleted count - Async calls do not wait, improving performance - Centralizes simple logic in _prepare_delete_all_request

Both sync and async delete_all_documents need to wait for completion so documents are actually deleted before the function returns

The is_async parameter is no longer needed since wait_for_completion is always True for delete_all_documents operations

anakin87

Looks OK to me, aligned with Elasticsearch.

@davidsbatista could you take a final look?

davidsbatista

LGTM! Thanks again @GunaPalanivel for one more contribution! We appreciated it!

anakin87 · 2026-01-07T13:49:22Z

@davidsbatista in case you merge and release a new version, let's use 5.0.0 since this is technically a breaking change.

GunaPalanivel requested a review from a team as a code owner December 18, 2025 16:33

GunaPalanivel requested review from vblagoje and removed request for a team December 18, 2025 16:33

github-actions bot added integration:opensearch type:documentation Improvements or additions to documentation labels Dec 18, 2025

GunaPalanivel mentioned this pull request Dec 18, 2025

feat: expose refresh parameter in ElasticsearchDocumentStore and OpenSearchDocumentStore #2613

Closed

3 tasks

GunaPalanivel added 4 commits December 19, 2025 20:59

Merge branch 'main' into feature/opensearch-expose-refresh-param

ef4d6f6

opensearch: expose refresh param — update types, defaults, and docstr…

a265738

…ings to match ES pattern (deepset-ai#2623)

opensearch: wrap long signatures to satisfy fmt-check (E501)

94be37c

style(opensearch): format with ruff

6a80680

anakin87 self-requested a review January 5, 2026 14:36

GunaPalanivel added 3 commits January 5, 2026 20:30

merge main: add async init support

c7f3ef1

fix: pass refresh param to _prepare_bulk_write_request in write_docum…

19dec4e

…ents_async

style: wrap long line in write_documents_async (E501)

a1171e6

anakin87 reviewed Jan 5, 2026

View reviewed changes

integrations/opensearch/src/haystack_integrations/document_stores/opensearch/document_store.py Outdated Show resolved Hide resolved

GunaPalanivel added 2 commits January 6, 2026 11:23

refactor: move wait_for_completion logic to _prepare_delete_all_reque

cf7d88b

fix: correct wait_for_completion logic in _prepare_delete_all_request

293c0de

- Sync calls: always wait_for_completion=True (to get result dict with deleted count) - Async calls: wait_for_completion=True only if refresh=True - Logic: wait_for_completion = (not is_async) or refresh

davidsbatista changed the title ~~feat: expose refresh parameter in OpenSearchDocumentStore~~ feat: expose refresh parameter in OpenSearchDocumentStore Jan 6, 2026

GunaPalanivel added 3 commits January 6, 2026 21:21

fix: set wait_for_completion=True always in delete_all_documents

abe2ce4

Both sync and async delete_all_documents need to wait for completion so documents are actually deleted before the function returns

style: remove unused is_async parameter from _prepare_delete_all_request

b75170f

The is_async parameter is no longer needed since wait_for_completion is always True for delete_all_documents operations

GunaPalanivel and others added 2 commits January 6, 2026 22:35

style: format with ruff

74f8bd3

Merge branch 'main' into feature/opensearch-expose-refresh-param

8ed4e40

anakin87 reviewed Jan 7, 2026

View reviewed changes

Merge branch 'main' into feature/opensearch-expose-refresh-param

5c79d18

davidsbatista approved these changes Jan 7, 2026

View reviewed changes

anakin87 changed the title ~~feat: expose refresh parameter in OpenSearchDocumentStore~~ feat!: expose refresh parameter in OpenSearchDocumentStore Jan 7, 2026

davidsbatista merged commit c36fbfa into deepset-ai:main Jan 7, 2026
6 checks passed

feat!: expose refresh parameter in OpenSearchDocumentStore #2623

feat!: expose refresh parameter in OpenSearchDocumentStore #2623

Uh oh!

Conversation

GunaPalanivel commented Dec 18, 2025 • edited by anakin87 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues

Description

Motivation

Changes

Test Updates

Additional Fixes

How was this tested?

Uh oh!

anakin87 commented Dec 18, 2025

Uh oh!

anakin87 commented Dec 19, 2025

Uh oh!

GunaPalanivel commented Dec 19, 2025

Uh oh!

GunaPalanivel commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GunaPalanivel commented Jan 5, 2026

Uh oh!

vblagoje commented Jan 5, 2026

Uh oh!

davidsbatista commented Jan 5, 2026

Uh oh!

anakin87 commented Jan 5, 2026

Uh oh!

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

davidsbatista left a comment

Choose a reason for hiding this comment

Uh oh!

anakin87 commented Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat!: expose refresh parameter in `OpenSearchDocumentStore` #2623

feat!: expose refresh parameter in `OpenSearchDocumentStore` #2623

GunaPalanivel commented Dec 18, 2025 •

edited by anakin87

Loading

GunaPalanivel commented Dec 19, 2025 •

edited

Loading