Is your feature request related to a problem?
The current predicate pushdown implementation in OpenSearchRelation.translateFilter handles basic comparison operators but leaves some filters unhandled, causing Spark to process them in memory after fetching data from OpenSearch.
Notably, In filters are explicitly marked as unhandled in unhandledFilters, meaning all values matching an IN clause are fetched and filtered on the Spark side.
What solution would you like?
Extend translateFilter to support additional filter types:
| Spark Filter |
OpenSearch Query |
In(field, values) |
terms query |
StringContains with wildcards |
wildcard query |
| Regex patterns |
regexp query |
What alternatives have you considered?
Leaving these filters to Spark. This works correctly but transfers unnecessary data over the network.
Do you have any additional context?
None.
Is your feature request related to a problem?
The current predicate pushdown implementation in
OpenSearchRelation.translateFilterhandles basic comparison operators but leaves some filters unhandled, causing Spark to process them in memory after fetching data from OpenSearch.Notably,
Infilters are explicitly marked as unhandled inunhandledFilters, meaning all values matching anINclause are fetched and filtered on the Spark side.What solution would you like?
Extend
translateFilterto support additional filter types:In(field, values)termsqueryStringContainswith wildcardswildcardqueryregexpqueryWhat alternatives have you considered?
Leaving these filters to Spark. This works correctly but transfers unnecessary data over the network.
Do you have any additional context?
None.