Support wildcard for replace command #4698

ahkcs · 2025-10-29T20:59:35Z

Description

Support wildcard for replace command

Related Issues

Resolves [RFC] Add replace Command to PPL (Calcite Engine) #3975

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Swiddis · 2025-10-29T23:05:05Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardReplaceUtils.java

+/**
+ * Utility for wildcard-based string replacement in PPL replace command.
+ *
+ * <p>Supports SPL-style wildcard matching where '*' matches zero or more characters. Captured


issue: No way to escape asterisks

That's a great catch! For now I am marking it as a limitation in the docs since the use case for literal asterisks is rare. Do you think we should implement escape instead or we can keep it as a limitation?

We can keep it as a limitation for now, but I'm a bit concerned about breaking a potential existing usage with no workaround. I don't know if who's using asterisks today, the number is probably low, but it'd be surprise me if it was 0.

Especially if we're already building the final pattern with regex split (L76), we might be able to just make the split have negative lookahead with Pattern.split("(?!\\\\)\\*")

Feel free to skip this if it's more complicated than that

Updated to support escaping asterisks

Swiddis · 2025-10-29T23:08:27Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardReplaceUtils.java

+    }
+
+    // Fast path: no wildcards = literal replacement
+    if (!pattern.contains("*")) {


question: What are the perf implications of reprocessing the pattern from scratch on every record?

Ideally it'd be faster to compile the pattern once and reuse it to reduce branching in the inner loop. But I don't know if it's a measurable difference.

That's a good catch! Added LRU pattern cache to compile once per unique pattern string instead of compiling on every row

Swiddis · 2025-10-29T23:12:48Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardReplaceUtils.java

+   * @param pattern The pattern with wildcards
+   * @return List of captured strings (one per wildcard), or null if no match
+   */
+  public static List<String> matchAndCapture(String input, String pattern) {


suggestion (if-minor): Compiling an equivalent Regex matcher might be faster and simpler.

That's a great suggestion! Refactored to use regex matching

ykmr1224 · 2025-10-30T16:22:53Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardReplaceUtils.java

+  /** Match pattern against input and capture wildcard portions. */
+  public static List<String> matchAndCapture(String input, String pattern) {
+    Pattern compiledPattern =
+        PATTERN_CACHE.computeIfAbsent(pattern, WildcardReplaceUtils::compileWildcardPattern);


What happens if pattern contains regex? I think we need to escape regex.

He's using Pattern.quote to escape the literal parts on line 80.

ykmr1224 · 2025-10-30T16:25:41Z

core/src/main/java/org/opensearch/sql/expression/function/udf/WildcardReplaceFunctionImpl.java

+import org.opensearch.sql.expression.function.UDFOperandMetadata;
+
+/** UDF for wildcard-based string replacement in PPL replace command. */
+public class WildcardReplaceFunctionImpl extends ImplementorUDF {


If we use regex to execute the replace, should we rather utilize existing function for regex replacement? In that case, we just need to compile pattern during planning, and don't need caching during execution.

I think that is a great improvement! I have updated, now wildcard patterns convert to regex at planning time. Also updated to use Clacite's REGEXP_REPLACE_3 operator instead of custom UDF, removed WildcardReplaceFunctionImpl, WildcardReplaceUtils, and runtime caching

LantaoJin · 2025-11-03T07:02:39Z

The bwc-tests failed due to no space left on device which can be fixed by #4716 , no related to this PR but better to merge upstream (after 4716 merged) then rerun.

ykmr1224 · 2025-11-04T00:23:15Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardUtils.java

+   * @param wildcardPattern wildcard pattern with '*' and escape sequences (\*, \\)
+   * @return regex pattern with capture groups
+   */
+  public static String convertWildcardPatternToRegex(String wildcardPattern) {


Let's add comprehensive unit tests for the methods in WildcardUtils.

Added unit tests in WildcardUtilsTest

ykmr1224 · 2025-11-04T00:23:47Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardUtils.java

+   * @param pattern wildcard pattern with escape sequences
+   * @return array of pattern parts
+   */
+  public static String[] splitWildcards(String pattern) {


Should it be private?

Changed to private

ykmr1224 · 2025-11-04T00:24:09Z

core/src/main/java/org/opensearch/sql/calcite/utils/WildcardUtils.java

+   * @param str string to count wildcards in
+   * @return number of unescaped wildcards
+   */
+  public static int countWildcards(String str) {


Changed to private

Signed-off-by: Kai Huang <[email protected]>

Signed-off-by: Kai Huang <[email protected]> # Conflicts: # integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java # Conflicts: # integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java

Signed-off-by: Kai Huang <[email protected]>

opensearch-trigger-bot · 2025-11-04T19:01:03Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4698-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f97adb00227d0b209becc4cea72501af221533e6
# Push it to GitHub
git push --set-upstream origin backport/backport-4698-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4698-to-2.19-dev.

(cherry picked from commit f97adb0) Signed-off-by: Kai Huang <[email protected]>

ahkcs marked this pull request as ready for review October 29, 2025 22:59

Swiddis added the enhancement New feature or request label Oct 29, 2025

Swiddis reviewed Oct 29, 2025

View reviewed changes

ykmr1224 reviewed Oct 30, 2025

View reviewed changes

ahkcs force-pushed the replace_wildcard branch from fd2f6c3 to 125d868 Compare October 30, 2025 21:14

ahkcs requested review from Swiddis and ykmr1224 October 30, 2025 22:59

ahkcs force-pushed the replace_wildcard branch from ecab069 to c2cfc01 Compare November 3, 2025 19:30

ykmr1224 reviewed Nov 4, 2025

View reviewed changes

ahkcs added 12 commits November 4, 2025 09:37

Support wildcard for replace command

31302d9

Signed-off-by: Kai Huang <[email protected]>

fixes

ab74440

Signed-off-by: Kai Huang <[email protected]>

fix

355751f

Signed-off-by: Kai Huang <[email protected]>

add limitation doc

b080c8a

Signed-off-by: Kai Huang <[email protected]>

refactor to use regex matching

3976883

Signed-off-by: Kai Huang <[email protected]>

Add Cache for perf

e4b28aa

Signed-off-by: Kai Huang <[email protected]>

handle asterisks

4724fd2

Signed-off-by: Kai Huang <[email protected]>

refactor the implementation

8ef3f1f

Signed-off-by: Kai Huang <[email protected]>

add ExplainIT

ffbe1d1

Signed-off-by: Kai Huang <[email protected]> # Conflicts: # integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java # Conflicts: # integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java

refactoring and trimming

0be5611

Signed-off-by: Kai Huang <[email protected]>

doc

e60720e

Signed-off-by: Kai Huang <[email protected]>

add tests

1ddce43

Signed-off-by: Kai Huang <[email protected]>

ahkcs force-pushed the replace_wildcard branch from c2cfc01 to 1ddce43 Compare November 4, 2025 17:52

Swiddis approved these changes Nov 4, 2025

View reviewed changes

ykmr1224 approved these changes Nov 4, 2025

View reviewed changes

Swiddis merged commit f97adb0 into opensearch-project:main Nov 4, 2025
39 of 57 checks passed

Swiddis added the backport 2.19-dev label Nov 4, 2025

opensearch-trigger-bot bot added the backport-failed label Nov 4, 2025

expani pushed a commit to vinaykpud/sql that referenced this pull request Nov 4, 2025

Support wildcard for replace command (opensearch-project#4698)

f298a11

ahkcs added a commit to ahkcs/sql that referenced this pull request Nov 4, 2025

Support wildcard for replace command (opensearch-project#4698)

361d994

(cherry picked from commit f97adb0) Signed-off-by: Kai Huang <[email protected]>

LantaoJin pushed a commit that referenced this pull request Nov 5, 2025

Support wildcard for replace command (#4698) (#4741)

8aa0ffc

(cherry picked from commit f97adb0) Signed-off-by: Kai Huang <[email protected]>

LantaoJin added the backport-manually Filed a PR to backport manually. label Nov 5, 2025

Support wildcard for replace command #4698

Support wildcard for replace command #4698

Uh oh!

Conversation

ahkcs commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin commented Nov 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ahkcs commented Oct 29, 2025 •

edited

Loading

Swiddis Oct 30, 2025 •

edited

Loading

Swiddis Oct 29, 2025 •

edited

Loading