PPL: Add `json_extract` function #3262

kenrickyap · 2025-01-23T16:33:02Z

Description

Implements json_extract function that would be useful to extract specific objects and scalars from an existing json_object.

Related Issues

Resolves #3211
Follow up issue for #3027

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Kenrick Yap <[email protected]>

Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

….java Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

…nsearch-project-sql into feature/json-valid Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: Andrew Carbonetto <[email protected]>

kenrickyap · 2025-01-23T16:41:07Z

would be ideal to merge:

PPL: Add json_object command #3242
PPL: Add json function and cast(x as json) function #3243
before merging this PR

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

currantw

Reviewed to 0f0b8d4a26fa47d2f9ae190d9050d9d87b2ab37c.

core/src/main/java/org/opensearch/sql/expression/DSL.java

core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java

docs/user/ppl/functions/json.rst

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

Signed-off-by: 14yapkc1 <[email protected]>

currantw

Reviewed to f030057

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

docs/user/ppl/functions/json.rst

Signed-off-by: 14yapkc1 <[email protected]>

Signed-off-by: 14yapkc1 <[email protected]> # Conflicts: # core/src/main/java/org/opensearch/sql/expression/DSL.java # core/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java # core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java # core/src/main/java/org/opensearch/sql/utils/JsonUtils.java # core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java # core/src/test/java/org/opensearch/sql/expression/operator/convert/TypeCastOperatorTest.java # docs/user/ppl/functions/json.rst # integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionsIT.java # ppl/src/main/antlr/OpenSearchPPLLexer.g4 # ppl/src/main/antlr/OpenSearchPPLParser.g4

Signed-off-by: 14yapkc1 <[email protected]> Signed-off-by: Kenrick Yap <[email protected]>

currantw

Reviewed to 2b08007.

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

docs/user/ppl/functions/json.rst

…pensearch-project-sql into feature/json-extract

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

docs/user/ppl/functions/json.rst

Signed-off-by: Kenrick Yap <[email protected]>

currantw

Reviewed to e57fa21.

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

docs/user/ppl/functions/json.rst

…tionsTest.java Co-authored-by: Taylor Curran <[email protected]> Signed-off-by: kenrickyap <[email protected]>

currantw

Reviewed to dfe5d3a. Looks good to me.

currantw

Reviewed to dfe5d3a. Looks good to me.

YANG-DB · 2025-01-30T19:54:22Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

can u plz add license header ?
thanks

Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: kenrickyap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

acarbonetto

a few comments to address

acarbonetto · 2025-02-05T21:59:28Z

core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java

+  private DefaultFunctionResolver jsonExtract() {
+    return define(
+        BuiltinFunctionName.JSON_EXTRACT.getName(),
+        impl(JsonUtils::extractJson, UNDEFINED, STRING, STRING));


this means we only accept a single path
Splunk function takes a variable length of paths: https://docs.splunk.com/Documentation/SCS/current/SearchReference/JSONFunctions#json_extract.28.26lt.3Bjson.26gt.3B.2C_.26lt.3Bpaths.26gt.3B.29

acarbonetto · 2025-02-05T22:02:13Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

+    try {
+      Object results = JsonPath.parse(jsonString).read(jsonPath);
+      return ExprValueUtils.fromObjectValue(results);
+    } catch (PathNotFoundException e) {


Suggested change

} catch (PathNotFoundException e) {

} catch (PathNotFoundException ignored) {

acarbonetto · 2025-02-05T22:02:30Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

+      return ExprValueUtils.fromObjectValue(results);
+    } catch (PathNotFoundException e) {
+      return LITERAL_NULL;
+    } catch (InvalidPathException e) {


please don't reuse variable names:

Suggested change

} catch (InvalidPathException e) {

} catch (InvalidPathException invalidPathException) {

acarbonetto · 2025-02-05T22:02:41Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

+    } catch (InvalidPathException e) {
+      final String errorFormat = "JSON path '%s' is not valid. Error details: %s";
+      throw new SemanticCheckException(String.format(errorFormat, path, e.getMessage()), e);
+    } catch (InvalidJsonException e) {


please don't reuse variable names:

Suggested change

} catch (InvalidJsonException e) {

} catch (InvalidJsonException invalidJsonException) {

acarbonetto · 2025-02-05T22:10:33Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+  @Test
+  void json_extract_search() {
+    ExprValue expected = new ExprIntegerValue(1);
+    execute_extract_json(expected, "{\"a\":1}", "$.a");


let's add a test for each scalar type: boolean, double, string, integer (already have) and null (not exactly the same case as the test case below).

acarbonetto · 2025-02-05T22:10:58Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+
+  @Test
+  void json_extract_search_arrays_out_of_bound() {
+    execute_extract_json(LITERAL_NULL, "{\"a\":[1,2,3]}", "$.a[4]");


move to `json_extract_returns_null

acarbonetto · 2025-02-05T22:12:27Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+    // null json
+    assertEquals(
+        LITERAL_NULL,
+        DSL.jsonExtract(DSL.literal(LITERAL_NULL), DSL.literal(new ExprStringValue("$.a")))


Suggested change

DSL.jsonExtract(DSL.literal(LITERAL_NULL), DSL.literal(new ExprStringValue("$.a")))

DSL.jsonExtract(DSL.literal(LITERAL_NULL), DSL.literal("$.a"))

acarbonetto · 2025-02-05T22:12:37Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+    // missing json
+    assertEquals(
+        LITERAL_MISSING,
+        DSL.jsonExtract(DSL.literal(LITERAL_MISSING), DSL.literal(new ExprStringValue("$.a")))


Suggested change

DSL.jsonExtract(DSL.literal(LITERAL_MISSING), DSL.literal(new ExprStringValue("$.a")))

DSL.jsonExtract(DSL.literal(LITERAL_MISSING), DSL.literal("$.a"))

same for rest of file...

14yapkc1 and others added 30 commits January 3, 2025 10:54

added implementation

70152eb

Signed-off-by: Kenrick Yap <[email protected]>

added doctest, integ-tests, and unit tests

76c3995

Signed-off-by: Kenrick Yap <[email protected]>

addressed PR comments

ce2c551

Signed-off-by: Kenrick Yap <[email protected]>

fixed unit tests

ad1bde3

Signed-off-by: Kenrick Yap <[email protected]>

addressed pr comments

ccf47a2

Signed-off-by: Kenrick Yap <[email protected]>

addressed PR comments

acc76a0

Signed-off-by: Kenrick Yap <[email protected]>

removed unused dependencies

519c6f2

Signed-off-by: Kenrick Yap <[email protected]>

linting

2e319fe

Signed-off-by: Kenrick Yap <[email protected]>

addressed pr comment and rolling back disabled test case

ee0820d

Signed-off-by: Kenrick Yap <[email protected]>

Merge branch 'main' into feature/json-valid

d44fc5a

Signed-off-by: Kenrick Yap <[email protected]>

removed disabled import

3407d4a

Signed-off-by: Kenrick Yap <[email protected]>

Update docs/user/ppl/functions/json.rst

7ef6cc9

Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

Update integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionIT…

e5e90ac

….java Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

nit

2187a5a

Signed-off-by: Kenrick Yap <[email protected]>

Merge branch 'feature/json-valid' of https://github.com/Bit-Quill/ope…

5e1e488

…nsearch-project-sql into feature/json-valid Signed-off-by: Kenrick Yap <[email protected]>

fixed integ test

3512b33

Signed-off-by: Kenrick Yap <[email protected]>

change text type to keyword

9fea606

Signed-off-by: Kenrick Yap <[email protected]>

addressed PR comments

fbc54bc

Signed-off-by: Kenrick Yap <[email protected]>

fix doc-test

31ad2a4

Signed-off-by: Kenrick Yap <[email protected]>

added null test

2b2a8f3

Signed-off-by: Kenrick Yap <[email protected]>

Merge branch 'main' into feature/json-valid

dc96563

SQL: adding error case unit tests for json_valid

1913bfe

Signed-off-by: Andrew Carbonetto <[email protected]>

json_valid: null and missing should return false

67d979d

Signed-off-by: Andrew Carbonetto <[email protected]>

PPL: Add json and cast to json functions

aa6b723

Signed-off-by: Andrew Carbonetto <[email protected]>

PPL: Update json cast for review

4c99235

Signed-off-by: Andrew Carbonetto <[email protected]>

Fix testes

9ccde7f

Signed-off-by: Andrew Carbonetto <[email protected]>

spotless

4306bf3

Signed-off-by: Andrew Carbonetto <[email protected]>

Fix tests

613137b

Signed-off-by: Andrew Carbonetto <[email protected]>

SPOTLESS

ab28872

Signed-off-by: Andrew Carbonetto <[email protected]>

Clean up for merge

3ec16e0

Signed-off-by: Andrew Carbonetto <[email protected]>

normanj-bitquill reviewed Jan 23, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java Outdated Show resolved Hide resolved

currantw reviewed Jan 23, 2025

View reviewed changes

addessed comments

f030057

Signed-off-by: 14yapkc1 <[email protected]>

currantw reviewed Jan 27, 2025

View reviewed changes

kenrickyap added 3 commits January 28, 2025 11:37

added addition edge cases for unit tests

2b08007

Signed-off-by: 14yapkc1 <[email protected]>

Merge branch 'feature/acarbo_json_cast_ppl' into feature/json-extract

6bd2f40

Signed-off-by: 14yapkc1 <[email protected]> Signed-off-by: Kenrick Yap <[email protected]>

currantw reviewed Jan 28, 2025

View reviewed changes

Merge branch 'feature/json-extract' of https://github.com/Bit-Quill/o…

0b9e9e4

…pensearch-project-sql into feature/json-extract

currantw reviewed Jan 28, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java Outdated Show resolved Hide resolved

andy-k-improving mentioned this pull request Jan 28, 2025

Feature: PPL - json_set function #3271

Open

7 tasks

currantw reviewed Jan 28, 2025

View reviewed changes

docs/user/ppl/functions/json.rst Outdated Show resolved Hide resolved

currantw reviewed Jan 28, 2025

View reviewed changes

docs/user/ppl/functions/json.rst Outdated Show resolved Hide resolved

kenrickyap and others added 2 commits January 29, 2025 11:11

addressed PR comments

6678be4

Signed-off-by: Kenrick Yap <[email protected]>

fix code coverage

e57fa21

Signed-off-by: Kenrick Yap <[email protected]>

currantw reviewed Jan 30, 2025

View reviewed changes

Update core/src/test/java/org/opensearch/sql/expression/json/JsonFunc…

112be65

…tionsTest.java Co-authored-by: Taylor Curran <[email protected]> Signed-off-by: kenrickyap <[email protected]>

currantw approved these changes Jan 30, 2025

View reviewed changes

YANG-DB reviewed Jan 30, 2025

View reviewed changes

YANG-DB previously approved these changes Jan 30, 2025

View reviewed changes

kenrickyap added 2 commits January 31, 2025 09:54

address comments

306ac97

Signed-off-by: Kenrick Yap <[email protected]>

fix build error

75e9cc3

Signed-off-by: Kenrick Yap <[email protected]>

kenrickyap force-pushed the feature/json-extract branch from dfe5d3a to 75e9cc3 Compare January 31, 2025 14:57

Merge branch 'main' into feature/json-extract

77827bb

Signed-off-by: kenrickyap <[email protected]>

kenrickyap dismissed YANG-DB’s stale review via 77827bb January 31, 2025 15:00

add header

80f44e2

Signed-off-by: Kenrick Yap <[email protected]>

YANG-DB approved these changes Feb 3, 2025

View reviewed changes

acarbonetto reviewed Feb 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PPL: Add `json_extract` function #3262

PPL: Add `json_extract` function #3262

kenrickyap commented Jan 23, 2025 •

edited

Loading

kenrickyap commented Jan 23, 2025

currantw left a comment

currantw left a comment

currantw left a comment

currantw left a comment

currantw left a comment

currantw left a comment

YANG-DB Jan 30, 2025

kenrickyap Jan 31, 2025

acarbonetto left a comment

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

acarbonetto Feb 5, 2025

	} catch (PathNotFoundException e) {
	} catch (PathNotFoundException ignored) {

	} catch (InvalidPathException e) {
	} catch (InvalidPathException invalidPathException) {

	} catch (InvalidJsonException e) {
	} catch (InvalidJsonException invalidJsonException) {

	DSL.jsonExtract(DSL.literal(LITERAL_NULL), DSL.literal(new ExprStringValue("$.a")))
	DSL.jsonExtract(DSL.literal(LITERAL_NULL), DSL.literal("$.a"))

	DSL.jsonExtract(DSL.literal(LITERAL_MISSING), DSL.literal(new ExprStringValue("$.a")))
	DSL.jsonExtract(DSL.literal(LITERAL_MISSING), DSL.literal("$.a"))

PPL: Add json_extract function #3262

Are you sure you want to change the base?

PPL: Add json_extract function #3262

Conversation

kenrickyap commented Jan 23, 2025 • edited Loading

Description

Related Issues

Check List

kenrickyap commented Jan 23, 2025

currantw left a comment

Choose a reason for hiding this comment

currantw left a comment

Choose a reason for hiding this comment

currantw left a comment

Choose a reason for hiding this comment

currantw left a comment

Choose a reason for hiding this comment

currantw left a comment

Choose a reason for hiding this comment

currantw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acarbonetto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PPL: Add `json_extract` function #3262

PPL: Add `json_extract` function #3262

kenrickyap commented Jan 23, 2025 •

edited

Loading