add json functions #3559

xinyual · 2025-04-17T07:03:30Z

Description

Here we add json related functions to align with spark

Here is the argument and description of functions

Functions	argument	description	return type	implementation
json	json(value: ANY)	return value if value is a json, otherwise, return null, e.g. json("123") = "123", json(json_object("1", "2")) = {"1": "2"}, json(1) = null	the first argument type	implement by ourselves
json_valid	json_valid(value: ANY)	return true if value is a json, otherwise, return false, e.g. json("123") = true, json(json_object("1", "2")) = true, json(1) = false	boolean	use calcite SqlStdOperatorTable.IS_JSON_VALUE
json_object	json_object(key1: String, value1: ANY, key2: string, value2 ....)	create an object using key value pairs, the key must be string	string	use calcite SqlStdOperatorTable.JSON_OBJECT
json_array	json_array(value1: ANY, value2: ANY, ...)	create an array using values	string	use SqlStdOperatorTable.JSON_ARRAY
json_array_length (this align with spark since splunk doesn't have this function)	json_array_length(value: STRING)	parse the string to json array and return size, if can't be transferred, return null	integer	implement by ourselves
json_extract	json_extract(target: ANY, path1: STRING, path2:STRING...)	it first transfer target to json, then extract value using paths. If only one path, return the value, otherwise, return the list of values. If one path cannot find value, return null as the result for this path. The path use "{}" to represent index for array, for e.g., demo = {"a": [{"b": 1}, {"b": 2}]}, json_extract(demo, "a{0}.b") = 1. "{}" means "{*}"	if one path, we return string, otherwise, return list of string	use calcite JsonFunctions::json_value, modify path to align with splunk
json_delete	json_delete(target: ANY, path1: STRING, path2:STRING...)	it first transfer target to json, then delete value using paths. return the object after deleting. If a path cannot find any value, skip it.	string. align with splunk	use calcite JsonFunctions::json_remove, modify path to align with splunk
json_set	json_set(target: ANY, path1: STRING, value1: ANY, path2:STRING, value2:ANY...)	it first transfer target to json, then set value to target path. The value can be set only when the parent node of target path is an object. Otherwise, skip it. For example, demo = {"a": [{"b": 1}, {"b": 2}]}, json_set(demo, "a{0}.b", 3) = {"a": [{"b": 3}, {"b": 2}]}. json_set(demo, "a{0}.b.d", 3) = {"a": [{"b": 1}, {"b": 2}]}.	string.align with splunk	use calcite JsonFunctions::json_set, modify path to align with splunk
json_append	json_append(target: ANY, path1: STRING, value1: ANY, path2:STRING, value2:ANY...)	it first transfer target to json, then append value to target path if the path points to an array. Otherwise, skip it. For example, demo = {"a": [{"b": 1}, {"b": 2}]}, json_append(demo, "a", 3) = {"a": [{"b": 1}, {"b": 2}, 3]}. json_append(demo, "a{0}.b", 3) = {"a": [{"b": 1}, {"b": 2}]}.	string. align with splunk	use calcite JsonFunctions::json_insert, modify path to align with splunk
json_extend	json_extend(target: ANY, path1: STRING, value1: ANY, path2:STRING, value2:ANY...)	it first transfer target to json, then extend value to target path if the path points to an array. Otherwise, skip it. For example, demo = {"a": [{"b": 1}, {"b": 2}]}, json_extend(demo, "a", json_array(1, 2)) = {"a": [{"b": 1}, {"b": 2}, 1, 2]}. json_extend(demo, "a{0}.b", 3) = {"a": [{"b": 1}, {"b": 2}]}.	string. align with splunk	use calcite JsonFunctions::json_insert, modify path to align with splunk
json_keys	json_keys(target: ANY)	it first transfer target to json object, then give the keys, otherwise, return null	string	use calcite JsonFunctions.jsonKeys

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#3565

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: xinyual <[email protected]>

LantaoJin · 2025-04-22T02:19:57Z

Why we didn't leverage the Calcite builtin JSON functions, for instance SqlStdOperatorTable.JSON_ARRAY

xinyual · 2025-04-22T03:23:40Z

Why we didn't leverage the Calcite builtin JSON functions, for instance SqlStdOperatorTable.JSON_ARRAY

I have listed the functions above and try to reuse calcite code

Signed-off-by: xinyual <[email protected]>

penghuo · 2025-05-22T20:57:48Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteJsonFunctionsIT.java

 import org.opensearch.sql.ppl.JsonFunctionsIT;

+@Ignore


why ignore IT?

Previous V2 json return the undefined type while we return a string instead.

docs/user/ppl/functions/json.rst

penghuo · 2025-05-22T21:13:56Z

docs/user/ppl/functions/json.rst

+
+Usage: `json_array_length(value)` parse the string to json array and return size, if can't be parsed, return null
+
+Argument type: value: STRING


Argument type: A JSON array

Already update

penghuo · 2025-05-22T21:14:51Z

docs/user/ppl/functions/json.rst

+Description
+>>>>>>>>>>>
+
+Usage: `json_array_length(value)` parse the string to json array and return size, if can't be parsed, return null


NULL is returned in case of any other valid JSON string, NULL or an invalid JSON.

Already update

Update doc,

json_array_length(value) parse the string to json array and return size,, null is returned in case of any other valid JSON string, null or an invalid JSON.

penghuo · 2025-05-22T21:17:31Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

+   | JSON_KEYS
+   | JSON_SET
+   | JSON_DELETE
+   | JSON_APPEND
+   | JSON_EXTEND


add docs for these functions.

Already update it.

penghuo · 2025-05-22T21:22:37Z

docs/user/ppl/functions/json.rst

+Description
+>>>>>>>>>>>
+
+Usage: `json_extract(json_string, path1, path2, ...)` it first transfer json_string to json, then extract value using paths. If only one path, return the value, otherwise, return the list of values. If one path cannot find value, return null as the result for this path. The path use "{<index>}" to represent index for array, "{}" means "{*}".


it first transfer json_string to json, then extract value using paths. If only one path, return the value, otherwise, return the list of values.

Extracts values using the specified JSON paths. If only one path is provided, it returns a single value. If multiple paths are provided, it returns a JSON Array in the order of the paths.

Already update it.

penghuo · 2025-05-22T21:24:18Z

docs/user/ppl/functions/json.rst

+
+Example::
+
+    > source=json_test | eval extract = json_extract('{\"a\": [{\"b\": 1}, {\"b\": 2}]}', 'a{}.b') | head 1 | fields extract


\" escaple is not necessary?

Yes. I update it.

penghuo · 2025-05-22T21:28:01Z

docs/user/ppl/functions/json.rst

+    +---------------------------------+
+    | test_json_array                 |
+    |---------------------------------|
+    | [[1,2],[{\"b\": 1}, {\"b\": 2}]]|


[[1,2],[{\"b\": 1}, {\"b\": 2}]] is not a valid JSON array.

I update it. Please check the doc.

Signed-off-by: xinyual <[email protected]>

penghuo · 2025-05-28T00:25:05Z

docs/user/ppl/functions/json.rst

+Description
+>>>>>>>>>>>
+
+Usage: `json_array_length(value)` parse the string to json array and return size, if can't be parsed, return null


Update doc,

json_array_length(value) parse the string to json array and return size,, null is returned in case of any other valid JSON string, null or an invalid JSON.

penghuo · 2025-05-28T00:32:28Z

docs/user/ppl/functions/json.rst

+Description
+>>>>>>>>>>>
+
+Usage: `json_extract(json_string, path1, path2, ...)` Extracts values using the specified JSON paths. If only one path is provided, it returns a single value. If multiple paths are provided, it returns a JSON Array in the order of the paths. If one path cannot find value, return null as the result for this path. The path use "{<index>}" to represent index for array, "{}" means "{*}".


Is JSON Path standard? add a section to explain it?

Not a standard json path. I would a section in this doc to describe this.

Already add the section at the top of doc. Please review it.

penghuo · 2025-05-28T00:34:44Z

docs/user/ppl/functions/json.rst

+
+Example::
+
+    > source=json_test | eval delete = json_delete('{"a": [{"b": 1}, {"b": 2}]}', 'a{0}.b') | head 1 | fields delete


json_delete('{"a": [{"b": 1}, {"b": 2}]}', 'a{0}.b') should delete is {"b": 1}? and keep {"b": 2}

Yes. Write wrong result, already fix it.

penghuo · 2025-05-28T00:37:52Z

docs/user/ppl/functions/json.rst

+    | {"a": [{"b": 3}]}       |
+    +-------------------------+
+
+    > source=json_test | eval jsonSet = json_set('{"a": [{"b": 1}, {"b": 2}]}', 'a{0}.b', 3, 'a{1}.b', 4) | head 1 | fields jsonSet


'a{1}.b', 4). should change {"b":2} to {"b":4}

Write wrong result, already fix it.

penghuo · 2025-05-28T00:46:08Z

docs/user/ppl/functions/json.rst

+    | {"a": [{"b": 1}, 3]}    |
+    +-------------------------+
+
+    > source=json_test | eval jsonAppend = json_append('{"a": [{"b": 1}, {"b": 2}]}', 'a{0}.b', 3, 'a{1}.b', 4) | head 1 | fields jsonAppend


a{0}.b is not an array, should skip append?

Yes. Write wrong query, already fix it.

penghuo · 2025-05-28T00:54:19Z

docs/user/ppl/functions/json.rst

+    | {"a": [{"b": 1}, 3]}    |
+    +-------------------------+
+
+    > source=json_test | eval jsonExtend = json_extend('{"a": [{"b": 1}, {"b": 2}]}', 'a{0}.b', 3, 'a{1}.b', 4) | head 1 | fields jsonExtend


a{0}.b is not an array, should skip append?

Yes. Write wrong query, already fix it.

penghuo · 2025-05-28T00:57:30Z

docs/user/ppl/functions/json.rst

+
+Example::
+
+    > source=json_test | eval jsonKeys = json_keys('{"a": 1, "b": 2}') | head 1 | fields jsonKeys


what if json object is nested {"a": {"b", 1}, "b":1}

Only the top level key will be returned. Still ["a", "b"] in this case. I update doc and add this case.

penghuo · 2025-05-28T01:00:49Z

docs/user/ppl/functions/json.rst

+Description
+>>>>>>>>>>>
+
+Usage: `json_keys(json_string)` Return the key list of the json_string as a string  if it's an object json string. Otherwise, return null.


"Return the key list of the json_string as a string if it's an object json string." -> Return the key list of the Json object as a Json array.

Signed-off-by: xinyual <[email protected]>

core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java

LantaoJin · 2025-06-04T09:57:51Z

docs/user/ppl/functions/json.rst

+----------
+
+Description
+>>>>>>>>>>>


please add Version and Limitation section for all new added functions.
Limitation: Only works when plugins.calcite.enabled=true

Signed-off-by: xinyual <[email protected]>

LantaoJin · 2025-06-05T11:38:22Z

CI failures CalcitePPLAppendcolTest > ... FAILED is not related.

xinyual · 2025-06-06T02:04:07Z

Force push to revert useless change and resolve DCO problems.

Signed-off-by: xinyual <[email protected]>

xinyual added 5 commits April 17, 2025 15:02

add json functions

1437f21

Signed-off-by: xinyual <[email protected]>

add new functions

54356af

Signed-off-by: xinyual <[email protected]>

add json related function

7959906

Signed-off-by: xinyual <[email protected]>

add all functions

aab67c2

Signed-off-by: xinyual <[email protected]>

add json functions

8bf3509

Signed-off-by: xinyual <[email protected]>

xinyual marked this pull request as ready for review April 21, 2025 09:20

xinyual requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin, noCharger and qianheng-aws as code owners April 21, 2025 09:20

LantaoJin added the calcite calcite migration releated label Apr 22, 2025

fix IT

df9ecbb

Signed-off-by: xinyual <[email protected]>

penghuo requested changes May 22, 2025

View reviewed changes

add docs

af2ace5

Signed-off-by: xinyual <[email protected]>

penghuo reviewed May 28, 2025

View reviewed changes

xinyual added 10 commits May 28, 2025 15:08

fix doc

e59d8fe

Signed-off-by: xinyual <[email protected]>

add json path

d30be16

Signed-off-by: xinyual <[email protected]>

optimize doc

8cc986a

Signed-off-by: xinyual <[email protected]>

optimize doc

8833a3f

Signed-off-by: xinyual <[email protected]>

optimize doc

aba660f

Signed-off-by: xinyual <[email protected]>

optimize doc

a199066

Signed-off-by: xinyual <[email protected]>

optimize doc

5cf3d9d

Signed-off-by: xinyual <[email protected]>

optimize doc

06100e3

Signed-off-by: xinyual <[email protected]>

optimize doc

64177d8

Signed-off-by: xinyual <[email protected]>

Merge remote-tracking branch 'origin/main' into addJsonFunctions

e411c4a

LantaoJin approved these changes Jun 4, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java Outdated Show resolved Hide resolved

LantaoJin requested changes Jun 4, 2025

View reviewed changes

penghuo previously approved these changes Jun 4, 2025

View reviewed changes

xinyual added 2 commits June 5, 2025 15:59

add version and limitation to docs

19b47a3

Signed-off-by: xinyual <[email protected]>

remove original implementation of V2

425afc8

Signed-off-by: xinyual <[email protected]>

xinyual dismissed penghuo’s stale review via 425afc8 June 5, 2025 08:00

xinyual added 4 commits June 5, 2025 16:01

revert useless change

10b102a

Signed-off-by: xinyual <[email protected]>

remove useless UTs

0af8cc8

Signed-off-by: xinyual <[email protected]>

Ignore useless IT

895f24c

Signed-off-by: xinyual <[email protected]>

Merge remote-tracking branch 'origin/main' into addJsonFunctions

b578d4d

Merge remote-tracking branch 'origin/main' into HEAD

2a0bad8

xinyual force-pushed the addJsonFunctions branch from 03931e1 to 2a0bad8 Compare June 6, 2025 02:03

xinyual added 3 commits June 6, 2025 10:05

update doc

b2ba7a8

Signed-off-by: xinyual <[email protected]>

merge from main

13d1880

Signed-off-by: xinyual <[email protected]>

add type checker for json

dad3178

Signed-off-by: xinyual <[email protected]>


		Usage: `json_array_length(value)` parse the string to json array and return size, if can't be parsed, return null

		Argument type: value: STRING


		Example::

		> source=json_test \| eval extract = json_extract('{\"a\": [{\"b\": 1}, {\"b\": 2}]}', 'a{}.b') \| head 1 \| fields extract


		Example::

		> source=json_test \| eval delete = json_delete('{"a": [{"b": 1}, {"b": 2}]}', 'a{0}.b') \| head 1 \| fields delete


		Example::

		> source=json_test \| eval jsonKeys = json_keys('{"a": 1, "b": 2}') \| head 1 \| fields jsonKeys

add json functions #3559

Are you sure you want to change the base?

add json functions #3559

Conversation

xinyual commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

LantaoJin commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xinyual commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xinyual commented Apr 17, 2025 •

edited

Loading

LantaoJin commented Apr 22, 2025 •

edited

Loading

xinyual commented Apr 22, 2025 •

edited

Loading