Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] Fix JSON text deserialization in ARRAYs and ROWs #5049

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[core] Fix JSON text deserialization in ARRAYs and ROWs #5049

wants to merge 2 commits into from
+366 −2

Conversation

atallahade
Copy link
Contributor

@atallahade atallahade commented Feb 10, 2025 •
edited
Loading

Purpose

Linked issue: relates to #4920

Fix JSON text deserialization in ARRAYs and ROWs.

Tests

Added test cases for deserialization of JSON strings with different types of elements.

Before the change:

$ mvn -pl paimon-common -Dtest=org.apache.paimon.utils.TypeUtilsTest test
[...]
[ERROR] Failures:
[ERROR]   TypeUtilsTest.testArrayStringCastFromString:101
expected: org.apache.paimon.data.GenericArray@93cd27db
 but was: org.apache.paimon.data.GenericArray@18d8f46e
[ERROR]   TypeUtilsTest.testCastFromString:71
expected: org.apache.paimon.data.GenericArray@6820ecbf
 but was: org.apache.paimon.data.GenericArray@9f40b0f7
[ERROR]   TypeUtilsTest.testRowCastFromString:268
expected: +I(org.apache.paimon.data.GenericMap@9fb34a0d,value)
 but was: +I(org.apache.paimon.data.GenericMap@9fb34a0d,"value")
[...]

The tests pass after the change.

API and Format

Documentation

AshishKhatkar
AshishKhatkar approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@AshishKhatkar AshishKhatkar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

@JingsongLi JingsongLi closed this Feb 11, 2025
@JingsongLi JingsongLi reopened this Feb 11, 2025
@JingsongLi
Copy link
Contributor

Thanks @atallahade for the contribution!

It seems this PR breaks some tests in Flink CDC.

@atallahade
Copy link
Contributor Author

atallahade commented Feb 11, 2025 •
edited
Loading

It seems this PR breaks some tests in Flink CDC.

looks like a transient issue? I see a similar error in another PR opened yesterday

@atallahade atallahade closed this Feb 11, 2025
@atallahade atallahade reopened this Feb 11, 2025
@atallahade
Copy link
Contributor Author

well, this is failing again with

Error:  Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 139.668 s <<< FAILURE! - in org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase
Error:  org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase.testAllTypes  Time elapsed: 90.131 s  <<< ERROR!
java.util.concurrent.TimeoutException: testAllTypes() timed out after 90 seconds
	at org.junit.jupiter.engine.extension.TimeoutInvocation.createTimeoutException(TimeoutInvocation.java:70)
	at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:59)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
	Suppressed: java.lang.InterruptedException: sleep interrupted
		at java.lang.Thread.sleep(Native Method)
		at org.apache.paimon.flink.action.cdc.CdcActionITCaseBase.waitForResult(CdcActionITCaseBase.java:173)
		at org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase.testAllTypesImpl(PostgresSyncTableActionITCase.java:449)
		at org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase.testAllTypesOnce(PostgresSyncTableActionITCase.java:329)
		at org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase.testAllTypes(PostgresSyncTableActionITCase.java:312)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:[498](https://github.com/apache/paimon/actions/runs/13263619120/job/37025585680?pr=5049#step:5:499))
		at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
		at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
		at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
		at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
		... 63 more

I'm trying to reproduce this locally to debug and fix potential issues but the test passes locally with:

$ mvn -pl paimon-flink/paimon-flink-cdc -Dtest=org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase test
$ mvn -pl paimon-flink/paimon-flink-cdc -Dtest=org.apache.paimon.flink.action.cdc.postgres.PostgresSyncTableActionITCase#testAllTypes test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AshishKhatkar AshishKhatkar AshishKhatkar approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@atallahade @JingsongLi @AshishKhatkar