Skip to content

Parquet: coerce_int96 does not work for int96 in nested types #15763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mbutrovich opened this issue Apr 18, 2025 · 1 comment · Fixed by #16058
Closed

Parquet: coerce_int96 does not work for int96 in nested types #15763

mbutrovich opened this issue Apr 18, 2025 · 1 comment · Fixed by #16058
Labels
bug Something isn't working

Comments

@mbutrovich
Copy link
Contributor

mbutrovich commented Apr 18, 2025

Describe the bug

The logic that coerces timestamps to a different resolution iterates through fields and uses their key in the Parquet schema as a key to match against the Arrow schema.

pub fn coerce_int96_to_resolution(

However, this is insufficient (consider a schema with structs each with an id field) to disambiguate nested fields. I think we might need a combination of Parquet ColumnDesc's path and a normalized Arrow schema to fix the mapping.

To Reproduce

CometFuzzTestSuite for INT96 reproduces the issue immediately. I will work on including an slt test in the fix PR.

Expected behavior

No response

Additional context

@mbutrovich mbutrovich added the bug Something isn't working label Apr 18, 2025
@Adez017
Copy link
Contributor

Adez017 commented Apr 18, 2025

take

@mbutrovich mbutrovich changed the title Parquet: coerce_int96 does not work for int96 in nested types with repeated names Parquet: coerce_int96 does not work for int96 in nested types, especially with repeated names May 15, 2025
@mbutrovich mbutrovich changed the title Parquet: coerce_int96 does not work for int96 in nested types, especially with repeated names Parquet: coerce_int96 does not work for int96 in nested types May 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants