Skip to content

Parquet dependency should be updated to 1.12+ to support the INT96 compatibility flag #99

@rjbarrington

Description

@rjbarrington

The current parquet dependency is for the 1.11.x version, which doesn't support INT96 types. Older datasets generated by Spark applications may have used this type. Parquet 1.12+ has added support for this. Updating to 1.12.3 allows the user to add "policy.fs.parquet.avro.readInt96AsFixed": "true", for example.

See https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#configuration, https://issues.apache.org/jira/browse/PARQUET-1928, https://github.com/apache/parquet-mr/blob/master/CHANGES.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions