Upcoming Changes (April 2024)

[joocer]

Changes to Mabel, should have no functional impact but may result in minor changes to execution times (some faster, some slower).

• When writing Parquet, Orso is used as the WAL (write-ahead-log)
  • This simplifies Mabel's handling of nulls in Parquet files
  • This makes schemas mandatory for Parquet files (this was enforced but not technically required earlier)
  • This enables the creation of the details for the manifest for these files
• The default file size is reduced to 62Mb, this is to account for inaccuracies in estimating the size of final file representations
• The frame.complete file now contains at least a partial manifest of the partition

These changes are designed to be relatively neutral to the data pipelines but enable improvements to Opteryx:

• Slightly smaller files should improve the cache hit rate
• Manifests are part of a metadata system to be used for indexing and cost-based optimization