v2.2.0

[dmetasoul01]

LakeSoul Release v2.2.0

v2.2.0 Release Notes

1. Native IO is by default enabled for Flink CDC Sink and Spark SQL. Native IO uses arrow-rs and Datafusion with special IO optimizations based on arrow-rs' object store. Benchmarks show 3x IO throughput improvement over parquet-mr and Hadoop filesystem. Native IO supports both HDFS and S3 object storage (including S3 protocol compatible storages). Native IO supports all data types in Spark and Flink and has passed both TPC-H and CHBenchmark correctness tests.
2. Snapshot read and incremental read support on Spark. LakeSoul's incremental read on spark supports both batch mode and microbatch streaming mode.
3. Default supported Spark's version has been upgraded to Spark 3.3.

v2.2.0 发布日志

1. Native IO 在 Flink 和 Spark 上默认启用。Native IO 使用 arrow-rs 和 [Datafusion] (https://github.com/apache/arrow-datafusion) 实现，并在 arrow-rs object store 上做了专门的性能优化。在实际测试中比 parquet-mr+hadoop filesystem 快 3 倍以上。Native IO 可以支持 HDFS 和 S3 存储，以及与 S3 兼容的存储系统。Native IO 经过了详细的测试，能够支持 Flink、Spark 所有数据类型，并通过了 TPC-H 和 CHBenchmark 的正确性校验。
2. 在 Spark 上支持了快照读和增量读功能。增量读功能可以支持 batch 模式和 micro batch streaming 模式。
3. 默认的 Spark 版本更新到 3.3.

What's Changed

• [Feature] Timestamp based snapshot read, rollback and cleanup by @dmetasoul01 in [Feature] Timestamp based snapshot read, rollback and cleanup #104
• [Flink] write timestamp to int64 instead of int96 in flink sink by @dmetasoul01 in [Flink] write timestamp to int64 instead of int96 in flink sink #106
• Only one partition and compaction to parquet scan by @F-PHantam in Only one partition and compaction to parquet scan #109
• Bump postgresql from 42.5.0 to 42.5.1 in /lakesoul-common by @dependabot in Bump postgresql from 42.5.0 to 42.5.1 in /lakesoul-common #111
• Incremental query by @lypnaruto in Incremental query #110
• Add Benchmarks by @dmetasoul01 in Add Benchmarks #115
• Flink serde optimization by @dmetasoul01 in Flink serde optimization #117
• Develop/native io spark by @Ceng23333 in Develop/native io spark #118
• Fix CI with Maven Test by @Ceng23333 in Fix CI with Maven Test #121
• Support Kafka multiple topics sync to LakeSoul by @F-PHantam in Support Kafka multiple topics sync to LakeSoul #122
• solve dependency problem of confluent jar by @F-PHantam in solve dependency problem of confluent jar #124
• fix maven-test with native-io by @Ceng23333 in fix maven-test with native-io #125
• [NativeIO] Native io parquet writer implementation by @dmetasoul01 in [NativeIO] Native io parquet writer implementation #128
• [Spark] Streaming Read by @lypnaruto in [Spark] Streaming Read #129
• [Spark] Upgrade Spark version to 3.3 for main branch by @dmetasoul01 in [Spark] Upgrade Spark version to 3.3 for main branch #132
• use Arrow Schema instead of HashMap for lakesoul_reader filter by @YuChangHui in use Arrow Schema instead of HashMap for lakesoul_reader filter #136
• [NativeIO] Native writer c and jnr-ffi interface by @dmetasoul01 in [NativeIO] Native writer c and jnr-ffi interface #137
• [NativeIO] fix native reader memory leak and double free by @dmetasoul01 in [NativeIO] fix native reader memory leak and double free #138
• [NativeIO] Native writer with primary keys sort support by @dmetasoul01 in [NativeIO] Native writer with primary keys sort support #141
• [NativeIO] Use ffi to pass arrow schema by @dmetasoul01 in [NativeIO] Use ffi to pass arrow schema #142
• [NativeIO][Flink] Implement Flink native writer by @dmetasoul01 in [NativeIO][Flink] Implement Flink native writer #143
• [NativeIO] fix callback object reference by @dmetasoul01 in [NativeIO] fix callback object reference #145
• [NativeIO] upgrade arrow-rs to 31 and datafusion to 17 by @dmetasoul01 in [NativeIO] upgrade arrow-rs to 31 and datafusion to 17 #148
• [NativeIO][Spark] Package native lib in lakesoul-spark jar by @dmetasoul01 in [NativeIO][Spark] Package native lib in lakesoul-spark jar #149
• [NativeIO] use maven profile for native packaging. default to local native build by @dmetasoul01 in [NativeIO] use maven profile for native packaging. default to local native build #150
• [NativeIO][Spark] Integrate nativeIO writer in lakesoul-spark by @F-PHantam in [NativeIO][Spark] Integrate nativeIO writer in lakesoul-spark #151
• [NativeIO] Implement Sorted Stream Merger by @Ceng23333 in [NativeIO] Implement Sorted Stream Merger #147
• fix ParquetNativeFilterSuite by @Ceng23333 in fix ParquetNativeFilterSuite #152
• [NativeIO][Bug] Fix flink writer panic by @dmetasoul01 in [NativeIO][Bug] Fix flink writer panic #154
• [NativeIO] optimize with smallvec for native merge by @dmetasoul01 in [NativeIO] optimize with smallvec for native merge #155
• [NativeIO][Flink] fix flink writer batch reset in java before write by @dmetasoul01 in [NativeIO][Flink] fix flink writer batch reset in java before write #157
• [NativeIO][Spark]Implement Interfaces for LakeSoulScanBuilder with Native-IO by @Ceng23333 in [NativeIO][Spark]Implement Interfaces for LakeSoulScanBuilder with Native-IO #156
• [NativeIO] Native hdfs object store by @dmetasoul01 in [NativeIO] Native hdfs object store #159
• Add python api for snapshot and incremental query by @lypnaruto in Add python api for snapshot and incremental query #160
• fix memory leak. add columnar supports by @dmetasoul01 in fix memory leak. add columnar supports #164
• [NativeIO] upgrade arrow version to 11 by @dmetasoul01 in [NativeIO] upgrade arrow version to 11 #173
• support date type for parmary key in flink cdc by @moresun in support date type for parmary key in flink cdc #174
• [NativeIO][Flink] Fix Flink CDC Data Sort Bug and Handle DataType Change Issues from Mysql to LakeSoul by @F-PHantam in [NativeIO][Flink] Fix Flink CDC Data Sort Bug and Handle DataType Change Issues from Mysql to LakeSoul #175
• fix native_io_timestamp_conversion for default case by @Ceng23333 in fix native_io_timestamp_conversion for default case #176
• Fix flink ci by @dmetasoul01 in Fix flink ci #177
• fix invalid LakeSoulSQLConf max_row_group_size in native_io_writer by @YuChangHui in fix invalid LakeSoulSQLConf max_row_group_size in native_io_writer #179
• fix snapshot query default start time by @YuChangHui in fix snapshot query default start time #182
• fix unexpectively close partitionColumnVectors on closeCurrentBatch by @Ceng23333 in fix unexpectively close partitionColumnVectors on closeCurrentBatch #185
• add support for non-pk steaming read in spark by @moresun in add support for non-pk steaming read in spark  #188
• upgrade jffi to 1.3.11 for centos 7 by @dmetasoul01 in upgrade jffi to 1.3.11 for centos 7 #189
• [Native-IO]add native_io support for empty schema and struct type by @Ceng23333 in [Native-IO]add native_io support for empty schema and struct type #180

Full Changelog: https://github.com/meta-soul/LakeSoul/commits/v2.2.0

---

This discussion was created from the release v2.2.0.