Iceberg/storage: storage implementation from iceberg s3

[laskoviymishka]

Implement Iceberg Storage

Feature Request

Develop an Iceberg Storage, focusing on reading data from Iceberg data-files.
As reference implementation took Delta.

---

Scope and Goals

✅ List table from iceberg catalog
✅ Infer schema and convert it to transfer internal type system.
✅ Read data convert arrow records to Change Item-s.
✅ Basic e2e tests with test-container for s3.

🚫 Replication and CDC handling are out of scope.
🚫 No sharding and performance tune in this phase.

---

Implementation Details

• Read format:

  • Utilize iceberg-go official library for meta catalogs
  • Apache arrow reader for data files reads

• Type System Support:

  • Primitives - mapped to transfer type system
  • Structural type - mapped to any

• Configuration Options:

  • Separate settings for catalogue type + catalog URI
  • Everything else as generic iceberg.Props map[string]string, this will support any settings out of icebergs out of the box

• Testing:

  • Basic tests should be in place
  • Docker compose e2e tests inside CI
  • Docker compose with for local tests run

---

References & Inspiration

• Delta