Skip to content

Conversation

@Josersanvil
Copy link
Contributor

@Josersanvil Josersanvil commented Nov 24, 2025

Copilot summary

This pull request introduces a new Ordeq integration package for Apache Iceberg, named ordeq-iceberg. The package provides IO classes for loading Iceberg catalogs and tables, and includes supporting configuration, type hints, and a comprehensive test suite. The changes also register the new package in the workspace.

New Ordeq-Iceberg Package:

  • Added ordeq-iceberg to the workspace and created its pyproject.toml with metadata, dependencies (including ordeq, pyarrow, and pyiceberg), build settings, test dependencies, and code coverage configuration. [1] [2]
  • Added ordeq_iceberg/__init__.py to expose the main IO classes: IcebergCatalog and IcebergTable.
  • Added py.typed file to indicate type hints are provided.

Core Functionality:

  • Implemented IcebergCatalog as a dataclass for loading Iceberg catalogs, supporting both enum and string catalog types.
  • Implemented IcebergTable as a dataclass for loading Iceberg tables from a catalog, handling both direct and IO-catalog inputs.

Testing and Examples:

  • Added fixtures for setting up and tearing down test Iceberg SQL catalogs.
  • Added unit tests for catalog loading and parameter handling in test_catalog.py.
  • Added integration tests for table loading with both catalog objects and IO inputs in test_table_load.py.
  • Added resource-based integration test and snapshot for loading a table, including logging and output verification. [1] [2] [3]

Copilot AI review requested due to automatic review settings November 24, 2025 08:48
@sbrugman sbrugman added the change New feature or request label Nov 24, 2025
@sbrugman sbrugman changed the title Add Ordeq Iceberg package ordeq-iceberg: Add Ordeq Iceberg package Nov 24, 2025
Copilot finished reviewing on behalf of Josersanvil November 24, 2025 08:54
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request introduces the initial implementation of the ordeq-iceberg package, providing integration between Ordeq and Apache Iceberg. The package enables working with Iceberg catalogs and tables through Ordeq's IO abstractions, with support for loading catalogs, reading tables, and creating tables with configurable behavior.

Key changes:

  • Core IO classes (IcebergCatalog, IcebergTable, IcebergTableCreate) for Iceberg integration
  • Custom error handling with IcebergIOError and IcebergTableAlreadyExistsError
  • Comprehensive test suite including unit tests and resource-based integration tests

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
src/ordeq_iceberg/__init__.py Exports main IO classes and error types for package API
src/ordeq_iceberg/catalog.py Implements IcebergCatalog for loading Iceberg catalogs with enum/string support
src/ordeq_iceberg/table.py Implements IcebergTable for loading tables from catalogs
src/ordeq_iceberg/table_create.py Implements IcebergTableCreate with configurable if-exists behavior
src/ordeq_iceberg/errors.py Defines custom exceptions for Iceberg operations
src/ordeq_iceberg/py.typed Marks package as type-hinted
pyproject.toml Configures build system, dependencies, and tooling for the package
tests/conftest.py Provides reusable pytest fixtures for catalog and resource directory setup
tests/test_catalog.py Tests catalog loading with different type formats
tests/test_table.py Tests table loading from catalogs
tests/test_table_create.py Tests table creation with various if-exists options
tests/test_resources.py Snapshot testing for resource examples
tests/resources/iceberg_test/create_and_load.py Example demonstrating table creation and loading workflow
tests/snapshots/iceberg_test/create_and_load.snapshot.md Expected output snapshot for the example resource

@Josersanvil Josersanvil force-pushed the feat/package/ordeq-iceberg branch 2 times, most recently from ff91f01 to 70945a8 Compare November 24, 2025 12:17
@Josersanvil Josersanvil requested a review from sbrugman November 26, 2025 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

change New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants