-
Notifications
You must be signed in to change notification settings - Fork 6
ordeq-iceberg: Add Ordeq Iceberg package
#416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
ordeq-iceberg: Add Ordeq Iceberg package
#416
Conversation
ordeq-iceberg: Add Ordeq Iceberg package
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request introduces the initial implementation of the ordeq-iceberg package, providing integration between Ordeq and Apache Iceberg. The package enables working with Iceberg catalogs and tables through Ordeq's IO abstractions, with support for loading catalogs, reading tables, and creating tables with configurable behavior.
Key changes:
- Core IO classes (
IcebergCatalog,IcebergTable,IcebergTableCreate) for Iceberg integration - Custom error handling with
IcebergIOErrorandIcebergTableAlreadyExistsError - Comprehensive test suite including unit tests and resource-based integration tests
Reviewed changes
Copilot reviewed 14 out of 16 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
src/ordeq_iceberg/__init__.py |
Exports main IO classes and error types for package API |
src/ordeq_iceberg/catalog.py |
Implements IcebergCatalog for loading Iceberg catalogs with enum/string support |
src/ordeq_iceberg/table.py |
Implements IcebergTable for loading tables from catalogs |
src/ordeq_iceberg/table_create.py |
Implements IcebergTableCreate with configurable if-exists behavior |
src/ordeq_iceberg/errors.py |
Defines custom exceptions for Iceberg operations |
src/ordeq_iceberg/py.typed |
Marks package as type-hinted |
pyproject.toml |
Configures build system, dependencies, and tooling for the package |
tests/conftest.py |
Provides reusable pytest fixtures for catalog and resource directory setup |
tests/test_catalog.py |
Tests catalog loading with different type formats |
tests/test_table.py |
Tests table loading from catalogs |
tests/test_table_create.py |
Tests table creation with various if-exists options |
tests/test_resources.py |
Snapshot testing for resource examples |
tests/resources/iceberg_test/create_and_load.py |
Example demonstrating table creation and loading workflow |
tests/snapshots/iceberg_test/create_and_load.snapshot.md |
Expected output snapshot for the example resource |
packages/ordeq-iceberg/tests/resources/iceberg_test/create_and_load.py
Outdated
Show resolved
Hide resolved
…e in create_save_table function
…atting, and refine test fixtures
… and schema; update tests for schema handling
ff91f01 to
70945a8
Compare
…date related logic
…remove IcebergTableCreate class
…r if_exists option
Copilot summary
This pull request introduces a new Ordeq integration package for Apache Iceberg, named
ordeq-iceberg. The package provides IO classes for loading Iceberg catalogs and tables, and includes supporting configuration, type hints, and a comprehensive test suite. The changes also register the new package in the workspace.New Ordeq-Iceberg Package:
ordeq-icebergto the workspace and created itspyproject.tomlwith metadata, dependencies (includingordeq,pyarrow, andpyiceberg), build settings, test dependencies, and code coverage configuration. [1] [2]ordeq_iceberg/__init__.pyto expose the main IO classes:IcebergCatalogandIcebergTable.py.typedfile to indicate type hints are provided.Core Functionality:
IcebergCatalogas a dataclass for loading Iceberg catalogs, supporting both enum and string catalog types.IcebergTableas a dataclass for loading Iceberg tables from a catalog, handling both direct and IO-catalog inputs.Testing and Examples:
test_catalog.py.test_table_load.py.