InMemory Catalog

[kevinjqliu]

Feature Request / Improvement

Feature Request: InMemory Catalog implementation for Python Iceberg. For testing, prototyping, and generally playing around with Iceberg.

It would be great to have a catalog implementation in-memory, and possibly write to local file system instead of s3. This would lower the barrier to entry to use the python iceberg library.

There's currently a NoopCatalog which is used for loading StaticTable

There's also a InMemoryCatalog implementation already used for testing. We can use this as the basis for implementation.

Inspired by Trino's Memory connector

[Fokko]

Forwarding my comment here as well: #289 (comment)

Maybe we should add this to the documentation of the SqlCatalog as well:

def catalog_memory(warehouse: Path) -> Generator[SqlCatalog, None, None]:
    props = {
        "uri": "sqlite+pysqlite:///:memory:",
        "warehouse": f"file://{warehouse}",
    }

iceberg-python/tests/catalog/test_sql.py

Lines 69 to 73 in 1befad7

	def catalog_memory(warehouse: Path) -> Generator[SqlCatalog, None, None]:
	props = {
	"uri": "sqlite+pysqlite:///:memory:",
	"warehouse": f"file://{warehouse}",
	}

The warehouse directory is where the files are stored.

[asheeshgarg]

@Fokko
Is this apache/iceberg#4518 supported as part of pycieberg?

[kevinjqliu]

@Fokko
Didn't know the in-memory sqlite option was available! That's awesome.
I was able to read/write using the SqlCatalog. Metadata is saved in memory using sqlite and data is saved on disk.

Pulling out your comment in #289

Thanks for working on this @kevinjqliu. The issues was created a long time ago, before we had the SqlCatalog with sqlite support. Sqlite can also work in memory rendering the InMemoryCatalog obsolete. Having two in-memory implementations in the codebase adds additional complexity in the codebase. My suggestion would be to replace the MemoryCatalog with the SqlCatalog. WDYT?

I agree that we don't need 2 in-memory catalog implementations. Let me see if I can repurpose #289

[kevinjqliu]

Looks like there's also a InMemoryCatalog in the Java lib

github.com/apache/iceberg/blob/0f509d2d678db2d7322dafded58ec0ca6d7fb268/core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java#L61

[kevinjqliu]

@Fokko I took some time to think this over. I understand your concern regarding 2 in-memory catalog implementations. The 2 implementations are just 2 different ways to achieve the same outcome; using Sqlite's in-memory database is an implementation detail.

I'm leaning towards having InMemoryCatalog as an option especially since there is a Java version of it already.

I think there's value in storing the pure Python Object in memory and making it easily accessible. The Sqlite's in-memory database is another level of indirection which can be difficult to reason about the internals of Iceberg.

[Fokko]

@kevinjqliu Alright, that's fair, I just wanted to make sure that we considered the option before making the InMemory one public 👍

[Fokko]

I must admit that I'm second-guessing the decision to add another catalog. This mostly comes from the recent discussions on the Java side where catalogs are being removed to avoid further proliferation.

[kevinjqliu]

@Fokko thats fair. I'll see if there are features in the PR I can pull out. We can leave the InMemory catalog under tests/ for now.