Import large table from bigquery to duckdb

[memeplex]

Hi all,

I would like to read a large table from bigquery and write it to a local duckdb table, without materializing the entire thing in memory as an intermediate step. I can't directly pass the ir.Table to duckdb_con.create_table, it seems like it tries to execute the expression inside the duckdb context. Most examples I've seen call to_pyarrow on the table, but I assume that loads the entire table in memory. I also tried to_pyarrow_batches() and got

Creating an ibis.memtable from a pyarrow.RecordBatchReader would load all data into memory.

What is the right way to do it?

Thanks!

Related: #11683, #4800, #8115.

[memeplex]

I came up with the solution along these lines:

t1 = ibis.examples.penguins.fetch()
t2 = con.create_table("penguins", schema=t1.schema(), overwrite=True)
for b in t1.to_pyarrow_batches(chunk_size=10):
    con.insert("penguins", b)

Is there a more convenient way to achieve it?