weakref finalizers errors

[kbzsl]

Sorry for my lack of knowledge, I would like to ask your help how to troubleshoot weakref finalizers errors and if possible to get some hints how to solve them. Or just some recommendation for a workaround or best practices.

I wrote a (quite) long ibis script transforming and combining different tables (backend: snowflake, but most probably this is irrelevant). This script is running fine, without any errors.
But when I tried to split up this long “spaghetti” code in smaller chunks, writing a separate function for each (bigger) transformation step (having as argument one or more ibis tables and returning an ibis table) there where multiple and random weakref errors generated.

I tried to implement a similar code structure:

def connect(region):
def transf1(table: ibis.Table) -> ibis.Table:
def transf2(table: ibis.Table) -> ibis.Table:
...

for r in REGION:
	con = connect(r)
	t = con.table(TABLE)
	t1 = transf1(t)
	t2 = transf1(t1)
	con.disconnect()

In my case it looks that these were just “irrelevant” errors, as the end result was ok.
I am reluctant to ignore totally these errors, because sometimes it can be a valid error. And if I understand well these cannot be handled simply by using the default (try except) error handling.

I was using: python 3.11, ibis-framework 10.8.0 and 11.0.0, snowflake-connector-python 3.18.0 and 4.0.0

[JonAnCla]

Thanks, are you able to share some logs/stack traces?

[kbzsl]

Here is one such example (just I removed some sensitive parts, like project name).

Please note that the connection was not closed (even though this is stated in the error log), because after this error the next function call (like t2 = transf1(t1)) was running without any problem (although it might generated a similar weekref error).
Some of these tables are cached, but I already checked if removing the .cache() makes any difference. I am exeprincing the same weakref errors with or without having .cahe().

Exception ignored in: <finalize object at 0x7fbb94b86f20; dead>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.11/site-packages/ibis/backends/__init__.py", line 948, in _finalize_cached_table
    self._drop_cached_table(name)
  File "/.venv/lib/python3.11/site-packages/ibis/backends/__init__.py", line 958, in _drop_cached_table
    self.drop_table(name, force=True)
  File "/.venv/lib/python3.11/site-packages/ibis/backends/sql/__init__.py", line 336, in drop_table
    with self._safe_raw_sql(drop_stmt):
  File "/opt/conda/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/.venv/lib/python3.11/site-packages/ibis/backends/snowflake/__init__.py", line 703, in _safe_raw_sql
    with contextlib.closing(self.raw_sql(query, **kwargs)) as cur:
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.11/site-packages/ibis/backends/snowflake/__init__.py", line 709, in raw_sql
    cur = self.con.cursor()
          ^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1063, in cursor
    Error.errorhandler_wrapper(
  File "/.venv/lib/python3.11/site-packages/snowflake/connector/errors.py", line 286, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.11/site-packages/snowflake/connector/errors.py", line 344, in hand_to_other_handler
    connection.errorhandler(connection, cursor, error_class, error_value)
  File "/.venv/lib/python3.11/site-packages/snowflake/connector/errors.py", line 217, in default_errorhandler
    raise error_class(
snowflake.connector.errors.DatabaseError: 250002 (08003): Connection is closed