feat: Implement option 'delete_rows' of argument 'if_exists' in 'DataFrame.to_sql' API.

[gmcrocetti]

• closes ENH: DataFrame.to_sql with if_exists='replace' should do truncate table instead of drop table #37210
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v3.0.0.rst file if fixing a bug or adding a new feature.

[gmcrocetti]

@WillAyd I chose the name delete_rows instead of delete_replace because the behavior of replace right now is of recreate - as you mentioned - and delete_rows means exactly what is going on behind the scenes.

@erfannariman tagging you due to your help/interest during the lifecycle of this issue.

[WillAyd]

I'm not sure that the test failures are related. Restarted so let's see...

My remaining feedback is rather minor; overall I think the implementation looks good.

@mroeschke care to take a look?

[gmcrocetti]

@WillAyd it seems new tests have introduced some sort of "lock" on pytest. It is affecting 3.11, 3.12 and "Future infer strings" using a single CPU (it works using multiple). Do you have any hunch ?

NVM. CI is fine now.

[WillAyd]

I think to make this test more manageable we can create a generic pd.SQLError and use that whenever these are raised instead. I don't think there's a lot of value to exposing the error messages from the underlying libraries to end users, and it would help simplify this test

[gmcrocetti]

Yes. I totally agree on that 💯 .
pandas should definitely provide a common interface for SQL-related errors. But I believe this is out of the current scope ?

[WillAyd]

I actually think we already have one in place - check pandas.errors.DatabaseError. If you catch where these problems are in the implementation and just do raise DatabaseError from exc then you can also just catch the DatabaseError in the test

[gmcrocetti]

yes, sure.
So as of now the only place raising pd.errors.DatabaseError is the method execute.
The new implementation delete_rows is using cursor.execute instead - which is at the driver level.
So the options we have to comply with what you suggested is:

1. raise pd.errors.DatabaseError from delete_rows. The issue being other methods (append, replace) would not raise it.
2. We start using self.execute but I tried in the past and it only works for SQLite because run_transaction closes the connection while ADBC and sqlalchemy don't.

Do you see another option or have a preference ?

[WillAyd]

Where within the class hierarchies does that execute get called? It seems like there is where you can catch the exception from the library and re-raise a more generic one

[WillAyd]

Hmm OK. We might need to fix that up as a precursor to this then - seems like something is off

[gmcrocetti]

Ok...so let's try to unravel it:
Here is the SQLAlchemy's implementation for execute.

There's no exception handling in this method as in SQLite and ADBC cases.

A problem with the current implementation is that the code that inserts is separated from the one deleting records.
So.... Even if delete_rows is implemented using self.execute it can happen that the insertion procedure doesn't - which is the case for SQLDatabase. I truly see the pain and wish we could implement the entire operation in a single method/function. But the codebase is not ready for that and would required some refactoring.

I really believe this is out of scope...please let me know what you think.

[WillAyd]

I appreciate wanting to get this in but I don't think the issue is out of scope - we would just be choosing to ignore it, which delays the problem for a later date.

Do you see a reasonable path to a pre-cursor that can fix some of the implementation issues with execute and get us back to a place where we consistently use the internal class methods, rather than cherry-picking calls to a third party execute?

[gmcrocetti]

I appreciate wanting to get this in but I don't think the issue is out of scope - we would just be choosing to ignore it, which delays the problem for a later date.

And just to clarify, the issue we're discussing is the lack of a common interface for database related errors ?

---

I have sent a patch. It touches parts of code that are out of my league but that are required for the new test test_delete_rows_is_atomic to work - using self.execute.

It is not meant to be a bullet proof and solve previously existing problems (but it definitely changes existing APIs) but might be a middle-ground to what is to come and solves it for the newly introduced delete_rows.

Please, let me know if that is feasible.

BTW, thanks a ton for the hard work reviewing it multiple times.

[gmcrocetti]

@WillAyd LMK as soon you have the time to review :). TY