Getting to API stability

[cpcloud]

I'd like to gather a list of things we'd want to do before stabilizing our non-internal APIs.

Those APIs include

1. The base backend API, along with various specific backend APIs
2. The user-facing public expression APIs
3. Anything that isn't officially public that we'd like to consider making officially public, like ibis.expr.types objects.

Out of scope:

1. Any modules and their contents under ibis that isn't explicitly mentioned or already public.
2. The backend test suite (this topic is worthy of a separate discussion)

I'll kick this off with my own list of things that I think are required for stability:

1. A more explicit contract around output data coming from both to_pandas() and to_pyarrow(). As part of that I would like to see us choose a specific pandas dtype "family" that we output.
2. An API for creating tables in different databases and schemas
3. Unification of behaviors that aren't consistent across backends to the extent possible, e.g., what does an "empty" window frame do? Is it BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW (the default for most functions in most databases that support window functions)?
4. A single page that people can go to see the list of currently deprecated APIs. We have the ability to show this information for specific APIs, but not across all APIs on a single page. Ideally this page would show alternatives.

[tokoko]

I don't know how relevant this is in this discussion, but one thing that always bothered me about read_x and to_x apis (read_csv, read_parquet and others) is that kwargs are treated as backend-specific. I think that could complicate further enrichment of those methods. For example, at some point to_parquet probably should have a partition_by parameter itself and not have to rely on the fact that most backends have identical naming for it. Adding those kinds of parameters could conflict with backend-specific arguments. Maybe it would be cleaner to introduce backend_arguments parameter and forward those to respective backend calls rather than forwarding the whole kwargs.

[lostmygithubaccount]

I do think it's relevant to this discussion. Personally, I'm against trying to standardize on these method across backends -- it will be a never ending game of whack-a-mole. But I'm also not the one who would have to implement and maintain this 🙃 and despite having the same kwargs, behavior may still differ slightly across backends and produce unexpected results

definitely worth discussing though, thanks for raising!

[tokoko]

I do agree that slight variations could get complicated, but that's the whole point of ibis, isn't it? I just think not even trying to standardize here hugely deviates from the "write once and execute anywhere" story. I don't see these methods as that much different from normal "transformation" apis. Some parts of reader/writer arguments will make sense to be standardized (e.g. how a delta table should be partitioned is part of the spec), some of them less so (e.g. how many parallel processes should a distributed engine use to write). We could argue about each and every one of them in the future, but all I'm saying is that the current approach might complicate incorporating even the most obvious arguments to the ibis side.