Skip to content

Recognize pandas nullable string columns as categorical factors - #281

Open
lllleolin-max wants to merge 1 commit into
matthewwardrop:mainfrom
lllleolin-max:codex/categorically-encode-pandas-string-dtypes
Open

lllleolin-max wants to merge 1 commit into
matthewwardrop:mainfrom
lllleolin-max:codex/categorically-encode-pandas-string-dtypes

Conversation

@lllleolin-max

Copy link
Copy Markdown

Pandas StringDtype columns using pd.NA are currently classified as numeric. A formula such as x then returns raw strings in the model matrix instead of dummy variables. Recognize StringDtype alongside object, str and categorical columns. Tests cover Python/Arrow storage, all three matrix outputs, missing observations and reuse of category state on new data.

Validation: the added regression tests fail on main (81e675b) and pass with this patch. Related tests: python -m pytest tests/materializers/test_pandas.py tests/materializers/test_equivalence.py -q. Changed files pass ruff check and ruff format.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant