Skip to content

Drop missing observations by position for duplicate pandas indexes - #276

Open
lllleolin-max wants to merge 1 commit into
matthewwardrop:mainfrom
lllleolin-max:codex/preserve-duplicate-index-observations
Open

lllleolin-max wants to merge 1 commit into
matthewwardrop:mainfrom
lllleolin-max:codex/preserve-duplicate-index-observations

Conversation

@lllleolin-max

Copy link
Copy Markdown

When a DataFrame has duplicate index labels and one observation is missing, dropping its index label also drops valid observations with the same label. This either loses data or makes generated columns disagree with the final index. Drop rows by position consistently in the Series helper, C() encoder and final pandas index. Regression tests compare numeric, categorical and explicit C() formulas against the same retained observations with unique indexes across pandas, NumPy and sparse output.

Validation: the added regression tests fail on main (81e675b) and pass with this patch. Related tests: python -m pytest tests/materializers/test_pandas.py tests/utils/test_null_handling.py tests/transforms/test_contrasts.py -q. Changed files pass ruff check and ruff format.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant