Route validation matching by Census layer (#120) - #164
Open
frankcampos wants to merge 1 commit into
Open
Conversation
Closes openstates#120. `find_matches()` hard-filtered every candidate to the Census place layer, so the county and state rows loaded by openstates#119 could never be reached: a `county:` or state-only OCDid returned an empty frame and was quarantined as a stub. Route by the OCDid's deepest segment instead. The `layer` column carries the TIGER file stem the row came from, so its suffix names the layer: place: OCDid -> rows whose layer ends _place county: OCDid -> rows whose layer ends _county state-only -> rows whose layer ends _state `_filter_to_place_layer` becomes `_filter_to_layer(df, layer)`, keeping the old PLACEFP heuristic as a fallback for a CSV with no `layer` column. A state-only OCDid matches on the state's full name from `state_lookup`, because the States tab carries "Washington" in NAMELSAD, not "wa". `territory:` now falls back alongside `district:`, so Puerto Rico and Guam resolve against their States-tab rows. Naming: every council district in a county matches that one county row, so they all shared a display name and the `{display_name}_{geoid}` filename collided. `_county_council_district_display_name` builds the name from the record's NAMELSAD rather than the OCDid slug, which keeps parishes and boroughs correct ("Orleans Parish Council District 2"). Verified against the live sheets on the Tennessee set: all 829 records now resolve to exactly one row and generate a full Division, up from 36 matched and 793 quarantined. No record falls into the 0 or 2+ branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DzZfcmwXPERr1rSL3e5fve
frankcampos
marked this pull request as ready for review
August 30, 2026 20:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #120. Follow-up to #119.
The problem
find_matches()hard-filtered every candidate to the Census place layer, so the county and state rows that #119 loaded could never be reached. Acounty:or state-only OCDid returned an empty frame and was quarantined as a stub.The change
Route by the OCDid's deepest segment. The
layercolumn carries the TIGER file stem the row came from, so its suffix names the layer:place:seattle_placeseattlecounty:king_countykingstate:wa_statewashington_filter_to_place_layerbecomes_filter_to_layer(df, layer). It keeps the old PLACEFP heuristic as a fallback for a CSV with nolayercolumn.state_lookup, because the States tab carriesWashingtonin NAMELSAD, notwa.territory:now falls back alongsidedistrict:, so Puerto Rico and Guam resolve against their States-tab rows. This is one line beyond the ticket, butgenerate_ocdids()emits those ids and the sheet has the rows.Naming fix in
generate_divisionEvery council district in a county matches that one county row, so all of them took the county's display name and collided on the
{display_name}_{geoid}filename — ten Blount County districts all wrote toblount_47009_*.yaml, separated only by a UUID._county_council_district_display_namebuilds the name from the record's NAMELSAD rather than the OCDid slug. Using NAMELSAD instead of appending the literal word "County" keeps Louisiana parishes and Alaska boroughs correct:Verification
pipeline summary
Replayed the Tennessee set (829 records) against the live sheets, then ran the real pipeline end to end:
Every record resolves to exactly one row. Nothing lands in the 0 or 2+ branch. Spot checks:
state:wa→ Washington (tl_2025_us_state),state:tn/county:davidson→ Davidson County (tl_2025_47_county),territory:pr→ Puerto Rico,state:wa/place:seattle→ Seattle city.173 tests pass.
ruff checkandruff format --checkare clean.Tests added
Five integration cases in
test_generate_pipeline_integration.py— one state, one county, and three cross-layer isolation cases — plus two decoy fixture rows (King city,Washington city) so a name-only match would visibly cross layers. Four unit cases intest_generate_division.pycover the county display name, including the parish case.