Skip to content

Route validation matching by Census layer (#120) - #164

Open
frankcampos wants to merge 1 commit into
openstates:mainfrom
frankcampos:120_fix_county_yaml_files
Open

Route validation matching by Census layer (#120)#164
frankcampos wants to merge 1 commit into
openstates:mainfrom
frankcampos:120_fix_county_yaml_files

Conversation

@frankcampos

@frankcampos frankcampos commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Closes #120. Follow-up to #119.

The problem

find_matches() hard-filtered every candidate to the Census place layer, so the county and state rows that #119 loaded could never be reached. A county: or state-only OCDid returned an empty frame and was quarantined as a stub.

The change

Route by the OCDid's deepest segment. The layer column carries the TIGER file stem the row came from, so its suffix names the layer:

OCDid layer suffix matched against
place:seattle _place seattle
county:king _county king
bare state:wa _state washington
  • _filter_to_place_layer becomes _filter_to_layer(df, layer). It keeps the old PLACEFP heuristic as a fallback for a CSV with no layer column.
  • A state-only OCDid matches on the state's full name from state_lookup, because the States tab carries Washington in NAMELSAD, not wa.
  • territory: now falls back alongside district:, so Puerto Rico and Guam resolve against their States-tab rows. This is one line beyond the ticket, but generate_ocdids() emits those ids and the sheet has the rows.

Naming fix in generate_division

Every council district in a county matches that one county row, so all of them took the county's display name and collided on the {display_name}_{geoid} filename — ten Blount County districts all wrote to blount_47009_*.yaml, separated only by a UUID.

_county_council_district_display_name builds the name from the record's NAMELSAD rather than the OCDid slug. Using NAMELSAD instead of appending the literal word "County" keeps Louisiana parishes and Alaska boroughs correct:

county:blount/council_district:10   ->  Blount County Council District 10
county:orleans/council_district:2   ->  Orleans Parish Council District 2

Verification

pipeline summary

image # all tests passed image

Replayed the Tennessee set (829 records) against the live sheets, then ran the real pipeline end to end:

before after
success 36 829
partial (quarantined stub) 793 0

Every record resolves to exactly one row. Nothing lands in the 0 or 2+ branch. Spot checks: state:wa → Washington (tl_2025_us_state), state:tn/county:davidson → Davidson County (tl_2025_47_county), territory:pr → Puerto Rico, state:wa/place:seattle → Seattle city.

173 tests pass. ruff check and ruff format --check are clean.

Tests added

Five integration cases in test_generate_pipeline_integration.py — one state, one county, and three cross-layer isolation cases — plus two decoy fixture rows (King city, Washington city) so a name-only match would visibly cross layers. Four unit cases in test_generate_division.py cover the county display name, including the parish case.

Closes openstates#120.

`find_matches()` hard-filtered every candidate to the Census place layer, so
the county and state rows loaded by openstates#119 could never be reached: a `county:`
or state-only OCDid returned an empty frame and was quarantined as a stub.

Route by the OCDid's deepest segment instead. The `layer` column carries the
TIGER file stem the row came from, so its suffix names the layer:

  place:  OCDid  -> rows whose layer ends _place
  county: OCDid  -> rows whose layer ends _county
  state-only     -> rows whose layer ends _state

`_filter_to_place_layer` becomes `_filter_to_layer(df, layer)`, keeping the
old PLACEFP heuristic as a fallback for a CSV with no `layer` column. A
state-only OCDid matches on the state's full name from `state_lookup`,
because the States tab carries "Washington" in NAMELSAD, not "wa".
`territory:` now falls back alongside `district:`, so Puerto Rico and Guam
resolve against their States-tab rows.

Naming: every council district in a county matches that one county row, so
they all shared a display name and the `{display_name}_{geoid}` filename
collided. `_county_council_district_display_name` builds the name from the
record's NAMELSAD rather than the OCDid slug, which keeps parishes and
boroughs correct ("Orleans Parish Council District 2").

Verified against the live sheets on the Tennessee set: all 829 records now
resolve to exactly one row and generate a full Division, up from 36 matched
and 793 quarantined. No record falls into the 0 or 2+ branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzZfcmwXPERr1rSL3e5fve
@frankcampos
frankcampos marked this pull request as ready for review August 30, 2026 20:03
@frankcampos frankcampos self-assigned this Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Match county: and state-only OCDids against States/Counties validation rows

1 participant