Skip to content

README.md

Latest commit

 

History

History
430 lines (303 loc) · 12.7 KB

README.md

File metadata and controls

430 lines (303 loc) · 12.7 KB
Raw

OpenStates Jurisdictions

This repository stores and generates Division and Jurisdiction YAML data for U.S.local governments.

It includes YAML files with metadata for every government entity in the United States, each referencing the geopolitical boundaries coextensive with that jurisdiction. 

Open States is a long-standing open-source project that provides state representative and legislative data for all U.S. state legislatures. We are working to extend that to counties, county subdivisions, municipalities, territories, school districts, and special districts.

Our data is derived from Census Data and mapped to Open Civic Data Division identifiers (another long-standing open-source project).

The OpenStates/Jurisdictions YAML files are intended for use by application builders. View sample files here

By providing comprehensive, accurate, stable and human-verified set of Jurisdiction data for local U.S. government entities, we are helping developers more easily and quickly provide their users with up-to-date representative information, ballot information, public notices, public meetings, and more.

Become a contributor! Join Civic Data on Unified.

Quick Navigation

Purpose Start Here
🚀 Getting Started Quickstart • Requirements • Installation
📚 Understanding the Project Core Concepts • FAQ • Data Models • Model Relationships
🤝 Contributing CONTRIBUTING.md • Ways to Contribute • Making Code Changes
🔧 Advanced Topics Using DuckDB • Running Tests • Common Commands • Reviewing PRs

Core Concepts

New to this project? This repository works with two core concepts:

  • Divisions = Geographic boundaries (the land areas)
  • Jurisdictions = Governing entities (the organizations that have authority)

For complete conceptual explanations and FAQ, see FAQ.md.

For technical details on data models and fields, see MODELS.md and docs/data_model_relationships.md for a visual overview.

What This Repo Contains

  • Source models and pipeline code under src/
  • Output YAML under:
    • divisions/<state>/local/
    • jurisdictions/<state>/local/
  • Tests under tests/
  • Human-facing docs under docs/

Requirements

System Requirements

Component Minimum Recommended Notes
Python 3.12+ 3.12+ Required for all workflows
uv Latest Latest Python package manager (install)

Optional Dependencies

  • DuckDB - Automatically instantiated in pipeline; can be used manually for data exploration
  • macOS - brew for system package management

Installation

Prerequisites: Install uv

uv is a fast Python package manager. Install it first:

macOS:

brew install uv

Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows (WSL2):

curl -LsSf https://astral.sh/uv/install.sh | sh

Or visit uv installation guide.

Platform-Specific Setup

macOS

# 1. Clone the repository
git clone https://github.com/openstates/jurisdictions.git
cd jurisdictions

# 2. Create virtual environment
uv venv .venv

# 3. Activate virtual environment
source .venv/bin/activate

# 4. Install dependencies
uv sync --all-extras

# 5. Verify installation
uv run ruff check .
uv run pytest -m "not integration and not slow"

Linux (Ubuntu/Debian)

# 1. Install system dependencies (if needed)
sudo apt-get update
sudo apt-get install -y python3.12 python3.12-venv python3.12-dev

# 2. Clone and setup
git clone https://github.com/openstates/jurisdictions.git
cd jurisdictions

# 3. Create virtual environment
uv venv .venv

# 4. Activate virtual environment
source .venv/bin/activate

# 5. Install dependencies
uv sync --all-extras

# 6. Verify installation
uv run ruff check .
uv run pytest -m "not integration and not slow"

Windows (WSL2 Recommended)

# In WSL2 terminal, follow Linux instructions above

# Or on native Windows (cmd/PowerShell):
# 1. Clone repository
git clone https://github.com/openstates/jurisdictions.git
cd jurisdictions

# 2. Create virtual environment
uv venv .venv

# 3. Activate virtual environment (PowerShell)
.\.venv\Scripts\Activate.ps1

# 4. Install dependencies
uv sync --all-extras

# 5. Verify
uv run ruff check .
uv run pytest -m "not integration and not slow"

Quickstart (New Contributor)

  1. Fork this repo on GitHub -> Owner (you) -> Create fork

  2. Clone and enter repo.

git clone <your-repo-url>
cd jurisdictions
  1. Create virtual environment and install dependencies.
uv venv .venv
source .venv/bin/activate
uv sync --all-extras
  1. Verify local setup.
uv run ruff check .
uv run pytest -m "not integration and not slow"
  1. Make changes and push
git checkout -b <your-new-branch>
<make changes>
git add .
git commit -m "Information about your changes."
git push origin <your-new-branch>
  1. Create pull request by going to https://github.com//jurisdictions/tree/ -> Contribute -> Open Pull Request

Running Tests

Test Organization

Tests are organized by scope and execution time:

Test Type Command Purpose When to Use
Fast Unit Tests uv run pytest -m "not integration and not slow" Quick validation of code Before commits, local development
All Unit Tests uv run pytest -m "not integration" Complete unit test coverage Before PR, CI validation
Integration Tests uv run pytest -m "integration" End-to-end pipeline testing After model changes, before merging
Full Suite uv run pytest Everything (includes slow tests) Final validation, CI/CD
Specific Test uv run pytest tests/path/to/test_file.py::test_name Single test function Debugging specific issues

Common Test Commands

# Fast local check (recommended before commits)
uv run pytest -m "not integration and not slow"

# All tests except slow ones
uv run pytest -m "not integration"

# Full test suite (takes longer)
uv run pytest

# Run tests with verbose output
uv run pytest -v

# Run tests and show output/print statements
uv run pytest -s

# Run only integration tests
uv run pytest -m "integration"

# Run tests matching a pattern
uv run pytest -k "test_division" -v

# Run with coverage report
uv run pytest --cov=src --cov-report=html

Using DuckDB

DuckDB is used in the pipeline for data exploration and validation. It's automatically instantiated when the pipeline runs, but you can also use it manually.

Automatic Instantiation

When you run the Stage 1 pipeline, DuckDB database is automatically created:

uv run python src/init_migration/main.py

This creates data/ocdid_pipeline.duckdb with OCD ID data.

Manual Usage

Explore the database:

import duckdb

# Connect to existing database
conn = duckdb.connect('data/ocdid_pipeline.duckdb')

# List all tables
print(conn.execute("SELECT * FROM information_schema.tables").fetchall())

# Query OCD ID data
result = conn.execute("""
    SELECT * FROM ocdid_data 
    WHERE state = 'ca' 
    LIMIT 5
""").fetchall()

for row in result:
    print(row)

conn.close()

Create a new database for analysis:

import duckdb

# Create in-memory database for testing
conn = duckdb.connect(':memory:')

# Create a table
conn.execute("""
    CREATE TABLE jurisdictions AS
    SELECT 'ca' as state, 'Los Angeles' as name
    UNION ALL
    SELECT 'wa' as state, 'Seattle' as name
""")

# Query it
result = conn.execute("SELECT * FROM jurisdictions").fetchall()
print(result)

conn.close()

DuckDB Resources

Common Commands

Development Workflow

# Create a new branch for your changes
git checkout -b feature/my-feature

# Install/update dependencies after pulling changes
uv sync --all-extras

# Run code quality checks
uv run ruff check .
uv run ruff format .

# Run tests before committing
uv run pytest -m "not integration and not slow"

Data Pipeline

# Run full Stage 1 pipeline
uv run python src/init_migration/main.py

# Run pipeline for specific states
uv run python src/init_migration/main.py --state ca,wa,tx

# Force re-run (bypass cache)
uv run python src/init_migration/main.py --force

# Run with verbose output
uv run python src/init_migration/main.py --state ca --verbose

Maintenance & Troubleshooting

# Clean up Python cache files
find . -type d -name __pycache__ -exec rm -r {} +
find . -type f -name "*.pyc" -delete

# Update all dependencies
uv sync --all-extras --upgrade

# Verify environment setup
python --version
uv --version

Ways to Contribute

Update YAML data directly - No coding required

See CONTRIBUTING.md for step-by-step instructions.

Pick up an existing issue - Coding required

See CONTRIBUTING.md for how to find and work on issues.

Contributing Guidance

Questions?

  • "How do I...?" Check the FAQ for common questions
  • "What's the data model?" See MODELS.md for technical details
  • "How do I contribute?" Read CONTRIBUTING.md
  • "Something's broken" Check docs/setup_uv.md for troubleshooting, or open an issue

🚀 YAML-Only Changes & Auto-Merge

For pull requests containing only YAML file changes in divisions/ or jurisdictions/:

  • ✅ PR automatically merges after approval
  • ✅ Branch automatically deletes
  • ✅ No manual merge step needed

How it works:

  1. Create PR with YAML-only changes
  2. Workflow verifies files are in safe paths
  3. Request reviewer approval
  4. Auto-merge triggers automatically

Learn more:

Notes

  • Use src package-root imports in code and tests (e.g., from src.models.division import Division)
  • Do not modify core model contracts in src/models/ without maintainer approval
  • See CONTRIBUTING.md for documentation expectations when making changes

Pulling & Reviewing External PRs

If you need to review changes from a pull request created on a forked repository, you can pull them locally:

Fetch a PR for Local Review

# Fetch the PR and create a local branch
git fetch origin pull/<PR_NUMBER>/head:<local-branch-name>

# Switch to that branch
git checkout <local-branch-name>

# Review the changes locally
# Run tests, lint, and other validations
uv run pytest -m "not integration and not slow"
uv run ruff check .

# Switch back when done
git checkout main

Example

# Fetch PR #42 for local review
git fetch origin pull/42/head:review-pr-42
git checkout review-pr-42

# Test the changes
uv run pytest -m "not integration and not slow"

# Return to main
git checkout main