fix(db): make alembic upgrade head work against an empty database

`uv run alembic upgrade head` on a fresh database failed at the first
`batch_op.drop_index(op.f("ix_fy_summaries_fiscal_year"))`: the migration
that creates `fy_summaries` names its index `ix_fy_summary_year`, so the
drop had nothing to drop and batch-mode reflection raised. Nothing caught
it -- CI never ran migrations from scratch, prod only applies new revisions
to a Neon database that already carries the history, and `init_db()`'s
`create_all()` masks migration gaps at runtime.

Fixing the reported drop exposed three more breaks in the same class, all
from the same root cause (migrations were never the complete schema):

- `account_classifications` was created by no migration at all, only by
  `create_all()`. Added guarded on reflection, following the existing
  `_create_transactions_table_if_missing` precedent.
- af63e055055a used non-batch `create_foreign_key` / `create_unique_constraint`,
  which SQLite cannot do, and dropped four more create_all-named indexes.
- The rollup backfill selected preference columns that only `create_all()`
  ever added. It now returns early when there are no users, since there is
  nothing to repair on a fresh database.

Reaching head was not enough on its own: the resulting schema still lacked
12 columns the ORM requires, so the app would fail on `no such column:
payday`. A reconciliation revision adds them, every add guarded on
reflection so it is a no-op on deployed databases. Verified against a
create_all-shaped database stamped at the old head: the incremental path
applies one revision, drift goes to zero, and rows are preserved.

No applied migration's effect is changed -- drops became conditional and
constraint DDL moved into batch mode, both no-ops where the objects
already exist as before.

Regression coverage, whose absence is why this survived: an integration
test upgrades a temp empty database and asserts it reaches head, that the
schema has every ORM table and column, and that re-running is idempotent.
Confirmed to fail with the original `No such index` error when the
migration fixes are reverted. Wired the same from-scratch upgrade into
ci.yml so every PR proves it.

Author: Sagargupta16

.github/workflows/ci.yml

@@ -54,6 +54,17 @@ jobs:
       - name: Pytest
         run: uv run --no-sync --locked --no-build pytest tests/ -q
 
+      # Prove a fresh environment can build its database from migrations alone.
+      # Nothing else covers this: migrate.yml only applies new revisions to a
+      # Neon database that already carries the whole history, and init_db()'s
+      # create_all() masks migration gaps at runtime. A broken chain here means
+      # a new contributor cannot bootstrap at all.
+      - name: Alembic upgrade from an empty database
+        env:
+          LEDGER_SYNC_DATABASE_URL: sqlite:///./ci_migration_check.db
+          LEDGER_SYNC_JWT_SECRET_KEY: ci-secret-key-at-least-32-characters-long
+        run: uv run --no-sync --locked --no-build alembic upgrade head
+
   security:
     uses: Sagargupta16/shared-workflows/.github/workflows/security-scan.yml@main
     permissions:

backend/pyproject.toml

@@ -134,6 +134,12 @@ select = [
 # Audited 2026-07-26, no real credential. Kept file-scoped on purpose so a
 # genuine hardcoded key added to any OTHER test fixture still fails CI.
 "tests/unit/test_auth.py" = ["S105", "S106"]
+# S603: this test shells out to `alembic upgrade head` to prove a from-scratch
+# database bootstrap works -- the argv is this interpreter plus string literals,
+# never user input. File-scoped rather than an inline noqa because ruff 0.15
+# (pinned in .pre-commit-config.yaml) raises S603 here while 0.16 does not, so
+# an inline directive fails one version or the other on RUF100.
+"tests/integration/test_migrations_from_scratch.py" = ["S101", "S603"]
 # Migration SQL is static DDL built from module-level constants and hardcoded
 # table lists, never from user input. The remaining S608 is a multi-line
 # f-string INSERT of default preference values whose diagnostic anchors to the

backend/src/ledger_sync/db/migrations/versions/20260206_0837_cc9fa860116e_add_user_id_to_aggregation_tables.py

@@ -18,6 +18,20 @@
 depends_on: str | Sequence[str] | None = None
 
 
+def _existing_indexes(table: str) -> set[str]:
+    """Reflect the index names currently on ``table``.
+
+    The autogenerated drop/recreate pairs below assume the index names that
+    ``create_all()`` produces (``ix_<table>_<column>``), but the migration that
+    created these tables named them differently (``ix_fy_summary_year``,
+    ``ix_merchant_name``, ``ix_monthly_summary_period``). On a database built
+    from migrations those names never exist, so the drops have to be
+    conditional or batch-mode reflection raises ``No such index``.
+    """
+    inspector = sa.inspect(op.get_bind())
+    return {idx["name"] for idx in inspector.get_indexes(table) if idx.get("name")}
+
+
 def upgrade() -> None:
     # ### commands auto generated by Alembic - please adjust! ###
 
@@ -69,16 +83,20 @@ def upgrade() -> None:
         batch_op.create_index(op.f("ix_financial_goals_user_id"), ["user_id"], unique=False)
         batch_op.create_foreign_key("fk_financial_goals_user_id", "users", ["user_id"], ["id"])
 
+    fy_indexes = _existing_indexes("fy_summaries")
     with op.batch_alter_table("fy_summaries") as batch_op:
         batch_op.add_column(sa.Column("user_id", sa.Integer(), nullable=False))
-        batch_op.drop_index(op.f("ix_fy_summaries_fiscal_year"))
+        if "ix_fy_summaries_fiscal_year" in fy_indexes:
+            batch_op.drop_index(op.f("ix_fy_summaries_fiscal_year"))
         batch_op.create_index(op.f("ix_fy_summaries_fiscal_year"), ["fiscal_year"], unique=False)
         batch_op.create_index(op.f("ix_fy_summaries_user_id"), ["user_id"], unique=False)
         batch_op.create_foreign_key("fk_fy_summaries_user_id", "users", ["user_id"], ["id"])
 
+    merchant_indexes = _existing_indexes("merchant_intelligence")
     with op.batch_alter_table("merchant_intelligence") as batch_op:
         batch_op.add_column(sa.Column("user_id", sa.Integer(), nullable=False))
-        batch_op.drop_index(op.f("ix_merchant_intelligence_merchant_name"))
+        if "ix_merchant_intelligence_merchant_name" in merchant_indexes:
+            batch_op.drop_index(op.f("ix_merchant_intelligence_merchant_name"))
         batch_op.create_index(
             op.f("ix_merchant_intelligence_merchant_name"), ["merchant_name"], unique=False
         )
@@ -87,9 +105,11 @@ def upgrade() -> None:
             "fk_merchant_intelligence_user_id", "users", ["user_id"], ["id"]
         )
 
+    monthly_indexes = _existing_indexes("monthly_summaries")
     with op.batch_alter_table("monthly_summaries") as batch_op:
         batch_op.add_column(sa.Column("user_id", sa.Integer(), nullable=False))
-        batch_op.drop_index(op.f("ix_monthly_summaries_period_key"))
+        if "ix_monthly_summaries_period_key" in monthly_indexes:
+            batch_op.drop_index(op.f("ix_monthly_summaries_period_key"))
         batch_op.create_index(op.f("ix_monthly_summaries_period_key"), ["period_key"], unique=False)
         batch_op.create_index(op.f("ix_monthly_summaries_user_id"), ["user_id"], unique=False)
         batch_op.create_index(

backend/src/ledger_sync/db/migrations/versions/20260207_1000_add_user_id_to_account_classifications.py

@@ -18,6 +18,49 @@
 depends_on: str | Sequence[str] | None = None
 
 
+ACCOUNT_TYPE_VALUES = (
+    "CASH",
+    "BANK_ACCOUNTS",
+    "CREDIT_CARDS",
+    "INVESTMENTS",
+    "LOANS",
+    "OTHER_WALLETS",
+)
+
+
+def _create_account_classifications_if_missing(inspector: sa.Inspector) -> None:
+    """Create ``account_classifications`` when no earlier migration has.
+
+    No migration ever created this table -- every deployed database got it from
+    ``init_db()``'s ``create_all()``, so the reference below (and in later
+    revisions) only resolved by luck. Creating it here, guarded on reflection,
+    lets a fresh database bootstrap from migrations alone while staying a no-op
+    everywhere the table already exists. Columns match what the ORM expected at
+    this revision; ``is_closed`` / ``closed_date`` arrive in closed_accounts_2026.
+    """
+    if "account_classifications" in inspector.get_table_names():
+        return
+
+    op.create_table(
+        "account_classifications",
+        sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
+        sa.Column("account_name", sa.String(length=255), nullable=False),
+        sa.Column(
+            "account_type",
+            sa.Enum(*ACCOUNT_TYPE_VALUES, name="accounttype"),
+            nullable=False,
+        ),
+        sa.Column("created_at", sa.DateTime(), nullable=False),
+        sa.Column("updated_at", sa.DateTime(), nullable=False),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    op.create_index(
+        "ix_account_classifications_account_name",
+        "account_classifications",
+        ["account_name"],
+    )
+
+
 def upgrade() -> None:
     """Add user_id to account_classifications, new indexes to transactions."""
     # --- account_classifications: add user_id column ---
@@ -27,6 +70,8 @@ def upgrade() -> None:
     # Check if user_id column already exists (idempotent)
     conn = op.get_bind()
     inspector = sa.inspect(conn)
+    _create_account_classifications_if_missing(inspector)
+    inspector.clear_cache()
     existing_columns = [col["name"] for col in inspector.get_columns("account_classifications")]
 
     if "user_id" not in existing_columns:

backend/src/ledger_sync/db/migrations/versions/20260221_1837_af63e055055a_add_created_at_updated_at_to_.py

@@ -18,6 +18,12 @@
 depends_on: str | Sequence[str] | None = None
 
 
+def _existing_indexes(table: str) -> set[str]:
+    """Reflect the index names currently on ``table``."""
+    inspector = sa.inspect(op.get_bind())
+    return {idx["name"] for idx in inspector.get_indexes(table) if idx.get("name")}
+
+
 def upgrade() -> None:
     # ### commands auto generated by Alembic - please adjust! ###
     op.create_table(
@@ -80,32 +86,47 @@ def upgrade() -> None:
     op.create_index(
         "ix_scheduled_user_active", "scheduled_transactions", ["user_id", "is_active"], unique=False
     )
-    op.drop_index(
-        op.f("ix_account_classifications_account_name"), table_name="account_classifications"
-    )
+    # The index/constraint names below are the ones create_all() produces. A
+    # database built from migrations names them differently (or not at all), so
+    # every drop is guarded on reflection and every constraint add goes through
+    # batch mode -- SQLite has no ALTER TABLE ... ADD CONSTRAINT.
+    account_indexes = _existing_indexes("account_classifications")
+    if "ix_account_classifications_account_name" in account_indexes:
+        op.drop_index(
+            op.f("ix_account_classifications_account_name"), table_name="account_classifications"
+        )
     op.create_index(
         op.f("ix_account_classifications_account_name"),
         "account_classifications",
         ["account_name"],
         unique=False,
     )
-    op.create_foreign_key(None, "account_classifications", "users", ["user_id"], ["id"])
-    op.drop_index(op.f("ix_net_worth_date"), table_name="net_worth_snapshots")
-    op.drop_index(op.f("ix_net_worth_user"), table_name="net_worth_snapshots")
+    with op.batch_alter_table("account_classifications") as batch_op:
+        batch_op.create_foreign_key(
+            "fk_account_classifications_user_id", "users", ["user_id"], ["id"]
+        )
+
+    net_worth_indexes = _existing_indexes("net_worth_snapshots")
+    for name in ("ix_net_worth_date", "ix_net_worth_user"):
+        if name in net_worth_indexes:
+            op.drop_index(op.f(name), table_name="net_worth_snapshots")
     op.create_index(
         "ix_net_worth_user_date", "net_worth_snapshots", ["user_id", "snapshot_date"], unique=False
     )
-    op.create_unique_constraint(
-        "uq_net_worth_user_date", "net_worth_snapshots", ["user_id", "snapshot_date"]
-    )
+    with op.batch_alter_table("net_worth_snapshots") as batch_op:
+        batch_op.create_unique_constraint("uq_net_worth_user_date", ["user_id", "snapshot_date"])
+
     op.add_column("tax_records", sa.Column("user_id", sa.Integer(), nullable=False))
-    op.drop_index(op.f("ix_tax_records_financial_year"), table_name="tax_records")
-    op.drop_index(op.f("ix_tax_records_fy"), table_name="tax_records")
+    tax_indexes = _existing_indexes("tax_records")
+    for name in ("ix_tax_records_financial_year", "ix_tax_records_fy"):
+        if name in tax_indexes:
+            op.drop_index(op.f(name), table_name="tax_records")
     op.create_index(
         "ix_tax_records_user_fy", "tax_records", ["user_id", "financial_year"], unique=False
     )
     op.create_index(op.f("ix_tax_records_user_id"), "tax_records", ["user_id"], unique=False)
-    op.create_foreign_key(None, "tax_records", "users", ["user_id"], ["id"])
+    with op.batch_alter_table("tax_records") as batch_op:
+        batch_op.create_foreign_key("fk_tax_records_user_id", "users", ["user_id"], ["id"])
     op.add_column("transactions", sa.Column("created_at", sa.DateTime(), nullable=False))
     op.add_column("transactions", sa.Column("updated_at", sa.DateTime(), nullable=False))
     # ### end Alembic commands ###

backend/src/ledger_sync/db/migrations/versions/20260727_1100_backfill_rollup_preference_splits.py

@@ -422,6 +422,14 @@ def _repair_fiscal(
 def upgrade() -> None:
     bind = op.get_bind()
 
+    user_ids = [row[0] for row in bind.execute(sa.text("SELECT id FROM users")).fetchall()]
+    if not user_ids:
+        # Nothing to repair. Checked before the preferences SELECT below, which
+        # names columns that only ``create_all()`` ever added -- on a database
+        # bootstrapped from migrations alone they do not exist until the
+        # reconciliation revision that follows this one.
+        return
+
     prefs_by_user: dict[int, dict[str, Any]] = {
         row["user_id"]: dict(row)
         for row in bind.execute(
@@ -434,7 +442,7 @@ def upgrade() -> None:
         ).mappings()
     }
 
-    for (user_id,) in bind.execute(sa.text("SELECT id FROM users")).fetchall():
+    for user_id in user_ids:
         prefs = prefs_by_user.get(user_id)
         # A missing preferences row falls back exactly like an empty one, so it
         # is in the victim class for both settings.

backend/src/ledger_sync/db/migrations/versions/20260727_1300_reconcile_create_all_only_columns.py

@@ -0,0 +1,144 @@
+"""add the columns that only ``create_all()`` ever created
+
+Revision ID: reconcile_create_all_2026
+Revises: capital_loss_categories_2026
+Create Date: 2026-07-27 13:00:00.000000
+
+``init_db()`` calls ``Base.metadata.create_all()`` on every startup, so a
+database that has ever booted the app has the full ORM schema whether or not a
+migration declared it. That masked a slow drift: twelve columns were added to the
+models without a matching migration, and every environment got them from
+``create_all`` instead. ``alembic upgrade head`` against an EMPTY database
+therefore reached head with a schema the app cannot query -- the first request
+touching preferences would fail on ``no such column: payday``.
+
+The columns, all reachable from live code paths:
+
+* ``user_preferences`` -- ``excluded_accounts``, ``fixed_expense_categories``,
+  ``monthly_investment_target``, ``notify_anomalies``, ``notify_budget_alerts``,
+  ``notify_days_ahead``, ``notify_upcoming_bills``, ``payday``,
+  ``preferred_tax_regime``, ``savings_goal_percent``
+* ``users`` -- ``last_login``
+* ``import_logs`` -- ``user_id`` (plus its index and CASCADE FK; the import
+  idempotency check is user-scoped through it)
+
+Every add is guarded on reflection, so this is a no-op on every already-deployed
+database and only does work on a from-scratch bootstrap. ``server_default``
+values mirror the ORM defaults so pre-existing rows would be valid either way.
+
+``import_logs.user_id`` is NOT NULL in the model. It is added nullable, then
+backfilled to the first user and tightened only when that is safe -- on a fresh
+database the table is empty, and on an existing one the column is already there
+and this whole block is skipped.
+
+Follows the repo convention of an empty ``downgrade()`` (restore from a
+database backup to roll back).
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "reconcile_create_all_2026"
+down_revision: str | None = "capital_loss_categories_2026"
+branch_labels: str | None = None
+depends_on: str | None = None
+
+
+# (table, column spec) -- server_default mirrors the ORM-side default.
+_MISSING_COLUMNS: list[tuple[str, sa.Column]] = [
+    (
+        "user_preferences",
+        sa.Column("excluded_accounts", sa.Text(), nullable=False, server_default="[]"),
+    ),
+    (
+        "user_preferences",
+        sa.Column("fixed_expense_categories", sa.Text(), nullable=False, server_default="[]"),
+    ),
+    (
+        "user_preferences",
+        sa.Column("monthly_investment_target", sa.Float(), nullable=False, server_default="0"),
+    ),
+    (
+        "user_preferences",
+        sa.Column("notify_anomalies", sa.Boolean(), nullable=False, server_default=sa.true()),
+    ),
+    (
+        "user_preferences",
+        sa.Column("notify_budget_alerts", sa.Boolean(), nullable=False, server_default=sa.true()),
+    ),
+    (
+        "user_preferences",
+        sa.Column("notify_days_ahead", sa.Integer(), nullable=False, server_default="7"),
+    ),
+    (
+        "user_preferences",
+        sa.Column("notify_upcoming_bills", sa.Boolean(), nullable=False, server_default=sa.true()),
+    ),
+    ("user_preferences", sa.Column("payday", sa.Integer(), nullable=False, server_default="1")),
+    (
+        "user_preferences",
+        sa.Column(
+            "preferred_tax_regime", sa.String(length=10), nullable=False, server_default="new"
+        ),
+    ),
+    (
+        "user_preferences",
+        sa.Column("savings_goal_percent", sa.Float(), nullable=False, server_default="20"),
+    ),
+    ("users", sa.Column("last_login", sa.DateTime(), nullable=True)),
+]
+
+
+def _columns(table: str) -> set[str]:
+    inspector = sa.inspect(op.get_bind())
+    return {col["name"] for col in inspector.get_columns(table)}
+
+
+def _indexes(table: str) -> set[str]:
+    inspector = sa.inspect(op.get_bind())
+    return {idx["name"] for idx in inspector.get_indexes(table) if idx.get("name")}
+
+
+def _add_import_logs_user_id() -> None:
+    """Add the user-scoping FK column to ``import_logs``."""
+    bind = op.get_bind()
+
+    if "user_id" not in _columns("import_logs"):
+        op.add_column("import_logs", sa.Column("user_id", sa.Integer(), nullable=True))
+        first_user = bind.execute(sa.text("SELECT id FROM users ORDER BY id LIMIT 1")).scalar()
+        if first_user is not None:
+            bind.execute(
+                sa.text("UPDATE import_logs SET user_id = :uid WHERE user_id IS NULL"),
+                {"uid": first_user},
+            )
+        # Only tighten to NOT NULL once no row can violate it. A database with
+        # import history but no users cannot happen (the FK is the owner link),
+        # but leaving the column nullable beats failing the upgrade.
+        orphans = bind.execute(
+            sa.text("SELECT COUNT(*) FROM import_logs WHERE user_id IS NULL"),
+        ).scalar()
+        if not orphans:
+            with op.batch_alter_table("import_logs") as batch_op:
+                batch_op.alter_column("user_id", existing_type=sa.Integer(), nullable=False)
+                batch_op.create_foreign_key(
+                    "fk_import_logs_user_id_cascade",
+                    "users",
+                    ["user_id"],
+                    ["id"],
+                    ondelete="CASCADE",
+                )
+
+    if "ix_import_logs_user_id" not in _indexes("import_logs"):
+        op.create_index("ix_import_logs_user_id", "import_logs", ["user_id"], unique=False)
+
+
+def upgrade() -> None:
+    for table, column in _MISSING_COLUMNS:
+        if column.name not in _columns(table):
+            op.add_column(table, column)
+
+    _add_import_logs_user_id()
+
+
+def downgrade() -> None:
+    """No downgrade -- restore from a database backup."""