fix(duplicate-service): replace in-memory/JSON store with Supabase (#…

[pranayukey200]

…2373)

Problem:

• DuplicateService kept ticket embeddings in a process-local list (self._tickets) and a local JSON file (case_history_cache.json).
• In multi-worker deployments (Uvicorn --workers N, HF Spaces) each worker had an isolated, diverging index — duplicate detection was non-deterministic depending on which worker handled the request.
• Ephemeral containers (HF Spaces, Docker) destroyed the local JSON on every restart, losing the entire embedding history.

Fix:

• Embeddings are now persisted in a new icket_embeddings Supabase table (ticket_id, text, embedding jsonb) shared by every worker.
• add_ticket() upserts the embedding; check_duplicate() fetches all rows and performs cosine-similarity in-process (same logic as before).
• Graceful fallback: when Supabase is not configured the service falls back to the previous in-memory + local-JSON behaviour so local development continues to work without any env changes.
• Legacy save_to_disk() method retained as a no-op shim for backwards compatibility with any external callers.

Migration:

• Run backend/supabase/create_ticket_embeddings_table.sql once in the Supabase SQL editor to create the required table and index.

Closes #2373

Summary by CodeRabbit

• New Features

  • Duplicate detection now supports persistent storage when configured with Supabase, ensuring ticket data persists across container restarts and server instances.

• Infrastructure

  • Added database table to store ticket embeddings in Supabase.

[vercel]

@pranayukey200 is attempting to deploy a commit to the ritesh Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

📝 Walkthrough

Walkthrough

DuplicateService refactored to persist ticket embeddings in Supabase ticket_embeddings table instead of local JSON and in-memory lists. Initialization detects Supabase availability and configures either persistent database backend or degraded in-memory+JSON fallback. add_ticket() and check_duplicate() now route dynamically between backends.

Changes

Supabase-backed Duplicate Detection

Layer / File(s)	Summary
Database schema and Supabase client setup backend/supabase/create_ticket_embeddings_table.sql, backend/services/duplicate_service.py (lines 4–46)	Creates ticket_embeddings table (UUID id, unique ticket_id, text, embedding as jsonb, created_at timestamp) with ticket_id index. Adds _build_supabase_client() helper to construct service-role client from SUPABASE_URL/SUPABASE_SERVICE_KEY, with graceful degradation when library unavailable.
Service initialization and backend configuration backend/services/duplicate_service.py (lines 63–94, 99–148)	__init__ sets up _use_supabase, _supabase, and _fallback_tickets attributes. load() switches from print() to logging, loads SentenceTransformer model, attempts Supabase initialization, and falls back to disk-backed store when Supabase unavailable; ALLOW_DEGRADED_STARTUP=1 allows continuing without model.
Public API methods and backend routing backend/services/duplicate_service.py (lines 153–197)	add_ticket() computes embedding and routes to Supabase upsert or fallback JSON append. check_duplicate() now returns standardized dict structure, routes based on backend availability, handles threshold override, and gracefully skips detection when model unavailable.
Supabase backend operations backend/services/duplicate_service.py (lines 202–272)	_supabase_upsert() converts tensor embedding to float list and upserts into ticket_embeddings with ticket_id conflict key. _check_duplicate_supabase() fetches stored embeddings, parses JSONB (handles string or list), computes cosine similarity, and returns best match above threshold; skips malformed rows with warnings.
Fallback backend and legacy support backend/services/duplicate_service.py (lines 277–351)	_load_fallback_from_disk() restores in-memory embeddings from legacy JSON cache. _save_fallback_to_disk() appends to JSON with defensive missing-file handling. _check_duplicate_fallback() computes similarity against in-memory store. save_to_disk() becomes conditional shim: writes to JSON only when Supabase inactive.
Environment configuration documentation backend/.env.example (lines 5–11)	Documents that SUPABASE_URL and SUPABASE_SERVICE_KEY enable persistent ticket_embeddings storage shared across Uvicorn workers and surviving container restarts; explains fallback to in-memory store for local development.

Sequence Diagram

sequenceDiagram
  participant Service as DuplicateService
  participant Model as SentenceTransformer
  participant Supabase as Supabase DB
  participant Fallback as In-Memory Store

  Service->>Model: load model
  Service->>Supabase: attempt connection
  
  alt Supabase configured
    Supabase-->>Service: connected
    Service->>Service: _use_supabase = true
  else Supabase unavailable
    Service->>Service: load fallback from disk
    Service->>Service: _use_supabase = false
  end

  rect rgba(100, 150, 200, 0.5)
  Note over Service: add_ticket(ticket_id, text)
  Service->>Model: embed(text)
  alt _use_supabase
    Service->>Supabase: upsert(ticket_id, embedding)
  else fallback
    Service->>Fallback: append
    Service->>Service: save to JSON cache
  end
  end

  rect rgba(100, 150, 200, 0.5)
  Note over Service: check_duplicate(text)
  Service->>Model: embed(text)
  alt _use_supabase
    Service->>Supabase: SELECT embeddings
    Supabase-->>Service: rows
  else fallback
    Service->>Fallback: iterate stored embeddings
  end
  Service->>Service: compute cosine similarity
  Service-->>Service: return best match if above threshold
  end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• ritesh-1918/HELPDESK.AI#1825: Overlaps on DuplicateService.load/add_ticket/check_duplicate/save_to_disk methods; prior PR fixes JSON atomicity and empty-store crashes in fallback path that this PR now conditionalizes around Supabase availability.
• ritesh-1918/HELPDESK.AI#2035: Modifies same DuplicateService core methods (load, add_ticket, check_duplicate, save_to_disk) to add thread-safety and atomic JSON writes; overlaps directly with this PR's fallback backend implementation.
• ritesh-1918/HELPDESK.AI#29: Related at the caller level—refactors /tickets/save endpoint to invoke duplicate_service.add_ticket(), so newly persisted tickets enter the refactored duplicate-detection flow and route through either Supabase or fallback backend.

Suggested labels

gssoc, gssoc:approved, level:advanced, type:bug, quality:clean

Suggested reviewers

• ritesh-1918

Poem

🐰 A rabbit's ode to shared memories

The workers once danced in isolation,
Each forgetting the tickets of the nation,
But now Supabase holds their collective mind,
Embeddings persistent and shared you'll find,
With fallback so graceful, degradation's kind! 🌟

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately identifies the main change: replacing in-memory/JSON storage with Supabase persistence for the duplicate service.
Linked Issues check	✅ Passed	The PR fully addresses the core objectives from issue #2373: eliminates per-process isolation by using Supabase as shared backend, prevents embedding loss in ephemeral containers, provides persistent storage, preserves local development fallback, and maintains backward compatibility.
Out of Scope Changes check	✅ Passed	All changes directly support the primary objective of replacing the in-memory/JSON storage with Supabase-backed persistence; no extraneous modifications were introduced.
Docstring Coverage	✅ Passed	Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

backend/services/duplicate_service.py (1)

205-211: ⚡ Quick win

Store embedding as a native JSON array, not a JSON-encoded string.

Line 210 serializes embedding_list with json.dumps(...); this weakens the jsonb contract and forces dual-path parsing at Line 253. Prefer writing the Python list directly and keep read-path type handling strict.

Suggested change

-                    "embedding": json.dumps(embedding_list),
+                    "embedding": embedding_list,

Also applies to: 251-254

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/services/duplicate_service.py` around lines 205 - 211, The upsert is
serializing the embedding into a JSON string which breaks the jsonb contract;
instead write the Python list directly so the DB stores a native JSON array —
change the upsert call in duplicate_service where embedding_list is built
(embedding_list: list[float] = embedding.cpu().tolist()) to pass
embedding=embedding_list (remove json.dumps) and make the same change for the
other upsert/read pair mentioned (the second upsert/read around the other
embedding insertion) so the read-path can assume a list type without dual-path
parsing.

backend/supabase/create_ticket_embeddings_table.sql (1)

18-19: ⚡ Quick win

Drop the duplicate index on ticket_id.

Line 11 already defines ticket_id as UNIQUE, which creates the lookup index needed for ON CONFLICT (ticket_id). Keeping Line 18-19 adds redundant write overhead.

Suggested change

 -- Index for fast lookup by ticket_id
-CREATE INDEX IF NOT EXISTS idx_ticket_embeddings_ticket_id
-    ON ticket_embeddings(ticket_id);
+-- (Removed) `ticket_id` UNIQUE constraint already creates the required index.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/supabase/create_ticket_embeddings_table.sql` around lines 18 - 19,
Remove the redundant index idx_ticket_embeddings_ticket_id on ticket_embeddings:
ticket_id is already declared UNIQUE (which creates the necessary lookup index
for ON CONFLICT (ticket_id)), so delete the CREATE INDEX statement referencing
idx_ticket_embeddings_ticket_id to avoid unnecessary write overhead.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/services/duplicate_service.py`:
- Around line 215-221: The except block in DuplicateService that currently logs
failures to upsert embeddings (logger.error with ticket_id) must not swallow
errors; modify the except Exception as exc handler in the upsert_embedding
method so that after logging the full error you either re-raise the original
exception (raise) or raise a specific DuplicateServiceError containing exc, so
the caller can detect/persist failure and avoid index drift; ensure the log
still includes exc and context before re-raising to preserve diagnostics.
- Around line 231-236: The Supabase query in _check_duplicate_supabase currently
does an unbounded .select on the ticket_embeddings table and can miss rows due
to Supabase's 1,000-row default cap; change the read to a paginated,
deterministic scan by adding an .order("ticket_id") (or other stable key) and
loop with .range(start, start + page_size - 1) (e.g., page_size=1000) requesting
successive pages until the returned response.data (rows) length is less than
page_size, aggregating rows into the existing rows variable before computing
embeddings/distance; ensure you keep using the same .select("ticket_id,
embedding") projection and handle response.errors per existing pattern.

---

Nitpick comments:
In `@backend/services/duplicate_service.py`:
- Around line 205-211: The upsert is serializing the embedding into a JSON
string which breaks the jsonb contract; instead write the Python list directly
so the DB stores a native JSON array — change the upsert call in
duplicate_service where embedding_list is built (embedding_list: list[float] =
embedding.cpu().tolist()) to pass embedding=embedding_list (remove json.dumps)
and make the same change for the other upsert/read pair mentioned (the second
upsert/read around the other embedding insertion) so the read-path can assume a
list type without dual-path parsing.

In `@backend/supabase/create_ticket_embeddings_table.sql`:
- Around line 18-19: Remove the redundant index idx_ticket_embeddings_ticket_id
on ticket_embeddings: ticket_id is already declared UNIQUE (which creates the
necessary lookup index for ON CONFLICT (ticket_id)), so delete the CREATE INDEX
statement referencing idx_ticket_embeddings_ticket_id to avoid unnecessary write
overhead.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1b1c2b97-6f4e-4188-8fa7-773b60779a54

📥 Commits

Reviewing files that changed from the base of the PR and between da8faf2 and 835786d.

📒 Files selected for processing (3)

• backend/.env.example
• backend/services/duplicate_service.py
• backend/supabase/create_ticket_embeddings_table.sql

[riteshbonthalakoti]

Hi @pranayukey200! Absolute pleasure to have you building with us. I've successfully merged your PR! 🚀

Please make sure to sign up under the company Ritesh PVT Limited when testing your features!

Let's keep building! 🔥