perf(#2442): cache ticket embeddings in storage to bypass startup inference - #2530
Conversation
|
@suhaniiz is attempting to deploy a commit to the ritesh Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughThis PR removes Git LFS pointers for two model files and updates the DuplicateService to cache and reuse embeddings from disk. Previously, embeddings were discarded; now they are serialized to the history cache and deserialized on load to avoid recomputation. ChangesEmbedding Caching and Model Cleanup
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
backend/services/duplicate_service.py (1)
111-112: ⚡ Quick winConsider catching more specific exceptions.
The broad
except Exceptioncatch here could mask unexpected errors. Consider catching more specific exceptions likeIOError,json.JSONDecodeError, orOSErrorto handle known failure modes while allowing unexpected errors to surface during development.♻️ Proposed refactor for more specific exception handling
with open(self.storage_file, "w") as f: json.dump(data, f, indent=2) print(f"[DuplicateService] Indexed ticket {ticket_id} to case history.") - except Exception as e: + except (IOError, OSError, json.JSONDecodeError) as e: print(f"[DuplicateService] Failed to save to disk: {e}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/services/duplicate_service.py` around lines 111 - 112, The broad except in DuplicateService that currently does "except Exception as e: print(...)" should be replaced with targeted handlers: catch OSError/IOError (for filesystem write errors) and json.JSONDecodeError (if JSON parsing/serialization is involved) and log each specific error in the same print/processLogger call; for any truly unexpected errors re-raise them so they surface during development. Locate the try/except block inside DuplicateService (the save-to-disk routine) and replace the blanket except Exception with specific except OSError/IOError as e: ... and except json.JSONDecodeError as e: ... and then optionally a final bare "except Exception:" that re-raises the exception.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/services/duplicate_service.py`:
- Around line 53-61: When deserializing cached embeddings (item["embedding"])
validate the vector length/shape before converting to a tensor: retrieve the
expected embedding dimension from the model (e.g., via a getter like
get_sentence_embedding_dimension() or by inspecting a known embedding shape),
check that len(item["embedding"]) == expected_dim, and if it mismatches or is
malformed log a warning and fall back to recomputing with
self.model.encode(text, convert_to_tensor=True) (then move to self.model.device
if available); ensure the validated tensor is the one passed into util.cos_sim()
so corrupted cached vectors cannot cause shape errors or silent accuracy issues.
---
Nitpick comments:
In `@backend/services/duplicate_service.py`:
- Around line 111-112: The broad except in DuplicateService that currently does
"except Exception as e: print(...)" should be replaced with targeted handlers:
catch OSError/IOError (for filesystem write errors) and json.JSONDecodeError (if
JSON parsing/serialization is involved) and log each specific error in the same
print/processLogger call; for any truly unexpected errors re-raise them so they
surface during development. Locate the try/except block inside DuplicateService
(the save-to-disk routine) and replace the blanket except Exception with
specific except OSError/IOError as e: ... and except json.JSONDecodeError as e:
... and then optionally a final bare "except Exception:" that re-raises the
exception.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9a235e7b-cb1c-4fc5-8c0b-ed65305806fe
📒 Files selected for processing (3)
backend/models/classifier/model.safetensorsbackend/models/ner/model.safetensorsbackend/services/duplicate_service.py
💤 Files with no reviewable changes (2)
- backend/models/classifier/model.safetensors
- backend/models/ner/model.safetensors
| # Fixed: Check if an embedding is cached, fallback safely if not found | ||
| if "embedding" in item and item["embedding"] is not None: | ||
| embedding = torch.tensor(item["embedding"], dtype=torch.float32) | ||
| # Move to model device if available | ||
| if hasattr(self.model, 'device'): | ||
| embedding = embedding.to(self.model.device) | ||
| else: | ||
| embedding = self.model.encode(text, convert_to_tensor=True) | ||
|
|
There was a problem hiding this comment.
Validate embedding shape when deserializing from cache.
When reconstructing embeddings from JSON (line 55), there's no validation that the embedding list has the correct length or shape. The all-MiniLM-L6-v2 model produces 384-dimensional embeddings, but if the cached data is corrupted or comes from a different model version, the reconstructed tensor could have the wrong shape, leading to errors in util.cos_sim() at line 167 or silent accuracy degradation.
🛡️ Proposed fix to validate embedding dimensions
for item in data:
text = item["text"]
# Fixed: Check if an embedding is cached, fallback safely if not found
if "embedding" in item and item["embedding"] is not None:
- embedding = torch.tensor(item["embedding"], dtype=torch.float32)
+ embedding_list = item["embedding"]
+ # Validate embedding dimension (all-MiniLM-L6-v2 produces 384-dim vectors)
+ if not isinstance(embedding_list, list) or len(embedding_list) != 384:
+ print(f"[DuplicateService] Invalid cached embedding for {item['ticket_id']}, re-encoding...")
+ embedding = self.model.encode(text, convert_to_tensor=True)
+ else:
+ embedding = torch.tensor(embedding_list, dtype=torch.float32)
# Move to model device if available
if hasattr(self.model, 'device'):
embedding = embedding.to(self.model.device)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/services/duplicate_service.py` around lines 53 - 61, When
deserializing cached embeddings (item["embedding"]) validate the vector
length/shape before converting to a tensor: retrieve the expected embedding
dimension from the model (e.g., via a getter like
get_sentence_embedding_dimension() or by inspecting a known embedding shape),
check that len(item["embedding"]) == expected_dim, and if it mismatches or is
malformed log a warning and fall back to recomputing with
self.model.encode(text, convert_to_tensor=True) (then move to self.model.device
if available); ensure the validated tensor is the one passed into util.cos_sim()
so corrupted cached vectors cannot cause shape errors or silent accuracy issues.
|
Hi @suhaniiz! Thanks for the amazing contribution. I've successfully merged your PR! 🚀 Please make sure to sign up under the company Ritesh PVT Limited when testing your features!
Let's build something epic! 🔥 |
e1d5daf
into
riteshbonthalakoti:gssoc
|
@ritesh-1918 , this pr was under GSSoC 2026, so kindly add the appropriate labels |
|
@ritesh-1918 ... |
|
@ritesh-1918 , please add appropriate labels |
|
@ritesh-1918 .... |
|
@ritesh-1918 , this pr is under gssoc 2026, please add appropriate labels otherwise it wont count to me |
2 participants
Description
Addresses issue #2442 by updating the storage schema and hydration layer. Previously, the
DuplicateServiceperformed an expensiveself.model.encode(text)operation for every cached historical ticket sequentially duringload(), inducing significant service availability latency on startup.This PR shifts the overhead into an$O(1)$ look-up pattern. The generated PyTorch embedding array is now serialized as a standard list of floats directly inside the JSON database when a ticket is created. On service reboot,
load()reads these float lists and maps them straight back into memory as float32 tensors, avoiding redundant deep learning inference entirely.Changes Made
save_to_diskto convert embedding arrays into standard Python lists for valid JSON serialization.add_ticketdirectly into the disk-writing mechanism.load()to read pre-compiled lists from cache and convert them back to tensors with runtime device parity checks.Testing Checklist
Summary by CodeRabbit
New Features
Chores