Eng 1740 fallback to database identities if identities have expired from redis #6896

JadeCara · 2025-10-30T21:32:29Z

Description Of Changes

🎯 Whenever we try to read identities or custom privacy request fields from Redis, fallback to getting the data from the database if the values have already expired.

Refer to the updates to get_cached_identity_data and get_cached_custom_privacy_request_fields in this closed PR Privacy request cache refresh by galvana · Pull Request #4610 · ethyca/fides

Code Changes

Added fallback query for identities and custom fields to privacy request
Added new function to verify if identity is cached for existing endpoint call
Added some tests for this fallback functionality.

Steps to Confirm

Run fidesplus pointed at this branch.
To allow us to manipulate the cache before our privacy request completes - Set auto approve to false using PATCH /api/v1/config with:

{
  "execution": {
    "require_manual_request_approval": true
  }
}

Create a privacy request and grab the id
using the local redis login redis-cli -a your_password
delete the identity keys DEL id-<<privacy_request_id here>>-identity-email
Verify its gone by running KEYS * to verify that key is gone
approve the privacy request and verify it runs and that the logs contain:

fidesplus-worker-generic-1  | 2025-11-04 03:14:02.788 | DEBUG    | fides.api.models.privacy_request.privacy_request:get_cached_identity_data:710 - Cache miss for request pri_08f0df9c-6cb7-44aa-9228-3f662e19657c, falling back to DB | {'privacy_request_id': 'pri_08f0df9c-6cb7-44aa-9228-3f662e19657c', 'privacy_request_source': 'Request Manager'}

Pre-Merge Checklist

vercel · 2025-10-30T21:32:34Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
fides-plus-nightly	Ready	Preview	Comment	Nov 4, 2025 10:04pm

1 Skipped Deployment

Project	Deployment	Preview	Comments	Updated (UTC)
fides-privacy-center	Ignored			Nov 4, 2025 10:04pm

src/fides/api/models/privacy_request/privacy_request.py

…identities-have-expired-from-redis

codecov · 2025-10-31T22:19:22Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.29%. Comparing base (eb3f579) to head (0849a89).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6896   +/-   ##
=======================================
  Coverage   87.29%   87.29%           
=======================================
  Files         523      523           
  Lines       34172    34188   +16     
  Branches     3932     3934    +2     
=======================================
+ Hits        29829    29845   +16     
  Misses       3487     3487           
  Partials      856      856

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…identities-have-expired-from-redis

greptile-apps

Greptile Overview

Greptile Summary

Implements fallback logic to retrieve identity and custom privacy request fields from the database when Redis cache expires, preventing privacy requests from failing when cached data is no longer available.

Key changes:

Added verify_cache_for_identity_data() method to check if identity data exists in cache without triggering fallback
Enhanced get_cached_identity_data() to automatically fetch from DB and re-cache when cache miss occurs
Enhanced get_cached_custom_privacy_request_fields() with similar fallback behavior
Updated restart endpoint to use verification method instead of retrieval method for cache checking
Added comprehensive test coverage for fallback scenarios

Test improvements:

Removed manual cache deletion in test fixtures, following best practices (f4371da8)
Added tests for both successful fallback and edge cases (no persisted identity)

Confidence Score: 5/5

This PR is safe to merge with minimal risk
The implementation is straightforward and defensive - adding fallback logic that prevents failures when cache expires. The changes are well-tested with multiple test cases covering edge cases. No breaking changes or risky refactoring. The endpoint change correctly delegates to a new verification method that doesn't trigger side effects.
No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
src/fides/api/models/privacy_request/privacy_request.py	4/5	Added fallback logic to retrieve identity and custom fields from DB when Redis cache expires, plus helper method to verify cache presence
src/fides/api/api/v1/endpoints/privacy_request_endpoints.py	5/5	Updated restart endpoint to use new `verify_cache_for_identity_data()` method instead of `get_cached_identity_data()` for cache verification
tests/ops/models/privacy_request/test_privacy_request.py	5/5	Added comprehensive tests for cache fallback behavior for identity and custom privacy request fields

_{4 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

erosselli · 2025-11-04T13:33:02Z

src/fides/api/models/privacy_request/privacy_request.py

+        prefix = f"id-{self.id}-identity-*"
+        cache: FidesopsRedis = get_cache()
+        keys = cache.keys(prefix)


maybe we can move these 3 lines to a private method that we can reuse in this verify_cache_for_identity_data function and also in get_cached_identity_data ?

erosselli · 2025-11-04T13:35:15Z

tests/ops/service/privacy_request/test_email_batch_send.py

-@pytest.fixture(scope="function")
-def third_privacy_request_awaiting_erasure_email_send(
-    db: Session, erasure_policy: Policy
-) -> PrivacyRequest:
-    """Add a third erasure privacy request w/ no identity in this state for these tests"""
-    privacy_request = _create_privacy_request_for_policy(
-        db,
-        erasure_policy,
-    )
-    privacy_request.status = PrivacyRequestStatus.awaiting_email_send
-    privacy_request.save(db)
-    yield privacy_request
-    privacy_request.delete(db)


what's the reason for deleting this fixture ?

It is un-used with the changes. The _create_privacy_request_for_policy persists and identity and in the tests we were deleting it. That doesn't really work for the tests anymore since we just go get it from the db now.

…identities-have-expired-from-redis

Jade Wibbels added 4 commits October 24, 2025 15:30

initial duplication detection

d340cb8

Merge branch 'main' of github.com:ethyca/fides

95e1545

initial

bd5976e

not that

cb0b578

.

690d996

JadeCara commented Oct 31, 2025

View reviewed changes

src/fides/api/models/privacy_request/privacy_request.py Outdated Show resolved Hide resolved

Jade Wibbels and others added 2 commits October 31, 2025 15:12

.

4fff3b3

Merge branch 'main' into ENG-1740-fallback-to-database-identities-if-…

7468b4b

…identities-have-expired-from-redis

vercel bot deployed to Preview – fides-plus-nightly October 31, 2025 21:22 View deployment

Jade Wibbels and others added 3 commits October 31, 2025 15:37

added naive tests

f7878c0

Merge branch 'main' into ENG-1740-fallback-to-database-identities-if-…

6a5d93d

…identities-have-expired-from-redis

linting

c0b4ea0

Jade Wibbels and others added 3 commits November 3, 2025 10:03

updated tests that relied on deleting cache to clear identities

2bb6ea0

.

d3d2c9b

Merge branch 'main' into ENG-1740-fallback-to-database-identities-if-…

40effd8

…identities-have-expired-from-redis

JadeCara marked this pull request as ready for review November 4, 2025 01:16

JadeCara requested a review from a team as a code owner November 4, 2025 01:16

JadeCara requested review from erosselli and removed request for a team November 4, 2025 01:16

greptile-apps bot reviewed Nov 4, 2025

View reviewed changes

vercel bot deployed to Preview – fides-plus-nightly November 4, 2025 01:21 View deployment

erosselli reviewed Nov 4, 2025

View reviewed changes

Jade Wibbels and others added 3 commits November 4, 2025 13:08

moved duplicate lines to function

7834138

Merge branch 'main' into ENG-1740-fallback-to-database-identities-if-…

f32f7cb

…identities-have-expired-from-redis

changelog

82f85a8

JadeCara requested a review from erosselli November 4, 2025 20:58

Merge branch 'main' into ENG-1740-fallback-to-database-identities-if-…

0849a89

…identities-have-expired-from-redis

vercel bot deployed to Preview – fides-plus-nightly November 4, 2025 22:04 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Eng 1740 fallback to database identities if identities have expired from redis #6896

Eng 1740 fallback to database identities if identities have expired from redis #6896

JadeCara commented Oct 30, 2025 •

edited

Loading

Uh oh!

vercel bot commented Oct 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

codecov bot commented Oct 31, 2025 •

edited

Loading

Uh oh!

greptile-apps bot left a comment

Uh oh!

erosselli Nov 4, 2025

Uh oh!

erosselli Nov 4, 2025

Uh oh!

JadeCara Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Eng 1740 fallback to database identities if identities have expired from redis #6896

Are you sure you want to change the base?

Eng 1740 fallback to database identities if identities have expired from redis #6896

Conversation

JadeCara commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

Uh oh!

vercel bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Greptile Overview

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

erosselli Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

erosselli Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

JadeCara Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JadeCara commented Oct 30, 2025 •

edited

Loading

vercel bot commented Oct 30, 2025 •

edited

Loading

codecov bot commented Oct 31, 2025 •

edited

Loading