[_]: fix/optimise-background-removal-index

[sg-gs]

What

Another step to speedup the Drive's database. We need an index to remove files metadata records of still existing users that are stored for >1 year (or any other given period) and these changes aim to offer an efficient solution to that.

Why

Because the current index is not efficient and it is also another thing to maintain and another expense to count on INSERT time. Check this, just to get 200 records:

EXPLAIN ANALYZE SELECT id, status, updated_at
from files where updated_at < NOW() - interval '1 year' and status = 'DELETED';
---
Seq Scan on files  (cost=0.00..62317363.52 rows=112393842 width=350) (actual time=127.480..1321875.634 rows=116543089 loops=1)
  Filter: ((status = 'DELETED'::enum_files_status) AND (updated_at < (now() - '1 year'::interval)))
  Rows Removed by Filter: 733607286
Planning Time: 0.413 ms
JIT:
  Functions: 4
  Options: Inlining true, Optimization true, Expressions true, Deforming true
  Timing: Generation 2.099 ms (Deform 1.282 ms), Inlining 8.262 ms, Optimization 71.281 ms, Emission 47.872 ms, Total 129.514 ms
Execution Time: 1327475.205 ms

We aim to make this table as efficient and light as possible. The current execution time is just not acceptable and the index is also weights 3.5GB.

How

Leveraging BRIN indexes to do that on an efficient manner, as the records removed one year ago are highly correlated with their physical location on the disk due to the fact that the updated_at is no longer updated after the files' statuses are set as DELETED.

Results

After adding this index, things change quite a lot (I have not added any EXPLAIN ANALYZE as they do not seem to work good with BRIN indexes):

• Before:
  • Index size: 3.5GB
  • Execution time: 1327475.205ms (~1327s)
• After:
  • Index size: 12MB
  • Execution time: 100ms

[sg-gs]

Migration run @jzunigax2

[jzunigax2]

I guess these changes must be necessary to fix the e2e failling test

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sg-gs]

Great, thanks! @jzunigax2