[_]: feat/allow-lastId-param-when-listing-files

[sg-gs]

What

Allow an optional lastId parameter on GET /files. This parameter will be used instead of offset. If both are provided, offset will be ignored when lastId is present. An index to leverage the current query is also being added.

Why

We have relied on OFFSET for a long time, and pagination is becoming a performance bottleneck. Large offsets grow almost linearly with the dataset size, making pagination for accounts with hundreds of thousands of files non-scalable.

The idea is to introduce a WHERE > clause based on the file UUID. When used together with sorting by the same UUID, this effectively switches the pagination strategy to cursor-based pagination, which is significantly more efficient and offers near-constant performance at the database level.

Proof

Same query, with real data. EXPLAIN ANALYZE with OFFSET:

EXPLAIN ANALYZE SELECT * 
FROM "files" AS "FileModel" 
WHERE "FileModel"."status" = 'DELETED'
	AND "FileModel"."user_id" = <some_user_id> 
	AND "FileModel"."updated_at" > NOW() - interval '3 months'
ORDER BY "FileModel"."uuid" ASC LIMIT 1000 OFFSET 5000;

Limit  (cost=3003.80..3003.81 rows=1 width=350) (actual time=126.134..126.499 rows=1000 loops=1)
  ->  Sort  (cost=2996.09..3003.80 rows=3085 width=350) (actual time=124.655..126.078 rows=6000 loops=1)
        Sort Key: uuid
        Sort Method: top-N heapsort  Memory: 5550kB
        ->  Index Scan using idx_user_status_updated_uuid on files "FileModel"  (cost=0.70..2817.30 rows=3085 width=350) (actual time=0.068..54.809 rows=25328 loops=1)
              Index Cond: ((user_id = <user_id>) AND (status = 'DELETED'::enum_files_status) AND (updated_at > (now() - '3 mons'::interval)))
Planning Time: 0.676 ms
Execution Time: 126.629 ms

By using the WHERE:

EXPLAIN ANALYZE SELECT * 
FROM "files" AS "FileModel" 
WHERE "FileModel"."status" = 'DELETED'
	AND "FileModel"."user_id" = <some_user_id> 
	AND "FileModel"."updated_at" > NOW() - interval '3 months'
	and uuid > '<some_uuid>'
ORDER BY "FileModel"."uuid" ASC LIMIT 1000;

Limit  (cost=2394.29..2396.79 rows=1000 width=350) (actual time=42.819..43.024 rows=1000 loops=1)
  ->  Sort  (cost=2394.29..2400.42 rows=2453 width=350) (actual time=42.817..42.957 rows=1000 loops=1)
        Sort Key: uuid
        Sort Method: top-N heapsort  Memory: 878kB
        ->  Index Scan using idx_user_status_updated_uuid on files "FileModel"  (cost=0.70..2259.79 rows=2453 width=350) (actual time=0.028..28.475 rows=20327 loops=1)
              Index Cond: ((user_id = <some_user_id>) AND (status = 'DELETED'::enum_files_status) AND (updated_at > (now() - '3 mons'::interval)) AND (uuid > 'some_uuid'::uuid))
Planning Time: 0.335 ms
Execution Time: 43.105 ms

How

Allow the new parameter and apply it in the WHERE clause so the database can leverage the index to skip records efficiently, instead of using OFFSET, which requires scanning and discarding rows explicitly. Also, adding a migration to leverage the current query makes this faster.

⚠️ Requires migration.

[sg-gs]

Running migration @jzunigax2

[sg-gs]

Migration run @jzunigax2. Let me think what's your opinion / review.

[sonarqubecloud]

Quality Gate failed

Failed conditions
62.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud