Skip to content

Pagination for /miners/:githubId/pulls and /issues #100

@aliangm

Description

@aliangm

Summary

Add cursor-based pagination and a hard result cap to the two miner endpoints. They currently return unbounded result sets.

Motivation

miners.service.ts:11-122 and miners.service.ts:125-225 return every row matching the author + since window with no LIMIT. A prolific contributor whose since is unset (defaults to 35 days, but can be passed in by the caller) can produce responses with hundreds of PRs, each carrying nested linked_issues, labels, review_summary JSON. The default 35-day window mitigates but does not bound — and the caller can pass any since.

The payload also embeds nested aggregations for every row (miners.service.ts:62-99), so memory and serialization cost scale super-linearly with result count.

Proposed Solution

API

Add to both endpoints:

GET /api/v1/miners/:githubId/pulls?since=...&cursor=...&limit=50
GET /api/v1/miners/:githubId/issues?since=...&cursor=...&limit=50
  • limit defaults to 50, capped at 200.
  • cursor is an opaque base64-encoded (created_at, repo_full_name, pr_number) tuple. Cursor decoding is server-side only — clients treat it as opaque.
  • Response shape adds next_cursor (null when no more rows).

Query change

Replace the ORDER BY p.created_at DESC clauses with:

ORDER BY p.created_at DESC, p.repo_full_name DESC, p.pr_number DESC
LIMIT $N

When cursor is present, add a keyset predicate using the decoded tuple. Keyset ordering is stable and uses idx_pull_requests_author plus the PK ordering.

Controller

Update miners.controller.ts:26-60 to accept cursor and limit query params via @ApiQuery annotations.

Backward compatibility

  • Default response is bounded at 50 rows. Existing callers that don't paginate get a smaller page but in the same shape (plus next_cursor).
  • This is technically a breaking change for callers relying on getting all rows. To soften: ship behind a config flag MINERS_PAGINATION_ENABLED and announce in CHANGELOG before flipping defaults.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions