Workday huge-tenant coverage: bounded detail-hydration truncates giant tenants

[prPMDev]

Context

The Workday adapter (shipped in 0.5.0) hydrates job descriptions via a per-posting detail endpoint after a filter-aware list pre-selection. To bound the N+1 cost on enterprise tenants, detail-hydration is hard-capped at MAX_DETAIL_FETCHES = 100 (src/adapters/workday.js), with list paging capped at LIST_PAGE_HARD_CAP = 100 (≤ 2000 list items scanned).

This is an intentional v1 simplification, not a complete solution.

Problem

On giant tenants (Salesforce ~1398 postings, Cisco ~939), a description filter query (matched by the library after the adapter returns) only sees the first ≤100 list-position candidates that survived list-evaluable pre-filters. Genuine matches sitting past list-position 100 are never hydrated, so they silently never appear. List-evaluable filters (titleFilter, location, postedWithinDays) narrow the set first and mitigate this, but a broad description-only filter on a huge tenant can still miss real matches with no signal to the caller.

v1 caps silently — there is deliberately no truncation-surfacing metadata yet.

Proper fix (design alongside #7)

This is fundamentally request-volume/politeness infrastructure and should be designed together with #7 (retry + rate-limit handling), since both govern how aggressively we may fan out detail calls:

• Smart pagination / early-exit once enough post-filter matches are found
• Rate-limited concurrency (bounded parallel detail fetches with backoff) so the cap can be raised safely
• Surfaced-truncation metadata (e.g. meta.truncated: true + scanned/total counts) so callers know results are incomplete

Tracked in source: src/adapters/workday.js has a // NOTE: near the cap pointing here and at #7.

Related: #7 (Retry logic and rate limit handling for all ATS APIs).

[prPMDev]

Design note before anyone picks this up

The cap is the easy thing to describe but the wrong thing to optimize in isolation. This needs more detailed thought, grounded in how people actually use the tool, not against the raw ~1398 number.

"Too many results" is itself a problem, not just a fetch-cost problem. Even if we could cheaply hydrate all 1398 Salesforce postings, returning ~1400 jobs to the caller (or into an AI assistant's context) is its own failure mode. The cap question and the result-volume question are the same question.

Does list-level pre-filtering already solve this in practice? Real queries are specific. Nobody asks for "all Salesforce jobs"; they ask for "customer service manager" or "staff PM, US, posted this week". The cheap list endpoint already carries title / location / postedOn, so titleFilter + location + recency pre-filter before any detail fetch. Obviously not all 1300 jobs are "customer service manager". So the real open question: for realistic queries, does that list pre-filter reliably bring the survivor set under the cap on its own (i.e. the cap almost never actually bites)? If yes, the fix is mostly "make the list pre-filter as strong as it can be, and surface clearly when it did not narrow enough", not "fetch more".

If pre-filter is not enough, sequential beats a bigger number. For the residual broad-query case, a sequential / lazy-streaming approach (hydrate in list order, stop once we have enough post-filter matches, page lazily) is likely better than raising a parallel cap. That ties directly into #7, since "how aggressively may we fan out" is the shared rate-limit/concurrency constraint.

Net: answer the realistic-usage question first (how often does the cap actually bite after a sensible pre-filter?), then choose between (a) strengthen + surface pre-filtering, or (b) sequential lazy hydration. Do not just raise the number.