Skip to content

fix(billing-cost-management-mcp-server): flag COH summaries that reach the 1,000-group limit - #4973

Open
mattkelsey wants to merge 3 commits into
awslabs:mainfrom
mattkelsey:fix/coh-summaries-group-limit
Open

mattkelsey wants to merge 3 commits into
awslabs:mainfrom
mattkelsey:fix/coh-summaries-group-limit

Conversation

@mattkelsey

@mattkelsey mattkelsey commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes

Summary

Changes

ListRecommendationSummaries returns at most 1,000 groups, the ones with the highest estimated savings, and no nextToken for the rest. With group_by=AccountId in an organization where more than 1,000 accounts have recommendations, list_recommendation_summaries fetched every page and reported complete_dataset: true. Callers then presented the top 1,000 accounts as every account, leaving out accounts that had matching recommendations.

  • A result that holds 1,000 groups is now marked complete_dataset: false, with has_more: false, no next_token, and a warning. The warning says groups may be missing and estimated_total_deduped_savings may cover only the returned groups. It also says to group by a dimension with fewer values for totals. It does not suggest a way to fetch the missing groups. total_recommendation_count is left out, as on any partial result.
  • With TagKey: grouping, the warning adds that NoTagKey may be missing, or may also count recommendations whose tag value was left out.
  • The tool description states the limit, and that neither a larger max_items nor a continuation can fetch past it. A continuation call can't count earlier pages, so its last page isn't marked; the description tells callers who page with max_items that 1,000 groups across all pages means the same.

Results under 1,000 groups are unchanged. A result of exactly 1,000 groups can still be complete, so the warning says groups may be missing.

User experience

Before: a 1,000-group summaries result looked complete, so the agent answered "which accounts have X?" with exactly 1,000 accounts and no caveat.

After: the same result leads with complete_dataset: false and a warning. The agent says the list covers the 1,000 groups with the most savings, and takes totals from a coarser grouping such as Region.

Testing

  • 9 new unit tests. 5 fail on the base and pass with the change: a 1,000-group result in one page, 1,000 groups across two pages, max_items=1000, the TagKey: warning, and the served tool description. 4 are regression guards that pass on both: 999 groups stay complete with total_recommendation_count, 999 TagKey: groups stay complete, a continuation is not flagged, and a max_items partial with a token keeps its own warning.
  • Full suite: 4,053 passed. ruff check, ruff format --check and pyright are clean.
  • Served description, checked over stdio through list_tools: the 5 new statements are present with the change and absent on the base, and the description suggests no way to fetch the missing groups.
  • I checked the limit, the missing token, and what the total covers against the service's behavior. The limit itself couldn't be reproduced in a test account, which has far fewer than 1,000 groups.
  • This touches the same description bullets as fix(billing-cost-management-mcp-server)!: request COH savings percentage, coerce boolean filters, fold summaries group_by #4906, so whichever merges second will need a small rebase.

Checklist

  • I have reviewed the contributing guidelines
  • I have performed a self-review of this change
  • Changes have been tested
  • Changes are documented

Is this a breaking change? (Y/N) N

RFC issue number:

Checklist:

  • Migration process documented
  • Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

…h the 1,000-group limit

ListRecommendationSummaries returns at most 1,000 groups, the ones with the
highest estimated savings, and no nextToken for the rest. Grouped by
AccountId in an organization with more than 1,000 accounts that have
recommendations, the tool fetched every page and reported
complete_dataset: true, so callers presented the top 1,000 accounts as
every account.

- A result that holds 1,000 groups is now marked complete_dataset: false,
  with has_more: false, no next_token, and a warning: groups may be
  missing, estimated_total_deduped_savings may cover only the returned
  groups, and the caller should group by a dimension with fewer values for
  totals, or narrow filters (or, for rows and counts, use
  list_recommendations) to cover every group. total_recommendation_count
  is left out, as on any partial result.
- With TagKey grouping the warning adds that NoTagKey may be missing, or
  may also count recommendations whose tag value was left out.
- The tool description states the limit, that a larger max_items or a
  continuation cannot fetch past it, and that a continuation's last page
  is not marked, since it cannot count earlier pages.

Results under 1,000 groups are unchanged.
@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.79%. Comparing base (c6951b9) to head (3740adf).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4973      +/-   ##
==========================================
- Coverage   93.79%   93.79%   -0.01%     
==========================================
  Files        1080     1080              
  Lines       94586    94595       +9     
  Branches    15402    15404       +2     
==========================================
+ Hits        88715    88723       +8     
  Misses       3498     3498              
- Partials     2373     2374       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

)
# ListRecommendationSummaries returns at most this many groups, the ones with the highest
# estimated savings, and no nextToken for the rest.
COH_SUMMARIES_MAX_GROUPS = 1000

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love hard-coding this, but there is not a source we can consume it from.

In botocore model there is max page size, but that is not the same as max groups. No model exists for this

- If the result has complete_dataset: false, it is partial: totals and counts undercount. Call again with a larger 'max_items', continue with 'next_token', or state that the result is partial. Never report the number of returned items as a total count.
- Fields from the first page only (summary totals such as estimated_total_deduped_savings, currency_code, metrics) are not repeated on continuation calls.
- list_recommendation_summaries returns total_recommendation_count (the sum of the groups' recommendation_count) only when every group was fetched; use it as the overall count instead of adding the groups. It is absent on a partial result, a continuation call, or when a group has no recommendation_count.
- list_recommendation_summaries returns at most 1,000 groups, those with the highest estimated savings, and no next_token for the rest, so a larger 'max_items' or a continuation cannot fetch them. A result of 1,000 groups is marked complete_dataset: false: groups (e.g. accounts) may be missing and estimated_total_deduped_savings may cover only the returned groups. For totals, group by a dimension with fewer values (such as Region). To cover every account, narrow 'filters' until fewer than 1,000 groups come back, or, for rows and counts, use list_recommendations with the same filters (its savings are not de-duplicated). A continuation call cannot see earlier pages, so its last page is not marked: when paging with 'max_items', 1,000 groups across all pages means the same.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove "for rows and counts, use list_recommendations with the same filters (its savings are not de-duplicated)"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a different alternative we can give it?

@pputra pputra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaving some comments

Matthew Kelsey added 2 commits October 30, 2026 21:43
… COH group-limit guidance

Review feedback: the tool description and the 1,000-group warning no longer
suggest a way to fetch every group (narrowing filters or calling
list_recommendations); no API returns every account well today. They state
the limit, that the result may be missing groups and its total may cover
only the returned groups, and that totals come from a grouping with fewer
values.
…up-limit

# Conflicts:
#	src/billing-cost-management-mcp-server/tests/tools/test_cost_optimization_hub_tools.py
wangyuhere
wangyuhere previously approved these changes Oct 30, 2026
@mattkelsey

Copy link
Copy Markdown
Contributor Author

Had to rebase sorry @wangyuhere

@pputra

pputra commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

no permission to approve, but LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants