UI 1st scan always stuck on the same place (aws codeartifact)

[a-khalilov]

Issue search

• I have searched the existing issues and this bug has not been reported yet

Which component is affected?

Prowler UI

Cloud Provider (if applicable)

AWS

Steps to Reproduce

I faced with error of Powler UI on the same point regardless of configuration and settings. I understand that the point is AWS CodeArtifact, but anyway it's fails always on the same place.

1. Installation

VERSION=$(curl -s https://api.github.com/repos/prowler-cloud/prowler/releases/latest | jq -r .tag_name)
curl -sLO "https://raw.githubusercontent.com/prowler-cloud/prowler/refs/tags/${VERSION}/docker-compose.yml"
# Environment variables can be customized in the .env file. Using default values in production environments is not recommended.
curl -sLO "https://raw.githubusercontent.com/prowler-cloud/prowler/refs/tags/${VERSION}/.env"
docker compose up -d

2. Provider AWS
3. Single account
4. Run scan and see error (18%, after 2h 30-40 minutes)

server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

5. Increase EC2 performance to x2gd.2xlarge. Run scan - see the same error on the same point
6. Use customized docker-compose.yml and postgresql.conf files. Run scan - see the same error on the same point.
7. prowler-cli - makes checks in all regions in 3-4 hours without errors. But output csv is massive - 256MB to be opened and work fast in browser.

Expected behavior

Prowler UI scan finish completely.

Actual Result with Screenshots or Logs

• Logs
  worker.log
  postgres.log

• UI Error

How did you install Prowler?

Cloning the repository from github.com (git clone)

Environment Resource

• TimeZone (UTC+4)
• Prowler APP/UI (docker-compose.yml)
• AWS EC2 :

• Instance type -x2gd.2xlarge
• Root volume - 20GB
• Additional EBS volume - 50 GB (Type -gp3, IOPS - 12000, Throughput -750)
• Instance build-in storage - 475 ssd (NVME)

docker-compose.yml

postgresql_configuration_file.txt

OS used

1. Amazon Linux (al2023-ami-2023.10.20260216.1-kernel-6.1-arm64)

Prowler version

5.18.3

Python version

Pip version

Context

AWS CodeArtifact has 3 repos

1. Maven central repo (official )- Total packages - 2175, Total versions - 8935
2. Maven custom repo - Total packages - 7466, Total versions - 441751
3. Npm store repo (official) - Total packages - 5026, Total versions - 17788

[github-actions]

✅ Issue Triage completed successfully!

[github-actions]

AI Assessment [Experimental]: Bug

Component: API (Celery worker executing scan tasks)
Provider: AWS
Severity: high
Complexity: medium
Agent Ready: Ready

Summary

The CodeArtifact service client makes one list_package_versions API call for every package in every repository, without pagination limits. With 441,751 versions across 7,466 packages reported, this generates hundreds of thousands of synchronous API calls during scan initialization, causing massive memory accumulation in the worker process and eventually crashes the PostgreSQL connection.

Extracted Issue Fields

• Reporter version: 5.18.3
• Install method: Docker Compose (git clone)
• Environment: AWS EC2 x2gd.2xlarge, PostgreSQL with 50GB volume

Duplicates & Related Issues

None found — this is the first report of CodeArtifact-specific performance issues.

---

Root Cause Analysis

Symptom

UI scan crashes at 18% completion with PostgreSQL error server closed the connection unexpectedly after 2.5+ hours. CLI scan completes successfully in 3-4 hours. Error occurs consistently when scanning AWS accounts with large CodeArtifact repositories.

Location

• File: prowler/providers/aws/services/codeartifact/codeartifact_service.py
• Function: _list_packages()
• Lines: 54-154 (specifically lines 87-115 — list_package_versions calls)

Cause

The _list_packages() method iterates through ALL packages from list_packages paginator and makes a synchronous, non-paginated list_package_versions() API call for each package (lines 87-115).

Per the reporter's context:

• 3 repositories
• Total: 7,466 packages + 2,175 packages + 5,026 packages = 14,667 packages
• Maven custom repo alone has 441,751 versions across 7,466 packages

This means the service client makes ~14,667 API calls during initialization, and list_package_versions() without pagination can return tens of thousands of versions per package in memory. With 441K versions in one repo, this causes:

1. Massive memory growth in the Celery worker process as packages are accumulated in the packages list (line 67)
2. Long blocking operation preventing progress updates, causing health check timeouts
3. PostgreSQL connection timeout — the worker holds the DB connection idle while making tens of thousands of AWS API calls
4. Worker process OOM or connection drop before findings are ever written to the database

The code retrieves full version lists but only uses versions[0] (the latest version) — this is a classic N+1 query pattern at the AWS API level, fetching orders of magnitude more data than needed.

Why CLI succeeds but UI fails

• CLI: Writes findings directly to files (CSV/JSON), no database connection to timeout, less memory-constrained
• UI/API: Holds PostgreSQL connection during entire scan, subject to connection timeouts and worker memory limits in containerized environment

Coding Agent Plan

Required Skills

Load these skills from AGENTS.md before starting:

• prowler-provider — Service client patterns and conventions
• prowler-test-sdk — SDK testing with pytest and moto
• pytest — Test fixtures, mocking, parametrize
• tdd — Test-Driven Development workflow (write tests first)

Test Specification

Write tests FIRST (TDD). The skills contain all testing conventions and patterns.

Test Scenario	Expected Result	Must FAIL today?
CodeArtifact repository with 100+ packages	Service initializes in <5 seconds, only fetches first page/max 100 versions per package	Yes (currently fetches all versions)
Package with 10,000+ versions	Only latest version info retrieved, memory usage <10MB	Yes (currently loads all 10K versions into memory)
Multiple repositories with different package counts	Each repository processed without accumulating unbounded memory	Yes (currently accumulates all package data in memory)

Test location: tests/providers/aws/services/codeartifact/codeartifact_service_test.py
Mock pattern: Moto @mock_aws with mocked list_packages and list_package_versions responses configured to return large datasets

Fix Specification

1. Add MaxResults=1 parameter to list_package_versions() calls (lines 87-115)

   • File: prowler/providers/aws/services/codeartifact/codeartifact_service.py
   • Change: Add maxResults=1 parameter to both list_package_versions() calls (with and without namespace)
   • Rationale: Code only uses versions[0] (latest version) — API supports limiting results to avoid fetching thousands of versions unnecessarily
   • Impact: Reduces data transferred from AWS API by 99%+ for repos with many versions

2. Remove paginator for list_package_versions (since we only need 1 result)

   • File: prowler/providers/aws/services/codeartifact/codeartifact_service.py
   • Change: Continue using direct regional_client.list_package_versions() call (not paginator) since maxResults=1 returns what we need
   • Rationale: Paginators are unnecessary when requesting a single item

3. Add pagination/limiting to list_packages if needed (currently uses paginator but may fetch thousands of packages)

   • File: prowler/providers/aws/services/codeartifact/codeartifact_service.py
   • Change: Consider adding maxResults parameter to list_packages paginator if scanning repos with 10,000+ packages
   • Rationale: Even with efficient version fetching, iterating 50K+ packages may take too long for UI scans
   • Note: This is optional — reporter has ~15K total packages which should be manageable with the version fix

Service Client Changes (if needed)

None — the Pydantic models already support the data structure. The fix only changes API call parameters, not the data model.

Acceptance Criteria

• list_package_versions() calls include maxResults=1 parameter
• Test with mocked large repository (10,000 packages, 50,000 versions each) completes in <10 seconds
• Memory usage during CodeArtifact service init does not exceed 100MB for large repos
• Existing functionality preserved: check still has access to latest version info
• All existing tests pass (pytest tests/providers/aws/services/codeartifact/ -x)
• New performance test passes demonstrating bounded memory/time

Files to Modify

File	Change Description
prowler/providers/aws/services/codeartifact/codeartifact_service.py	Add maxResults=1 to list_package_versions() calls (lines 87-115)
tests/providers/aws/services/codeartifact/codeartifact_service_test.py	Add performance test with large mocked dataset

Edge Cases

• Repository with zero packages (already handled)
• Package with zero versions (code handles with if latest_version_information.get("versions"))
• Package with namespace vs. without namespace (both code paths need the fix)
• Multiple regions with CodeArtifact repositories (fix applies per-region)
• API throttling with high package counts (existing retry logic should handle)

---

Note for maintainers: While this assessment identifies the root cause and provides a fix plan, the actual implementation should be reviewed for:

1. Whether maxResults=1 is sufficient or if sorting by PUBLISHED_TIME + maxResults=1 guarantees the latest version
2. Whether to add optional filtering to exclude certain repos from scans (e.g., via scan filters)
3. Consider adding progress logging every N packages to help diagnose similar issues in future

AI generated by Issue Triage

[jfagoagas]

Hi @a-khalilov In the past we implemented some changes to prevent this but seems either something changed on the AWS side or those changes were not enough. We don't have an AWS CodeaArtifact environment large enough to test this. Could you please test the above PR and let us know if that works for you?

Theoretically calling list_package_versions with sortBy="PUBLISHED_TIME", maxResults=1" should make the service to retrieve just the latest version which is was Prowler needs to run the check.

Thanks for using Prowler 🚀

[jfagoagas]

Hi @a-khalilov were you able of testing the above PR?

Thanks!

[a-khalilov]

Hi @jfagoagas
First of all thanks for attention to my issue.
I tried to test it - but faced so many blockers/issues trying to build and install Prowler app manually (not docker-compose) according to instruction.
Still on that point, need a little bit more time to understand that process.

[jfagoagas]

Hi @a-khalilov please tell me which issues you faced. Another option to test it is to install it with pip install prowler and then prowler aws --service codeartifact

If you prefer you could join our Prowler Community Slack at https://goto.prowler.com/slack