Skip to content

perf(RHINENG-24466): add batch-scoped caches for MQ ingestion#3844

Draft
rodrigonull wants to merge 1 commit intomasterfrom
perf/batch-caches
Draft

perf(RHINENG-24466): add batch-scoped caches for MQ ingestion#3844
rodrigonull wants to merge 1 commit intomasterfrom
perf/batch-caches

Conversation

@rodrigonull
Copy link
Copy Markdown
Member

@rodrigonull rodrigonull commented Mar 26, 2026

Overview

This PR is being created to address RHINENG-24466.

Introduce ThreadLocalBatchCache base class and two batch-scoped caches to reduce redundant DB queries during MQ message processing:

  • StalenessCache: eliminates ~3 identical Staleness SELECTs per host
  • UngroupedGroupCache: eliminates 1 Group SELECT per message

Both caches are activated via context managers in _process_batch and automatically cleared at batch boundaries. Values are plain dicts (AttrDict), not ORM objects, to avoid memory accumulation.

PR Checklist

  • Keep PR title short, ideally under 72 characters
  • Descriptive comments provided in complex code blocks
  • Include raw query examples in the PR description, if adding/modifying SQL query
  • Tests: validate optimal/expected output
  • Tests: validate exceptions and failure scenarios
  • Tests: edge cases
  • Recovers or fails gracefully during potential resource outages (e.g. DB, Kafka)
  • Uses type hinting, if convenient
  • Documentation, if this PR changes the way other services interact with host inventory
  • Links to related PRs

Secure Coding Practices Documentation Reference

You can find documentation on this checklist here.

Secure Coding Checklist

  • Input Validation
  • Output Encoding
  • Authentication and Password Management
  • Session Management
  • Access Control
  • Cryptographic Practices
  • Error Handling and Logging
  • Data Protection
  • Communication Security
  • System Configuration
  • Database Security
  • File Management
  • Memory Management
  • General Coding Practices

Summary by Sourcery

Introduce thread-local, batch-scoped caches for MQ ingestion to avoid redundant staleness and ungrouped-group database lookups within a processing batch.

New Features:

  • Add a reusable ThreadLocalBatchCache base class for thread-local, batch-scoped caching.
  • Add StalenessCache and UngroupedGroupCache to cache staleness configs and ungrouped host groups per org within a batch.

Enhancements:

  • Wrap MQ batch processing in staleness and ungrouped-group cache contexts to reduce duplicate database queries.
  • Update staleness and ungrouped-group retrieval routines to first consult the relevant batch cache before hitting the database and to store results in the cache.

Tests:

  • Add unit tests verifying that StalenessCache and UngroupedGroupCache behave as context managers and prevent redundant database queries within a batch.

@github-actions
Copy link
Copy Markdown
Contributor

SC Environment Impact Assessment

Overall Impact:NONE

No SC Environment-specific impacts detected in this PR.

What was checked

This PR was automatically scanned for:

  • Database migrations
  • ClowdApp configuration changes
  • Kessel integration changes
  • AWS service integrations (S3, RDS, ElastiCache)
  • Kafka topic changes
  • Secrets management changes
  • External dependencies

@rodrigonull rodrigonull force-pushed the perf/batch-caches branch 3 times, most recently from ef58793 to 01794c1 Compare March 31, 2026 10:54
Introduce ThreadLocalBatchCache base class and two batch-scoped caches
to reduce redundant DB queries during MQ message processing:

- StalenessCache: eliminates ~3 identical Staleness SELECTs per host
- UngroupedGroupCache: eliminates 1 Group SELECT per message

Both caches are activated via context managers in _process_batch and
automatically cleared at batch boundaries. Values are plain dicts
(AttrDict), not ORM objects, to avoid memory accumulation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant