Skip to content

pipeline: don't preallocate bucket queues - #4800

Merged
blotus merged 2 commits into
crowdsecurity:masterfrom
AdzerKI:queue-no-prealloc
Nov 10, 2026
Merged

blotus merged 2 commits into
crowdsecurity:masterfrom
AdzerKI:queue-no-prealloc

Conversation

@AdzerKI

@AdzerKI AdzerKI commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What and why

NewQueue allocated the whole queue (capacity or cache_size events) for every new bucket, so queue memory grew with live buckets times their capacity, not with the events they hold. With a cache_size: 150 scenario and ~15.7k live buckets, these allocations alone held ~940 MiB of heap. The queue now starts empty and grows in Add, which already caps it at L. Buckets that fill up end with the same array size as before: the old code also reallocated on the L+1-th event.

Fixes #4799

How it was tested

# go test ./pkg/pipeline/ ./pkg/leakybucket/...
ok  	github.com/crowdsecurity/crowdsec/pkg/pipeline	0.008s
ok  	github.com/crowdsecurity/crowdsec/pkg/leakybucket	9.096s

# golangci-lint v2.13.2 run ./pkg/pipeline/...
0 issues.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

@AdzerKI: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

  • /kind feature
  • /kind enhancement
  • /kind refactoring
  • /kind fix
  • /kind chore
  • /kind dependencies
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

@AdzerKI: There are no area labels on this PR. You can add as many areas as you see fit.

  • /area agent
  • /area local-api
  • /area cscli
  • /area appsec
  • /area security
  • /area configuration
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.81%. Comparing base (849c7a2) to head (d48749b).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4800      +/-   ##
==========================================
- Coverage   71.82%   71.81%   -0.01%     
==========================================
  Files         545      545              
  Lines       37163    37163              
==========================================
- Hits        26691    26689       -2     
- Misses      10471    10473       +2     
  Partials        1        1              
Flag Coverage Δ
bats 45.27% <100.00%> (+<0.01%) ⬆️
unit-linux 49.78% <100.00%> (-0.01%) ⬇️
unit-windows 37.79% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdzerKI

AdzerKI commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/kind fix
/area agent

Comment thread pkg/pipeline/queue_test.go Outdated
"github.com/stretchr/testify/require"
)

func TestQueueGrowsOnDemand(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this test, it does not provide anything of value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in d48749b.

Comment thread pkg/pipeline/queue.go
}
q := &Queue{
Queue: make([]Event, 0, l),
Queue: make([]Event, 0),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a one line comment to make sure we don't reintroduce this in the future.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in d48749b.

Signed-off-by: AdzerKI <silver_cat@list.ru>
@blotus
blotus merged commit a78fba4 into crowdsecurity:master Nov 10, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bucket queues preallocate all their slots when the bucket is created

2 participants