Skip to content

Conversation

@vasil-pashov
Copy link
Collaborator

@vasil-pashov vasil-pashov commented Dec 30, 2025

Reference Issues/PRs

What does this implement or fix?

Peakmem params are tuned for the machine that runs the tests. It has 16 CPU threads and 24 IO threads. Having too much segments on leads to variability of ~20% in the processing pipeline because of the scheduling. 3_000_000 rows are 30 segments, a bit more than the number of IO threads but still not enough to cause large variability

Any other comments?

Checklist

Checklist for code changes...
  • Have you updated the relevant docstrings, documentation and copyright notice?
  • Is this contribution tested against all ArcticDB's features?
  • Do all exceptions introduced raise appropriate error messages?
  • Are API changes highlighted in the PR description?
  • Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

@vasil-pashov vasil-pashov added no-release-notes This PR shouldn't be added to release notes. patch Small change, should increase patch version labels Dec 30, 2025
Copy link
Collaborator

@poodlewars poodlewars left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How have you tested that this reduces the variability?

Needs a rebase - it clashes with my change that removes to 20M rows case.

@vasil-pashov
Copy link
Collaborator Author

How have you tested that this reduces the variability?

I had a script that ran the benchmark 1000 times and gathered the lowest and highest reported memory.

@vasil-pashov vasil-pashov force-pushed the vasil.pashov/resample_peakmem branch from 24fce06 to 73356c5 Compare January 5, 2026 09:19
@vasil-pashov vasil-pashov merged commit 54dfdd7 into master Jan 6, 2026
388 of 389 checks passed
@vasil-pashov vasil-pashov deleted the vasil.pashov/resample_peakmem branch January 6, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-release-notes This PR shouldn't be added to release notes. patch Small change, should increase patch version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants