fix(pipelines): reserve memory for TiDB ghpr build - #5069
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
![ti-chi-bot[bot]](https://avatars.githubusercontent.com/in/214286?s=60&v=4){kind=small}
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary
This PR increases the memory request for the TiDB ghpr_build Jenkins pod from 16Gi to 32Gi to prevent eviction due to node memory pressure, while keeping the memory limit and Bazel concurrency unchanged. The change is minimal and focused, addressing a real stability issue identified from a failed build. Overall, the approach is straightforward and appropriate for the problem described.
Code Improvements
- File:
pipelines/pingcap/tidb/latest/ghpr_build/pod.yaml(line 14)- The increase from 16Gi to 32Gi for the memory request is a direct fix addressing memory eviction issues. However, consider also adding a short comment above the resource block explaining the reason for this higher memory request, so future maintainers understand the rationale without looking up the PR. For example:
# Increased memory request to 32Gi to prevent eviction during build (see PR #XYZ) resources: requests: cpu: "2" memory: 32Gi
- This small addition improves maintainability and knowledge sharing.
- The increase from 16Gi to 32Gi for the memory request is a direct fix addressing memory eviction issues. However, consider also adding a short comment above the resource block explaining the reason for this higher memory request, so future maintainers understand the rationale without looking up the PR. For example:
Best Practices
-
Testing coverage:
- The PR description mentions running
.ci/verify-k8s-pod-yaml.shand other scripts, which is good to validate YAML correctness and policy compliance. However, consider adding a note or automated check to monitor actual pod memory usage over time to proactively catch resource request mismatches.
- The PR description mentions running
-
Documentation:
- Add a reference or link to the Jenkins build logs (already partially included in the description) inside the pod.yaml as a comment, so it's easier to trace why this change was made.
No critical issues identified given the scope and nature of the change. The patch is minimal and suitable for the problem described.
Jenkins Replay Status
PR validation: pre-commit, Pod YAML, PR content policy, secret scan, and storage class policy all passed. Summary: 0 success, 0 failure, 1 skipped. |
1 participant
Summary
Root cause
Jenkins staging build #12 was evicted by kubelet during make bazel_build because the node was under memory pressure. The golang container was using 23,219,056Ki, above its 16Gi request. Reserving 32Gi covers the observed usage and aligns this build with the existing pull_build_next_gen template.
Failed build: https://do.pingcap.net/jenkins-staging/job/pingcap/job/tidb/job/ghpr_build/12/
Tests
A Jenkins replay will be tracked in a PR comment.