TRT-1912: Treat flake as failures in component readiness #2172
Conversation
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
bot
added
the
do-not-merge/hold
|
@xueqzhan: This pull request references TRT-1912 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xueqzhan The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-robot
added
the
needs-rebase
openshift-merge-robot
removed
the
needs-rebase
|
@xueqzhan: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I tried it locally, and AFAICS it works great. Flipping the flag changes the results, both in CR and when drilling down to a test result. I haven't reviewed the code and don't know that we want major changes to sippy right before the break, but it would give David something to enjoy fiddling with :) |
| @@ -1102,7 +1099,7 @@ func (c *componentReportGenerator) matchBaseRegression(testID crtype.ReportTestI | |||
| baseRegression = regressionallowances.IntentionalRegressionFor(baseRelease, testID.ColumnIdentification, testID.TestID) | |||
|
|
|||
| // This could go away if we remove the option for ignoring fallback | |||
| if baseRegression != nil && baseRegression.PreviousPassPercentage() > getTestStatusSuccessRate(baseStats) { | |||
| if baseRegression != nil && baseRegression.PreviousPassPercentage(c.FlakeAsFailure) > c.getTestStatusPassRate(baseStats) { | |||
There was a problem hiding this comment.
Shouldn't we update places like these as well so that flakes can be counted as failures when c.FlakeAsFailure is enabled?
There was a problem hiding this comment.
Ahh, nevermind so you are separating out the success, failure and flake and then checking the flag in getPassRate
/hold for now.