Skip to content

Commit 515a7b9

Browse files
fix: prevent div-by-zero in evaluator when base_refusals is 0 (#225)
* fix: prevent div-by-zero in evaluator when base_refusals is 0 When a model refuses all prompts from the start, base_refusals is 0. Return refusals directly in that case so ablations that introduce new refusals are still penalized correctly. * fix: cast refusals to float for type consistency" before hitting commit changes Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent e26da5e commit 515a7b9

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

src/heretic/evaluator.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,9 @@ def get_score(self) -> tuple[tuple[float, float], float, int]:
110110
kl_divergence_scale = self.settings.kl_divergence_scale
111111
kl_divergence_target = self.settings.kl_divergence_target
112112

113-
refusals_score = refusals / self.base_refusals
113+
refusals_score = (
114+
refusals / self.base_refusals if self.base_refusals > 0 else float(refusals)
115+
)
114116

115117
if kl_divergence >= kl_divergence_target:
116118
kld_score = kl_divergence / kl_divergence_scale

0 commit comments

Comments
 (0)