gpt 5.5 review#3
Open
AndyDu0921 wants to merge 8 commits into
Open
Conversation
… + registry v2 + proposal-only self-evolution - PredictionKernel: deterministic fusion kernel shared by async/sync/CLI, wraps run_core_fusion() with immutable context + feature snapshot + provenance - ScoreMatrixFusion: extracted shared DC+NegBin+Weibull score matrix fusion, eliminating ~130 lines of duplicate code from predict_match/predict_sync - EvaluationRegistry v2: strict/diagnostic/rejected three-tier sample classification with horizon_bucket, leakage_status, as_of_time - ShadowCandidateModels: 8 registered candidates (3 implemented: dynamic_dixon_coles, dynamic_bivariate_poisson, bayesian_weighted_dynamic), all shadow-only, no production mutation - CandidateExperiments: paired Brier/LogLoss/RPS/ECE evaluation with Bootstrap CI, three-tier gate (rejected/needs_more_evidence/candidate_only), walk-forward time safety - ModelChangeProposals: generic proposal ledger for weights/calibrator/model/feature_rule, idempotent by SHA-256 fingerprint, coexists with model_weight_proposals - PlayerAvailability: shadow xG modifier from availability records, zero-impact on missing data - Accuracy Engine ORM models: FeatureSnapshot, ExperimentRun, CandidatePrediction, ModelChangeProposal - ProcessEvaluator: fix shot_volume_delta false zero-error when pre-match expected shots unavailable - PredictionPipeline: integrate PredictionKernel in both predict_match() and predict_sync() paths Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…accuracy engine tables) - d4e5f6a7b8c9: add score-level fields to prediction_learning_log (score_log_loss, score_exact_hit, score_top3_hit, dc/negbin/weibull_score_log_loss) - e5f6a7b8c9d0: add V4.8 accuracy-engine tables — feature_snapshots, experiment_runs, candidate_predictions, model_change_proposals - a8b9c0d1e2f3-b1c2d3e4f5a6-c7d8e9f0a1b2-d3a9d4c1f2ab: made idempotent with _has_table/_has_column/_has_index guards for safe local SQLite drift recovery Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ 17 new tests (494 passed) Scripts: - build_evaluation_registry.py: CLI to build leak-aware evaluation registry - run_candidate_experiments.py: paired champion/challenger experiment runner - diagnose_evaluation_registry.py: read-only sample quality diagnostics - generate_model_change_proposals.py: bridge experiment results to proposal ledger - run_accuracy_experiments.py: unified accuracy experiment orchestrator - backtest_full_pipeline.py, collect_stacking_training_data.py, grid_search_score_params.py: add V4.8 deprecation notices pointing to new experiment runner Tests (17 new): - test_prediction_kernel: kernel matches run_core_fusion() output - test_shadow_candidate_models: dynamic DC, Dirichlet calibration, unknown candidate rejection - test_evaluation_registry: schedule-only finished rows, fallback snapshot, local schema alignment - test_model_candidates: all shadow_only, registration, KeyError on unknown - test_model_change_proposals: insufficient logs rejected, idempotent persist, walk-forward required - test_player_availability: empty dataset, key forward absence, goalkeeper absence - test_score_matrix_fusion: NegBin source addition, single-source fallback - test_accuracy_engine_schema: 5 audit tables exist after migration - test_process_evaluator: shot_volume_delta reason field Backend: 494 passed, 4 skipped Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- version.py: VERSION 4.7.0-alpha → 4.8.0-alpha, BUILD_NAME updated - README: test badge 483→494, registry sample counts, shadow candidate system, bilingual sync - CHANGELOG: V4.8.0-alpha entry — Accuracy Engine, PredictionKernel, shadow experiments, proposal-only self-evolution - MAGIC_NUMBERS: V4.8 registry schema version, shadow candidate configs - PRD + Technical Audit: version references updated - .gitignore: add *.db.backup-* for local DB backup exclusion Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…didate tournament ## V4.9.0-alpha — 2026-07-05 ### Added - **Accuracy Data OS repair report**: evaluation_registry_repair_report.v1 produces deterministic repair actions for diagnostic/rejected samples without creating snapshots, probabilities, artifacts, or reports. - **Structured pre-match intelligence**: information_state_signals.v1 converts player availability into source/time/confidence-scored shadow signals, and excludes future evidence from strict features. - **FeatureSnapshot v2**: strict feature payloads now include information-state signals, player availability shadow payloads, schedule context, and data quality fields, while still excluding actual goals. - **V4.9 shadow candidates**: dynamic Bayesian weighted goal model, international covariate hybrid, Dirichlet calibration, proper-scoring stacking, and player availability shadow are available as shadow/proposal-only candidates. - **Data-repair proposal type**: model_change_proposals now records data-repair, calibrator, and stacking proposal categories separately. - **Legacy experiment wrappers**: old backtest/grid/stacking scripts delegate to unified accuracy experiment runner; no longer overwrite old artifacts. ### KO Weight Rebalance (V4.8.1 → V4.9) - Elo: 0.22→0.24 (63% KO correct, stable anchor) - Pi: 0.18→0.22 (69% KO correct, best non-market, +0.18 marginal) - Weibull: 0.10→0.05 (38% KO correct, bimodal, unreliable) - market_max: 0.30→0.35 (79% KO correct, strongest component) ### Engine Tuning (from R32 full review) - xG calibration factor: 1.20→1.35 (KO stage 50-80% underestimation) - KO draw floor: 0.12→0.18 (4/4 KO draws missed at previous floor) - KO draw guard thresholds: warning 0.22→0.20, max_blend 0.60→0.80 - Market consensus: boost 0.05→0.08, cap 0.40→0.45, CV 0.02→0.03 - Score matrix weights: DC 0.40→0.45, NegBin 0.35→0.38, Weibull 0.25→0.17 - Weibull draw_noise: skip→shadow (KO draw rate 25% makes signal valuable) - BacktestGate.knockout_mode(): relaxed params for KO (8 min samples) - KO learning weights: WEIBULL_EXTREME 0.80→0.90, LUCKY 0.30→0.50 ### Pipeline Improvements - Shared PredictionKernel adapter eliminates inline component construction duplicate (~100 lines DRY'd across predict_match + predict_sync) - KO-aware draw floor (0.18 KO vs 0.12 group) applied per-stage - Post-calibration KO draw guard at step 13.7b - Pre-match snapshot fallback in run_postmatch_complete.py - Non-UUID match_id support in verification pipeline - --allow-placeholder-runs escape hatch for legacy backfill ### Verified - Backend tests: 516 passed, 4 skipped - Evaluation registry: 81 total samples, 25 strict, 46 diagnostic, 10 rejected - Shadow experiments: all candidates remain proposal-only (<30 strict samples) - Feature snapshots: 66 audit rows, persistence is idempotent Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## PAR-FRA Post-Match (July 5 2026) - Result: Paraguay 0-1 France (Mbappe 70' penalty, VAR) - **GOOD_PREDICTION** — Brier=0.0573 (KO best ever) - **Exact score HIT**: 0-1 (P=23.8%) — first KO exact hit - **7/7 components + market all correct** — first unanimous KO consensus - Weibull best: Brier=0.0040 (95.4% France, single-component record) - Pi worst: Brier=0.4441 (33.4% Paraguay, overestimated upset) - xG MAE=0.4274, Dominance Index=0.8205 (France dominant) - Learning Weight=1.00, Process=SUPPORTED - Calibration extreme (-8.2pp PAR) was over-aggressive but didn't affect direction - DB writes: postmatch_process_eval + prediction_learning_log + postmatch_eval + match_team_statistics ## CAN-MAR Post-Match (other AI) - Result: Canada 0-3 Morocco - Direction correct, Brier=0.2673 - run_postmatch_complete.py executed via other AI ## run_postmatch_complete.py Improvements (other AI) - Added _is_uuid_like() helper for non-UUID match IDs - FIFA JSON API payload parsing as score verification fallback - Official competition tier for FIFA source URLs - Snapshot lookup gated by UUID check for non-UUID match IDs ## Calibrator Rebuilt - calibrator_wc.json: 21 WC samples, ECE=0.1466 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…predictions ## Fix: match_id ForeignKey type mismatch (model → service → consumer) - match_result_verification.py: Mapped[UUID] → Mapped[str](String(36)) Fixes FK constraint failure for non-UUID IDs (WC26 schedule uses "195" etc.) - result_verification.py: All method signatures UUID → str, add _match_id_key() Normalizes UUID/hex/numeric string IDs into the canonical matches.id format - run_postmatch_complete.py: Remove _verification_match_uuid() + uuid5 dead code No longer needed — verification now accepts any string match_id ## Golden test fixtures regenerated (V4.7.0 → V4.9.0) - balanced_sync, strong_vs_strong, strong_vs_weak, wc_md3_motivation - Reflects V4.8.1 KO weight config + xG 1.35 + NegBin r=8.0 parameter changes - All 516 tests pass ## PAR-FRA audit correction - FIFA API cross-reference: IdMatch=400021533, MatchNumber=89 (not 92) - Corrected: xG source (FOX boxscore), yellow cards (PAR 1, FRA 3 per FIFA) ## New content - 2026-07-05 article: Brazil-Norway + Mexico-England R16 prediction analysis - Brazil vs Norway prediction report (match #197) - Mexico vs England prediction report (match #198) ## Security review - No credentials, SQL injection, or dangerous execution found - All DB queries use parameterized placeholders Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## 核心主题 ### 1. V4.9 合规策略放松 (6 files) - 市场赔率/博彩商名称现在允许作为研究证据引用 - 投注建议、带单、保证性结论仍然封锁 - 全链路一致更新: LLM prompt → 过滤层 → 输出策略 → 审计脚本 - public_safety_filter: CREATOR_SAFE_FORBIDDEN 从21个通用博彩词缩减为18个具体投注建议词 - output_policy: creator_safe 模式保留 market_applied/weight/divergence/odds_info - analysis_prompts + analysis router + article_generator 统一新规则措辞 ### 2. 评估注册表修复 (2 files) - evaluation_registry: 修复赛后快照泄露bug — 新增 _should_use_prediction_snapshot_primary() 当 pre_match_snapshot 是赛后补拍(延时回填)时，优先使用干净的 prediction_snapshot - evaluation_registry_repair: v2 schema — 新增 priority/blocking_level/repair_order/promotability_reason + 提取 HARD_BLOCK_REASONS/ACTION_PRIORITY/ACTION_REPAIR_ORDER 常量 + 新增 _row_priority/_blocking_level/_promotability_reason/_action_groups 辅助函数 ### 3. 模型变更提案状态管理 (1 file) - model_change_proposals: 新增 update_model_change_proposal_status() + ALLOWED_PROPOSAL_STATUSES 验证 + MANUAL_APPROVAL_STATUSES (shadow/promotion需人工确认) + 完整 review_history 审计跟踪，production_mutation=False 保障 ### 4. 准确度实验基础设施 — 新增 14 文件 - accuracy_experiment_preflight: 只读预检门 — DB完整性/严格样本计数/修复机会 - accuracy_todo_backlog: 确定性 P0/P1/P2 改进待办队列 - db_integrity_audit: SQLite FK修复 — 空值规范化/孤立隔离/队名重映射，quarantine审计表 - strict_sample_repair_queue: 本地证据修复分类器 - 配套 CLI 脚本 (5) + 测试 (5) - test_magic_numbers_doc: 验证 MAGIC_NUMBERS.md 与 engine.py 常量一致性 - run_accuracy_experiments.py: 新增 --force 逃生门 + 预检阻断逻辑 ### 5. 文档同步与清理 - MAGIC_NUMBERS.md: V4.9 参数对齐 (xG=1.35, KO draw floor=0.18, Market consensus gate 参数) - COMPLIANCE_AND_OUTPUT_POLICY.md: 完整策略重写 - README/CHANGELOG: 534测试/29严格样本/版本号/功能列表更新 - 删除 3 份过时文档: POSTMATCH_SOP/EXTERNAL_REVIEW_SUMMARY/PRD_ARCHITECTURE_COMPLETE ## 审计结果 - 全量 534 测试通过 (新增 18 测试) - 零回归、零凭据泄露、零SQL注入 - 仅发现非阻塞质量问题: 4处代码重复(_has_table/_norm/_parse_dt), process_gap 计算污染, 12个过时替换项 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Compliance checklist
Test results