Skip to content

fix(evaluator): ignore invalid LLM judge metrics - #573

Open
Yushu-Li wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
Yushu-Li:fix/finite-llm-judge-metrics
Open

Yushu-Li wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
Yushu-Li:fix/finite-llm-judge-metrics

Conversation

@Yushu-Li

@Yushu-Li Yushu-Li commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

A judge response such as {"quality": NaN} currently enters the weighted metric average and can turn the final combined_score into NaN even when another judge supplies a valid score. Infinity, overflowing JSON exponents and integers too large to convert to float have similar failure modes; a boolean field also becomes a numeric fitness component.

I validate numeric feedback per field before adding its value or weight. Non-finite or unrepresentable values are skipped, so valid peer scores and other valid fields in the same response survive. Boolean fields remain artifacts alongside other nonnumeric feedback. Finite scores keep their existing range and weighting.

Validation:

  • Six additional test methods cover both judge positions, NaN/Infinity/overflow, boolean artifacts, measured-fitness fallback and the full evaluate_program() combination path; five methods fail on the unchanged implementation (14 failing subcases), while the finite-range control passes.
  • All 16 focused ensemble/disabled-judge tests pass, with mocked responses and no model calls.
  • All 1003 local unit tests pass; Black, isort and whitespace checks pass.

This is limited to LLM judge feedback; it does not introduce general validation of user evaluator metrics.

I scoped this contribution and used Codex for implementation, regression tests and refinement.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant