Improve evals

[dgageot]

• Validate LLM as a judge upfront
• Disable thinking for llm as a judge
• Remove duplicated creation of the judge provider
• Remove handoffs from evals

[docker-agent]

Review Summary

Assessment: 🟡 NEEDS ATTENTION

This PR improves evaluation robustness by validating the judge model upfront and streamlining error handling. The changes are well-structured and achieve the stated goals of failing fast on configuration issues.

Key Finding

Medium Severity: The new error handling in CheckRelevance changes the system's fault tolerance behavior. While the fail-fast approach is intentional for catching configuration errors early, it also means transient errors (network blips, rate limits) will abort entire evaluations rather than being retried or logged as warnings.

Summary

• ✅ Judge validation added upfront (good defensive programming)
• ✅ Thinking disabled for judge providers (appropriate for structured evaluation)
• ✅ Duplicate judge provider creation removed
• ⚠️ Error handling now fails fast on any error, including transient ones
• ✅ Progress display improvements
• ✅ Handoff metrics removed

The code quality is good overall. The one concern is whether the reduced fault tolerance for transient errors is acceptable for your use case.