Add unit tests for scoring.score() dispatch + fix _wis bug#52
Merged
jc-macdonald merged 2 commits intomainfrom Apr 9, 2026
Merged
Add unit tests for scoring.score() dispatch + fix _wis bug#52jc-macdonald merged 2 commits intomainfrom
jc-macdonald merged 2 commits intomainfrom
Conversation
Test all 8 supported metrics: crps, wis, interval, energy, rmse, mae, coverage, brier. Known-answer tests for rmse, mae, and coverage; smoke tests (finite float) for scoringrules-backed metrics. Test ValueError on unknown metric. WIS test is xfail(strict=True) — _wis passes 4 positional args to sr.weighted_interval_score which expects 5 (missing median). Closes #5
_wis was passing only 4 positional args (truth, predictions[..., 0], predictions[..., 1], alpha) but sr.weighted_interval_score expects 5: (obs, median, lower, upper, alpha). predictions now expected to have shape (..., 3) with columns [median, lower, upper]. Remove xfail marker from WIS test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unit tests for
score()dispatch — exercises all 8 supported metrics and the unknown-metric error path. Also fixes a bug in_wis.Closes #5
Changes
Commit 1: dispatch tests
ValueErrorwith "Unknown metric" messageCommit 2: fix
_wisbug_wispassed 4 positional args tosr.weighted_interval_scorewhich expects 5:(obs, median, lower, upper, alpha).predictionsnow expected to have shape(..., 3)with columns[median, lower, upper].Stats
tests/test_scoring.py(59 total)scoring.pycoverage: 44% → 98%just cipasses (format, lint, mypy strict, coverage)