Summary
Implement a Scorer protocol specialised for rank-selection methods. Given a batch of simulation runs (each with a true rank and a selected rank), it returns a metrics dictionary:
{
"exact_rate": float, # fraction where selected == true
"mae": float, # mean |selected - true|
"over_rate": float, # fraction where selected > true
"under_rate": float, # fraction where selected < true
"type_i_rate": float, # (null scenarios) fraction selecting rank > 0
"power_rate": float, # (signal scenarios) fraction selecting rank >= true
}
Motivation
pp-eigentest's analysis/metrics.py already computes these via summarize_selection() and summarize_batch_with_time(). This protocol formalises that interface so trade-study can orchestrate rank-selection sweeps through run_grid() / Study without coupling to pp-eigentest internals.
Design
- Conforms to the existing
Scorer protocol in protocols.py
- Input: list of
(true_rank: int, selected_rank: int) pairs (or structured array)
- Output:
dict[str, float] of the metrics above
- Should support scenario-equal weighting (aggregate across scenario groups with equal weight per group)
Acceptance criteria
Summary
Implement a
Scorerprotocol specialised for rank-selection methods. Given a batch of simulation runs (each with a true rank and a selected rank), it returns a metrics dictionary:{ "exact_rate": float, # fraction where selected == true "mae": float, # mean |selected - true| "over_rate": float, # fraction where selected > true "under_rate": float, # fraction where selected < true "type_i_rate": float, # (null scenarios) fraction selecting rank > 0 "power_rate": float, # (signal scenarios) fraction selecting rank >= true }Motivation
pp-eigentest's
analysis/metrics.pyalready computes these viasummarize_selection()andsummarize_batch_with_time(). This protocol formalises that interface so trade-study can orchestrate rank-selection sweeps throughrun_grid()/Studywithout coupling to pp-eigentest internals.Design
Scorerprotocol inprotocols.py(true_rank: int, selected_rank: int)pairs (or structured array)dict[str, float]of the metrics aboveAcceptance criteria
RankSelectionScorerclass implementingScorerprotocolscenario_groupsparameter for scenario-equal aggregation