Skip to content

RankSelectionScorer protocol for rank-selection method evaluation #40

@jc-macdonald

Description

@jc-macdonald

Summary

Implement a Scorer protocol specialised for rank-selection methods. Given a batch of simulation runs (each with a true rank and a selected rank), it returns a metrics dictionary:

{
    "exact_rate": float,   # fraction where selected == true
    "mae": float,          # mean |selected - true|
    "over_rate": float,    # fraction where selected > true
    "under_rate": float,   # fraction where selected < true
    "type_i_rate": float,  # (null scenarios) fraction selecting rank > 0
    "power_rate": float,   # (signal scenarios) fraction selecting rank >= true
}

Motivation

pp-eigentest's analysis/metrics.py already computes these via summarize_selection() and summarize_batch_with_time(). This protocol formalises that interface so trade-study can orchestrate rank-selection sweeps through run_grid() / Study without coupling to pp-eigentest internals.

Design

  • Conforms to the existing Scorer protocol in protocols.py
  • Input: list of (true_rank: int, selected_rank: int) pairs (or structured array)
  • Output: dict[str, float] of the metrics above
  • Should support scenario-equal weighting (aggregate across scenario groups with equal weight per group)

Acceptance criteria

  • RankSelectionScorer class implementing Scorer protocol
  • Computes all 6 metrics listed above
  • Optional scenario_groups parameter for scenario-equal aggregation
  • Unit tests with known inputs/outputs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions