Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,19 @@ Numerical changes are marked with [NUMERICAL].

## [Unreleased]

## [0.3.0] - 2026-03-24

### Fixed
- [NUMERICAL] `ts_covariance`: unified to ddof=0 (population), consistent with all other
variance/std operators. Fixes the broken identity `cov/(std_x*std_y) == corr` which
previously had ~5-20% error due to mixed ddof. Cross-validated against numpy (diff < 1e-15).

### Changed
- OPERATORS.md rewritten as pure operator reference manual (signatures, behavior, edge cases)
- Design rationale moved to CLAUDE.md Section 4.1 (developer-facing)
- Fixed incorrect signatures in docs: `trade_when`, `scale`, `bucket`
- Fixed README example code to use only columns present in sample data

## [0.2.0] - 2026-03-23

### Added
Expand All @@ -24,7 +37,7 @@ Numerical changes are marked with [NUMERICAL].

### Fixed
- [NUMERICAL] `ts_product`: silently returned null for negative inputs; now correctly handles negative values via sign-magnitude decomposition
- [NUMERICAL] `ts_covariance`: used ddof=0 (population) inconsistent with `ts_corr` (ddof=1); aligned to ddof=1 (sample)
- [NUMERICAL] `ts_covariance`: added explicit ddof parameter (was using Polars default)
- [NUMERICAL] `divide()`: no zero-denominator protection; now returns null where abs(divisor) < 1e-10
- [NUMERICAL] `inverse()`: no zero protection; now returns null where abs(x) < 1e-10
- `ts_regression` rettype=7 (MSE): implicit Inf-to-null on window=2; now has explicit guard
Expand Down
102 changes: 47 additions & 55 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ elvers/
tests/
conftest.py Test fixtures (make_ts, make_factor)
test_*.py One test file per operator module
OPERATORS.md Operator specification: numerical conventions, per-operator behavior, design rationale
CLAUDE.md Development standards (this file)
```

---
Expand Down Expand Up @@ -92,13 +94,34 @@ git push -u origin fix/bug-name

### 4.1 Numerical Correctness (Highest Priority)

Operator behavior reference: [OPERATORS.md](OPERATORS.md).
The rules below are for writing new code:

- All divisions MUST have explicit zero guards:
`pl.when(denom.abs() < 1e-10).then(None).otherwise(num / denom)`
- NEVER rely on the Factor constructor's implicit Inf-to-null conversion as normal logic flow
- Statistical convention: ddof=0 (population) for std/variance, ddof=1 (sample) for corr/cov.
This is consistent across the entire library.
- Null semantics: null propagates naturally through Polars expressions. Boundary cases
(zero denominator, constant window, insufficient data) must be handled explicitly.
(zero denominator, constant window, insufficient data) must be handled explicitly

#### Design Decisions (rationale for current conventions)

- **NaN/Inf unified to null**: eliminates the NaN-infection problem (`NaN + 1 = NaN`)
that silently corrupts downstream computations. The Factor constructor converts on
creation so the entire library operates on a single missing-value type.
- **ddof=0 everywhere**: rolling windows and cross-sections operate on the full observed
population, not a sample from a larger one. ddof=0 is semantically correct and avoids
n=1 division-by-zero (ddof=1 divides by n-1=0).
- **ts_corr/ts_autocorr use ddof=1 internally**: Polars `rolling_corr(ddof=0)` has a bug
where ddof only applies to the covariance numerator, not the variance denominator,
producing values outside [-1, 1]. Reported: https://github.com/pola-rs/polars/issues/16161.
Correlation is ddof-invariant (cancels in ratio), so ddof=1 output is correct.
- **rank range (0, 1] not [0, 1]**: a rank of 0 is ambiguous (could mean "missing" or
"lowest"). Strictly positive range ensures every ranked value is distinguishable from null.
- **Zero guard threshold 1e-10**: conservative enough to catch near-zero denominators,
small enough not to interfere with legitimate small values in financial data.
- **ts_product sign-magnitude decomposition**: naive `exp(sum(log(x)))` fails for negative
inputs because `log(x)` is undefined for x < 0. Separating sign and magnitude handles
this correctly.

### 4.2 Operator Writing Rules

Expand Down Expand Up @@ -178,7 +201,8 @@ negative values. Previously returned null, now returns correct product.

## 7. Code Review Rules

- All PRs require at least one reviewer approval before merge
- When the team has multiple developers, enable "Require approvals" in branch protection
- Currently (single-developer mode): CI status checks are required, review approval is optional
- Reviewer must verify:
1. Tests pass and cover the change
2. Numerical correctness (manually verify at least one expected value)
Expand Down Expand Up @@ -220,26 +244,26 @@ pytest tests/ -v
ruff check elvers/

# 2. Update version number (single source of truth)
# Edit elvers/__init__.py: __version__ = "0.2.0"
# Edit elvers/__init__.py: __version__ = "X.Y.Z"

# 3. Update CHANGELOG.md
# Move items from [Unreleased] to [0.2.0] - YYYY-MM-DD
# Move items from [Unreleased] to [X.Y.Z] - YYYY-MM-DD

# 4. Commit the release
git add elvers/__init__.py CHANGELOG.md
git commit -m "release: v0.2.0"
git commit -m "release: vX.Y.Z"
git push origin dev

# 5. Create PR: dev -> main on GitHub
# Title: "release: v0.2.0"
# Title: "release: vX.Y.Z"
# Wait for CI to pass and review approval
# Squash merge on GitHub

# 6. Tag on main (after PR merged)
git checkout main
git pull origin main
git tag v0.2.0
git push origin v0.2.0
git tag vX.Y.Z
git push origin vX.Y.Z

# 7. Automated (triggered by tag push):
# - CI runs full test suite again
Expand All @@ -251,14 +275,14 @@ git push origin v0.2.0

### What Happens Automatically

When you push a tag like `v0.2.0`:
When you push a tag like `vX.Y.Z`:

1. `.github/workflows/publish.yml` triggers
2. Runs full test suite on Python 3.10-3.13 (safety net)
3. If tests pass: builds package, publishes to PyPI
4. Creates a GitHub Release page at github.com/quantbai/elvers/releases
with auto-generated release notes from commit messages
5. Users can now `pip install elvers==0.2.0`
5. Users can now `pip install elvers==X.Y.Z # specific version`

### What You See on GitHub After Release

Expand All @@ -268,29 +292,15 @@ When you push a tag like `v0.2.0`:

---

## 10. One-Time Setup (for repository admin)

### PyPI Trusted Publisher (required for automated publishing)

1. Go to https://pypi.org -> Your projects -> elvers -> Publishing
2. Add a new publisher:
- Owner: quantbai
- Repository: elvers
- Workflow name: publish.yml
- Environment: (leave blank)
## 10. Setup

### GitHub Branch Protection (strongly recommended)
### Infrastructure (already configured)

1. GitHub repo -> Settings -> Branches -> Add rule
2. Branch name pattern: `main`
3. Enable:
- "Require a pull request before merging"
- "Require approvals" (1 minimum)
- "Require status checks to pass before merging"
- Select required status check: "test"
4. Save changes
- PyPI Trusted Publisher: configured for quantbai/elvers -> publish.yml
- GitHub Branch Protection on main: require PR, require CI status checks
- GitHub Actions: ci.yml (push/PR) + publish.yml (tag-triggered release)

### Local Development Setup (every developer)
### Local Development Setup (every new developer)

```bash
git clone https://github.com/quantbai/elvers.git
Expand All @@ -302,33 +312,15 @@ pre-commit install

---

## 11. Commands Reference
## 11. Quick Reference

```bash
# === Setup ===
pip install -e ".[dev]" # Install with dev dependencies
pre-commit install # Install git hooks

# === Daily Development ===
pip install -e ".[dev]" # Setup
pre-commit install # Git hooks
pytest tests/ -v # Run all tests
pytest tests/test_timeseries.py -v # Single file
pytest tests/test_timeseries.py::TestTsProduct -v # Single class
ruff check elvers/ # Lint check
ruff check elvers/ --fix # Auto-fix lint issues
pytest tests/test_timeseries.py::TestTsProduct -v # Single test class
ruff check elvers/ --fix # Lint + auto-fix
ruff format elvers/ # Format code

# === Git ===
git status # See what changed
git diff # See actual changes
git add <files> # Stage specific files (never git add -A)
git commit -m "type(scope): msg" # Commit with convention
git push origin <branch> # Push to remote
git log --oneline -10 # Recent history

# === Release ===
python -m build # Build package locally (for testing)
git tag v0.2.0 # Create version tag
git push origin v0.2.0 # Push tag (triggers publish)
```

---
Expand Down
Loading
Loading