Skip to content

feat(scheduler): add error resilience with retry backoff to Monitor#719

Merged
alexluong merged 3 commits intomainfrom
redis-connection
Mar 4, 2026
Merged

feat(scheduler): add error resilience with retry backoff to Monitor#719
alexluong merged 3 commits intomainfrom
redis-connection

Conversation

@alexluong
Copy link
Collaborator

@alexluong alexluong commented Mar 3, 2026

Adds error resilience to the scheduler's Monitor function to handle transient Redis/RSMQ errors gracefully instead of failing immediately.

Changes

  • Retry with exponential backoff on transient RSMQ errors (default: 5 retries, 5s max backoff)
  • Increase default pollBackoff from 100ms to 200ms
  • Add Warn logging for each retry attempt
  • Extract rsmq.Client interface for testability

Testing

  • Test transient error recovery
  • Test max consecutive errors exhaustion
  • Test context cancellation during backoff

- Add rsmq.Client interface in rsmq package for mocking
- Add retry with exponential backoff on transient RSMQ errors
- Add maxConsecutiveErrors config (default 5) to limit retries
- Increase default pollBackoff from 100ms to 200ms
- Add tests for transient error recovery, max retries, and context cancellation
- Add WithLogger option to pass *logging.Logger
- Log Warn on each transient error during Monitor retry
- Update deliverymq retry scheduler to pass logger
@vercel
Copy link

vercel bot commented Mar 3, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
outpost-docs Ready Ready Preview, Comment Mar 3, 2026 6:48am
outpost-website Ready Ready Preview, Comment Mar 3, 2026 6:48am

Request Review

@alexluong alexluong merged commit 2b63811 into main Mar 4, 2026
5 checks passed
@alexluong alexluong deleted the redis-connection branch March 4, 2026 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants