Vox is a sophisticated forecasting bot for the Metaculus AI Tournament that uses a 5-agent committee architecture with prediction market priors from the ADJ Political Index API.
- Multi-Agent Committee: 5 specialized forecasting agents with distinct personas
- Prediction Market Priors: Integrates ADJ API for real-time market prices as Bayesian priors
- Peer Review Workflow: Agents critique and revise forecasts before aggregation
- Multi-Source Research: Combines ADJ markets, AskNews, and Perplexity
- Logit-Space Aggregation: Final forecasts aggregated in probability space for better calibration
┌─────────────────────────────────────────────────────────────────┐
│ Vox Forecaster │
├─────────────────────────────────────────────────────────────────┤
│ Research Layer │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ ADJ API │ │ AskNews │ │ Perplexity│ │
│ │ (priors) │ │ (news) │ │ (research)│ │
│ └───────────┘ └───────────┘ └───────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Agent Committee (5 agents with peer review) │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ base_rate_analyst│ │ market_prior │ │
│ │ (weight 1.0) │ │ (weight 1.5) │ │
│ └──────────────────┘ └──────────────────┘ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ status_quo_anchor│ │ devils_advocate │ │
│ │ (weight 1.0) │ │ (weight 1.0) │ │
│ └──────────────────┘ └──────────────────┘ │
│ ┌──────────────────┐ │
│ │ synthesizer │ │
│ │ (weight 1.5) │ │
│ └──────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
| Agent | Weight | Focus |
|---|---|---|
| base_rate_analyst | 1.0 | Historical frequency, reference class forecasting |
| market_prior | 1.5 | Prediction market prices as Bayesian priors |
| status_quo_anchor | 1.0 | Conservative default, regression to mean |
| devils_advocate | 1.0 | Challenge consensus, find overlooked risks |
| synthesizer | 1.5 | Combine perspectives, resolve disagreements |
git clone https://github.com/your-username/vox.git
cd vox
poetry installcp .env.template .env
# Edit .env with your API keysRequired:
METACULUS_TOKEN- Your bot's Metaculus API token
Recommended:
ADJ_API_KEY- For prediction market priorsASKNEWS_CLIENT_ID+ASKNEWS_SECRET- For news researchPERPLEXITY_API_KEY- For deep research
# Test on example questions
poetry run python main.py --mode test_questions
# Run on tournament questions
poetry run python main.py --mode tournament
# Use simpler single-agent mode (faster)
poetry run python main.py --simple --mode test_questions
# Run without submitting to Metaculus
poetry run python main.py --no-publish --mode test_questionspoetry run python test_setup.pyVox uses the ADJ Political Index API for prediction market data:
- Semantic Search: Find markets related to forecasting questions
- Direct Match Priors: When similarity > 70%, use market price as primary prior
- Cross-Platform Aggregation: Reference rates combine Kalshi, Polymarket, etc.
- Caching: 5-minute TTL with request deduplication
Example:
from research import AdjClient
async with AdjClient() as client:
# Find related markets
results = await client.search("Will Trump win 2024?", entity_type="market")
# Get market prior for a question
prior, section = await client.get_market_prior(question)-
Research Phase
- Query ADJ for prediction market priors
- Fetch recent news from AskNews
- Get deep research from Perplexity
- All sources run in parallel
-
Initial Forecasts
- All 5 agents generate independent forecasts
- Each follows their persona's approach
-
Peer Review
- Each agent reviews another agent's forecast
- Provides constructive critique
-
Revision
- Agents revise based on peer feedback
- May maintain or adjust their forecast
-
Aggregation
- Forecasts combined in logit space
- Weighted by agent weights (market_prior, synthesizer = 1.5x)
The included workflow runs every 30 minutes:
- Fork this repository
- Go to Settings → Secrets and variables → Actions
- Add your API keys as repository secrets:
METACULUS_TOKENADJ_API_KEYASKNEWS_CLIENT_ID,ASKNEWS_SECRETPERPLEXITY_API_KEY
- Enable Actions in the Actions tab
- Go to https://metaculus.com/aib
- Create a bot account
- Click "Show My Token"
Contact the ADJ team for API access to prediction market data.
- Create account at https://my.asknews.app
- Email
rob [at] asknews [.app]with your bot username - Generate credentials at https://my.asknews.app/en/settings/api-credentials
- Create account at https://perplexity.ai
- Go to Settings → Account → API
- Generate API key and add credits
Email ben [at] metaculus [.com] with:
- Your bot username
- Description of your bot
- Requested models and budget
Key settings in config.py:
cache_ttl_seconds: int = 300 # Cache TTL for API responses
committee_size: int = 5 # Number of agents
agent_timeout: int = 120 # LLM call timeout
max_concurrent_questions: int = 2 # Parallel question limit- Add system prompt to
bot/prompts.py:
NEW_AGENT_SYSTEM = """Your agent's approach..."""- Register in
bot/agents.py:
AGENT_CONFIGS["new_agent"] = AgentConfig(
name="new_agent",
system_prompt=AGENT_PROMPTS["new_agent"],
weight=1.0,
)- Create client in
research/new_source.py - Integrate in
research/integrated_search.py - Add to
integrated_research()function
- Fine-tune on Metaculus data: Train calibration on historical forecasts
- Dynamic agent weights: Adjust weights based on past accuracy
- Question decomposition: Break complex questions into simpler ones
- Monte Carlo simulations: Combine with scenario modeling
- Extremize predictions: Apply calibration transforms based on bot history
- Metaculus AI Tournament
- forecasting-tools package
- Metaculus Discord - #build-a-forecasting-bot
MIT License - See LICENSE file for details.