v0.2.0 - Tool Testing Framework

What's New in v0.2.0

🛠️ Comprehensive Tool Testing Framework

This release introduces a powerful tool testing framework with three complementary approaches:

1. Tool Expectation DSL

Declarative API for specifying tool expectations within conversation flows:

from testllm import expect_tools, conversation_flow

flow = (
    conversation_flow("booking")
    .tool_step(
        "Find flights to NYC",
        criteria=["Should present options"],
        tool_expectations=expect_tools()
            .expect_call("search_flights")
            .with_arguments_containing(destination="NYC")
            .returning({"flights": [{"id": "F1", "price": 299}]})
    )
)

2. Tool Interceptor

Framework-agnostic tool call interception and mocking:

from testllm import ToolInterceptor

interceptor = ToolInterceptor()
interceptor.register_mock("search_flights", {"flights": [...]})
interceptor.register_schema("book_flight", {...})

3. Response Simulation

Test agent behavior under various tool response scenarios:

from testllm import simulate_tool

simulator = (
    simulate_tool("search_flights")
    .on_success({"flights": [...]})
    .on_failure("Service unavailable")
    .on_timeout(30000)
)

🔄 Default Evaluator Change

• Google Gemini is now the default evaluator (free tier, no credit card required)
• Easy setup: python -m testllm.setup
• Still supports Anthropic Claude and Mistral as alternatives

🗑️ Removed

• OpenAI support has been removed from the project

📦 New Modules

• testllm.tool_testing - Complete tool testing framework
• testllm.setup - CLI setup wizard for API keys

✅ Tests

• 180 tests all passing

Release v0.1.5

Changes

• Changed package name from test-llm to testkitLLM to resolve PyPI naming conflicts
• All imports remain unchanged as testllm

Installation

pip install testkitLLM

v0.1.3

🚀 testLLM v0.1.3 - Production Release!

The first testing framework designed specifically for LLM-based agents.

✨ Features

• 🧠 Semantic Testing: Use LLM evaluators instead of brittle string matching
• 🔄 Conversation Flows: Test multi-step agent interactions
• 📊 Behavioral Patterns: Pre-built patterns for common AI behaviors
• 🔌 PyTest Integration: Works seamlessly with existing test suites
• ⚡ Fast Evaluation: Mistral Large for 3-5x faster testing than Claude

📦 Installation

pip install testllm

🔧 Quick Setup

# Add to your .env file
MISTRAL_API_KEY=your_key_here
ANTHROPIC_API_KEY=your_key_here  # optional

🎯 Quick Example

from testllm import semantic_test, LocalAgent

# Test your agent semantically
test = semantic_test("greeting", "Test friendly responses")
test.add_scenario("Hello!", ["Response should be welcoming"])
results = test.execute_sync(your_agent)

📚 Documentation

  - https://github.com/Wayy-Research/testLLM/tree/main/examples
  - https://github.com/Wayy-Research/testLLM/blob/main/ARCHITECTURE.md

---
Ready to build reliable AI systems? Start testing your LLM agents today! 🎉