Skip to main content

​
Install Merit

Merit requires Python 3.12 or higher. Install directly from GitHub using uv (recommended) or pip: 
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install Merit from GitHub
uv pip install git+https://github.com/appMerit/merit.git

​
Write Your First Test

Create a file called test_chatbot.py:
test_chatbot.py
import merit

def chatbot(prompt: str) -> str:
    """Your AI system under test."""
    return f"Hello, {prompt}!"

def merit_chatbot_responds():
    """Test that chatbot responds correctly."""
    response = chatbot("World")
    assert response == "Hello, World!"
    assert len(response) > 0

​
Run Your Tests

merit
You’ll see output like: 
Merit Test Runner
=================

Collected 1 test

test_chatbot.py::merit_chatbot_responds ✓

==================== 1 passed in 0.12s ====================

​
Add AI-Powered Assertions

Now let’s add an LLM-as-a-Judge assertion. First, set up your Merit API credentials:
.env
# Required for AI predicates
MERIT_API_BASE_URL=https://api.appmerit.com
MERIT_API_KEY=your_merit_api_key_here
AI predicates call the Merit cloud service. You need a Merit API key - contact us to get access.
Update your test to use AI predicates:
test_chatbot.py
import merit
from merit.predicates import has_facts

def chatbot(prompt: str) -> str:
    return "Paris is the capital of France. It's known for the Eiffel Tower."

async def merit_chatbot_facts():
    """Test that chatbot provides accurate facts."""
    response = chatbot("Tell me about Paris")
    
    # AI-powered assertion checks if facts are present
    assert await has_facts(response, "Paris is the capital of France")
Run again: 
merit

​
Use Parametrization

Test multiple inputs easily:
test_chatbot.py
import merit

def chatbot(prompt: str) -> str:
    return f"Hello, {prompt}!"

@merit.parametrize(
    "name,expected",
    [
        ("World", "Hello, World!"),
        ("Alice", "Hello, Alice!"),
        ("Bob", "Hello, Bob!"),
    ],
)
def merit_chatbot_greetings(name: str, expected: str):
    """Test multiple greetings."""
    response = chatbot(name)
    assert response == expected
This runs 3 tests automatically, one for each parameter set.

​
Next Steps

Core Concepts

Learn about resources, test cases, and more

AI Predicates

Explore all LLM-as-a-Judge assertions

Advanced Features

Tags, filters, repeat tests, and tracing

Error Analysis

Analyze failures with Merit Analyzer