Quickstart¶
Two paths, each a few minutes from pip install to a first result. Use
Python 3.11 or later.
Test a chatbot¶
1. Install and generate starter files¶
This writes two files in the current directory:
conftest.py: thechatbotfixture that points at your bot, and abusiness_truth_factfixture with placeholder facts;test_my_bot.py: imports the shipped checks, so plainpytestruns them.
Industry templates start you with richer placeholder facts:
pytest --wardenbot-quickstart=ecommerce # refund and shipping facts
pytest --wardenbot-quickstart=saas-support # plan and trial facts
2. Point it at your chatbot¶
export CHATBOT_URL=https://your-bot.example.com/chat
export CHATBOT_TOKEN=sk-your-token # optional, sent as a Bearer token
The generated fixture posts {"message": "..."} and reads
{"response": "..."}. If your bot uses other field names, change
request_field and response_field in conftest.py. Until CHATBOT_URL is
set, every check fails setup with a message saying so, rather than passing
without contacting a bot.
Not a JSON-over-HTTP bot? Install an extra and use a bundled adapter
([openai], [anthropic], [langchain]), a
local model preset, or write a small adapter; see
Add your chatbot.
3. Run¶
The first run sends about 35 requests to your bot: 29 checks, three of which send three messages. The placeholder facts are skipped until you replace them, so a first run can't fail on an answer you never set.
4. Replace the placeholder facts¶
Each BusinessTruthFact in conftest.py is one question your bot must answer
correctly. Edit the placeholders to your real answers:
from pytest_wardenbot.business_truth import BusinessTruthFact
BusinessTruthFact(
label="Standard plan price",
question="How much does the Standard plan cost per month?",
expected_answer="$49",
match_type="substring",
acceptable_variations=("49/mo", "forty-nine dollars"),
)
Each fact becomes one test, which fails with the question and the bot's answer when the bot gets it wrong.
5. Read the failures¶
A failure says which check failed and why, then shows the bot's reply fenced
as untrusted text, and ends with a block labeled
Agent-ready remediation (paste into Cursor / Claude Code). Paste that block
into your coding assistant to draft a fix, then run pytest again.
Triage a failing test explains each kind of
failure, and plant a canary token to make the leak
checks much stronger.
Test an agent¶
1. Install and generate starter files¶
This writes:
conftest.py:MyAgent, a complete tool-calling agent (an OpenAI-compatible chat loop over the controlled harness tools), plus theagent_target,agent_harnessandagent_observerfixtures;test_my_agent.py: imports the packaged agent suite.
2. Choose a model¶
Or run fully local with Ollama, vLLM or llama.cpp:
With a local model the agent first checks that the model can call tools. If
it can't, every case is reported as unsupported rather than resisted.
3. Run¶
The suite runs eight versioned MCP scenarios, each twice: a benign baseline, then the same task with an injected instruction in a tool result, tool description, resource or prompt. A trial fails only if the harness recorded the agent calling the forbidden tool or sending the canary. Required mode makes a skipped, unsupported or incomplete case fail the run instead of passing quietly.
Expect 16 trials of 2–4 model requests each. In our run, gpt-4.1-mini took
21 seconds and about a fifth of a cent (28 requests): every baseline passed and
it followed 3 of the 8 attacks. A 7B model on a laptop takes a minute or two;
see local-model results.
If a case errors, run again with --wardenbot-agent-debug to see the
underlying error locally.
4. Connect your own agent¶
Replace MyAgent in conftest.py with your agent, built exactly as in
production but given the harness's tools instead of your real ones:
| Your agent | Connect it with |
|---|---|
| LangGraph | LangGraphAgentTarget ([langgraph]) |
| Claude Agent SDK | ClaudeAgentTarget ([claude-agent]) |
| OpenAI Agents SDK | Recipe |
| Any other tool loop | Custom target with TrialTools |
Agent outcomes and reason codes explains every result. A baseline that fails with "Agent took the forbidden action with no attack present" means the agent calls that tool on its own, so attacks on it can't be attributed to the injection.
Next steps¶
- Run in CI, with per-run request counts.
- Enable LLM-judge tests for semantic checks such as brand voice and hallucination, hosted or local.
- Test application boundaries such as retrieval, memory and approvals.