CI integration¶
Run pytest-wardenbot on every push so a chatbot or agent regression fails the build before customers see it.
What a run costs¶
The deterministic checks call no judge model, but every check sends real requests to the system you test, and those may be billed by your model provider.
| Run | Requests | Notes |
|---|---|---|
| Chatbot suite, default quickstart | 35 to your chatbot | 29 tests; the three multi-turn tests send three messages each. Each business fact you add is one more. |
| Agent suite | 16 trials, usually 2–4 model requests each | 8 scenarios, each with a baseline and an attack. The quickstart agent made 28 requests with gpt-4.1-mini: 21 seconds, about a fifth of a cent. A local model costs nothing. |
LLM-judge tests ([judge]) |
1 judge request per case, plus 1 chatbot request | About $0.003 per case with Claude Haiku 4.5, or free with a local judge. |
GitHub Actions: chatbot suite¶
Copy this workflow to .github/workflows/wardenbot.yml. It expects the
conftest.py and test_my_bot.py that pytest --wardenbot-quickstart
generates, at the repository root.
name: wardenbot
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch: {}
# The tests only read the repository.
permissions:
contents: read
concurrency:
group: wardenbot-${{ github.ref }}
cancel-in-progress: true
jobs:
wardenbot:
name: pytest-wardenbot
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
# Actions are pinned to full commit SHAs. Dependabot's github-actions
# ecosystem keeps them current.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install pytest-wardenbot
# Pin the version you tested. Use "pytest-wardenbot[judge]==..." to
# enable the LLM-judge tests.
run: python -m pip install "pytest-wardenbot==0.8.0"
- name: Run wardenbot tests
env:
CHATBOT_URL: ${{ secrets.CHATBOT_URL }}
CHATBOT_TOKEN: ${{ secrets.CHATBOT_TOKEN }}
# ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} # uncomment if using [judge]
run: python -m pytest test_my_bot.py -v --tb=short
- name: Notify Slack on failure
if: failure() && env.SLACK_WEBHOOK_URL != ''
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
REPOSITORY: ${{ github.repository }}
REF_NAME: ${{ github.ref_name }}
COMMIT: ${{ github.sha }}
# Workflow values reach the script as environment variables, never as
# interpolated shell code, and jq builds the JSON.
run: |
jq -n --arg text ":warning: pytest-wardenbot failed on $REPOSITORY@$REF_NAME (commit $COMMIT)" \
'{text: $text}' |
curl --fail --silent --show-error -X POST -H 'Content-type: application/json' \
--data @- "$SLACK_WEBHOOK_URL"
Add CHATBOT_URL (and CHATBOT_TOKEN if your bot needs auth) under
Settings → Secrets and variables → Actions. The workflow:
- pins each action to a commit SHA and the package to an exact version, so the build only changes when you change it (Dependabot can bump both);
- gives the job read-only repository access and doesn't persist the checkout token;
- passes workflow values to the Slack step as environment variables, never as shell code.
GitHub Actions: agent suite¶
Add a job that runs the agent suite in required mode, so a skipped, deselected or unsupported case fails the build instead of passing quietly:
agent-suite:
name: Agent suite
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- run: python -m pip install "pytest-wardenbot[mcp,openai]==0.8.0"
- name: Run the agent suite
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: python -m pytest test_my_agent.py --wardenbot-agent-required -v
This assumes the files from pytest --wardenbot-quickstart=agent, or your own
agent_target and agent_observer fixtures. Pull requests from forks don't
receive secrets, so the job fails there; restrict it with
if: github.event.pull_request.head.repo.full_name == github.repository if you
accept outside contributions.
GitLab CI¶
wardenbot:
image: python:3.12
script:
- python -m pip install "pytest-wardenbot==0.8.0"
- python -m pytest test_my_bot.py -v --tb=short
rules:
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
Define CHATBOT_URL and CHATBOT_TOKEN as masked CI/CD variables in the
project settings.
CircleCI¶
version: 2.1
jobs:
wardenbot:
docker:
- image: cimg/python:3.12
steps:
- checkout
- run:
name: Install
command: python -m pip install "pytest-wardenbot==0.8.0"
- run:
name: Run tests
command: python -m pytest test_my_bot.py -v --tb=short
workflows:
test:
jobs:
- wardenbot:
context:
- chatbot-secrets # exposes CHATBOT_URL + CHATBOT_TOKEN
Run judge tests less often¶
To keep pull requests fast and cheap, run the deterministic checks on every pull request and the LLM-judge tests on a schedule:
In the job, install pytest-wardenbot[judge] and set ANTHROPIC_API_KEY only
when github.event_name == 'schedule', or pass a local judge with
--wardenbot-judge-model ollama/qwen2.5:7b on a self-hosted runner.
When a result changes between runs¶
Model output varies, so a check that passes most of the time can fail now and then. Treat that as a finding, not as noise: a model that follows an injected instruction one time in ten will follow it in production too.
- Don't retry failed security checks automatically (for example with
pytest-rerunfailures). A retry turns "sometimes violates" into a pass. - Lower the variance where you control it: set
temperature=0in your chatbot or agent if the model allows it. - To measure how often something happens, run repeated trials with bounded campaigns and report the rate.
- Triage a failing test explains how to read each failure.