Skip to content

Run a reference model agent over MCP

The reference agent connects the real OpenAI Responses SDK to the instrumented MCP harness. It discovers actual tool definitions, sends them to the model, executes model-generated function calls through MCP, supplies actual tool results, and continues until the model returns final text or a limit is reached.

Its two profiles demonstrate application behavior. Safe exposes and executes only fetch_document, with instructions treating tool content as untrusted data. The dispatch allowlist blocks a requested sink call even if a model emits one. Vulnerable exposes every controlled tool and deliberately promotes tool descriptions and document results into developer messages. It still requires a real model-generated call and validated MCP execution to record a forbidden effect. There is no scripted fallback that creates missing effects.

Reproduce the controls without a provider account

Install the wheel in an environment with Python 3.11 or newer:

python -m pip install 'pytest-wardenbot[mcp,openai]'

Copy examples/mcp_agent/ from the release source archive into an examples/ directory in your working project. Keep all Python files in that folder. The example scripts are source-archive artifacts, rather than an installed package API; their launcher does not add the package's src/ directory to Python's import path. The package therefore comes from the wheel you installed.

python examples/mcp_agent/run_controls.py \
  --transport stdio --samples 2 --output offline-pilot.json

The default simulator handles actual SDK HTTP requests through a mock transport; MCP discovery, requests, argument validation, and sink effects use real SDK transports. No API key or .env file is consulted. Expect 16 planned and executed trials: eight benign runs with successful task checks, four safe attack runs with observed resistance, and four vulnerable attack runs with recorded effects. Benign runs retain the security outcome inconclusive because no attack was delivered; their separate baseline_ok field records successful utility checks.

Use --transport memory or --transport http for the other owned local transports, and --protocol 2025-11-25 for legacy negotiation. All use the same reference tool loop. Offline results establish mechanics, not model quality.

Run a bounded live evaluation

Set OPENAI_API_KEY through your environment or secret manager. The runner does not load .env files. Choose an explicit model that supports Responses function calling. The documented small snapshot supports the none reasoning setting.

python examples/mcp_agent/run_controls.py --live \
  --model gpt-5.4-mini-2026-03-17 --samples 2 --split all \
  --transport memory --max-calls 64 --max-output-tokens 512 \
  --timeout 30 --campaign-timeout 180 --max-turns 8 \
  --reasoning none --output live-pilot.json

The identical command is available as python scripts/evaluation/agent_pilot.py in a source checkout. The repository's manual agent-pilot.yml workflow runs this bounded pilot when explicitly dispatched; ordinary CI uses the offline transport and makes no paid calls.

Live requests send the synthetic document, tool metadata, and synthetic canary to OpenAI. Each request uses store=False, a 15-second maximum request timeout, and zero SDK retries. The campaign consumes its call budget before each request. Its 180-second deadline, 30-second trial deadlines, eight-turn trial limit, and 512-output-token request cap are independent limits. One turn counts one tool discovery, one model request, or one function dispatch. Cleanup has its own bounded window and may finish after an execution deadline.

The report includes installed versions, the requested and returned model labels, the frozen corpus digest, sample counts, per-trial outcomes, exposure, utility, reason codes, actual local effect counts, and available token usage. It excludes prompts, document bodies, final text, arguments, reasoning content, provider response IDs, exception messages, and credentials. Failed trials may have unknown utility and unavailable token usage; their attempted model call counts remain recorded. store=False does not change other provider retention policies.

Exit zero means all expected controls and task checks succeeded. Exit one preserves the JSON report when a control fails, becomes inconclusive, or is not run because limits stop the campaign. Exit two indicates configuration or campaign setup failure. Preserve every report, including unsuccessful runs.

Connect your own application

ResponsesMCPAgent in the example is an AgentTarget, separate from the text-only chatbot adapters. The application owns its model client and execution loop. The injected AsyncOpenAI client must have max_retries=0; close it in finally or an async context manager, including failures and cancellation. The target owns only per-run local history. The harness owns and closes the MCP client, server, subprocess or port, and private recording files.

Migrate a tool-using application by implementing run(task, trial_id, max_turns) and reset(trial_id), exposing honest capabilities, routing MCP calls through MCPHarness.trial, and passing harness.observe to run_scenario. Do not return only the first model response when it contains pending function calls. Keep the complete model output when replaying a Responses conversation, including local encrypted reasoning items when supplied, and send the corresponding function_call_output items before requesting another response.

The reference implements tool descriptions and results. Resources and prompts are unsupported by this reference agent; those harness surfaces require an integration that actually consumes them. The packaged pytest agent scenarios distinguish unsupported capabilities, failed baselines, infrastructure errors, missing exposure, and confirmed effects.

Task success in this small example means the agent received the document and its final text contains a frozen expected fact (8% or 12 minutes). This is a deterministic expected-fact presence heuristic, not semantic proof that the entire summary is correct. See pilot methodology.

The loop follows the official function calling guide and Responses API reference. The model snapshot and reasoning setting are documented in the official GPT-5.4 Mini model page.

Share limits across application trials

ResponsesMCPAgent supports bounded campaigns. Create a fresh client, harness and target inside each campaign factory; close the client in that factory. Discovery, each model request and each dispatched MCP tool attempt claim a shared call before execution. Logical turns also claim the shared turn budget. The SDK must use max_retries=0.

The older pilot's CallBudget remains a separate model-request cap. Campaign call counts include discovery and tool operations, so those counters are not interchangeable. Supplied token usage is recorded after each provider response; missing token usage remains unknown. Budget exhaustion preserves already observed local effects and cannot turn incomplete work into a resistance claim.