Latest commit
History
Folders and files
Repository files navigation
Consumer Agent Testing Tool (CATT)
Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.
Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract , including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.
Contributing | Security | Changelog | MIT license
A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis: documented , unofficial , assumed or unknown . This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.
What you can test today
The connector lane records the basis for each Muse-specific behavior. Tool classes and approval prompts follow documented guidelines. The muse-cli projection and template client follow unofficial mirrors; the native projection is an assumed generic baseline. The neutral cli projection supports both targets as a CATT treatment. Instinct runs have a separate source record and label the shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter; no public Instinct MCP integration contract was found.
Run the local booking example under Instinct without a model or account:
uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \ --driver scripted --script examples/connector/fixture-booking.calls.json \ --start " python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none "
Set emulation_target: instinct in your scenario or pass --emulation-target instinct . Both targets use native or cli projection and the generic MCP client. Instinct defaults to plain-text replies and does not run Muse guideline checks. Its local wallet fixture covers payment behavior; live payment rails, trusted-network protocol, Vault and Files remain outside the shared contract .
Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.
Setup
You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS), uv , and Node.js 20 or newer with npm (the installed packages declare >=20 ; tested with 25.9). The recorded release evidence covers macOS. From a clone of this repository, install the Python and Node dependencies and the Chromium build the browser lanes use:
uv sync --extra openrouter --extra dev npm --prefix tooling ci --ignore-scripts npm --prefix tooling/payment-ui ci --ignore-scripts PLAYWRIGHT_BROWSERS_PATH= " $PWD /.cache/ms-playwright " node tooling/payment-ui/node_modules/playwright/cli.js install chromium
The connector lane, the readiness checks and the scripted runs need only uv sync . Node and Chromium are for the MCP Apps and browser lanes.
On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the CI workflow but have not been qualified here.
sudo apt-get update && sudo apt-get install -y nodejs npm # or Node 20+ from nodesource or nvm curl -LsSf https://astral.sh/uv/install.sh | sh uv sync --extra openrouter --extra dev npm --prefix tooling ci --ignore-scripts npm --prefix tooling/payment-ui ci --ignore-scripts PLAYWRIGHT_BROWSERS_PATH= " $PWD /.cache/ms-playwright " \ node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium
Three features rely on macOS Seatbelt ( sandbox-exec ) and report unsupported or blocked on Linux rather than running unconfined: the --offline readiness mode, the network-denied validator workers behind run --surface api-tools , and the local-model memory profile. The connector lane, the scripted runs, the readiness checks and the Docker-backed stdio targets do not depend on it.
uv run consumer-agent-testing-tool doctor
It checks Node against the floors the installed packages declare, the tooling/payment-ui packages against their pinned versions, and Chromium and its headless shell under .cache/ms-playwright . It reports the optional protocol tooling, and checks Docker only for development lanes or configured stdio MCP and CLI targets. It makes no model requests, even when an existing configuration or --model-profile selects a model. Each failed or blocked check prints one line on stderr with its reason and, for a setup problem, the command that fixes it.
To explicitly run the provider preflight, which sends two short model requests and may incur charges, add --model-preflight :
uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight
Scripted runs need no model account, and the scripted commands in all three quickstarts need no configuration file. Agent runs need a model profile and an API key; Model providers shows how to create one. Supply credentials through environment variables, and keep them out of scenarios and shared evidence. consumer-agent-testing-tool.yaml , the default settings file, is local and git-ignored, and no profile is shipped.
Quickstart: connector lane
This quickstart stays on loopback and needs no account. Complete Setup , then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it:
uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none
Replay the example's scripted calls, with no model:
uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \ --driver scripted --script examples/connector/fixture-booking.calls.json
The script searches rooms, gets a quote, books and gives a final answer. The host fills the guest's email from the scenario's private: values, which the model never sees, and the scripted user answers each approval card from the scenario's permissions: rules. After the run, the host checks the fixture's private oracle (one confirmed booking, no duplicate charges) and the booking's status through get_booking . It also checks that each sent call matched its approval card ( approval.dispatches_match , approval.disclosure_bound ) and that the final answer made no false claim and relayed the total and the confirmation ( answer.no_false_claims , answer.relay_complete ).
The command prints the run directory and Connector report: .../report.html , and exits 0 when every required check passed. Runs are sealed under runs/connector-<id>/ . Share the output of export , not the run directory: its private/ folder holds host-private values.
For an agent run, create agent.yaml with the init command in Model providers , set OPENROUTER_API_KEY in your shell, and run:
uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml
The model preflight runs first and prints its own evidence path. The agent then sees the four allowed tools as native functions, with the host-private email removed from their schemas, and writes the other arguments itself. This uses paid model inference. To change the treatment without editing the scenario, --projection cli offers the same tools through one exec command for either target, --exposure search-only loads their parameters through tool_search , --output-schema strict refuses results that break a tool's outputSchema as the official MCP SDK does, and --trials N runs independent trials and prints pass^k. The run lock records the projection, exposure, client-profile and output-schema overrides under overrides .
connector-run exits 0 when every required check passed; 1 when a check failed, the answer made a false claim or an agent limit was reached; 2 for a setup, harness or reporting error; and 3 when the trial is blocked or unsupported, for example by an uncertain write to reconcile, an unobservable check or a client-profile limit. To start from your own server, run scaffold ; to replay a recorded, shareable agent trial without a model, turn it into a calls file with scaffold --from-run . The connector lane reference covers scenarios, flags and evidence. The examples list each shipped scenario with its fixture command, and lane extensions covers routes, custom connectors, the browser sub-agent, wallets, schedules and the other extensions.
Quickstart: a popular MCP server
The connector lane works with any MCP server over Streamable HTTP, not only the bundled test service. This quickstart points it at "Everything" ( @modelcontextprotocol/server-everything , MIT), the reference server the MCP project maintains for client authors. Its weather tool returns simulated data, so it needs no account or key. Start it in a second terminal; the first run downloads the pinned release from npm:
PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp
See what the host infers from the live catalog. scaffold runs the handshake and tools/list , calls no tool, and writes a scenario skeleton with suggested tool classes from the server's annotations:
uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines
The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client:
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --driver scripted --script examples/connector/everything-weather.calls.json uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --projection muse-cli --client-profile muse-template-unofficial-2026-10 \ --driver scripted --script examples/connector/everything-weather-cli.calls.json
Both exit 0. The scenario offers the model three of the server's tools; a reviewed allowlist keeps the rest off its list, including get-env , which returns the server process's environment variables, and gzip-file-as-resource , which fetches any URL it is given. The server has no private oracle, so the outcome check calls get-structured-content itself after the agent stops, and the answer checks look for the temperature and conditions the server returned. With a model profile, let an agent do the same task three times and print pass^k (paid inference). The scenario declares no reset URL, so connector-run warns that the trials share the server's state; this weather tool keeps none:
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --config agent.yaml --trials 3
Readiness checks
check also runs readiness suites that read catalogs and documents instead of running an agent. With the connector-lane quickstart's fixture still running, review its catalog and the example manifest against the pinned Muse connector guidelines:
uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \ --manifest examples/connector/readiness.yaml
The guidelines review lists the server's tools and calls none; --drift-check fetches the live guidelines, and only when given. Poppy discovery only reads. Its probes run only with --manifest and act on the company's own hosts: they start sessions, sign in the declared test account and confirm one declared operation. The gadget suite reads one local file. The contract suite checks a target named in the configuration file ( --config ). For a kind: mcp target without declared contract_cases , its operation-contracts check is unsupported and the run exits 3.
A clean readiness result is not an admission. Meta reviews connectors by hand, Poppy is draft 0.1 with no production Muse evidence, and the Gadget SDK's terms say it is not a developer platform. Readiness checks describes the rules, manifests, --offline behavior and the mcp-contract-2026-07-28 profile.
Quickstart: Coverage Cat sandbox
From a source checkout or unpacked source distribution, complete Setup , then run:
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted
The example creates a fictional shopper's draft in Coverage Cat's hosted sandbox at https://www.coveragecat.com/api/consumer/mcp . It needs no Coverage Cat account, operator credentials or model API key.
The first umbrella_consumer_prefill call sets sandbox: true , as specified in the consumer MCP manifest . That argument activates the server's sandbox. The scenario's test_only: true flag merely records the test author's intent.
The host fixes the tool arguments and keeps the returned access token private. It permits two tools, umbrella_consumer_prefill and consumer_intake_issues , and checks that:
Prefill returns sandbox: true .
Follow-up retains sandbox: true .
Follow-up returns the original application ID.
The server reports application_submitted: false .
Each run creates a new draft and stops before submission. A failed or missing sandbox check stops the workflow. These checks cover draft creation and follow-up, not payment widgets, browser behavior or independent backend checks.
To let a model choose the steps, create agent.yaml with the init command in Model providers , set OPENROUTER_API_KEY in your shell, and run:
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml \ --driver agent --config agent.yaml
The agent driver is a scripted action menu : Spark Open picks one ready scenario step per turn, and the host runs it with the scenario's arguments. The model never sees the server's tools, schemas or raw results, only the result fields the scenario exposes, and the evidence is labeled scripted action menu . The host supplies the test inputs and checks the results. This version uses paid model inference through Together, with ZDR enabled and fallback disabled by default.
Verified 15 October 2026: both scripted and Spark Open agent runs passed all four sandbox checks. Both model responses identified Together; both requests used the configured ZDR policy. See the recorded quickstart results .
Each command prints an outcome line, such as Surface run completed (exit 0) , then Surface report: .../report.html . Open the report to inspect the workflow and its checks. The same directory contains result.json for the structured result and events.jsonl for the redacted event timeline. Agent runs also store model and tool records under agent/ .
Replace the placeholders below with the run directories printed by your commands:
uv run consumer-agent-testing-tool report runs/surface-YOUR_RUN_ID uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID
Share the generated export. Keep the full run folder private. While a run can resume, checkpoint.private.json holds the credentials, tool results and host inputs it needs; a completed run, or one that failed with exit 1 and left no write unconfirmed, keeps only a scrubbed record.
To test your own MCP service, copy the example and replace its endpoint, allowed origins, tool policies, test data and response checks. Surface workflows explains UI steps, the agent driver and resuming runs.
Extend through submission and sandbox checkout
The checkout example adds review, submission, Monoline offer selection, consent, bind follow-up, document upload and simulated payment-method setup. Its 14 steps contain 62 assertions. All seven tools use the consumer umbrella MCP, and every service step must return the same sandbox intake.
Current integration status, 15 October 2026: blocked at submission. Coverage Cat returns quote fields that its MCP output schema excludes. It also returns minimum_auto_limits as a string where the schema requires an object. The host rejects the response. The draft quickstart passes, but the full checkout test has not passed end to end . See the failure report and separate diagnostics .
Run this command to reproduce the failure or retest after the service contract is corrected:
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \ --driver scripted \ --replies examples/coveragecat-sandbox-checkout-replies.json
The reply files supply a fictional shopper's answers : approval to submit, credit consent, the Monoline $2M offer at $389, sufficient auto limits, declined excess UM/UIM, and “No” to both carrier questions. These fixtures do not authorize transactions for a real shopper. A changed quote or missing or negative approval stops the next write. The two generated declarations PDFs are labelled SYNTHETIC TEST ONLY .
Once submission works, the example pauses with exit code 3 at sandbox_document_review_poll . Only at that checkpoint , wait at least the reported 60 seconds and resume the printed run:
sleep 60 uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \ --driver scripted --resume runs/surface-YOUR_RUN_ID \ --replies examples/coveragecat-sandbox-checkout-after-review.json
The host pauses for your reply but does not measure how long you wait. The after-review fixture approves the simulated payment callback.
For an agent-driven run, replace --driver scripted with --driver agent --config agent.yaml on both commands. A resumed run must keep its original scenario, driver and model configuration.
The sandbox supplies mock payment credentials. The example sends the returned seti_sandbox_... with the fixture pm_sandbox_consumer_agent_testing_tool , then requires payment_method_on_file , waiting_on_carrier , and no charge confirmation. Billing checks inspect setup metadata rather than rendered fields. No Stripe form opens, no card details are collected, and no fallback URL is used. This test establishes neither premium collection nor active insurance. Test the embedded payment UI and Stripe test-mode setup separately.
Test your own service
The quickstarts above are the pieces of one workflow. For an MCP server you own:
Write a scenario. scaffold URL --output my-connector.yaml reads the live catalog ( --auth-env VAR for an API key, or --auth-from FILE with a scenario that holds an OAuth block) and writes a skeleton. Replace every TODO(scaffold) : the user's goal in their own words, the allowlist of tools a reviewed connector would expose, tool classes your documentation settles, approval rules for the scripted user, host-private values such as the user's email, and ch