<!-- source: https://deepwiki.com/openclaw/openclaw/8.5-qa-lab-and-scenario-packs -->

# QA Lab & Scenario Packs

- .agents/skills/telegram-crabbox-e2e-proof/SKILL.md
- .github/actions/docker-e2e-plan/action.yml
- .github/codex/prompts/mantis-telegram-desktop-proof.md
- .github/workflows/full-release-validation.yml
- .github/workflows/mantis-discord-smoke.yml
- .github/workflows/mantis-discord-status-reactions.yml
- .github/workflows/mantis-discord-thread-attachment.yml
- .github/workflows/mantis-slack-desktop-smoke.yml
- .github/workflows/mantis-telegram-desktop-proof.yml
- .github/workflows/mantis-telegram-live.yml
- .github/workflows/npm-telegram-beta-e2e.yml
- .github/workflows/openclaw-live-and-e2e-checks-reusable.yml
- .github/workflows/openclaw-release-checks.yml
- .github/workflows/opengrep-precise-full.yml
- .github/workflows/opengrep-precise.yml
- .github/workflows/package-acceptance.yml
- .github/workflows/qa-live-transports-convex.yml
- .github/workflows/sandbox-common-smoke.yml
- docs/concepts/personal-agent-benchmark-pack.md
- docs/concepts/qa-e2e-automation.md
- extensions/qa-lab/api.ts
- extensions/qa-lab/src/cli.runtime.test.ts
- extensions/qa-lab/src/cli.runtime.ts
- extensions/qa-lab/src/cli.test.ts
- extensions/qa-lab/src/cli.ts
- extensions/qa-lab/src/docker-runtime.ts
- extensions/qa-lab/src/live-transports/discord/cli.runtime.ts
- extensions/qa-lab/src/live-transports/discord/discord-live.runtime.test.ts
- extensions/qa-lab/src/live-transports/discord/discord-live.runtime.ts
- extensions/qa-lab/src/live-transports/shared/live-artifacts.test.ts
- extensions/qa-lab/src/live-transports/shared/live-artifacts.ts
- extensions/qa-lab/src/live-transports/shared/live-gateway.runtime.ts
- extensions/qa-lab/src/live-transports/shared/live-transport-cli.runtime.test.ts
- extensions/qa-lab/src/live-transports/shared/live-transport-cli.runtime.ts
- extensions/qa-lab/src/live-transports/slack/cli.runtime.ts
- extensions/qa-lab/src/live-transports/slack/slack-live.runtime.test.ts
- extensions/qa-lab/src/live-transports/slack/slack-live.runtime.ts
- extensions/qa-lab/src/live-transports/telegram/cli.runtime.ts
- extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.test.ts
- extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.ts
- extensions/qa-lab/src/live-transports/whatsapp/cli.runtime.ts
- extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.test.ts
- extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.ts
- extensions/qa-lab/src/providers/mock-openai/server.test.ts
- extensions/qa-lab/src/providers/mock-openai/server.ts
- extensions/qa-lab/src/scenario-catalog.test.ts
- extensions/qa-lab/src/scenario-catalog.ts
- extensions/qa-lab/src/scenario-packs.test.ts
- extensions/qa-lab/src/scenario-packs.ts
- extensions/qa-lab/src/suite-runtime-agent-tools.test.ts
- extensions/qa-lab/src/suite-runtime-agent-tools.ts
- extensions/qa-matrix/src/runners/contract/runtime.test.ts
- extensions/qa-matrix/src/runners/contract/runtime.ts
- extensions/qa-matrix/src/runners/contract/scenario-catalog.ts
- extensions/qa-matrix/src/runners/contract/scenario-runtime-e2ee.ts
- extensions/qa-matrix/src/runners/contract/scenario-runtime-room.ts
- extensions/qa-matrix/src/runners/contract/scenario-runtime-shared.test.ts
- extensions/qa-matrix/src/runners/contract/scenario-runtime-shared.ts
- extensions/qa-matrix/src/runners/contract/scenario-runtime-state-files.ts
- extensions/qa-matrix/src/runners/contract/scenario-runtime.ts
- extensions/qa-matrix/src/runners/contract/scenario-types.ts
- extensions/qa-matrix/src/runners/contract/scenarios.test.ts
- extensions/qa-matrix/src/substrate/client.test.ts
- extensions/qa-matrix/src/substrate/fault-proxy.test.ts
- extensions/qa-matrix/src/substrate/fault-proxy.ts
- qa/scenarios/agents/subagent-fanout-synthesis.md
- qa/scenarios/index.md
- qa/scenarios/personal/failure-recovery.md
- qa/scenarios/personal/no-fake-progress.md
- qa/scenarios/plugins/kitchen-sink-live-openai.md
- qa/scenarios/runtime/first-hour-20-turn.md
- qa/scenarios/runtime/gateway-restart-inflight-run.md
- qa/scenarios/runtime/soak-100-turn.md
- qa/scenarios/runtime/streaming-final-integrity.md
- qa/scenarios/runtime/tools/apply-patch.md
- qa/scenarios/runtime/tools/fs-read.md
- scripts/e2e/openwebui-docker.sh
- scripts/e2e/telegram-user-crabbox-proof.ts
- scripts/e2e/telegram-user-driver.py
- scripts/lib/docker-e2e-plan.mjs
- scripts/lib/docker-e2e-scenarios.mjs
- scripts/list-prod-store-packages.mjs
- test/scripts/docker-e2e-plan.test.ts
- test/scripts/list-prod-store-packages.test.ts
- test/scripts/mantis-telegram-desktop-proof-workflow.test.ts
- test/scripts/sandbox-common-smoke-workflow.test.ts

The QA Lab is a specialized testing infrastructure within OpenClaw designed for high-fidelity automation and manual debugging of agent behaviors. Unlike standard unit tests, the QA Lab operates by spawning isolated Gateway instances, simulating real messaging transport layers, and executing complex multi-turn "Scenario Packs" that validate end-to-end agent logic across various model providers.

## Overview & Purpose

OpenClaw employs a multi-tiered testing strategy where the QA Lab serves as the bridge between developer unit tests and full production deployment. It complements existing suites by providing:

- Isolated Gateway Runtimes: Each QA scenario runs in its own ephemeral Gateway environment with unique configuration and credential patches extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.test.ts 196-215
- Transport Simulation & Realism: Uses adapters to drive real channels (Telegram, WhatsApp, Matrix) or fully synthetic ones via qa-channel extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.ts 34-38
- Model Frontier Testing: Validates agent performance on "frontier" models using real credentials leased via a central broker mechanism extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.ts 1-25

### Component Architecture

The QA system is primarily implemented within the `qa-lab` extension, which orchestrates scenario execution, runtime setup, credential leasing, and runtime integration with live transport lanes.

#### QA Lab Execution Lifecycle

```

```

Sources: [extensions/qa-lab/src/cli.runtime.ts69-82](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/cli.runtime.ts#L69-L82) [extensions/qa-lab/src/gateway-child.js1-16](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/gateway-child.js#L1-L16) [extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.ts16-33](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/live-transports/telegram/telegram-live.runtime.ts#L16-L33)

## Scenario Packs & Structure

Scenario Packs are collections of structured test cases under the `./qa` directory. These suites describe typical agent interaction flows and expected behavior outcomes. The scenarios are authored in Markdown and enhanced with metadata for execution control and coverage tracking.

### Scenario Definition

Key properties for each scenario include:

- Coverage ID: A unique string used for feature coverage tracking, e.g., memory.recall extensions/qa-lab/src/scenario-catalog.test.ts 45
- Standard ID: Identifies alignment to a baseline cross-platform standard scenario for parity comparison, such as canary or allowlist-block extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.ts 204-223
- Execution Flow: Defines step-wise conversational turns or tool invocations, typically described in rich markdown structures extensions/qa-lab/src/scenario-catalog.test.ts 32-36
- Runtime Parity Tier: Metadata tagging scenarios into categories such as standard , optional , or soak extensions/qa-lab/src/scenario-catalog.test.ts 109-120

### Executable vs. Manual Suites

OpenClaw supports two main types of QA suite runs:

1. Automated Suite: Invoked via qa run or transport-specific commands like qa telegram or qa whatsapp extensions/qa-lab/src/cli.runtime.ts 80-84 Runs scenarios programmatically, exercising the agent runtime fully. Results are collected into Markdown and JSON summary reports extensions/qa-lab/src/cli.runtime.test.ts 122-138
2. Manual Lab: Launched with qa ui . Presents a local QA lab server for interactive transcript inspection and manual scenario drives extensions/qa-lab/src/cli.runtime.ts 37-38

Sources: [extensions/qa-lab/src/scenario-catalog.ts1-10](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/scenario-catalog.ts#L1-L10) [extensions/qa-lab/src/cli.runtime.ts1-83](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/cli.runtime.ts#L1-L83) [extensions/qa-lab/src/scenario-catalog.test.ts12-46](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/scenario-catalog.test.ts#L12-L46)

## Gateway Child Orchestration

### Environment Isolation

Each QA scenario runs inside a **child Gateway process** that serves as the System Under Test (SUT). This isolation is critical to ensure clean test state and reproducibility.

- Temporary state directories and configs are spun up dynamically using mkdtemp extensions/qa-matrix/src/runners/contract/scenarios.test.ts 1-3
- The child Gateway process is launched with controlled lifecycle functions such as startMatrixQaOpenClawCli extensions/qa-matrix/src/runners/contract/scenarios.test.ts 17-26

### Gateway-to-Lab Communication

The QA Lab orchestrates via an RPC and CLI bridge that connects the local test runner with the Gateway SUT.

```

```

Sources: [extensions/qa-matrix/src/runners/contract/scenarios.test.ts14-26](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-matrix/src/runners/contract/scenarios.test.ts#L14-L26) [extensions/qa-matrix/src/runners/contract/scenario-runtime-cli.js1-44](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-matrix/src/runners/contract/scenario-runtime-cli.js#L1-L44)

## Shared Live-Credential Leasing (Convex Credential Broker)

Testing "frontier" providers and real-world models securely requires sharing precious, limited secrets across parallel CI jobs. OpenClaw implements a **Convex-backed credential broker**.

- Acquisition: The QA runtime calls acquireQaCredentialLease to get credentials for a specific role (e.g., Telegram bot tokens) extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.ts 22
- Persistence: Leases are kept alive during scenarios via startQaCredentialLeaseHeartbeat extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.ts 23
- Management: Commands like qa credentials add and qa credentials list allow admins to manage the secret pool extensions/qa-lab/src/qa-credentials-admin.runtime.ts 47-51

Sources: [extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.ts1-28](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.ts#L1-L28) [extensions/qa-lab/src/qa-credentials-admin.runtime.ts1-53](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/qa-credentials-admin.runtime.ts#L1-L53)

## Mock Provider Integration

The QA Lab includes a `mock-openai` server to facilitate testing without consuming real API credits or requiring external connectivity.

- Variant Resolution: The mock server can simulate different providers (OpenAI, Anthropic) based on the model string via resolveProviderVariant extensions/qa-lab/src/providers/mock-openai/server.ts 65-92
- Scenario Dispatch: It supports specialized prompt regexes to trigger specific QA behaviors, such as reasoning-only recovery or tool progress checks extensions/qa-lab/src/providers/mock-openai/server.ts 145-164

```

```

Sources: [extensions/qa-lab/src/providers/mock-openai/server.ts1-106](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/providers/mock-openai/server.ts#L1-L106) [extensions/qa-lab/src/providers/mock-openai/server.test.ts1-162](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/providers/mock-openai/server.test.ts#L1-L162)

## Mantis & Proof Workflows

For bugs that require visual evidence or interaction with native desktop clients (like Telegram Desktop), OpenClaw uses the **Mantis** system. Mantis orchestrates "proof" workflows that capture screenshots or video of agent interactions in a controlled environment.

### Desktop Proof Infrastructure

Mantis utilizes **Crabbox** for ephemeral desktop environments. A specialized GitHub Action `mantis-telegram-desktop-proof.yml` automates the process of:

1. Acquiring a Crabbox lease scripts/e2e/telegram-user-crabbox-proof.ts 49
2. Deploying the OpenClaw Gateway and a "User Driver" script scripts/e2e/telegram-user-crabbox-proof.ts 135
3. Executing the scenario and recording the desktop scripts/e2e/telegram-user-crabbox-proof.ts 64
4. Publishing a motion GIF or video as evidence to the PR scripts/e2e/telegram-user-crabbox-proof.ts 183

Sources: [.github/workflows/mantis-telegram-desktop-proof.yml1-124](https://github.com/openclaw/openclaw/blob/fbdf5937/.github/workflows/mantis-telegram-desktop-proof.yml#L1-L124) [scripts/e2e/telegram-user-crabbox-proof.ts1-196](https://github.com/openclaw/openclaw/blob/fbdf5937/scripts/e2e/telegram-user-crabbox-proof.ts#L1-L196) [test/scripts/mantis-telegram-desktop-proof-workflow.test.ts78-162](https://github.com/openclaw/openclaw/blob/fbdf5937/test/scripts/mantis-telegram-desktop-proof-workflow.test.ts#L78-L162)

## Comparison with Other Test Suites

OpenClaw implements a multi-layered testing approach:

Sources: [extensions/qa-lab/src/cli.runtime.ts69-82](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/cli.runtime.ts#L69-L82) [extensions/qa-lab/src/scenario-catalog.test.ts1-110](https://github.com/openclaw/openclaw/blob/fbdf5937/extensions/qa-lab/src/scenario-catalog.test.ts#L1-L110) [docs/help/testing.md10-80](https://github.com/openclaw/openclaw/blob/fbdf5937/docs/help/testing.md?plain=1#L10-L80)
