OpenTester

MCP-First Testing Execution Infrastructure

OpenTester is a testing execution engine designed for AI coding tools (Claude Code, Cursor, OpenCode, etc.). It provides a unified DSL format and MCP interface, enabling Agents to generate, execute, and manage test cases, achieving an automated "code-test-fix" workflow.

Core Positioning

┌─────────────────────────────────────────┐
│  AI Agent (Claude Code / Cursor / ...)  │
│  ├─ Generate DSL test cases             │
│  ├─ Decide testing strategies           │
│  └─ Analyze failure reasons             │
├─────────────────────────────────────────┤
│  OpenTester (MCP Server)                │
│  ├─ Validate DSL syntax                 │
│  ├─ Execute tests (CLI/Web)             │
│  ├─ Store cases/projects                │
│  └─ Return structured results           │
├─────────────────────────────────────────┤
│  Web UI (Auxiliary Observation Panel)   │
│  ├─ View execution progress             │
│  ├─ Debug cases (create/edit)           │
│  └─ View history reports                │
└─────────────────────────────────────────┘

Design Principles:

• Agent Intelligence: Test generation and failure analysis are handled by the Agent

• OpenTester Execution: Focuses on DSL validation and test execution

• MCP-First: All core features exposed through MCP

• Web UI Auxiliary: Visual monitoring and debugging, not required

Related MCP server: EvalView

Installation

Option 1: Install from PyPI (Recommended)

pip install opentester

# Or with uv
uv pip install opentester

Option 2: Install from Source

# Clone repository
git clone https://github.com/kznr02/OpenTester.git
cd OpenTester

# Install backend
uv pip install ./backend

# Install frontend dependencies (optional)
cd frontend
npm install

Option 3: Direct Run (Development)

cd backend
uv run opentester start

Quick Start

Start Services

# Start both FastAPI + MCP (foreground mode with prefixed logs)
opentester start

# Daemon mode (detached background with log files)
opentester start --daemon

# Start only API
opentester start --api

# Start only MCP
opentester start --mcp

# Custom ports
opentester start --api-port 8080 --mcp-port 8081

# Check status
opentester status

# Stop services
opentester stop

# Environment check
opentester doctor

Foreground Mode Log Output:

[API] INFO:     Started server process [12345]
[API] INFO:     Waiting for application startup.
[MCP] INFO:     Started MCP server on port 8001
[API] INFO:     Application startup complete.

Configure Claude Code (Streamable HTTP)

Add MCP configuration in .claude/settings.json:

{
  "mcpServers": {
    "opentester": {
      "url": "http://localhost:8001/mcp"
    }
  }
}

Note: Current runtime transport is Streamable HTTP.

Usage Example

Conversation with Claude Code:

You: Help me test the login functionality

Claude: I'll create tests for the login feature...
        [Generate DSL test case]
        [Call MCP: validateDSL] ✓ Validation passed
        [Call MCP: createProject] ✓ Project created
        [Call MCP: saveCase] ✓ Case saved
        [Call MCP: runCase]
        Executing...
        ✓ All passed

Project Structure

OpenTester/
├── backend/              # FastAPI + Python
│   ├── opentester/
│   │   ├── api/         # REST API (for Web UI)
│   │   ├── core/        # Execution engine, storage
│   │   ├── models/      # Pydantic models (DSL, Project)
│   │   └── mcp/         # MCP Server (core)
│   └── pyproject.toml
├── frontend/            # React + TypeScript + Vite
│   └── src/             # Web UI (observation panel)
├── docs/
│   ├── SKILL_PROMPT.md  # Agent Skill Prompt template
│   ├── DSL_SPEC.md      # DSL syntax specification
│   └── MCP.md           # MCP interface documentation
├── README.md            # This document
└── LICENSE              # MIT License

Core Features

1. DSL Validation

After Agent generates DSL, OpenTester validates syntax correctness:

version: "1.0"
meta:
  name: "Login Test"
steps:
  - action: exec
    command: "curl http://localhost:3000/login"
  - action: assert
    assertion:
      type: stdout_contains
      expected: "token"

2. Test Execution

Supports execution targets:

• CLI: subprocess command execution (implemented)

• WEB: Playwright-based browser automation (implemented)

• GUI: experimental executor routing (disabled by default)

• TUI: experimental executor routing (disabled by default)

3. Web Testing MVP + AI DOM Analysis

• Web actions are executed by WebExecutor (Playwright)

• Browser console, page errors, and failed requests are normalized into diagnostic_events for each execution step

• Supports AI-assisted locator flow via ai_locator in DSL steps

• Execution can pause in paused_waiting_for_ai status and wait for AI selector submission

• Supports DOM snapshot + optional screenshot capture for AI analysis

• Web UI and REST clients can retrieve persisted browser diagnostics from the execution diagnostics endpoints for post-run investigation

3. Project Management

• Projects stored in XDG data directory (~/.local/share/opentester/ on Linux, ~/Library/Application Support/opentester/ on macOS, %LOCALAPPDATA%\opentester\ on Windows)

• PRD content persisted with projects (provided by Agent)

• Test case version management

• Template library for reusable DSL patterns

4. Real-time Monitoring

• WebSocket real-time execution progress push

• Web UI visual display

• Execution history traceability

MCP Tools

See MCP Interface Documentation for details.

Documentation

See documentation for detailed guides:

• Architecture - System architecture and design decisions

• MCP Interface - MCP tool specifications

• DSL Specification - YAML-based test definition language

• Development Guide - Development setup and contribution

Agents using OpenTester need to include DSL generation specifications. Refer to SKILL_PROMPT.md

DSL Specification

YAML-based test definition language. See DSL_SPEC.md for details.

Architecture

See ARCHITECTURE.md for system architecture, design principles, and architectural decisions.

Development Guide

Developers refer to DEVELOPMENT.md

Web UI

Auxiliary features:

• View project list and details

• Edit DSL cases (Monaco editor)

• Monitor execution progress

• View history reports

Note: Web UI is not the main entry point. All core features are provided through MCP.

Ports

• FastAPI (Web UI / REST API): http://localhost:8000

• MCP Server: http://localhost:8001/mcp

• API Docs: http://localhost:8000/docs

• Web UI Dev Server: http://localhost:5173

Data Storage

OpenTester follows the XDG Base Directory Specification:

• Projects: <XDG_DATA_HOME>/opentester/projects/{project_id}.json

  • Linux: ~/.local/share/opentester/projects/

  • macOS: ~/Library/Application Support/opentester/projects/

  • Windows: %LOCALAPPDATA%\opentester\projects\

• Executions: <XDG_DATA_HOME>/opentester/executions/

• Templates: <XDG_DATA_HOME>/opentester/templates/

• Logs: <XDG_DATA_HOME>/opentester/logs/daemon/ (daemon mode service logs)

• Config: ~/.config/opentester/ (or $XDG_CONFIG_HOME/opentester/)

Distribution

Refer to DISTRIBUTION.md for:

• PyPI publishing

• PyInstaller packaging

• Docker images

• System package managers

Quick build executable:

cd backend
pip install pyinstaller
pyinstaller opentester.spec
# Output: dist/opentester.exe (Windows) or dist/opentester (Linux/Mac)

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Documentation

• English Documentation

• 中文文档

License

MIT License

OpenTester - MCP-First Testing Execution Platform