<!-- source: https://deepwiki.com/MervinPraison/PraisonAI/11.1-testing-framework -->

## Test Examples and Patterns

### Example Test Classes

The testing framework includes several example test classes that demonstrate different testing patterns:

### Example Test Methods

```

```

Sources: [src/praisonai/tests/test.py15-100](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L15-L100)

# Testing Framework

The PraisonAI testing framework provides comprehensive test infrastructure for validating agent functionality, framework integrations, and API interactions. The framework is built on pytest with custom orchestration through `test_runner.py` and automated execution via GitHub Actions workflows. Test execution is organized into pattern-based categories with strict separation between mock tests (no API costs) and real API tests (explicit confirmation required).

## Purpose and Scope

The testing system validates:

- Core agent functionality through unit tests with mocked LLM responses
- Framework integrations (AutoGen, CrewAI) through isolated integration tests
- Protocol implementations (MCP, A2A) with mocked external services
- Real API behavior through manual-trigger workflows with cost controls
- Performance characteristics via benchmark tests
- Code coverage across all execution paths

Key design principles:

- Zero API costs by default : All automated workflows use mocked responses
- Explicit cost confirmation : Real API tests require manual trigger and boolean confirmation
- Fast feedback : Unit tests complete in <1 second each
- Test isolation : Root configuration backup/restore prevents test interference
- Pattern-based execution : Single command for running specific test categories

## Test Architecture Overview

The testing framework consists of three primary layers:

1. Test Runner Layer : test_runner.py provides command-line interface with pattern-based routing to pytest
2. Test Discovery Layer : pytest discovers tests across tests/unit/ , tests/integration/ , and tests/e2e/ directories
3. Execution Layer : pytest executes tests with markers ( @pytest.mark.real , @pytest.mark.slow ) controlling behavior

Central orchestration occurs through the `run_tests()` function in `test_runner.py`, which maps patterns to pytest command configurations and enforces safety checks for cost-incurring tests.

### Test Organization Structure

```

```

Sources: [src/praisonai/tests/test_runner.py1-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L1-L363) [src/praisonai/tests/test.py1-108](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L1-L108) [src/praisonai/tests/unit/test_cli_features.py1-425](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_cli_features.py#L1-L425)

### CI/CD Workflow Architecture

```

```

Sources: [.github/workflows/unittest.yml1-516](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L1-L516) [.github/workflows/test-core.yml1-288](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L1-L288) [.github/workflows/test-comprehensive.yml1-196](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L1-L196) [.github/workflows/test-extended.yml1-175](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L1-L175) [.github/workflows/test-frameworks.yml1-154](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-frameworks.yml#L1-L154) [.github/workflows/test-real.yml1-168](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L1-L168) [.github/workflows/coverage.yml1-62](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/coverage.yml#L1-L62)
</old_str>

<new_str>

## Test Architecture Overview

The testing framework is organized into multiple categories with dedicated CI/CD workflows and supports testing across different Python versions and AI frameworks. The central test orchestration is handled by `test_runner.py`, which provides pattern-based test selection and execution.

### Test Organization Structure

```

```

Sources: [src/praisonai/tests/test_runner.py1-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L1-L363) [src/praisonai/tests/test.py1-108](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L1-L108) [src/praisonai/tests/unit/test_cli_features.py1-425](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_cli_features.py#L1-L425)

### CI/CD Workflow Architecture

```

```

Sources: [.github/workflows/unittest.yml1-516](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L1-L516) [.github/workflows/test-core.yml1-288](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L1-L288) [.github/workflows/test-comprehensive.yml1-196](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L1-L196) [.github/workflows/test-extended.yml1-175](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L1-L175) [.github/workflows/test-frameworks.yml1-154](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-frameworks.yml#L1-L154) [.github/workflows/test-real.yml1-168](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L1-L168) [.github/workflows/coverage.yml1-62](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/coverage.yml#L1-L62)

## Test Categories and Organization

### Test Runner (test_runner.py)

The `test_runner.py` script implements pattern-based test routing through the `run_tests(pattern, verbose, coverage)` function which constructs pytest commands based on pattern selection.

**Pattern-to-Command Mapping:**

**Command Line Interface:**

The `main()` function parses arguments and enforces safety checks before test execution:

```

```

**Safety Check Implementation:**

For patterns containing "real" or "e2e", the runner enforces interactive confirmation:

```

```

For "full" execution patterns, a stronger confirmation is required:

```

```

Sources: [src/praisonai/tests/test_runner.py157-276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L157-L276) [src/praisonai/tests/test_runner.py278-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L278-L363)

### Pytest Markers

pytest markers control test execution through the `-m` flag. Markers are defined in test methods using decorators and filtered during execution.

**Marker Definitions:**

**Implementation in TestPraisonAIFramework:**

```

```

**Implementation in TestPraisonAICommandLine:**

```

```

**Marker Filtering in test_runner.py:**

The `run_tests()` function applies marker filters based on pattern:

```

```

Sources: [src/praisonai/tests/test.py16-49](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L16-L49) [src/praisonai/tests/test.py68-74](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L68-L74) [src/praisonai/tests/test_runner.py224-236](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L224-L236)

### Mock vs Real Tests

The framework maintains a strict separation between mock tests and real API tests to prevent accidental costs.

**Mock Test Characteristics:**

- Use mocked LLM responses and API calls
- Execute in standard CI/CD workflows (push, PR)
- Fast execution (< 1 second per test)
- No API key requirements
- Default for all test categories

**Real Test Characteristics:**

- Make actual API calls to LLM providers
- Execute only via manual workflow dispatch
- Require valid API keys and cost confirmation
- Located in tests/e2e/ directory
- Marked with @pytest.mark.real

### Test Directory Structure

```

```

Sources: [src/praisonai/tests/test_runner.py202-256](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L202-L256) [src/praisonai/tests/test.py1-108](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L1-L108) [src/praisonai/tests/unit/test_cli_features.py1-425](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_cli_features.py#L1-L425)

### Unit Tests

Unit tests focus on individual components and classes with minimal external dependencies. They use mocked API calls to ensure fast execution and reliable results.

Sources: [src/praisonai/tests/unit/test_cli_features.py1-425](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_cli_features.py#L1-L425)

### Integration Tests

Integration tests validate component interactions and framework integrations with controlled environments using mocked external services.

Sources: [src/praisonai/tests/test_runner.py206-222](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L206-L222)

### Framework-Specific Tests

Framework tests validate integration with external AI frameworks using the `TestPraisonAIFramework` class.

### Framework Test Implementation

```

```

Sources: [src/praisonai/tests/test.py15-49](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L15-L49)

## CI/CD Pipeline Architecture

The testing framework uses multiple GitHub Actions workflows for different testing scenarios and triggers.

### Workflow Triggers and Matrix

```

```

Sources: [.github/workflows/unittest.yml3](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L3-L3) [.github/workflows/test-core.yml3-7](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L3-L7) [.github/workflows/test-comprehensive.yml3-24](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L3-L24) [.github/workflows/test-extended.yml3-11](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L3-L11) [.github/workflows/test-frameworks.yml3-22](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-frameworks.yml#L3-L22) [.github/workflows/test-real.yml6-23](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L6-L23)

### Test Execution Environment

CI/CD workflows establish consistent test environments through standardized setup steps.

**UV Package Manager Installation:**

All workflows begin by installing the `uv` package manager for fast dependency resolution:

```

```

**Dependency Installation:**

Workflows install the full dependency set including optional feature groups:

```

```

**Environment Variable Configuration:**

Environment variables are set for API access and debugging:

```

```

Environment variable purposes:

- OPENAI_API_KEY : API authentication (fallback prevents unset errors)
- OPENAI_API_BASE : Custom API endpoint support
- OPENAI_MODEL_NAME : Model override for testing
- LOGLEVEL : Verbose logging for debugging
- PYTHONPATH : Module resolution for praisonai-agents

**Test Isolation Through Config Backup:**

The `unittest.yml` workflow implements root configuration file backup to prevent test interference:

```

```

The restoration step ensures cleanup even if tests fail:

```

```

Sources: [.github/workflows/unittest.yml19-22](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L19-L22) [.github/workflows/unittest.yml25-32](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L25-L32) [.github/workflows/unittest.yml34-40](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L34-L40) [.github/workflows/unittest.yml42-55](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L42-L55) [.github/workflows/unittest.yml505-516](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L505-L516)

## Test Execution Framework

### test_runner.py Implementation

The `test_runner.py` module provides the `run_tests(pattern, verbose, coverage)` function which constructs and executes pytest commands based on pattern selection.

**Function Signature and Command Construction:**

```

```

**Pattern-to-Command Logic:**

The function implements a decision tree for pattern matching:

```

```

**Real Test Safety Checks:**

Before executing real tests, the function performs API key validation:

```

```

**Full Execution Mode:**

For comprehensive real tests, the function enables real-time output:

```

```

Sources: [src/praisonai/tests/test_runner.py157-276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L157-L276)

### pytest Configuration

pytest is invoked with standardized arguments configured in `test_runner.py` and CI/CD workflows.

**Standard Pytest Arguments:**

The `run_tests()` function appends standard flags to all test commands:

```

```

**Coverage Configuration:**

When the `--coverage` flag is provided, pytest-cov arguments are added:

```

```

In CI/CD workflows, coverage is configured for branch coverage and multiple report formats:

```

```

**Additional Pytest Plugins:**

Workflows install pytest plugins for enhanced functionality:

```

```

Plugin purposes:

- pytest-asyncio : Support for async def test functions
- pytest-cov : Code coverage measurement
- pytest-timeout : Timeout enforcement for long-running tests

**Test Suite Organization in run_test_suite():**

The legacy `run_test_suite()` function defines test categories with dedicated pytest configurations:

```

```

Sources: [src/praisonai/tests/test_runner.py30-49](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L30-L49) [src/praisonai/tests/test_runner.py259-263](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L259-L263) [src/praisonai/tests/test_runner.py173-174](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L173-L174) [.github/workflows/unittest.yml30](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L30-L30) [.github/workflows/unittest.yml476](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L476-L476)

### Test Runner Execution Flow

**Title: test_runner.py Command Construction and Safety Check Flow**

```

```

Sources: [src/praisonai/tests/test_runner.py278-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L278-L363) [src/praisonai/tests/test_runner.py157-276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L157-L276) [src/praisonai/tests/test_runner.py306-341](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L306-L341)

### Test Execution Flow

```

```

Sources: [.github/workflows/test-core.yml224-227](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L224-L227) [.github/workflows/test-comprehensive.yml70-95](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L70-L95) [src/praisonai/tests/test.py15-49](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L15-L49)

### API Key Management and Safety

API key handling implements defense-in-depth with validation at multiple levels: GitHub Secrets, environment variables, shell validation, Python validation, and actual API testing.

### API Key Flow in CI/CD Workflows

```

```

Sources: [.github/workflows/unittest.yml34-75](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L34-L75) [.github/workflows/test-core.yml41-79](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L41-L79) [.github/workflows/test-real.yml65-78](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L65-L78)

### API Key Validation Process

The workflows include dedicated steps for API key validation:

**Multi-Stage API Key Validation:**

The workflows implement a four-stage validation pipeline:

**Stage 1: Bash Environment Check**

```

```

**Stage 2: Python Environment Check**

```

```

**Stage 3: API Connectivity Test**

```

```

**Stage 4: Framework Integration Check**

The workflow includes debugging steps to trace how API keys flow through PraisonAI:

```

```

Sources: [.github/workflows/unittest.yml57-75](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L57-L75) [.github/workflows/unittest.yml76-92](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L76-L92) [.github/workflows/unittest.yml339-356](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L339-L356) [.github/workflows/unittest.yml303-337](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L303-L337)

## Coverage Reporting

### Codecov Integration

The framework integrates with Codecov for comprehensive coverage tracking across multiple test workflows.

**Coverage Configuration:**

- Branch coverage enabled with --cov-branch
- XML report generation for Codecov upload
- Multiple coverage flags for different test categories: quick-validation : Fast unit tests core-tests : Core functionality tests comprehensive-tests : Full test suite unit-tests : Unit test specific coverage

**Coverage Commands:**

```

```

### Artifact Management

Test results and coverage reports are preserved as GitHub Actions artifacts with configurable retention periods:

### Coverage Workflow

```

```

Sources: [.github/workflows/unittest.yml494-502](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L494-L502) [.github/workflows/test-core.yml266-285](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L266-L285) [.github/workflows/coverage.yml52-60](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/coverage.yml#L52-L60)

## Performance Testing

### Benchmark Categories

The extended test workflow includes performance benchmarks for key operations:

**Agent Creation Performance:**

```

```

**Import Performance:**

```

```

**Memory Usage:**

```

```

### Performance Metrics Collection

```

```

Sources: [.github/workflows/test-extended.yml100-144](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L100-L144) [.github/workflows/test-extended.yml158-168](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L158-L168)

## Real API Testing with Cost Controls

### Safety Mechanisms

Real API tests include multiple safety mechanisms to prevent accidental costs:

1. Manual Trigger Only : workflow_dispatch exclusively
2. Cost Confirmation : Boolean input requirement ( confirm_costs )
3. Framework Selection : Explicit framework choice ( autogen , crewai , all , none )
4. API Key Verification : Pre-test validation
5. Minimal Test Design : Reduced API call volume
6. Safety Fallback Job : Runs when confirmation is missing

### test-real.yml Workflow Configuration

**Workflow Inputs:**

```

```

**Safety Job Condition:**

```

```

**Safety Check Job:**

```

```

Sources: [.github/workflows/test-real.yml6-23](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L6-L23) [.github/workflows/test-real.yml26-29](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L26-L29) [.github/workflows/test-real.yml146-168](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L146-L168)

### Cost Warning System

**Pre-Execution Warning:**

```

```

**Test Runner Confirmation:**
The `test_runner.py` script includes interactive confirmations for real tests:

```

```

Sources: [.github/workflows/test-real.yml36-42](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L36-L42) [src/praisonai/tests/test_runner.py306-341](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L306-L341)

### Cost Control Architecture

```

```

Sources: [.github/workflows/test-real.yml28](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L28-L28) [.github/workflows/test-real.yml35-40](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L35-L40) [.github/workflows/test-real.yml146-166](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L146-L166) [.github/workflows/test-real.yml6-23](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L6-L23)

## Legacy Test Integration

### TestPraisonAICommandLine Class

The framework includes command-line interface testing through the `TestPraisonAICommandLine` class, which validates CLI functionality with proper environment variable propagation.

**CLI Test Methods:**

- test_praisonai_command() : Tests basic CLI execution with framework flags
- test_praisonai_init_command() : Validates agent file generation
- Environment variable inheritance for subprocess execution

**Command Patterns:**

```

```

Sources: [src/praisonai/tests/test.py52-84](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L52-L84)
