<!-- source: https://deepwiki.com/MervinPraison/PraisonAI/11-testing-and-quality-assurance -->

# Testing and Quality Assurance

- .github/ISSUE_TEMPLATE/bug_report.md
- .github/actions/claude-code-action/action.yml
- .github/actions/claude-issue-triage-action/action.yml
- .github/actions/gemini/action.yml
- .github/workflows/auto-issue-comment.yml
- .github/workflows/auto-pr-comment.yml
- .github/workflows/build-image.yml
- .github/workflows/claude-issue-triage.yml
- .github/workflows/claude.yml
- .github/workflows/coverage.yml
- .github/workflows/docker-publish.yml
- .github/workflows/gemini-issue-automated-triage.yml
- .github/workflows/gemini-issue-review.yml
- .github/workflows/gemini-pr-review.yml
- .github/workflows/python-package.yml
- .github/workflows/test-comprehensive.yml
- .github/workflows/test-core.yml
- .github/workflows/test-extended.yml
- .github/workflows/test-frameworks.yml
- .github/workflows/test-real.yml
- .github/workflows/unittest.yml
- src/praisonai-agents/autoagents-tavily.py
- src/praisonai-agents/praisonaiagents/agents/autoagents.py
- src/praisonai-agents/praisonaiagents/llm/model_capabilities.py
- src/praisonai/praisonai/README.md
- src/praisonai/tests/test.py
- src/praisonai/tests/test_runner.py
- src/praisonai/tests/unit/test_cli_features.py

## Purpose and Scope

This document describes PraisonAI's testing framework, CI/CD pipeline, and quality assurance processes. It covers test organization, automated testing workflows, AI-powered code review systems, and development practices for maintaining code quality.

**Related documentation:**

- For deployment infrastructure, see Deployment and Infrastructure
- For package structure and dependencies, see Package System
- For development environment setup, see Development Environment

## Test Infrastructure Overview

PraisonAI uses a sophisticated testing infrastructure built around pytest, GitHub Actions, and AI-assisted automation. The system is designed to provide fast feedback for developers while protecting against API costs through tiered testing strategies.

### Test Runner Architecture

```

```

**Sources:** [src/praisonai/tests/test_runner.py1-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L1-L363) [.github/workflows/unittest.yml1-517](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L1-L517)

The test runner provides pattern-based routing to different test suites:

### Test Execution Flow

```

```

**Sources:** [src/praisonai/tests/test_runner.py157-276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L157-L276) [.github/workflows/unittest.yml42-56](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L42-L56) [.github/workflows/unittest.yml505-516](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L505-L516)

The test execution flow includes multiple safety mechanisms:

1. Config file backup - Root agents.yaml is backed up before tests to prevent default file resolution interference unittest.yml 42-56
2. API key validation - Real tests verify API keys are present before execution test_runner.py 184-188
3. User confirmation - Real tests require explicit user confirmation test_runner.py 316-319
4. Full execution gates - Tests that run praisonai.run() require typing 'EXECUTE' to proceed test_runner.py 334-337

## Test Categories and Organization

### Test Directory Structure

```
tests/
|-- unit/                          # Fast, isolated unit tests (<1s each)
|   |-- test_core_agents.py       # Core Agent class tests
|   |-- cli/                       # CLI feature tests
|   |   `-- test_cli_features.py  # GuardrailHandler, MetricsHandler, etc.
|   `-- ...
|-- integration/                   # Integration tests (mocked LLM calls)
|   |-- autogen/                   # AutoGen framework integration
|   |-- crewai/                    # CrewAI framework integration
|   |-- test_mcp_integration.py   # MCP protocol tests
|   `-- test_rag_integration.py   # RAG system tests
|-- e2e/                           # Real end-to-end tests (actual API calls)
|   |-- autogen/                   # Real AutoGen tests
|   `-- crewai/                    # Real CrewAI tests
`-- test_runner.py                 # Central test orchestration
```

**Sources:** [src/praisonai/tests/test_runner.py19-73](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L19-L73)

### Test Types and Markers

```

```

**Sources:** [src/praisonai/tests/test.py16-104](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L16-L104) [src/praisonai/tests/test_runner.py177-191](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L177-L191)

Test markers control execution behavior:

- @pytest.mark.real - Tests that make actual API calls test.py 16-43 Requires valid OPENAI_API_KEY environment variable Only runs when explicitly invoked with -m real flag Triggers cost warnings and user confirmation prompts test_runner.py 307-320
- @pytest.mark.slow - Long-running tests (>10s) Excluded from fast test runs Typically involves complex multi-agent workflows

### Unit Tests

Unit tests focus on isolated component functionality with no external dependencies. Example structure:

```

```

**Sources:** [src/praisonai/tests/unit/test_cli_features.py16-66](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_cli_features.py#L16-L66)

### Integration Tests

Integration tests validate complex features with mocked LLM calls:

```

```

**Sources:** [src/praisonai/tests/test.py23-28](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test.py#L23-L28)

### Framework-Specific Tests

Tests for AutoGen and CrewAI integration are organized separately:

**Sources:** [.github/workflows/test-frameworks.yml1-154](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-frameworks.yml#L1-L154)

## CI/CD Pipeline Architecture

### Workflow Overview

```

```

**Sources:** [.github/workflows/unittest.yml1-4](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L1-L4) [.github/workflows/test-core.yml1-8](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L1-L8) [.github/workflows/test-comprehensive.yml1-24](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L1-L24) [.github/workflows/test-frameworks.yml1-23](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-frameworks.yml#L1-L23) [.github/workflows/test-real.yml1-24](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L1-L24)

### Workflow Execution Matrix

**Sources:** [.github/workflows/unittest.yml8](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L8-L8) [.github/workflows/test-core.yml12](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L12-L12) [.github/workflows/test-comprehensive.yml29](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L29-L29) [.github/workflows/test-real.yml29](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-real.yml#L29-L29)

### Environment Setup and Dependencies

All workflows follow a consistent setup pattern:

```

```

**Sources:** [.github/workflows/unittest.yml11-93](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L11-L93) [.github/workflows/test-core.yml18-105](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L18-L105)

Dependency installation pattern used across all workflows:

```

```

**Sources:** [.github/workflows/unittest.yml25-32](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L25-L32)

### Environment Variable Management

```

```

**Sources:** [.github/workflows/unittest.yml34-40](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L34-L40) [.github/workflows/test-core.yml41-52](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L41-L52)

The environment setup includes extensive debugging steps to validate API key availability:

```

```

**Sources:** [.github/workflows/unittest.yml58-75](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L58-L75) [.github/workflows/test-core.yml58-78](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L58-L78)

## AI-Powered Code Review and Automation

### AI Assistant Architecture

```

```

**Sources:** [.github/workflows/claude.yml1-160](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/claude.yml#L1-L160) [.github/workflows/gemini-pr-review.yml1-248](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/gemini-pr-review.yml#L1-L248) [.github/workflows/gemini-issue-review.yml1-203](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/gemini-issue-review.yml#L1-L203) [.github/workflows/auto-issue-comment.yml1-26](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/auto-issue-comment.yml#L1-L26) [.github/workflows/auto-pr-comment.yml1-23](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/auto-pr-comment.yml#L1-L23)

### Claude Code Action

The Claude integration provides extensive tool permissions for automated code modifications:

```

```

**Sources:** [.github/workflows/claude.yml38-57](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/claude.yml#L38-L57)

Claude workflows follow this pattern:

1. Trigger detection - @claude mention in comments claude.yml 16-20
2. Context extraction - GitHub event parsing actions/claude-code-action/action.yml 86-219
3. Tool execution - Claude can execute git, gh, pytest commands claude.yml 38-57
4. Automatic review - Post-execution review comment claude.yml 59-160

### Gemini Integration Workflows

#### PR Review Process

```

```

**Sources:** [.github/workflows/gemini-pr-review.yml38-186](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/gemini-pr-review.yml#L38-L186)

The Gemini PR review includes structured output format:

```

```

**Sources:** [.github/workflows/gemini-pr-review.yml206-247](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/gemini-pr-review.yml#L206-L247)

#### Issue Resolution Process

```

```

**Sources:** [.github/workflows/gemini-issue-review.yml88-162](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/gemini-issue-review.yml#L88-L162)

Gemini issue resolution includes mandatory workflow steps:

```

```

**Sources:** [.github/workflows/gemini-issue-review.yml100-162](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/gemini-issue-review.yml#L100-L162)

### Auto-Comment Triggers

Automated triggers request AI review on new issues and PRs:

```

```

**Sources:** [.github/workflows/auto-issue-comment.yml1-26](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/auto-issue-comment.yml#L1-L26) [.github/workflows/auto-pr-comment.yml1-23](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/auto-pr-comment.yml#L1-L23)

## Coverage and Quality Metrics

### Coverage Collection Flow

```

```

**Sources:** [.github/workflows/unittest.yml495-503](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L495-L503) [.github/workflows/test-core.yml268-277](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L268-L277) [.github/workflows/test-comprehensive.yml148-156](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L148-L156) [.github/workflows/coverage.yml54-62](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/coverage.yml#L54-L62)

### Coverage Configuration

All workflows use consistent coverage collection:

```

```

**Sources:** [.github/workflows/unittest.yml474-476](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L474-L476)

Coverage uploads to Codecov with specific flags:

**Sources:** [.github/workflows/unittest.yml500](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L500-L500) [.github/workflows/test-core.yml274](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L274-L274) [.github/workflows/test-comprehensive.yml153](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-comprehensive.yml#L153-L153) [.github/workflows/coverage.yml59](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/coverage.yml#L59-L59)

### Test Artifacts

Workflows upload test results and coverage reports as artifacts:

```

```

**Sources:** [.github/workflows/unittest.yml495-503](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L495-L503) [.github/workflows/test-core.yml279-288](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L279-L288)

## Development Workflow

### Running Tests Locally

```

```

**Sources:** [src/praisonai/tests/test_runner.py278-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L278-L363)

### Test Execution Commands

```

```

**Sources:** [src/praisonai/tests/test_runner.py278-363](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L278-L363)

### Running Real API Tests

Real tests require explicit user confirmation due to API costs:

```

```

**Sources:** [src/praisonai/tests/test_runner.py306-341](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L306-L341)

The test runner includes safety confirmations:

```

```

**Sources:** [src/praisonai/tests/test_runner.py306-341](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L306-L341)

### Pre-commit Testing Best Practices

Recommended workflow before committing:

```

```

**Sources:** [src/praisonai/tests/test_runner.py157-276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py#L157-L276)

## Performance Monitoring

### Performance Test Categories

```

```

**Sources:** [.github/workflows/test-extended.yml1-175](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L1-L175)

### Performance Benchmark Implementation

```

```

**Sources:** [.github/workflows/test-extended.yml114-155](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L114-L155)

### Example Script Testing

Extended workflow tests key example scripts with timeout protection:

```

```

**Sources:** [.github/workflows/test-extended.yml51-76](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L51-L76)

### Performance Reports

Performance test results are uploaded as artifacts:

```

```

**Sources:** [.github/workflows/test-extended.yml158-175](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-extended.yml#L158-L175)

## Test Environment Configuration

### Config File Backup Mechanism

To prevent test interference from root configuration files, tests implement a backup/restore mechanism:

```

```

**Sources:** [.github/workflows/unittest.yml42-56](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L42-L56) [.github/workflows/unittest.yml505-516](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L505-L516)

This mechanism solves the issue where PraisonAI's default file resolution (looking for `agents.yaml` in the working directory) could cause tests specifying explicit YAML files to load the wrong configuration.

### Environment Variable Debugging

Extensive debugging steps validate environment setup before test execution:

```

```

**Sources:** [.github/workflows/unittest.yml58-93](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L58-L93) [.github/workflows/test-core.yml58-105](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L58-L105) [.github/workflows/test-core.yml107-123](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L107-L123) [.github/workflows/test-core.yml125-195](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L125-L195)

Example debugging output structure:

```

```

**Sources:** [.github/workflows/unittest.yml58-78](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/unittest.yml#L58-L78) [.github/workflows/test-core.yml58-78](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/test-core.yml#L58-L78)

## Summary

PraisonAI's testing infrastructure provides:

1. Multi-tier test strategy - Fast unit tests, comprehensive integration tests, and gated real API tests
2. Pattern-based test routing - Centralized test_runner.py with intuitive pattern selection
3. Cost protection - Multiple confirmation gates before running tests with actual API calls
4. AI-powered automation - Claude and Gemini integrations for code review and issue resolution
5. Comprehensive coverage - Codecov integration with multiple workflow uploads
6. Performance monitoring - Nightly benchmarks for startup time, memory usage, and agent creation speed
7. Developer-friendly workflows - Extensive debugging, config backup/restore, and clear error messages

The testing system emphasizes:

- Fast feedback - Unit tests run on every push/PR (~15 min)
- Safety - Real tests require explicit confirmation and are manual-only
- Automation - AI assistants handle routine reviews and issue triaging
- Visibility - Centralized coverage reporting and test artifacts

**Sources:** All workflow files in [.github/workflows/](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/.github/workflows/) [src/praisonai/tests/test_runner.py](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/test_runner.py) test examples in [src/praisonai/tests/](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/)
