<!-- source: https://deepwiki.com/MervinPraison/PraisonAI/11.4-performance-monitoring -->

# Performance Monitoring

- examples/python/performance_monitoring_demo.py
- src/praisonai-agents/TELEMETRY_SUMMARY.md
- src/praisonai-agents/debug_auto_instrument.py
- src/praisonai-agents/debug_telemetry.py
- src/praisonai-agents/debug_telemetry_double.py
- src/praisonai-agents/praisonaiagents/telemetry/README.md
- src/praisonai-agents/praisonaiagents/telemetry/__init__.py
- src/praisonai-agents/praisonaiagents/telemetry/integration.py
- src/praisonai-agents/praisonaiagents/telemetry/performance_cli.py
- src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py
- src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py
- src/praisonai-agents/praisonaiagents/telemetry/telemetry.py
- src/praisonai-agents/telemetry_analysis.md
- src/praisonai-agents/test_telemetry_performance.py

This document describes PraisonAI Agents' telemetry and performance monitoring systems. It covers the privacy-first telemetry architecture, user-friendly performance monitoring tools for tracking function execution and API calls, flow analysis capabilities, and programmatic APIs for integrating monitoring into agent workflows.

The performance monitoring system is **disabled by default** and requires explicit opt-in to enable. For information about the testing infrastructure, see [Testing Framework](/MervinPraison/PraisonAI/11.1-testing-framework). For CI/CD integration, see [CI/CD Pipeline](/MervinPraison/PraisonAI/11.2-cicd-pipeline).

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/__init__.py1-28](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/__init__.py#L1-L28) [src/praisonai-agents/praisonaiagents/telemetry/README.md1-127](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/README.md?plain=1#L1-L127)

## Telemetry Architecture Overview

PraisonAI Agents implements a dual-layer telemetry and performance monitoring architecture. The system is designed with privacy-first principles and is **disabled by default**, requiring explicit opt-in via environment variables.

**System Architecture**

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/__init__.py1-196](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/__init__.py#L1-L196) [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py1-98](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L1-L98) [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py1-46](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L1-L46)

### Privacy-First Design Principles

The telemetry system follows strict privacy guarantees:

**Data Collection Boundaries:**

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py78-88](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L78-L88) [src/praisonai-agents/praisonaiagents/telemetry/README.md5-21](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/README.md?plain=1#L5-L21)

## MinimalTelemetry System

The `MinimalTelemetry` class at [telemetry.py78-637](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/telemetry.py#L78-L637) provides the foundation for anonymous usage tracking with zero overhead when disabled.

### Telemetry Initialization and Control

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py38-76](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L38-L76) [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py99-169](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L99-L169)

### Tracking Methods

The `MinimalTelemetry` class provides five tracking methods:

**Async Mode for Streaming:**

The `async_mode` parameter in `track_agent_execution()` at [telemetry.py206-260](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/telemetry.py#L206-L260) defers PostHog capture to prevent blocking in streaming scenarios:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py206-397](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L206-L397)

### Shutdown and Resource Management

The shutdown mechanism at [telemetry.py465-535](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/telemetry.py#L465-L535) handles cleanup with interpreter shutdown detection:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py465-637](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L465-L637)

## PerformanceMonitor System

The `PerformanceMonitor` class at [performance_monitor.py37-705](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_monitor.py#L37-L705) provides user-friendly tools for tracking function performance, API calls, and execution flow without requiring PostHog or telemetry.

### PerformanceMonitor Architecture

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py37-708](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L37-L708)

### Function Performance Tracking

The `@monitor_function` decorator at [performance_monitor.py104-180](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_monitor.py#L104-L180) tracks function execution with minimal overhead:

```

```

**Collected Statistics:**

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py70-81](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L70-L81) [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py215-239](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L215-L239)

### API Call Tracking

The `track_api_call()` context manager at [performance_monitor.py182-213](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_monitor.py#L182-L213) monitors API performance:

```

```

**API Call Statistics:**

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py84-91](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L84-L91) [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py241-264](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L241-L264)

### Performance Reports

The `generate_performance_report()` method at [performance_monitor.py471-554](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_monitor.py#L471-L554) generates comprehensive reports:

```

```

**Example Output:**

```
================================================================================
PRAISONAI PERFORMANCE MONITORING REPORT
================================================================================
Generated at: 2024-01-15T10:30:00

 FUNCTION PERFORMANCE SUMMARY
----------------------------------------
Total Functions Monitored: 15
Total Function Calls: 342
Total Errors: 3
Overall Success Rate: 99.1%

 SLOWEST FUNCTIONS (Top 5)
1. data_processing
   Average: 0.125s | Max: 0.450s | Calls: 50

 API CALL PERFORMANCE SUMMARY
----------------------------------------
Total APIs Monitored: 3
Total API Calls: 156
Overall API Success Rate: 97.4%

 SLOWEST API CALLS (Top 5)
1. openai:/v1/chat/completions
   Average: 0.350s | Max: 1.200s | Success Rate: 98.5%
```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py471-554](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L471-L554)

## Performance Analysis Utilities

The performance analysis utilities at [performance_utils.py1-599](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_utils.py#L1-L599) provide advanced capabilities for flow analysis and bottleneck detection.

### FunctionFlowAnalyzer

The `FunctionFlowAnalyzer` class at [performance_utils.py42-373](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_utils.py#L42-L373) analyzes execution patterns:

```

```

**Bottleneck Detection Thresholds:**

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py32-38](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py#L32-L38) [src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py42-373](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py#L42-L373)

### PerformanceAnalyzer

The `PerformanceAnalyzer` class at [performance_utils.py375-574](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_utils.py#L375-L574) provides trend analysis:

**Trend Analysis Flow:**

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py375-574](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py#L375-L574)

### Flow Visualization

The flow visualization at [performance_utils.py276-372](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_utils.py#L276-L372) supports multiple formats:

**Text Format Example:**

```
 FUNCTION EXECUTION FLOW VISUALIZATION
============================================================

Thread Thread 123456:
------------------------------
Medium START workflow_step_1
  Medium START workflow_step_2
    Medium START workflow_step_4
    ok END   workflow_step_4 (0.030s)
  ok END   workflow_step_2 (0.050s)
  Medium START workflow_step_3
  ok END   workflow_step_3 (0.080s)
ok END   workflow_step_1 (0.100s)
```

**Mermaid Format:**

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py305-372](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py#L305-L372)

## CLI Interface

The `PerformanceCLI` class at [performance_cli.py35-397](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/performance_cli.py#L35-L397) provides command-line access to performance monitoring.

### Available Commands

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_cli.py287-397](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_cli.py#L287-L397)

### Command Details

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_cli.py43-284](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_cli.py#L43-L284)

## Integration Mechanisms

The integration module at [integration.py1-651](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/integration.py#L1-L651) provides automatic instrumentation for agents and workflows.

### Auto-Instrumentation Architecture

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/integration.py48-196](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/integration.py#L48-L196) [src/praisonai-agents/praisonaiagents/telemetry/integration.py198-516](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/integration.py#L198-L516)

### Wrapped Method Behavior

The instrumentation wraps agent methods at [integration.py198-391](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/integration.py#L198-L391) with minimal overhead:

**Agent Method Wrapping:**

**Workflow Method Wrapping:**

**Streaming Support:**

For streaming responses at [integration.py262-298](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/integration.py#L262-L298) the wrapper detects generators and tracks completion only after streaming finishes:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/integration.py262-322](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/integration.py#L262-L322)

### Performance Mode

Performance mode at [integration.py597-606](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/integration.py#L597-L606) disables telemetry overhead:

```

```

When performance mode is enabled, all telemetry event queueing is skipped, providing zero overhead for production workloads.

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/integration.py597-606](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/integration.py#L597-L606)

## Programmatic API

The performance monitoring system provides a programmatic API for integration into custom workflows.

### Basic Usage Pattern

```

```

**Sources:** [examples/python/performance_monitoring_demo.py22-30](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/examples/python/performance_monitoring_demo.py#L22-L30) [examples/python/performance_monitoring_demo.py268-288](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/examples/python/performance_monitoring_demo.py#L268-L288)

### API Component Map

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/__init__.py40-100](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/__init__.py#L40-L100) [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py707-752](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L707-L752) [src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py576-599](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py#L576-L599)

### Integration with Agents

Integrate performance monitoring with agents and tasks:

```

```

**Sources:** [examples/python/performance_monitoring_demo.py193-233](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/examples/python/performance_monitoring_demo.py#L193-L233)

### External APM Integration

Export metrics for external APM tools like DataDog or New Relic:

```

```

**Metric Types:**

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py593-704](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_monitor.py#L593-L704)

## Environment Variables

The monitoring system respects multiple environment variables for control.

### Control Variables

**Decision Flow:**

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/telemetry.py38-76](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/telemetry.py#L38-L76) [src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py32](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/performance_utils.py#L32-L32)

### Usage Examples

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/telemetry/README.md22-42](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/telemetry/README.md?plain=1#L22-L42)

## Profile Data Schema

The profile report uses a versioned schema (v1.0) for stability and backward compatibility.

### Mandatory Fields

All profile reports contain these fields regardless of profiling layer:

**Sources:** [src/praisonai/tests/unit/test_unified_profiler.py259-288](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_unified_profiler.py#L259-L288)

### Optional Fields (Layer-Dependent)

**Sources:** [src/praisonai/tests/unit/test_unified_profiler.py259-288](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_unified_profiler.py#L259-L288)

### InvocationInfo

```

```

**Sources:** [src/praisonai/tests/unit/test_unified_profiler.py290-301](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_unified_profiler.py#L290-L301)

### FunctionStat

Function statistics include:

- name: str - Function name with module path
- calls: int - Number of times called
- total_time_ms: float - Total time spent in function
- cumulative_time_ms: float - Time including callees
- per_call_ms: float - Average time per call

### CallGraph

Call graph representation:

- callers: Dict[str, List[str]] - Map of function -> callers
- callees: Dict[str, List[str]] - Map of function -> callees
- edges: List[Tuple[str, str]] - List of (caller, callee) edges

**Sources:** [src/praisonai/tests/unit/test_unified_profiler.py303-319](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_unified_profiler.py#L303-L319)

### RequestTiming (Network Profile)

Network request timing data:

- url: str - Request URL
- method: str - HTTP method
- start_time: float - Request start
- first_byte_time: float - Time to first byte
- end_time: float - Request end
- status_code: int - HTTP status
- ttfb_ms: float - Time to first byte (ms)
- total_ms: float - Total request time (ms)

**Sources:** [src/praisonai/tests/unit/test_unified_profiler.py321-339](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/tests/unit/test_unified_profiler.py#L321-L339)
