<!-- source: https://deepwiki.com/MervinPraison/PraisonAI/2.3-llm-integration -->

# LLM Integration

- .github/ISSUE_TEMPLATE/bug_report.md
- .github/actions/claude-code-action/action.yml
- .github/actions/claude-issue-triage-action/action.yml
- .github/actions/gemini/action.yml
- .github/workflows/auto-issue-comment.yml
- .github/workflows/auto-pr-comment.yml
- .github/workflows/claude-issue-triage.yml
- .github/workflows/claude.yml
- .github/workflows/docker-publish.yml
- .github/workflows/gemini-issue-automated-triage.yml
- .github/workflows/gemini-issue-review.yml
- .github/workflows/gemini-pr-review.yml
- src/praisonai-agents/autoagents-tavily.py
- src/praisonai-agents/praisonaiagents/agent/agent.py
- src/praisonai-agents/praisonaiagents/agents/autoagents.py
- src/praisonai-agents/praisonaiagents/llm/llm.py
- src/praisonai-agents/praisonaiagents/llm/model_capabilities.py

## Purpose and Scope

This document explains the LLM integration layer in PraisonAI, which provides a unified interface for interacting with 100+ language model providers through LiteLLM. The `LLM` class handles model configuration, streaming responses, tool calling, rate limiting, and provider-specific optimizations.

For agent-level configuration and execution patterns, see [Agent Class](/MervinPraison/PraisonAI/2.1-agent-class). For tool integration details, see [Tool Integration](/MervinPraison/PraisonAI/5.1-tool-integration). For environment variable configuration, see [Environment Variables](/MervinPraison/PraisonAI/8.3-environment-variables).

## LLM Class Architecture

The `LLM` class serves as a wrapper around LiteLLM, providing a consistent interface for all language model interactions. It handles initialization, response generation, streaming, tool calling, and provider-specific quirks.

### Core Class Structure

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py91-2428](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L91-L2428)

### Agent-LLM Integration

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/agent/agent.py398-3714](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/agent/agent.py#L398-L3714) [src/praisonai-agents/praisonaiagents/llm/llm.py290-2428](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L290-L2428)

## Provider Support via LiteLLM

PraisonAI supports 100+ language model providers through LiteLLM integration. The `LLM` class uses a cached module reference to avoid repeated import overhead.

### LiteLLM Configuration

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py91-426](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L91-L426)

### Model Naming Conventions

The `model` parameter accepts various formats depending on the provider:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py290-468](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L290-L468)

### Provider Detection Methods

The LLM class includes methods to detect specific providers and their capabilities:

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py446-1220](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L446-L1220)

## Model Configuration

### Initialization Parameters

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py290-426](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L290-L426)

### Configuration Examples

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py290-426](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L290-L426)

### Context Window Management

The LLM class defines context window sizes for various models to prevent context overflow:

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py107-147](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L107-L147)

## Streaming and Response Generation

### Response Generation Flow

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1222-2428](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1222-L2428)

### Streaming Modes

The LLM class supports three streaming modes:

1. Full Streaming (stream=True, verbose=True) : Progressive display with display_generating() llm.py 1733-1752
2. Silent Streaming (stream=True, verbose=False) : Accumulate chunks without display llm.py 1754-1770
3. Non-Streaming (stream=False) : Single API call llm.py 1928-1986

### Provider-Specific Streaming

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1180-1895](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1180-L1895)

### Delta Processing

The `_process_stream_delta()` method extracts content and tool calls from streaming chunks:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py991-1022](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L991-L1022)

## Tool Calling System

### Tool Formatting Pipeline

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py951-2115](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L951-L2115)

### Tool Call Formats

The LLM class supports three tool call formats:

1. Standard (OpenAI/Anthropic) : tool_calls field in response llm.py 2022
2. JSON (Ollama) : JSON object in response text llm.py 2026-2057
3. XML (Qwen) : <tool_call> tags with JSON content llm.py 2061-2115

### Sequential Tool Calling

The tool calling system supports sequential execution across multiple iterations:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py778-2336](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L778-L2336)

### Tool Call Repair

The LLM class includes automatic repair for malformed tool calls (especially for Ollama):

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py371-2164](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L371-L2164)

## Rate Limiting and Retry Logic

### Retry System Architecture

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py489-673](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L489-L673)

### Retry Flow Example

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py547-594](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L547-L594)

### Rate Limiter Integration

The LLM class supports external rate limiter instances:

```

```

The rate limiter is invoked before each API call via `rate_limiter.acquire()` [llm.py566-567](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/llm.py#L566-L567) and during wait periods via `rate_limiter.wait_for_retry()` [llm.py586-589](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/llm.py#L586-L589)

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py383-594](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L383-L594)

## Provider-Specific Handling

### Ollama Optimizations

Ollama requires special handling due to weaker tool calling support and response formatting differences.

#### Ollama Detection and Configuration

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py436-2057](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L436-L2057)

#### Ollama-Specific Constants

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py150-172](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L150-L172)

#### Ollama Tool Call Example

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py844-2057](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L844-L2057)

### Gemini Optimizations

#### Structured Output Support

Gemini models (2.0-flash-exp and later) support native structured outputs, avoiding the need for JSON prompt injection:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1238-1260](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1238-L1260)

#### Gemini Internal Tools

Gemini supports three internal tools without explicit definitions:

These are passed directly to LiteLLM without formatting [llm.py1428-1433](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/llm.py#L1428-L1433)

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1428-1433](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1428-L1433)

#### Gemini Streaming

Gemini has issues with streaming when tools are present, so streaming is automatically disabled:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1718-1720](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1718-L1720)

### Anthropic/Claude Features

#### Prompt Caching

Anthropic models support prompt caching to reduce costs and latency for repeated system prompts:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1262-1276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1262-L1276)

#### Web Fetch

Anthropic models support web fetch for retrieving content from specific URLs:

```

```

The web fetch capability is checked via `_supports_web_fetch()` [llm.py688-699](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/llm.py#L688-L699) which validates the model is Anthropic/Claude.

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py310-699](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L310-L699)

#### Claude Memory Tool

Claude models can use a special memory tool for persistent context:

```

```

Memory tool calls are detected and executed internally [llm.py750-776](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/llm.py#L750-L776)

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py313-776](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L313-L776)

### OpenAI Features

#### System Message Handling

Legacy o1 models don't support system messages, so they must be skipped:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1166-1178](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1166-L1178)

#### Reasoning Steps

OpenAI's o1 models support extended reasoning with `reasoning_steps`:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py1601-1676](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L1601-L1676)

## Advanced Features

### Web Search and Fetch

The LLM class supports two web capabilities:

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py310-699](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L310-L699) [src/praisonai-agents/praisonaiagents/agent/agent.py1280-1295](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/agent/agent.py#L1280-L1295)

### Prompt Caching

Prompt caching reduces costs and latency by caching portions of prompts across requests:

#### Supported Providers

- OpenAI (GPT-4o and later)
- Anthropic (all Claude models)
- Bedrock
- Deepseek

#### Configuration

```

```

The `_supports_prompt_caching()` method checks provider support [llm.py701-712](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/llm.py#L701-L712)

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py312-1276](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L312-L1276)

### Claude Memory

Claude Memory Tool provides persistent context across conversations:

#### Initialization

```

```

#### Memory Tool Detection

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py313-776](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L313-L776)

### Token Tracking and Metrics

#### Token Metrics Collection

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py65-1653](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L65-L1653)

#### Metrics Configuration

```

```

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py363-380](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L363-L380)

### Logging and Debug Configuration

The LLM class includes centralized debug logging via `_log_llm_config()`:

```

```

All initialization parameters, get_response calls, and LLM instance state are logged when `LOGLEVEL=DEBUG` or `verbose >= 10`.

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py229-1530](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L229-L1530)

## Summary

The LLM integration layer provides:

1. Unified Interface : Single LLM class for 100+ providers via LiteLLM
2. Streaming Support : Provider-aware streaming with automatic fallback
3. Tool Calling : Multi-format tool parsing (standard, JSON, XML) with repair logic
4. Rate Limiting : Automatic retry with smart delay parsing
5. Provider Optimization : Special handling for Ollama, Gemini, Claude, and OpenAI
6. Advanced Features : Web search/fetch, prompt caching, Claude memory, token tracking

The `LLM` class serves as the foundation for all agent interactions, abstracting provider differences while exposing advanced capabilities through a consistent API.

**Sources:** [src/praisonai-agents/praisonaiagents/llm/llm.py91-2428](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/llm/llm.py#L91-L2428) [src/praisonai-agents/praisonaiagents/agent/agent.py398-3714](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai-agents/praisonaiagents/agent/agent.py#L398-L3714)
