<!-- source: https://deepwiki.com/MervinPraison/PraisonAI/7.4-browser-automation -->

# Browser Automation

- .gitignore
- README.md
- src/praisonai/README.md

This page documents PraisonAI's browser automation system, which enables AI agents to control web browsers to accomplish tasks like searching, navigating, and extracting information from websites. The system supports four automation engines (Extension, CDP, Playwright, Hybrid) with features including automatic retry on failures, screenshot-based element detection, session persistence, and real-time video recording.

For information about the CLI interface and other user-facing commands, see [CLI Interface](/MervinPraison/PraisonAI/7.1-cli-interface). For web-based interfaces, see [Web UI (Chainlit)](/MervinPraison/PraisonAI/7.3-web-ui-(chainlit)).

## Architecture Overview

The browser automation system consists of multiple engines and supporting components that work together to provide flexible, production-ready web automation capabilities.

### System Components

```

```

Sources: [src/praisonai/praisonai/browser/agent.py1-828](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L1-L828) [src/praisonai/praisonai/browser/server.py1-661](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L1-L661) [src/praisonai/praisonai/browser/cdp_agent.py1-1612](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L1-L1612) [src/praisonai/praisonai/browser/launcher.py1-364](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/launcher.py#L1-L364) [src/praisonai/praisonai/browser/sessions.py1-387](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/sessions.py#L1-L387)

### Four Automation Engines

Sources: [src/praisonai/praisonai/browser/cli.py229-343](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L229-L343)

## BrowserAgent: AI Decision Maker

The `BrowserAgent` class processes observations (page state, screenshots, elements) and returns structured actions using LLM reasoning. It uses a comprehensive system prompt with error detection, loop prevention, and multi-step goal tracking.

### Core Agent Flow

```

```

Sources: [src/praisonai/praisonai/browser/agent.py347-828](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L347-L828)

### BrowserAgent Class

The agent is initialized with model selection and step limits:

```

```

**Core Methods:**

Sources: [src/praisonai/praisonai/browser/agent.py354-428](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L354-L428)

### Structured Output: BrowserAction Model

The agent uses a Pydantic model for structured LLM responses with extensive error detection fields:

**Core Action Fields:**

- thought (str): Brief reasoning for the action
- action (Literal): One of: type , submit , click , scroll , navigate , done , wait , clear_input
- selector (Optional[str]): CSS selector for target element
- value (Optional[str]): Text value for type action
- done (bool): Whether task is complete
- summary (Optional[str]): Summary when done=True

**Goal Progress Tracking:**

- goal_progress (Optional[int]): Percentage completion (0-100)
- page_summary (Optional[str]): Description of current page state
- on_track (Optional[bool]): Whether progressing as expected

**Error Detection (Critical):**

- error_detected (Optional[bool]): True if any error noticed
- error_description (Optional[str]): Description of the error
- input_field_value (Optional[str]): Exact text visible in input fields
- expected_vs_actual (Optional[str]): Expected vs actual state
- blockers (Optional[str]): Popups, consent dialogs, etc.
- retry_reason (Optional[str]): Why retrying a failed action

Sources: [src/praisonai/praisonai/browser/agent.py14-91](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L14-L91)

### Action Normalization

The agent normalizes freeform LLM responses to valid CDP action types:

**Normalization Mapping:**

Sources: [src/praisonai/praisonai/browser/agent.py94-169](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L94-L169)

### System Prompt Features

The `BROWSER_AGENT_SYSTEM_PROMPT` includes:

1. Error Detection Instructions : Detect garbled text, wrong pages, blockers
2. Error Recovery Rules : Use clear_input for garbled text, retry with different selectors
3. Anti-Loop Rules : Never repeat same action twice, use submit after type
4. Multi-Step Goal Parsing : Split goals by commas and "and", track completion
5. Cookie Consent Handling : Prioritize consent buttons before other actions
6. Valid Selector Format : Only use CSS selectors from element list
7. Completion Criteria : Mark done only when ALL goal parts complete

Sources: [src/praisonai/praisonai/browser/agent.py170-342](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L170-L342)

## BrowserServer: WebSocket Bridge

The `BrowserServer` class provides a FastAPI WebSocket server that bridges the Chrome extension to PraisonAI agents, managing multiple concurrent sessions.

### Server Architecture

```

```

Sources: [src/praisonai/praisonai/browser/server.py24-537](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L24-L537)

### Message Protocol

The WebSocket protocol uses JSON messages with a `type` field:

**Client -> Server Messages:**

**Server -> Client Messages:**

Sources: [src/praisonai/praisonai/browser/server.py171-203](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L171-L203)

### Server Lifecycle

**Initialization:**

```

```

**Async Usage:**

```

```

Sources: [src/praisonai/praisonai/browser/server.py36-523](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L36-L523)

### Session Broadcasting

When an observation is processed, the server broadcasts actions to ALL clients with the matching `session_id`. This allows the CLI client to receive real-time updates while the extension executes actions:

```

```

Sources: [src/praisonai/praisonai/browser/server.py412-421](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L412-L421)

## CDP Engine: Direct Browser Control

The `CDPBrowserAgent` class provides browser automation using Chrome DevTools Protocol directly, without requiring the extension. This is the recommended engine for headless automation and CI/CD environments.

### CDPBrowserAgent Class

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py92-1612](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L92-L1612)

### Key Features

**1. Retry Logic with Alternative Selectors:**

The CDP agent automatically retries failed actions with fallback selectors:

```

```

When an action fails, the agent tries:

1. Original selector from LLM
2. Site-specific fallback selectors (e.g., GOOGLE_SEARCH_SELECTORS )
3. Generic element patterns
4. Text-based matching via JavaScript

Sources: [src/praisonai/praisonai/browser/cdp_agent.py106-159](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L106-L159) [src/praisonai/praisonai/browser/cdp_agent.py544-631](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L544-L631)

**2. Cookie Consent Detection:**

The agent detects and prioritizes cookie consent dialogs:

```

```

Consent buttons are extracted first and marked with `type: 'consent_button'` for priority handling.

Sources: [src/praisonai/praisonai/browser/cdp_agent.py46-79](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L46-L79) [src/praisonai/praisonai/browser/cdp_agent.py268-360](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L268-L360)

**3. Enhanced Element Extraction:**

The `_get_page_state()` method builds multiple selector strategies per element:

**Selector Priority (Highest to Lowest):**

1. #id - Element ID
2. tag[name="value"] - Name attribute
3. [data-testid="value"] - Test ID
4. [aria-label="value"] - Accessibility label
5. [jsname="value"] - Google-specific attribute
6. a[href="value"] - Link href
7. tag.className - First CSS class

Sources: [src/praisonai/praisonai/browser/cdp_agent.py249-471](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L249-L471)

**4. Stuck Detection:**

The agent detects infinite loops using action history analysis:

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py472-510](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L472-L510)

**5. Video Recording:**

Real-time video recording via CDP screencast:

```

```

Frames are captured via `Page.screencastFrame` events and encoded to GIF using FFmpeg.

Sources: [src/praisonai/praisonai/browser/cdp_agent.py1163-1295](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L1163-L1295)

### CDP Action Execution

The agent translates high-level actions to CDP commands:

Sources: [src/praisonai/praisonai/browser/cdp_agent.py633-904](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L633-L904)

## BrowserLauncher: Chrome Startup

The `BrowserLauncher` class launches Chrome with the PraisonAI extension loaded and connects to the bridge server.

### Launcher Flow

```

```

Sources: [src/praisonai/praisonai/browser/launcher.py86-364](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/launcher.py#L86-L364)

### Chrome Launch Arguments

```

```

**Profile Management:**

- Default: Persistent profile at ~/.praisonai/browser_profile
- Temp profile: use_temp_profile=True creates and deletes temp directory

Sources: [src/praisonai/praisonai/browser/launcher.py218-245](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/launcher.py#L218-L245)

### Chrome Executable Discovery

The launcher searches for Chrome in platform-specific locations:

**macOS Priority:**

1. Chrome for Testing (supports --load-extension in Chrome 137+)
2. Chromium
3. Branded Chrome (may not work with extensions on 137+)

**Linux:**

1. google-chrome-stable
2. chromium
3. /snap/bin/chromium

**Windows:**

1. %ProgramFiles%\Google\Chrome\Application\chrome.exe
2. %LocalAppData%\Google\Chrome\Application\chrome.exe

Sources: [src/praisonai/praisonai/browser/launcher.py20-64](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/launcher.py#L20-L64)

## SessionManager: Persistence Layer

The `SessionManager` class provides SQLite persistence for browser automation sessions with thread-safe operations.

### Database Schema

```

```

Sources: [src/praisonai/praisonai/browser/sessions.py59-99](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/sessions.py#L59-L99)

### SessionManager API

**Core Methods:**

**Thread Safety:**

The manager uses thread-local connections for safe multi-threaded access:

```

```

Sources: [src/praisonai/praisonai/browser/sessions.py19-387](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/sessions.py#L19-L387)

### Step Recording

Steps are recorded with full context including retry information:

```

```

Sources: [src/praisonai/praisonai/browser/sessions.py255-299](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/sessions.py#L255-L299)

## CLI Commands

The browser automation CLI provides commands for running agents, managing sessions, and viewing history.

### Available Commands

```

```

Sources: [src/praisonai/praisonai/browser/cli.py1-783](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L1-L783)

### Run Command

Execute browser automation with a goal:

**Basic Usage:**

```

```

**Engine Selection:**

```

```

**Advanced Options:**

```

```

**Options:**

Sources: [src/praisonai/praisonai/browser/cli.py345-616](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L345-L616)

### Start Command

Start the WebSocket bridge server:

```

```

**Options:**

- --port (default: 8765): WebSocket port
- --host (default: 0.0.0.0): Bind host
- --model (default: gpt-4o-mini): LLM model
- --max-steps (default: 20): Max steps per session
- --verbose : Enable verbose logging

Sources: [src/praisonai/praisonai/browser/cli.py24-62](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L24-L62)

### Sessions Command

List browser automation sessions:

```

```

**Output includes:**

- Session ID (first 8 chars)
- Goal (truncated to 40 chars)
- Status (running, completed, failed, stopped)
- Step count
- Start time
- Current URL

Sources: [src/praisonai/praisonai/browser/cli.py64-123](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L64-L123)

### History Command

Show detailed step-by-step history for a session:

```

```

**Output for each step:**

- Step number
- Action type
- Selector used
- Text entered (if applicable)
- Thought/reasoning (truncated to 100 chars)

Sources: [src/praisonai/praisonai/browser/cli.py125-185](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L125-L185)

### Clear Command

Delete session history:

```

```

Sources: [src/praisonai/praisonai/browser/cli.py187-227](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L187-L227)

### Tabs Command

Manage Chrome tabs via CDP:

```

```

Requires Chrome running with `--remote-debugging-port=9222`.

Sources: [src/praisonai/praisonai/browser/cli.py618-705](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cli.py#L618-L705)

## Advanced Features

### 1. Vision-Based Element Detection

When enabled with `--vision` or `enable_vision=True`, the agent uses screenshot analysis for element location:

```

```

Vision mode is automatically enabled for models that support it: GPT-4, GPT-4o, Gemini, Claude.

Sources: [src/praisonai/praisonai/browser/agent.py463-535](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L463-L535)

### 2. Automatic Action Retry

The CDP engine retries failed actions with progressively broader selectors:

**Retry Strategy:**

1. Attempt 1 : Original LLM-suggested selector
2. Attempt 2 : Site-specific fallback selector (e.g., GOOGLE_SEARCH_SELECTORS )
3. Attempt 3 : Generic element pattern matching
4. Attempt 4 : Text-based JavaScript evaluation

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py1003-1146](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L1003-L1146)

### 3. Action Verification

When `verify_actions=True`, the CDP agent captures before/after screenshots to verify action effects:

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py118-120](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L118-L120)

### 4. Real-Time Video Recording

The CDP engine can record video of the automation session:

```

```

Frames are encoded to GIF using FFmpeg with the specified FPS.

Sources: [src/praisonai/praisonai/browser/cdp_agent.py1163-1295](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L1163-L1295)

### 5. Multi-Step Goal Parsing

The agent automatically parses multi-step goals and tracks completion:

```

```

The prompt includes explicit goal breakdown:

```
 GOAL BREAKDOWN (complete ALL parts):
   1. go to google
   2. search for ai
   3. click the first link

WARNING DO NOT mark done until ALL parts above are complete!
```

Sources: [src/praisonai/praisonai/browser/agent.py596-621](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L596-L621)

### 6. Cookie Consent Priority

The system detects overlays/dialogs and prioritizes consent buttons:

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py314-360](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L314-L360)

## Error Handling and Recovery

The browser automation system includes comprehensive error detection and recovery mechanisms.

### Error Detection Categories

```

```

Sources: [src/praisonai/praisonai/browser/agent.py182-248](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L182-L248)

### Recovery Actions

**1. Garbled Input Text:**

```
Error Detected: input_field_value = "wikipediawikipedia"
Recovery: Use clear_input action, then retype
```

**2. Wrong Page After Click:**

```
Error Detected: expected_vs_actual = "Expected: Wikipedia | Actual: Google"
Recovery: Navigate directly to correct URL or click different element
```

**3. Consent Dialog Blocking:**

```
Error Detected: overlay_info.detected = true
Recovery: Click consent button from prioritized element list
```

**4. Repeated Failures:**

```
Error Detected: Same selector failed 3+ times
Recovery: Try site-specific fallback selector or mark stuck
```

Sources: [src/praisonai/praisonai/browser/agent.py206-218](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L206-L218)

## Performance Considerations

### Import Time Optimization

The browser module uses lazy imports to minimize startup time:

```

```

Sources: [src/praisonai/praisonai/browser/agent.py14-91](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L14-L91)

### Token Usage Optimization

**Memory Disabled by Default:**

The `BrowserAgent` disables memory to prevent token explosion:

```

```

This prevents chat history from growing across multiple steps, keeping token usage bounded.

Sources: [src/praisonai/praisonai/browser/agent.py395-404](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/agent.py#L395-L404)

**Element List Limiting:**

Element extraction is limited to prevent context overflow:

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py434-441](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L434-L441)

## Integration Examples

### Programmatic Usage (Extension Mode)

```

```

Sources: [src/praisonai/praisonai/browser/launcher.py150-175](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/launcher.py#L150-L175)

### Programmatic Usage (CDP Mode)

```

```

Sources: [src/praisonai/praisonai/browser/cdp_agent.py1297-1377](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L1297-L1377)

### Server-Based Usage

Start the server in one process:

```

```

Connect from client:

```

```

Sources: [src/praisonai/praisonai/browser/server.py539-661](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L539-L661)

## Configuration

### Environment Variables

No specific environment variables are required. The system uses standard LLM API keys:

- OPENAI_API_KEY - For GPT models
- ANTHROPIC_API_KEY - For Claude models
- GOOGLE_API_KEY - For Gemini models

### Storage Locations

Sources: [src/praisonai/praisonai/browser/sessions.py31-36](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/sessions.py#L31-L36) [src/praisonai/praisonai/browser/launcher.py95](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/launcher.py#L95-L95)

### Port Configuration

Sources: [src/praisonai/praisonai/browser/server.py38-42](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/server.py#L38-L42) [src/praisonai/praisonai/browser/cdp_agent.py108](https://github.com/MervinPraison/PraisonAI/blob/ab80807b/src/praisonai/praisonai/browser/cdp_agent.py#L108-L108)
