<!-- source: https://deepwiki.com/openclaw/openclaw/3.4.6-image-and-media-tools -->

# Image & Media Tools

- docs/automation/tasks.md
- docs/providers/together.md
- docs/tools/image-generation.md
- docs/tools/media-overview.md
- docs/tools/music-generation.md
- docs/tools/pdf.md
- docs/tools/video-generation.md
- extensions/alibaba/video-generation-provider.test.ts
- extensions/byteplus/video-generation-provider.test.ts
- extensions/byteplus/video-generation-provider.ts
- extensions/document-extract/document-extractor.test.ts
- extensions/document-extract/document-extractor.ts
- extensions/minimax/provider-http.test-helpers.ts
- extensions/minimax/video-generation-provider.test.ts
- extensions/minimax/video-generation-provider.ts
- extensions/openai/video-generation-provider.test.ts
- extensions/openai/video-generation-provider.ts
- extensions/qwen/video-generation-provider.test.ts
- extensions/runway/video-generation-provider.test.ts
- extensions/runway/video-generation-provider.ts
- extensions/together/onboard.ts
- extensions/together/openclaw.plugin.json
- extensions/together/together.live.test.ts
- extensions/together/video-generation-provider.test.ts
- extensions/together/video-generation-provider.ts
- extensions/xai/video-generation-provider.test.ts
- extensions/xai/video-generation-provider.ts
- src/agents/generated-attachments.ts
- src/agents/internal-events.ts
- src/agents/model-fallback.run-embedded.e2e.test.ts
- src/agents/openclaw-tools.registration.ts
- src/agents/openclaw-tools.ts
- src/agents/openclaw-tools.update-plan.test.ts
- src/agents/subagent-announce-delivery.test.ts
- src/agents/subagent-announce-delivery.ts
- src/agents/subagent-announce-dispatch.test.ts
- src/agents/subagent-announce-dispatch.ts
- src/agents/tools/image-generate-background.test.ts
- src/agents/tools/image-generate-tool.test.ts
- src/agents/tools/image-generate-tool.ts
- src/agents/tools/image-tool.test.ts
- src/agents/tools/image-tool.ts
- src/agents/tools/media-generate-background-shared.test.ts
- src/agents/tools/media-generate-background-shared.ts
- src/agents/tools/media-generate-background.test-support.ts
- src/agents/tools/media-tool-shared.ts
- src/agents/tools/music-generate-background.test.ts
- src/agents/tools/music-generate-tool.test.ts
- src/agents/tools/music-generate-tool.ts
- src/agents/tools/pdf-tool.test.ts
- src/agents/tools/pdf-tool.ts
- src/agents/tools/skill-workshop-tool.test.ts
- src/agents/tools/skill-workshop-tool.ts
- src/agents/tools/video-generate-background.test.ts
- src/agents/tools/video-generate-tool.test.ts
- src/agents/tools/video-generate-tool.ts
- src/channels/plugins/package-state-probes.test.ts
- src/channels/plugins/package-state-probes.ts
- src/media-understanding/shared.test.ts
- src/media-understanding/shared.ts
- src/media/document-extractors.runtime.ts
- src/media/pdf-extract.test.ts
- src/media/pdf-extract.ts
- src/plugin-sdk/test-helpers/provider-http-mocks.ts
- src/plugins/document-extractor-types.ts
- src/video-generation/dashscope-compatible.ts

This section documents the image and media tools implemented within OpenClaw. It covers the infrastructure for image generation, video synthesis, music creation, media understanding, and Text-to-Speech (TTS) capabilities.

## 1. Purpose and Scope

OpenClaw provides a comprehensive system to handle media inputs and outputs within AI agent workflows. The purpose of these tools is to:

- Generate Media : Create images, music, and video from natural language prompts using various providers src/agents/tools/video-generate-tool.ts 14-17 src/agents/tools/music-generate-tool.ts 15-18
- Understand Content : Extract information from images, PDFs, and media files via vision-capable models or specialized extractors src/agents/tools/image-tool.ts 27-30 src/agents/tools/pdf-tool.ts 52-54
- Voice Interaction : Synthesize speech (TTS) and handle transcription for audio inputs src/media-understanding/shared.ts 46-58 src/media-understanding/shared.ts 60-79
- Provider Failover : Automatically cycle through configured providers if the primary one fails src/agents/tools/image-tool.ts 45-47
- Asynchronous Processing : Support long-running media generation tasks (e.g., video, music) via background workers and session "wake" events src/agents/tools/media-generate-background-shared.ts 69-76 src/agents/tools/music-generate-background.ts 68-74
- Network Security : Protect media fetching and provider requests using a strict SSRF guard and DNS pinning src/media-understanding/shared.ts 27-30

**Sources:** [src/agents/tools/video-generate-tool.ts14-17](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/video-generate-tool.ts#L14-L17) [src/agents/tools/music-generate-tool.ts15-18](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-tool.ts#L15-L18) [src/agents/tools/image-tool.ts27-30](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-tool.ts#L27-L30) [src/media-understanding/shared.ts46-58](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L46-L58) [src/media-understanding/shared.ts60-79](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L60-L79) [src/agents/tools/media-generate-background-shared.ts69-76](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/media-generate-background-shared.ts#L69-L76) [src/agents/tools/music-generate-background.ts68-74](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-background.ts#L68-L74) [src/media-understanding/shared.ts27-30](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L27-L30)

## 2. Image Generation & Editing

The image generation system uses a unified runtime that abstracts multiple providers (OpenAI, Google, OpenRouter, Fal, etc.) behind a common interface.

### 2.1 Image Generation Runtime

The runtime abstracts provider-specific APIs into a standard interface.

- OpenAI Compatibility : Many providers are wrapped using OpenAI-compatible logic, handling JSON or multipart requests src/agents/tools/image-generate-tool.ts 187-209
- Geometry Normalization : The runtime maps requested sizes and aspect ratios to provider capabilities src/agents/tools/image-generate-tool.ts 114-130
- Vision Integration : Tools can determine if a model has vision capabilities and route requests accordingly src/agents/tools/image-tool.ts 57-58

### 2.2 Image Generation Tool ( image_generate )

The `createImageGenerateTool` function constructs the tool for agents. It supports:

- Actions : generate (new image), status (active task), or list (inspect models) src/agents/tools/image-generate-tool.ts 134-139
- Reference Images : Supports passing images via path, URL, or data URL, which are decoded using decodeDataUrl src/agents/tools/image-generate-tool.ts 67 src/agents/tools/image-generate-tool.ts 141-150
- Storage : Generated images are saved to the managed media directory via saveMediaBuffer src/agents/tools/image-generate-tool.ts 33

### 2.3 Bridge: Language Space to Code Entities

Title: Image Generation Execution Flow

```

```

**Sources:** [src/agents/tools/image-generate-tool.ts33](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-generate-tool.ts#L33-L33) [src/agents/tools/image-generate-tool.ts134-139](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-generate-tool.ts#L134-L139) [src/agents/tools/image-generate-tool.ts187-209](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-generate-tool.ts#L187-L209) [src/agents/tools/image-tool.ts57-58](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-tool.ts#L57-L58)

## 3. Video & Music Generation Pipeline

Video and music generation are treated as asynchronous processes due to high latency.

### 3.1 Task Lifecycle

When an agent calls `video_generate` or `music_generate`, the system creates a background task:

1. Submission : OpenClaw submits the request to the provider (e.g., OpenAI, Runway, Luma, Google Lyria) src/agents/tools/music-generate-tool.ts 15-19
2. Background Execution : Workers monitor job status. The createDefaultMediaGenerateBackgroundScheduler ensures the task is enqueued and tracked src/agents/tools/media-generate-background-shared.ts 70-76
3. Session Wake : Once completed, the system wakes the original session with an internal completion event via wakeMusicGenerationTaskCompletion src/agents/tools/music-generate-background.ts 118 This uses deliverSubagentAnnouncement to notify the requester src/agents/subagent-announce-delivery.ts 204-215

### 3.2 Media Tool Configuration

The system determines which media tools are enabled based on explicit model configs like `musicGenerationModel` [src/agents/tools/music-generate-tool.ts146-160](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-tool.ts#L146-L160) It verifies provider availability through `listRuntimeMusicGenerationProviders` [src/agents/tools/music-generate-tool.ts158](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-tool.ts#L158-L158)

**Sources:** [src/agents/tools/music-generate-tool.ts15-19](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-tool.ts#L15-L19) [src/agents/tools/media-generate-background-shared.ts70-76](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/media-generate-background-shared.ts#L70-L76) [src/agents/tools/music-generate-background.ts118](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-background.ts#L118-L118) [src/agents/subagent-announce-delivery.ts204-215](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/subagent-announce-delivery.ts#L204-L215) [src/agents/tools/music-generate-tool.ts146-160](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-tool.ts#L146-L160)

## 4. Media Infrastructure & Security

### 4.1 Media Understanding

OpenClaw provides specialized utilities for content extraction and understanding:

- describeImageWithModel : Central interface for image analysis src/agents/tools/image-tool.ts 27-28
- loadWebMedia : The central utility for fetching media from URLs or local paths, handling buffer loading and MIME detection src/agents/tools/image-tool.ts 98-101
- pdf_tool : Specialized tool for extracting text and structure from PDF documents src/agents/tools/pdf-tool.ts 52-54

### 4.2 Storage & MIME Handling

- MIME Normalization : The system uses resolveAudioTranscriptionUploadFileName to ensure correct extensions for audio uploads src/media-understanding/shared.ts 46-58
- File Persistence : Media is stored using saveMediaBuffer , ensuring that generated or edited assets are available for the session src/agents/tools/music-generate-tool.ts 12

### 4.3 Security & SSRF

All media tools that fetch remote assets implement SSRF protection via `fetchWithSsrFGuard` [src/media-understanding/shared.ts28](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L28-L28)

- Timeouts : Managed by fetchWithTimeout and createProviderOperationDeadline src/media-understanding/shared.ts 36-37 src/media-understanding/shared.ts 103-122
- Operation Deadlines : The system enforces global timeouts across multi-step media generation via resolveProviderOperationTimeoutMs src/media-understanding/shared.ts 124-138
- Polling Logic : pollProviderOperationJson is used to safely check provider status until a task is complete or fails src/media-understanding/shared.ts 168-182

### 4.4 Bridge: Media Processing Pipeline

Title: Media Retrieval and Processing Logic

```

```

**Sources:** [src/agents/tools/image-tool.ts27-28](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-tool.ts#L27-L28) [src/agents/tools/image-tool.ts98-101](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/image-tool.ts#L98-L101) [src/media-understanding/shared.ts28](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L28-L28) [src/media-understanding/shared.ts103-122](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L103-L122) [src/media-understanding/shared.ts124-138](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L124-L138) [src/media-understanding/shared.ts168-182](https://github.com/openclaw/openclaw/blob/fbdf5937/src/media-understanding/shared.ts#L168-L182) [src/agents/tools/music-generate-tool.ts12](https://github.com/openclaw/openclaw/blob/fbdf5937/src/agents/tools/music-generate-tool.ts#L12-L12)
