<!-- source: https://deepwiki.com/qhkm/zeptoclaw/9.3-context-overflow-and-compaction -->

# Context Overflow and Compaction

- .claude/settings.json
- .gitignore
- src/agent/compaction.rs
- src/agent/middleware/cache.rs
- src/agent/middleware/compaction.rs
- src/agent/middleware/context_build.rs
- src/agent/middleware/feedback.rs
- src/agent/middleware/token_budget.rs
- src/error.rs
- src/providers/error_classifier.rs
- src/providers/fallback.rs
- src/utils/sanitize.rs

This page describes ZeptoClaw's context management system, which prevents token limit overflows during long conversations. The system monitors token usage, triggers compaction at configurable thresholds, and performs pre-compaction memory flushes to preserve important facts.

## Overview

ZeptoClaw implements a graduated context compaction strategy with three urgency levels (**Normal**, **Emergency**, and **Critical**) that activate at increasing token thresholds. The system is primarily implemented as a middleware component in the agent pipeline [src/agent/middleware/compaction.rs1-8](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L1-L8) When the conversation context approaches the configured limit, the system first attempts to save important facts to long-term memory via a "Memory Flush" turn, then applies progressively more aggressive truncation or summarization strategies.

**Sources:** [src/agent/compaction.rs1-12](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L1-L12) [src/agent/middleware/compaction.rs1-8](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L1-L8)

## System Architecture

The compaction system is integrated into the `AgentLoop` via the `Agent Middleware Pipeline` [src/agent/middleware/compaction.rs25-45](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L25-L45) It utilizes the `ContextMonitor` to evaluate the current session's token weight against the `CompactionConfig` limits.

```

```

**Diagram: Context Overflow Detection and Compaction Pipeline**

**Sources:** [src/agent/middleware/compaction.rs40-94](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L40-L94) [src/agent/compaction.rs1-12](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L1-L12)

## Token Estimation and Sanitization

ZeptoClaw uses heuristic-based estimation to avoid the overhead of full tokenizer libraries for every loop iteration. Additionally, it performs aggressive sanitization of tool outputs to prevent "bloat" before messages are even stored in the session.

### Tool Result Sanitization

The `sanitize_tool_result` function [src/utils/sanitize.rs35](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/utils/sanitize.rs#L35-L35) performs pre-processing on raw tool outputs:

- Base64 Removal : Replaces data:...;base64,... URIs with placeholders src/utils/sanitize.rs 36-41
- Hex Blob Stripping : Replaces long hex strings (>= 200 chars) with placeholders src/utils/sanitize.rs 43-48
- Dynamic Budgeting : The compute_tool_result_budget function calculates a fair share of the remaining context window for each tool result src/utils/sanitize.rs 91-128

### Estimation Heuristics

The system applies multipliers to different content types:

- General Text : Uses an approximate bytes-per-token ratio src/utils/sanitize.rs 17
- Tool Results : Subject to strict truncation via shrink_tool_results src/agent/compaction.rs 194-210

**Sources:** [src/utils/sanitize.rs10-28](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/utils/sanitize.rs#L10-L28) [src/utils/sanitize.rs35-74](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/utils/sanitize.rs#L35-L74) [src/agent/compaction.rs194-210](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L194-L210)

## Three-Tier Compaction Urgency

The `CompactionMiddleware` checks the urgency via the `ContextMonitor` and applies the following tiers [src/agent/middleware/compaction.rs58-80](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L58-L80):

### Memory Flush turn

On `Normal` urgency, the system attempts a memory flush turn [src/agent/middleware/compaction.rs61-68](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L61-L68) This allows the agent to use the `longterm_memory` tool to persist important facts from the messages that are about to be compacted.

**Sources:** [src/agent/middleware/compaction.rs58-80](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L58-L80) [src/agent/compaction.rs46-83](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L46-L83) [src/agent/compaction.rs116-166](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L116-L166)

## Implementation Details

### Key Functions and Classes

### Provider Error Handling

The `ProviderError::ContextOverflow` is specifically marked as **not** retryable by the generic `RetryProvider` [src/error.rs69-73](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/error.rs#L69-L73) This is because retrying the same oversized request would result in a deterministic failure. Instead, the `AgentLoop` must catch this error, perform compaction, and then retry.

**Sources:** [src/error.rs41](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/error.rs#L41-L41) [src/error.rs69-73](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/error.rs#L69-L73) [src/agent/middleware/compaction.rs45-56](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L45-L56)

## Data Flow: Compaction Trigger

The following sequence shows how the system reacts when a context limit is approached or exceeded.

```

```

**Sources:** [src/agent/middleware/compaction.rs58-90](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/middleware/compaction.rs#L58-L90) [src/agent/compaction.rs73-81](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L73-L81) [src/agent/compaction.rs116-120](https://github.com/qhkm/zeptoclaw/blob/2c743d31/src/agent/compaction.rs#L116-L120)
