> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/nearai/ironclaw/llms.txt
> Use this file to discover all available pages before exploring further.

# Container Orchestration

> Manage persistent Docker containers for sandboxed job execution with worker processes

## Overview

IronClaw's container orchestration extends the sandbox system to support persistent Docker containers running full agent worker processes. Unlike ephemeral command containers, orchestrated containers maintain state and communicate with the main agent via an internal HTTP API.

## Architecture

```text theme={null}
┌───────────────────────────────────────────────┐
│              Orchestrator                       │
│                                                 │
│  Internal API (default :50051, configurable)    │
│    POST /worker/{id}/llm/complete               │
│    POST /worker/{id}/llm/complete_with_tools    │
│    GET  /worker/{id}/job                        │
│    GET  /worker/{id}/credentials                │
│    POST /worker/{id}/status                     │
│    POST /worker/{id}/complete                   │
│                                                 │
│  ContainerJobManager                            │
│    create_job() -> container + token             │
│    stop_job()                                    │
│    list_jobs()                                   │
│                                                 │
│  TokenStore                                     │
│    per-job bearer tokens (in-memory only)       │
│    per-job credential grants (in-memory only)   │
└───────────────────────────────────────────────┘
           │                              │
           ▼                              ▼
  ┌──────────────────────┐   ┌──────────────────────┐
  │  Worker Container    │   │  Claude Container  │
  │                      │   │                    │
  │  ironclaw worker    │   │  claude-bridge    │
  │  + full tools       │   │  + claude CLI     │
  │  + LLM via proxy    │   │  + native API     │
  └──────────────────────┘   └──────────────────────┘
```

## Job Modes

### Worker Mode

Standard IronClaw worker with proxied LLM calls:

```rust theme={null}
pub enum JobMode {
    Worker,        // Full IronClaw agent loop
    ClaudeCode,    // Claude CLI bridge
}
```

**Worker Container**:

* Runs `ironclaw worker` command
* LLM requests proxied to orchestrator
* Full tool access (Bash, Read, Write, etc.)
* Multi-turn agent loop
* Custom tools via WASM/MCP

**Use Cases**:

* Long-running background jobs
* Isolated project work
* Multi-step workflows
* Testing in clean environment

### Claude Code Mode

Bridge to the official Claude CLI:

```bash theme={null}
# Container runs:
ironclaw claude-bridge \
  --job-id <uuid> \
  --orchestrator-url http://host:50051 \
  --max-turns 50 \
  --model sonnet
```

**Claude Container**:

* Spawns `claude` CLI directly
* Native Claude Code tool use
* Anthropic API or OAuth authentication
* Tool allowlist for security
* Automatic session management

**Use Cases**:

* Use Claude's native computer use
* Access Claude-specific features
* Compare IronClaw vs Claude behavior
* Development and testing

## Container Job Manager

### Creating Jobs

```rust theme={null}
use ironclaw::orchestrator::{
    ContainerJobManager, JobMode, ContainerJobConfig,
    CredentialGrant,
};
use uuid::Uuid;

let config = ContainerJobConfig {
    image: "ironclaw-worker:latest".to_string(),
    memory_limit_mb: 2048,
    cpu_shares: 1024,
    orchestrator_port: 50051,
    ..Default::default()
};

let manager = ContainerJobManager::new(config, token_store);

let job_id = Uuid::new_v4();
let token = manager.create_job(
    job_id,
    "Build and test the project",
    Some(project_dir),
    JobMode::Worker,
    vec![],  // credential grants
).await?;

println!("Job {} started with token: {}", job_id, token);
```

### Stopping Jobs

```rust theme={null}
manager.stop_job(job_id).await?;
```

Stopping a job:

1. Stops the container (10 second grace period)
2. Removes the container
3. Revokes the auth token
4. Updates job state to `Stopped`

### Listing Jobs

```rust theme={null}
let jobs = manager.list_jobs().await;
for job in jobs {
    println!(
        "{}: {} ({})",
        job.job_id,
        job.task_description,
        job.state
    );
}
```

## Authentication

### Bearer Token System

Each job gets a unique bearer token:

```rust theme={null}
pub struct TokenStore {
    tokens: Arc<RwLock<HashMap<Uuid, String>>>,
    grants: Arc<RwLock<HashMap<Uuid, Vec<CredentialGrant>>>>,
}

impl TokenStore {
    pub async fn create_token(&self, job_id: Uuid) -> String {
        let token = format!("irc_{}", Uuid::new_v4());
        self.tokens.write().await.insert(job_id, token.clone());
        token
    }
    
    pub async fn validate(&self, token: &str) -> Option<Uuid> {
        let tokens = self.tokens.read().await;
        tokens.iter()
            .find(|(_, t)| *t == token)
            .map(|(id, _)| *id)
    }
}
```

**Security Properties**:

* Tokens are never logged or serialized
* Stored in-memory only (lost on restart)
* Automatically revoked when job completes
* Single token per job

### Credential Grants

Jobs can be granted access to specific credentials:

```rust theme={null}
pub struct CredentialGrant {
    pub name: String,           // e.g., "github"
    pub credential_type: String, // e.g., "api_token"
    pub value: String,          // Actual secret
}
```

Workers request credentials via the orchestrator API:

```http theme={null}
GET /worker/{job_id}/credentials
Authorization: Bearer irc_...
```

Response:

```json theme={null}
{
  "github": "ghp_...",
  "npm": "npm_..."
}
```

## Orchestrator API

### Endpoints

#### POST /worker/{id}/llm/complete

Proxy LLM completion request:

```http theme={null}
POST /worker/{job_id}/llm/complete
Authorization: Bearer irc_...
Content-Type: application/json

{
  "messages": [
    {"role": "user", "content": "What is 2+2?"}
  ],
  "max_tokens": 4096
}
```

Response:

```json theme={null}
{
  "content": "2 + 2 = 4",
  "finish_reason": "stop",
  "input_tokens": 15,
  "output_tokens": 8
}
```

#### GET /worker/{id}/job

Get job metadata:

```http theme={null}
GET /worker/{job_id}/job
Authorization: Bearer irc_...
```

Response:

```json theme={null}
{
  "job_id": "...",
  "task_description": "Build and test",
  "project_dir": "/workspace/project"
}
```

#### POST /worker/{id}/status

Update worker status:

```http theme={null}
POST /worker/{job_id}/status
Authorization: Bearer irc_...
Content-Type: application/json

{
  "message": "Running tests (iteration 5)",
  "iteration": 5
}
```

#### POST /worker/{id}/complete

Mark job complete:

```http theme={null}
POST /worker/{job_id}/complete
Authorization: Bearer irc_...
Content-Type: application/json

{
  "success": true,
  "message": "All tests passed. Deployed to staging."
}
```

## Container Configuration

### Worker Container

```dockerfile theme={null}
FROM debian:bookworm-slim

# Install dev tools
RUN apt-get update && apt-get install -y \
    git build-essential nodejs npm \
    python3 python3-pip gh

# Install Rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

# Copy IronClaw binary
COPY ironclaw /usr/local/bin/

# Non-root user
USER 1000
WORKDIR /workspace

ENTRYPOINT ["ironclaw"]
```

**Environment Variables** (injected by orchestrator):

```bash theme={null}
IRONCLAW_WORKER_TOKEN=irc_...
IRONCLAW_JOB_ID=...
IRONCLAW_ORCHESTRATOR_URL=http://172.17.0.1:50051
IRONCLAW_WORKSPACE=/workspace
```

### Claude Code Container

Same base image plus:

```dockerfile theme={null}
# Install Claude CLI
RUN npm install -g @anthropic-ai/claude-code@latest

# Create Claude config directory
RUN mkdir -p /home/sandbox/.claude
```

**Environment Variables**:

```bash theme={null}
# Worker vars
IRONCLAW_WORKER_TOKEN=irc_...
IRONCLAW_JOB_ID=...
IRONCLAW_ORCHESTRATOR_URL=http://172.17.0.1:50051

# Claude auth (one of):
ANTHROPIC_API_KEY=sk-ant-...  # Direct API key
CLAUDE_CODE_OAUTH_TOKEN=...  # OAuth token

# Claude config
CLAUDE_CODE_ALLOWED_TOOLS=bash,read,write,glob
```

### Volume Mounts

Project directories are bind-mounted:

```bash theme={null}
# Host path validation
~/.ironclaw/projects/my-project  # OK
/tmp/untrusted                   # REJECTED
../escape                        # REJECTED (canonicalized first)
```

**Mount Configuration**:

```rust theme={null}
let canonical = validate_bind_mount_path(&project_dir, job_id)?;
binds.push(format!("{}:/workspace:rw", canonical.display()));
```

### Resource Limits

```rust theme={null}
pub struct ContainerJobConfig {
    pub memory_limit_mb: u64,     // Default: 2048 (worker)
    pub cpu_shares: u32,           // Default: 1024
    pub claude_code_memory_limit_mb: u64,  // Default: 4096
}
```

Claude containers get more memory due to heavier node\_modules.

### Security

```rust theme={null}
let host_config = HostConfig {
    // Capability dropping
    cap_drop: Some(vec!["ALL".to_string()]),
    cap_add: Some(vec!["CHOWN".to_string()]),
    
    // Security options
    security_opt: Some(vec![
        "no-new-privileges:true".to_string()
    ]),
    
    // Tmpfs for ephemeral storage
    tmpfs: Some([
        ("/tmp".to_string(), "size=512M".to_string())
    ].into_iter().collect()),
    
    // Network bridge
    network_mode: Some("bridge".to_string()),
    
    // Host access via host.docker.internal
    extra_hosts: Some(vec![
        "host.docker.internal:host-gateway".to_string()
    ]),
    
    ..Default::default()
};
```

## Lifecycle Management

### Container States

```rust theme={null}
pub enum ContainerState {
    Creating,   // Container creation in progress
    Running,    // Worker executing
    Stopped,    // Completed or manually stopped
    Failed,     // Error occurred
}
```

### State Transitions

```text theme={null}
         create_job()
              │
              ▼
          Creating
              │
              │ (container starts)
              ▼
           Running
              │
              ├────────────> Failed (error)
              │
              ├────────────> Stopped (stop_job())
              │
              │ (complete_job())
              ▼
           Stopped
```

### Cleanup

Automatic cleanup on completion:

```rust theme={null}
pub async fn complete_job(
    &self,
    job_id: Uuid,
    result: CompletionResult,
) -> Result<()> {
    // Store result
    self.containers.write().await
        .get_mut(&job_id)
        .map(|h| h.completion_result = Some(result));
    
    // Stop container
    docker.stop_container(&container_id, timeout).await?;
    docker.remove_container(&container_id, force).await?;
    
    // Revoke token
    self.token_store.revoke(job_id).await;
    
    Ok(())
}
```

### Manual Cleanup

Remove completed job from memory:

```rust theme={null}
manager.cleanup_job(job_id).await;
```

## Configuration

```bash theme={null}
# Orchestrator API port
ORCHESTRATOR_PORT=50051

# Worker container image
WORKER_IMAGE=ironclaw-worker:latest

# Resource limits
WORKER_MEMORY_MB=2048
WORKER_CPU_SHARES=1024

# Claude Code config
CLAUDE_CODE_MEMORY_MB=4096
CLAUDE_CODE_MAX_TURNS=50
CLAUDE_CODE_MODEL=sonnet
CLAUDE_CODE_ALLOWED_TOOLS=bash,read,write

# Authentication (one of):
ANTHROPIC_API_KEY=sk-ant-...  # Direct API
CLAUDE_CODE_OAUTH_TOKEN=...  # OAuth
```

## Building Worker Images

### Standard Worker

```bash theme={null}
docker build -f Dockerfile.worker -t ironclaw-worker:latest .
```

### Custom Worker

Add project-specific tools:

```dockerfile theme={null}
FROM ironclaw-worker:latest

# Add custom tools
RUN apt-get update && apt-get install -y \
    terraform kubectl helm

# Add custom scripts
COPY scripts/ /usr/local/bin/
```

Build:

```bash theme={null}
docker build -t ironclaw-worker:custom .
```

Configure:

```bash theme={null}
WORKER_IMAGE=ironclaw-worker:custom
```

## Troubleshooting

### Container Creation Fails

```
Error: Failed to create container: image not found
```

**Solution**:

```bash theme={null}
# Build worker image
docker build -f Dockerfile.worker -t ironclaw-worker:latest .

# Or pull from registry
docker pull ghcr.io/your-org/ironclaw-worker:latest
docker tag ghcr.io/your-org/ironclaw-worker:latest ironclaw-worker:latest
```

### Orchestrator Connection Failed

```
Error: Failed to connect to orchestrator
```

**Solutions**:

1. Check orchestrator is running: `lsof -i :50051`
2. Verify firewall allows port 50051
3. For Linux, ensure `172.17.0.1` is accessible from containers
4. For macOS/Windows, ensure `host.docker.internal` resolves

### Token Validation Failed

```
Error: Invalid bearer token
```

**Causes**:

1. Orchestrator restarted (tokens are in-memory only)
2. Job completed and token was revoked
3. Token expired or corrupted

**Solution**: Recreate the job.

### Volume Mount Rejected

```
Error: Project directory outside allowed base
```

**Cause**: Security validation prevents mounting paths outside `~/.ironclaw/projects/`.

**Solution**:

```bash theme={null}
# Move project to allowed location
mv /tmp/project ~/.ironclaw/projects/my-project

# Then create job
ironclaw job create --project ~/.ironclaw/projects/my-project
```

## Source Code

Key files:

* `src/orchestrator/mod.rs` - Module overview
* `src/orchestrator/job_manager.rs` - Container lifecycle management
* `src/orchestrator/api.rs` - HTTP API implementation
* `src/orchestrator/auth.rs` - Token store and credential grants
* `Dockerfile.worker` - Worker container definition