agcanvas/README.md

# agcanvas

A system-level interactive canvas for agent-human collaboration. Draw shapes, paste SVGs from Figma, render Mermaid diagrams, and let AI agents understand your designs via MCP or WebSocket.

## What is this?

agcanvas bridges the gap between visual design and code generation. It's a **collaboration surface** for humans and AI agents — draw your ideas, paste designs, and let agents see exactly what's on the canvas.

```
┌──────────────────────────────────────────────────────────────┐
│  Figma / Draw / Mermaid         agcanvas                     │
│  ┌─────┐  ┌─────┐  ┌─────┐   ┌──────────────────────────┐   │
│  │ SVG │  │Shapes│  │Merm.│   │  Canvas (pan/zoom)       │   │
│  │Paste│  │ Draw │  │ Diag│   │  ┌────────┐ ┌────────┐  │   │
│  └──┬──┘  └──┬───┘  └──┬──┘   │  │ Parsed │ │ Agent  │  │   │
│     └────────┴─────────┘      │  │  Tree  │ │ Server │  │   │
│              │                │  └────────┘ └───┬────┘  │   │
│              ▼                └──────────────────┼──────┘   │
│         Visual Canvas                           │          │
│                                                 │          │
│  AI Agent ◄──── MCP (stdio) ◄── agcanvas-mcp ──┘          │
│  AI Agent ◄──── WebSocket (JSON) ───────────────┘          │
│  - Sees structure (element tree)                           │
│  - Reads drawing shapes                                    │
│  - Generates code                                          │
└──────────────────────────────────────────────────────────────┘
```

## Features

### Canvas & Drawing
- **SVG Paste** — Copy frames from Figma, paste directly (Cmd+V)
- **Shape Drawing** — Rectangles, ellipses, lines, arrows, text directly on canvas
- **Selection & Editing** — Select, move, resize shapes with corner handles
- **Mermaid Diagrams** — Write Mermaid syntax, render as SVG on canvas
- **Sessions/Tabs** — Multiple canvases in tabs, each with independent state, creator tracking (human vs agent), descriptions, and timestamps
- **Pan/Zoom** — Smooth canvas navigation

### AI Agent Integration
- **MCP Server** — `agcanvas-mcp` bridge for Claude Code, OpenCode, and Codex
- **WebSocket Protocol** — Direct JSON API on `ws://127.0.0.1:9876`
- **Structure Parsing** — SVG → typed element tree (groups, rects, circles, paths, text, images)
- **Semantic Description** — Human-readable canvas description for LLMs
- **Bidirectional Events** — Push notifications to agents when canvas state changes
- **Code Generation** — Stubs for React, HTML, Tailwind, Svelte, Vue

## Installation

### From source

```bash
git clone https://github.com/yourusername/agcanvas.git
cd agcanvas
cargo build --release
```

This builds two binaries:
- `target/release/agcanvas` — The desktop app
- `target/release/agcanvas-mcp` — The MCP server bridge

### Install to PATH

After building, symlink (or copy) the binaries so they're available system-wide:

**macOS / Linux:**

```bash
sudo ln -sf "$(pwd)/target/release/agcanvas" /usr/local/bin/agcanvas
sudo ln -sf "$(pwd)/target/release/agcanvas-mcp" /usr/local/bin/agcanvas-mcp
```

Or install to a user-local directory (no sudo):

```bash
mkdir -p ~/.local/bin
ln -sf "$(pwd)/target/release/agcanvas" ~/.local/bin/agcanvas
ln -sf "$(pwd)/target/release/agcanvas-mcp" ~/.local/bin/agcanvas-mcp
```

> Make sure `~/.local/bin` is in your `PATH`. Add `export PATH="$HOME/.local/bin:$PATH"` to your `~/.zshrc` or `~/.bashrc` if needed.

**Windows (PowerShell, run as Administrator):**

```powershell
New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.local\bin\agcanvas.exe" -Target "$(Get-Location)\target\release\agcanvas.exe" -Force
New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.local\bin\agcanvas-mcp.exe" -Target "$(Get-Location)\target\release\agcanvas-mcp.exe" -Force
```

> Make sure `%USERPROFILE%\.local\bin` is in your system `PATH`. Or use an existing PATH directory like `C:\Users\<you>\AppData\Local\Microsoft\WindowsApps`.

**Verify:**

```bash
agcanvas --help
agcanvas-mcp --help
```

### Requirements

- Rust 1.70+
- macOS / Linux / Windows

## Usage

### Basic workflow

1. **Open agcanvas**
   ```bash
   cargo run --release -p agcanvas
   ```

2. **Draw shapes** — Select a tool from the toolbar (or press a shortcut key) and drag on the canvas

3. **Paste SVG from Figma** — Copy a frame in Figma, then Cmd+V

4. **Render Mermaid** — Click the Mermaid button in the toolbar, write your diagram, click Render

5. **Navigate** — Pan (middle-click drag or Cmd+drag), Zoom (scroll wheel), Reset (Cmd+0)

### Keyboard shortcuts

| Action | Shortcut |
|--------|----------|
| Select tool | V |
| Rectangle tool | R |
| Ellipse tool | E |
| Line tool | L |
| Arrow tool | A |
| Text tool | T |
| Delete selected | Delete / Backspace |
| Cancel / back to Select | Escape |
| Paste SVG | Cmd+V |
| New Tab | Cmd+T |
| Close Tab | Cmd+W |
| Reset zoom | Cmd+0 |

## MCP Server (AI Agent Integration)

`agcanvas-mcp` is a standalone MCP server that bridges AI coding tools to the agcanvas desktop app. It communicates with agcanvas over WebSocket and exposes canvas data as MCP tools.

### Setup for Claude Code

Add to your project's `.mcp.json` (or `~/.claude/mcp.json` for global):

```json
{
  "mcpServers": {
    "agcanvas": {
      "command": "agcanvas-mcp",
      "args": ["--port", "9876"]
    }
  }
}
```

### Setup for OpenCode

Add to your `opencode.json` (project-level) or `~/.config/opencode/opencode.json` (global):

```json
{
  "mcp": {
    "agcanvas": {
      "type": "local",
      "command": ["agcanvas-mcp", "--port", "9876"],
      "enabled": true
    }
  }
}
```

### Setup for Codex

Same MCP config format — add the `agcanvas` entry to your Codex MCP configuration.

> **Note:** Make sure `agcanvas-mcp` is in your PATH (e.g., `~/.local/bin`), or use the full path to the binary. agcanvas must be running for the MCP tools to work.

### MCP Tools

| Tool | Description |
|------|-------------|
| `list_sessions` | List all open tabs/sessions with creator info, descriptions, timestamps. Supports sorting by name, created_at, created_by, element_count |
| `create_session` | Create a new session/tab from an agent, with name, description, and creator identity |
| `update_session` | Update an existing session's name or description |
| `get_element_tree` | Get the full parsed SVG element tree (structured JSON) |
| `describe_canvas` | Get a human-readable description of the canvas |
| `get_element_by_id` | Look up a specific element by ID |
| `get_elements_at_point` | Find elements at an (x, y) coordinate |
| `get_drawing_elements` | Get all user-drawn shapes (rects, ellipses, lines, arrows, text) |
| `generate_code` | Generate code stubs (html, react, tailwind, svelte, vue) |
| `create_drawing_element` | Create a shape on the canvas (Rectangle, Ellipse, Line, Arrow, Text) |
| `update_drawing_element` | Update an existing drawing element's shape or style |
| `delete_drawing_element` | Delete a drawing element by ID |
| `clear_drawing_elements` | Clear all drawing elements from the canvas |

All tools accept an optional `session_id` parameter. If omitted, the active session is used.

## WebSocket Protocol

agcanvas also exposes a direct WebSocket server on `ws://127.0.0.1:9876` for custom integrations.

### Connecting

```python
import websocket
import json

ws = websocket.create_connection("ws://127.0.0.1:9876")
```

### Requests

All requests support an optional `session_id` parameter. If omitted, the active session is used.

#### List sessions

```json
{"type": "ListSessions"}
{"type": "ListSessions", "sort_by": "created_at", "sort_order": "desc"}
```

Sort fields: `name`, `created_at` (default), `created_by`, `element_count`. Order: `asc` (default), `desc`.

Response:
```json
{
  "type": "Sessions",
  "sessions": [
    {"id": "session-1", "name": "Tab 1", "has_svg": true, "element_count": 15, "description": null, "created_by": {"type": "Human"}, "created_at": 1707500000},
    {"id": "session-2", "name": "Agent Work", "has_svg": false, "element_count": null, "description": "Architecture diagram", "created_by": {"type": "Agent", "name": "Claude"}, "created_at": 1707500100}
  ],
  "active_session": "session-1"
}
```

#### Create session (agent)

```json
{"type": "CreateSession", "name": "My Session", "description": "Working on auth flow", "created_by_name": "Claude"}
```

Response:
```json
{"type": "SessionCreated", "session": {"id": "session-3", "name": "My Session", ...}}
```

#### Update session

```json
{"type": "UpdateSession", "session_id": "session-1", "name": "Renamed", "description": "Updated description"}
```

Response:
```json
{"type": "SessionUpdated", "session": {"id": "session-1", "name": "Renamed", ...}}
```

#### Get full element tree

```json
{"type": "GetTree"}
{"type": "GetTree", "session_id": "session-2"}
```

#### Get semantic description

```json
{"type": "Describe"}
```

#### Get drawing elements (user-drawn shapes)

```json
{"type": "GetDrawingElements"}
{"type": "GetDrawingElements", "session_id": "session-1"}
```

#### Get element by ID

```json
{"type": "GetElementById", "id": "button-primary"}
```

#### Query elements at point

```json
{"type": "GetElementsAtPoint", "x": 150.0, "y": 200.0}
```

#### Generate code

```json
{"type": "GenerateCode", "target": "react", "element_id": null}
```

Targets: `html`, `react`, `tailwind`, `svelte`, `vue`

#### Ping

```json
{"type": "Ping"}
```

### Events (GUI → Agent)

agcanvas pushes events to all connected agents when state changes:

| Event | Trigger |
|-------|---------|
| `Connected` | Agent connects (includes current session state) |
| `SessionCreated` | New tab created |
| `SessionClosed` | Tab closed |
| `SessionActivated` | User switches tabs |
| `SvgLoaded` | SVG pasted or loaded |
| `SvgCleared` | Canvas cleared |

### Element types

```rust
enum ElementKind {
    Group { name: Option<String> },
    Rectangle { rx: Option<f32>, ry: Option<f32> },
    Circle { cx: f32, cy: f32, r: f32 },
    Ellipse { cx: f32, cy: f32, rx: f32, ry: f32 },
    Path { d: String },
    Text { content: String, font_size: f32 },
    Image { href: String },
    Line { x1: f32, y1: f32, x2: f32, y2: f32 },
}
```

## Architecture

```
crates/
├── agcanvas/                 # Desktop app
│   └── src/
│       ├── main.rs           # Entry point, window setup
│       ├── app.rs            # Main app state, UI, toolbar, drawing interaction
│       ├── session.rs        # Session/tab state management
│       ├── element_tree.rs   # Structured element representation
│       ├── clipboard.rs      # System clipboard integration
│       ├── mermaid.rs        # Mermaid → SVG rendering
│       ├── drawing/
│       │   ├── element.rs    # DrawingElement, Shape, ShapeStyle, hit testing
│       │   ├── tool.rs       # Tool enum, DragState, ResizeHandle
│       │   └── render.rs     # Shape rendering via egui Painter
│       ├── canvas/
│       │   ├── state.rs      # Pan/zoom transformation state
│       │   └── interaction.rs # Mouse/keyboard input handling
│       ├── svg/
│       │   ├── parser.rs     # SVG → ElementTree conversion
│       │   └── renderer.rs   # SVG → pixels (resvg/tiny-skia)
│       └── agent/
│           ├── protocol.rs   # JSON message types
│           └── server.rs     # WebSocket server
└── agcanvas-mcp/             # MCP server bridge
    └── src/
        ├── main.rs           # CLI entry point, stdio transport
        ├── bridge.rs         # WebSocket client → agcanvas
        └── tools.rs          # MCP tool definitions
```

### Dependencies

| Crate | Purpose |
|-------|---------|
| `eframe`/`egui` | GUI framework |
| `usvg` | SVG parsing |
| `resvg`/`tiny-skia` | SVG rendering |
| `mermaid-rs-renderer` | Mermaid → SVG |
| `arboard` | Clipboard access |
| `tokio-tungstenite` | WebSocket (both server and client) |
| `rmcp` | MCP server SDK (Anthropic official) |
| `serde`/`serde_json` | Serialization |

## Roadmap

- [x] Multi-frame support (sessions/tabs)
- [x] Shape drawing (rectangle, ellipse, line, arrow, text)
- [x] Selection, move, resize with handles
- [x] Mermaid diagram rendering
- [x] MCP server bridge for AI coding tools
- [x] Agent draw commands (modify canvas from agent)
- [x] Session metadata (creator tracking, descriptions, timestamps, sorting)
- [ ] Real code generation (not just stubs)
- [ ] Export to file
- [ ] Diff view (before/after agent changes)
- [ ] Plugin system for code generators
- [ ] Undo/redo
- [ ] Multi-select and group operations

## License

MIT