A system-level interactive canvas for agent-human collaboration. Draw shapes, paste SVGs from Figma, render Mermaid diagrams, and let AI agents understand your designs via MCP or WebSocket.

What is this?

agcanvas (short for Augmented Canvas) bridges the gap between visual design and code generation. It's a collaboration surface for humans and AI agents — draw your ideas, paste designs, and let agents see exactly what's on the canvas.

┌──────────────────────────────────────────────────────────────┐
│  Figma / Draw / Mermaid         agcanvas                     │
│  ┌─────┐  ┌─────┐  ┌─────┐   ┌──────────────────────────┐   │
│  │ SVG │  │Shapes│  │Merm.│   │  Canvas (pan/zoom)       │   │
│  │Paste│  │ Draw │  │ Diag│   │  ┌────────┐ ┌────────┐  │   │
│  └──┬──┘  └──┬───┘  └──┬──┘   │  │ Parsed │ │ Agent  │  │   │
│     └────────┴─────────┘      │  │  Tree  │ │ Server │  │   │
│              │                │  └────────┘ └───┬────┘  │   │
│              ▼                └──────────────────┼──────┘   │
│         Visual Canvas                           │          │
│                                                 │          │
│  AI Agent ◄──── MCP (stdio) ◄── agcanvas-mcp ──┘          │
│  AI Agent ◄──── WebSocket (JSON) ───────────────┘          │
│  - Sees structure (element tree)                           │
│  - Reads drawing shapes                                    │
│  - Generates code                                          │
└──────────────────────────────────────────────────────────────┘

Features

Canvas & Drawing

SVG Paste — Copy frames from Figma, paste directly (Cmd+V)
Shape Drawing — Rectangles, ellipses, lines, arrows, text directly on canvas
Selection & Editing — Select, move, resize shapes with corner handles
Mermaid Diagrams — Write Mermaid syntax, render as SVG on canvas
Sessions/Tabs — Multiple canvases in tabs, each with independent state, creator tracking (human vs agent), descriptions, and timestamps
Pan/Zoom — Smooth canvas navigation
Session Persistence — Auto-saves workspace to ~/Library/Application Support/agcanvas/, restores all tabs on launch
Command Palette — Cmd+K to search and execute any command with fuzzy matching

AI Agent Integration

MCP Server — agcanvas-mcp bridge for Claude Code, OpenCode, and Codex
WebSocket Protocol — Direct JSON API on ws://127.0.0.1:9876
Structure Parsing — SVG → typed element tree (groups, rects, circles, paths, text, images)
Semantic Description — Human-readable canvas description for LLMs
Bidirectional Events — Push notifications to agents when canvas state changes
Code Generation — Stubs for React, HTML, Tailwind, Svelte, Vue

Installation

From source

git clone https://github.com/yourusername/agcanvas.git
cd agcanvas
cargo build --release

This builds two binaries:

target/release/agcanvas — The desktop app
target/release/agcanvas-mcp — The MCP server bridge

Install to PATH

After building, symlink (or copy) the binaries so they're available system-wide:

macOS / Linux:

sudo ln -sf "$(pwd)/target/release/agcanvas" /usr/local/bin/agcanvas
sudo ln -sf "$(pwd)/target/release/agcanvas-mcp" /usr/local/bin/agcanvas-mcp

Or install to a user-local directory (no sudo):

mkdir -p ~/.local/bin
ln -sf "$(pwd)/target/release/agcanvas" ~/.local/bin/agcanvas
ln -sf "$(pwd)/target/release/agcanvas-mcp" ~/.local/bin/agcanvas-mcp

Make sure ~/.local/bin is in your PATH. Add export PATH="$HOME/.local/bin:$PATH" to your ~/.zshrc or ~/.bashrc if needed.

Windows (PowerShell, run as Administrator):

New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.local\bin\agcanvas.exe" -Target "$(Get-Location)\target\release\agcanvas.exe" -Force
New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.local\bin\agcanvas-mcp.exe" -Target "$(Get-Location)\target\release\agcanvas-mcp.exe" -Force

Make sure %USERPROFILE%\.local\bin is in your system PATH. Or use an existing PATH directory like C:\Users\<you>\AppData\Local\Microsoft\WindowsApps.

Verify:

agcanvas --help
agcanvas-mcp --help

macOS `.app` Bundle

agcanvas compiles into a native macOS application. The release binary (target/release/agcanvas) runs directly, but if you want a proper .app bundle you can double-click in Finder or drag to /Applications, use cargo-bundle:

Install cargo-bundle:
```
cargo install cargo-bundle
```

Add bundle metadata to crates/agcanvas/Cargo.toml:

[package.metadata.bundle]
name = "Augmented Canvas"
identifier = "com.agcanvas.app"
icon = ["assets/icon.icns"]
category = "public.app-category.developer-tools"
short_description = "Interactive canvas for agent-human collaboration"

Bundle it:
```
cargo bundle --release -p agcanvas
```
This produces target/release/bundle/osx/Augmented Canvas.app.

Install:

cp -r "target/release/bundle/osx/Augmented Canvas.app" /Applications/

Note: The raw cargo build --release binary already works as a native macOS app — cargo-bundle just wraps it in a .app bundle with an icon, metadata, and Finder integration. No code changes are needed.

Requirements

Rust 1.70+
macOS / Linux / Windows

Usage

Basic workflow

Open agcanvas
```
cargo run --release -p agcanvas
```
Draw shapes — Select a tool from the toolbar (or press a shortcut key) and drag on the canvas
Paste SVG from Figma — Copy a frame in Figma, then Cmd+V
Render Mermaid — Click the Mermaid button in the toolbar, write your diagram, click Render
Navigate — Pan (middle-click drag or Cmd+drag), Zoom (scroll wheel), Reset (Cmd+0)

Keyboard shortcuts

Action	Shortcut
Select tool	V
Rectangle tool	R
Ellipse tool	E
Line tool	L
Arrow tool	A
Text tool	T
Delete selected	Delete / Backspace
Cancel / back to Select	Escape
Paste SVG	Cmd+V
New Tab	Cmd+T
Close Tab	Cmd+W
Save workspace	Cmd+S
Command palette	Cmd+K
Reset zoom	Cmd+0

MCP Server (AI Agent Integration)

agcanvas-mcp is a standalone MCP server that bridges AI coding tools to the agcanvas desktop app. It communicates with agcanvas over WebSocket and exposes canvas data as MCP tools.

Setup for Claude Code

Add to your project's .mcp.json (or ~/.claude/mcp.json for global):

{
  "mcpServers": {
    "agcanvas": {
      "command": "agcanvas-mcp",
      "args": ["--port", "9876"]
    }
  }
}

Setup for OpenCode

Add to your opencode.json (project-level) or ~/.config/opencode/opencode.json (global):

{
  "mcp": {
    "agcanvas": {
      "type": "local",
      "command": ["agcanvas-mcp", "--port", "9876"],
      "enabled": true
    }
  }
}

Setup for Codex

Same MCP config format — add the agcanvas entry to your Codex MCP configuration.

Note: Make sure agcanvas-mcp is in your PATH (e.g., ~/.local/bin), or use the full path to the binary. agcanvas must be running for the MCP tools to work.

MCP Tools

Tool	Description
`list_sessions`	List all open tabs/sessions with creator info, descriptions, timestamps. Supports sorting by name, created_at, created_by, element_count
`create_session`	Create a new session/tab from an agent, with name, description, and creator identity
`update_session`	Update an existing session's name or description
`get_element_tree`	Get the full parsed SVG element tree (structured JSON)
`describe_canvas`	Get a human-readable description of the canvas
`get_element_by_id`	Look up a specific element by ID
`get_elements_at_point`	Find elements at an (x, y) coordinate
`get_drawing_elements`	Get all user-drawn shapes (rects, ellipses, lines, arrows, text)
`generate_code`	Generate code stubs (html, react, tailwind, svelte, vue)
`create_drawing_element`	Create a shape on the canvas (Rectangle, Ellipse, Line, Arrow, Text)
`update_drawing_element`	Update an existing drawing element's shape or style
`delete_drawing_element`	Delete a drawing element by ID
`clear_drawing_elements`	Clear all drawing elements from the canvas

All tools accept an optional session_id parameter. If omitted, the active session is used.

WebSocket Protocol

agcanvas also exposes a direct WebSocket server on ws://127.0.0.1:9876 for custom integrations.

Connecting

import websocket
import json

ws = websocket.create_connection("ws://127.0.0.1:9876")

Requests

All requests support an optional session_id parameter. If omitted, the active session is used.

List sessions

{"type": "ListSessions"}
{"type": "ListSessions", "sort_by": "created_at", "sort_order": "desc"}

Sort fields: name, created_at (default), created_by, element_count. Order: asc (default), desc.

Response:

{
  "type": "Sessions",
  "sessions": [
    {"id": "session-1", "name": "Tab 1", "has_svg": true, "element_count": 15, "description": null, "created_by": {"type": "Human"}, "created_at": 1707500000},
    {"id": "session-2", "name": "Agent Work", "has_svg": false, "element_count": null, "description": "Architecture diagram", "created_by": {"type": "Agent", "name": "Claude"}, "created_at": 1707500100}
  ],
  "active_session": "session-1"
}

Create session (agent)

{"type": "CreateSession", "name": "My Session", "description": "Working on auth flow", "created_by_name": "Claude"}

Response:

{"type": "SessionCreated", "session": {"id": "session-3", "name": "My Session", ...}}

Update session

{"type": "UpdateSession", "session_id": "session-1", "name": "Renamed", "description": "Updated description"}

Response:

{"type": "SessionUpdated", "session": {"id": "session-1", "name": "Renamed", ...}}

Get full element tree

{"type": "GetTree"}
{"type": "GetTree", "session_id": "session-2"}

Get semantic description

{"type": "Describe"}

Get drawing elements (user-drawn shapes)

{"type": "GetDrawingElements"}
{"type": "GetDrawingElements", "session_id": "session-1"}

Get element by ID

{"type": "GetElementById", "id": "button-primary"}

Query elements at point

{"type": "GetElementsAtPoint", "x": 150.0, "y": 200.0}

Generate code

{"type": "GenerateCode", "target": "react", "element_id": null}

Targets: html, react, tailwind, svelte, vue

Ping

{"type": "Ping"}

Events (GUI → Agent)

agcanvas pushes events to all connected agents when state changes:

Event	Trigger
`Connected`	Agent connects (includes current session state)
`SessionCreated`	New tab created
`SessionClosed`	Tab closed
`SessionActivated`	User switches tabs
`SvgLoaded`	SVG pasted or loaded
`SvgCleared`	Canvas cleared

Element types

enum ElementKind {
    Group { name: Option<String> },
    Rectangle { rx: Option<f32>, ry: Option<f32> },
    Circle { cx: f32, cy: f32, r: f32 },
    Ellipse { cx: f32, cy: f32, rx: f32, ry: f32 },
    Path { d: String },
    Text { content: String, font_size: f32 },
    Image { href: String },
    Line { x1: f32, y1: f32, x2: f32, y2: f32 },
}

Architecture

crates/
├── agcanvas/                 # Desktop app
│   └── src/
│       ├── main.rs           # Entry point, window setup
│       ├── app.rs            # Main app state, UI, toolbar, drawing interaction
│       ├── session.rs        # Session/tab state management
│       ├── persistence.rs    # Workspace save/load (~/.agcanvas/)
│       ├── command_palette.rs # Cmd+K command palette with fuzzy search
│       ├── element_tree.rs   # Structured element representation
│       ├── clipboard.rs      # System clipboard integration
│       ├── mermaid.rs        # Mermaid → SVG rendering
│       ├── drawing/
│       │   ├── element.rs    # DrawingElement, Shape, ShapeStyle, hit testing
│       │   ├── tool.rs       # Tool enum, DragState, ResizeHandle
│       │   └── render.rs     # Shape rendering via egui Painter
│       ├── canvas/
│       │   ├── state.rs      # Pan/zoom transformation state
│       │   └── interaction.rs # Mouse/keyboard input handling
│       ├── svg/
│       │   ├── parser.rs     # SVG → ElementTree conversion
│       │   └── renderer.rs   # SVG → pixels (resvg/tiny-skia)
│       └── agent/
│           ├── protocol.rs   # JSON message types
│           └── server.rs     # WebSocket server
└── agcanvas-mcp/             # MCP server bridge
    └── src/
        ├── main.rs           # CLI entry point, stdio transport
        ├── bridge.rs         # WebSocket client → agcanvas
        └── tools.rs          # MCP tool definitions

Dependencies

Crate	Purpose
`eframe`/`egui`	GUI framework
`usvg`	SVG parsing
`resvg`/`tiny-skia`	SVG rendering
`mermaid-rs-renderer`	Mermaid → SVG
`arboard`	Clipboard access
`tokio-tungstenite`	WebSocket (both server and client)
`rmcp`	MCP server SDK (Anthropic official)
`dirs`	Platform data directory paths
`serde`/`serde_json`	Serialization

Roadmap

Multi-frame support (sessions/tabs)
Shape drawing (rectangle, ellipse, line, arrow, text)
Selection, move, resize with handles
Mermaid diagram rendering
MCP server bridge for AI coding tools
Agent draw commands (modify canvas from agent)
Session metadata (creator tracking, descriptions, timestamps, sorting)
Session persistence (auto-save/restore workspace)
Command palette (Cmd+K)
Real code generation (not just stubs)
Export to file
Diff view (before/after agent changes)
Plugin system for code generators
Undo/redo
Multi-select and group operations

License

MIT

README.md

agcanvas — Augmented Canvas

What is this?

Features

Canvas & Drawing

AI Agent Integration

Installation

From source

Install to PATH

macOS .app Bundle

Requirements

Usage

Basic workflow

Keyboard shortcuts

MCP Server (AI Agent Integration)

Setup for Claude Code

Setup for OpenCode

Setup for Codex

MCP Tools

WebSocket Protocol

Connecting

Requests

List sessions

Create session (agent)

Update session

Get full element tree

Get semantic description

Get drawing elements (user-drawn shapes)

Get element by ID

Query elements at point

Generate code

Ping

Events (GUI → Agent)

Element types

Architecture

Dependencies

Roadmap

License

macOS `.app` Bundle