agcanvas — Augmented Canvas

A system-level interactive canvas for agent-human collaboration. Draw shapes, paste SVGs from Figma, render Mermaid diagrams, and let AI agents understand your designs via MCP or WebSocket.

What is this?

agcanvas (short for Augmented Canvas) bridges the gap between visual design and code generation. It's a collaboration surface for humans and AI agents — draw your ideas, paste designs, and let agents see exactly what's on the canvas.

┌──────────────────────────────────────────────────────────────┐
│  Figma / Draw / Mermaid         agcanvas                     │
│  ┌─────┐  ┌─────┐  ┌─────┐   ┌──────────────────────────┐   │
│  │ SVG │  │Shapes│  │Merm.│   │  Canvas (pan/zoom)       │   │
│  │Paste│  │ Draw │  │ Diag│   │  ┌────────┐ ┌────────┐  │   │
│  └──┬──┘  └──┬───┘  └──┬──┘   │  │ Parsed │ │ Agent  │  │   │
│     └────────┴─────────┘      │  │  Tree  │ │ Server │  │   │
│              │                │  └────────┘ └───┬────┘  │   │
│              ▼                └──────────────────┼──────┘   │
│         Visual Canvas                           │          │
│                                                 │          │
│  AI Agent ◄──── MCP (stdio) ◄── agcanvas-mcp ──┘          │
│  AI Agent ◄──── WebSocket (JSON) ───────────────┘          │
│  - Sees structure (element tree)                           │
│  - Reads drawing shapes                                    │
│  - Generates code                                          │
└──────────────────────────────────────────────────────────────┘

Features

Canvas & Drawing

• SVG Paste — Copy frames from Figma, paste directly (Cmd+V)
• Shape Drawing — Rectangles, ellipses, lines, arrows, text directly on canvas
• Boolean Shape Operations — Union, intersection, difference, XOR on overlapping shapes via agent API
• Selection & Editing — Select, move, resize shapes with corner handles
• Mermaid Diagrams — Write Mermaid syntax, renders as interactive drawing elements (rectangles, arrows, text) that can be selected, moved, and resized. Supports edge labels (-->|Yes|)
• Export to PNG — Export canvas as high-DPI PNG via File menu (Cmd+Shift+E) or MCP tool
• Sessions/Tabs — Multiple canvases in tabs, each with independent state, creator tracking (human vs agent), descriptions, and timestamps
• Visual Undo Tree — Git-like branching history with checkout, fork, and tree visualization (Cmd+H)
• Pan/Zoom — Smooth canvas navigation with Pan tool (H), middle-click drag, and zoom reset (click zoom %)
• Batch Commands — Agents can send multiple operations in a single request for faster workflows
• Session Persistence — Auto-saves workspace to ~/Library/Application Support/agcanvas/, restores all tabs on launch
• Command Palette — Cmd+K to search and execute any command with fuzzy matching

AI Agent Integration

• MCP Server — agcanvas-mcp bridge for Claude Code, OpenCode, and Codex
• WebSocket Protocol — Direct JSON API on ws://127.0.0.1:9876
• Structure Parsing — SVG → typed element tree (groups, rects, circles, paths, text, images)
• Semantic Description — Human-readable canvas description for LLMs
• Bidirectional Events — Push notifications to agents when canvas state changes
• Code Generation — Stubs for React, HTML, Tailwind, Svelte, Vue

Installation

From source

git clone https://github.com/yourusername/agcanvas.git
cd agcanvas
cargo build --release

This builds two binaries:

• target/release/agcanvas — The desktop app
• target/release/agcanvas-mcp — The MCP server bridge

Install to PATH

After building, symlink (or copy) the binaries so they're available system-wide:

macOS / Linux:

sudo ln -sf "$(pwd)/target/release/agcanvas" /usr/local/bin/agcanvas
sudo ln -sf "$(pwd)/target/release/agcanvas-mcp" /usr/local/bin/agcanvas-mcp

Or install to a user-local directory (no sudo):

mkdir -p ~/.local/bin
ln -sf "$(pwd)/target/release/agcanvas" ~/.local/bin/agcanvas
ln -sf "$(pwd)/target/release/agcanvas-mcp" ~/.local/bin/agcanvas-mcp

Make sure ~/.local/bin is in your PATH. Add export PATH="$HOME/.local/bin:$PATH" to your ~/.zshrc or ~/.bashrc if needed.

Windows (PowerShell, run as Administrator):

New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.local\bin\agcanvas.exe" -Target "$(Get-Location)\target\release\agcanvas.exe" -Force
New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.local\bin\agcanvas-mcp.exe" -Target "$(Get-Location)\target\release\agcanvas-mcp.exe" -Force

Make sure %USERPROFILE%\.local\bin is in your system PATH. Or use an existing PATH directory like C:\Users\<you>\AppData\Local\Microsoft\WindowsApps.

Verify:

agcanvas --help
agcanvas-mcp --help

macOS .app Bundle

agcanvas compiles into a native macOS application (Apple Silicon and Intel). Use the bundling script to create an Augmented Canvas.app you can open from Finder or drag to /Applications:

./scripts/bundle-macos.sh

This builds a release binary and packages it into target/release/bundle/Augmented Canvas.app.

To install directly to /Applications:

./scripts/bundle-macos.sh --install

To add a custom icon, place an AppIcon.icns file in assets/ before bundling.

Note: The raw cargo build --release binary already runs as a native macOS app — the bundle script wraps it in a .app with Info.plist, Finder integration, and HiDPI support. No third-party tools required.

Requirements

• Rust 1.70+
• macOS / Linux / Windows

Usage

Basic workflow

1. Open agcanvas

   cargo run --release -p agcanvas
   

2. Draw shapes — Select a tool from the toolbar (or press a shortcut key) and drag on the canvas

3. Paste SVG from Figma — Copy a frame in Figma, then Cmd+V

4. Render Mermaid — Click the Mermaid button in the toolbar, write your diagram, click Render

5. Navigate — Pan (middle-click drag or Cmd+drag), Zoom (scroll wheel), Reset (Cmd+0)

Keyboard shortcuts

Action	Shortcut
Select tool	V
Pan tool	H
Rectangle tool	R
Ellipse tool	E
Line tool	L
Arrow tool	A
Text tool	T
Delete selected	Delete / Backspace
Cancel / back to Select	Escape
Paste SVG	Cmd+V
Export as PNG	Cmd+Shift+E
New Tab	Cmd+T
Close Tab	Cmd+W
Save workspace	Cmd+S
Command palette	Cmd+K
Toggle history panel	Cmd+H
Undo	Cmd+Z
Redo	Cmd+Shift+Z
Reset zoom	Cmd+0

MCP Server (AI Agent Integration)

agcanvas-mcp is a standalone MCP server that bridges AI coding tools to the agcanvas desktop app. It communicates with agcanvas over WebSocket and exposes canvas data as MCP tools.

Setup for Claude Code

Add to your project's .mcp.json (or ~/.claude/mcp.json for global):

{
  "mcpServers": {
    "agcanvas": {
      "command": "agcanvas-mcp",
      "args": ["--port", "9876"]
    }
  }
}

Setup for OpenCode

Add to your opencode.json (project-level) or ~/.config/opencode/opencode.json (global):

{
  "mcp": {
    "agcanvas": {
      "type": "local",
      "command": ["agcanvas-mcp", "--port", "9876"],
      "enabled": true
    }
  }
}

Setup for Codex

Same MCP config format — add the agcanvas entry to your Codex MCP configuration.

Note: Make sure agcanvas-mcp is in your PATH (e.g., ~/.local/bin), or use the full path to the binary. agcanvas must be running for the MCP tools to work.

MCP Tools

Tool	Description
list_sessions	List all open tabs/sessions with creator info, descriptions, timestamps. Supports sorting by name, created_at, created_by, element_count
create_session	Create a new session/tab from an agent, with name, description, and creator identity
update_session	Update an existing session's name or description
get_element_tree	Get the full parsed SVG element tree (structured JSON)
describe_canvas	Get a human-readable description of the canvas
get_element_by_id	Look up a specific element by ID
get_elements_at_point	Find elements at an (x, y) coordinate
get_drawing_elements	Get all user-drawn shapes (rects, ellipses, lines, arrows, text)
generate_code	Generate code stubs (html, react, tailwind, svelte, vue)
create_drawing_element	Create a shape on the canvas (Rectangle, Ellipse, Line, Arrow, Text)
update_drawing_element	Update an existing drawing element's shape or style
delete_drawing_element	Delete a drawing element by ID
clear_drawing_elements	Clear all drawing elements from the canvas
boolean_op	Perform boolean operations (union, intersection, difference, xor) on two or more shapes
render_mermaid	Render a Mermaid diagram as interactive drawing elements on the canvas
export_canvas	Export the canvas as a high-DPI PNG image
batch	Send multiple operations in one request for faster agent workflows

All tools accept an optional session_id parameter. If omitted, the active session is used.

WebSocket Protocol

agcanvas also exposes a direct WebSocket server on ws://127.0.0.1:9876 for custom integrations.

Connecting

import websocket
import json

ws = websocket.create_connection("ws://127.0.0.1:9876")

Requests

All requests support an optional session_id parameter. If omitted, the active session is used.

List sessions

{"type": "ListSessions"}
{"type": "ListSessions", "sort_by": "created_at", "sort_order": "desc"}

Sort fields: name, created_at (default), created_by, element_count. Order: asc (default), desc.

Response:

{
  "type": "Sessions",
  "sessions": [
    {"id": "session-1", "name": "Tab 1", "has_svg": true, "element_count": 15, "description": null, "created_by": {"type": "Human"}, "created_at": 1707500000},
    {"id": "session-2", "name": "Agent Work", "has_svg": false, "element_count": null, "description": "Architecture diagram", "created_by": {"type": "Agent", "name": "Claude"}, "created_at": 1707500100}
  ],
  "active_session": "session-1"
}

Create session (agent)

{"type": "CreateSession", "name": "My Session", "description": "Working on auth flow", "created_by_name": "Claude"}

Response:

{"type": "SessionCreated", "session": {"id": "session-3", "name": "My Session", ...}}

Update session

{"type": "UpdateSession", "session_id": "session-1", "name": "Renamed", "description": "Updated description"}

Response:

{"type": "SessionUpdated", "session": {"id": "session-1", "name": "Renamed", ...}}

Get full element tree

{"type": "GetTree"}
{"type": "GetTree", "session_id": "session-2"}

Get semantic description

{"type": "Describe"}

Get drawing elements (user-drawn shapes)

{"type": "GetDrawingElements"}
{"type": "GetDrawingElements", "session_id": "session-1"}

Get element by ID

{"type": "GetElementById", "id": "button-primary"}

Query elements at point

{"type": "GetElementsAtPoint", "x": 150.0, "y": 200.0}

Generate code

{"type": "GenerateCode", "target": "react", "element_id": null}

Targets: html, react, tailwind, svelte, vue

Boolean operation on shapes

{"type": "BooleanOp", "operation": "union", "element_ids": ["elem-1", "elem-2"], "consume_sources": true}

Operations: union, intersection, difference, xor. Set consume_sources to true to delete the source shapes after the operation. Optional style object to override the result's appearance.

Render Mermaid diagram

{"type": "RenderMermaid", "mermaid_source": "flowchart LR\n    A-->|Yes| B[OK]", "x": 0, "y": 0}

Renders Mermaid syntax into interactive drawing elements (rectangles, arrows, text). Supports edge labels (-->|Yes|), decision diamonds, and all flowchart/sequence/class diagram types.

Export canvas as PNG

{"type": "ExportCanvas", "path": "/tmp/canvas.png", "scale": 2.0, "background": "#1e1e1e"}

Batch operations

{"type": "Batch", "requests": [{"type": "Ping"}, {"type": "GetTree"}]}

Ping

{"type": "Ping"}

Events (GUI → Agent)

agcanvas pushes events to all connected agents when state changes:

Event	Trigger
Connected	Agent connects (includes current session state)
SessionCreated	New tab created
SessionClosed	Tab closed
SessionActivated	User switches tabs
SvgLoaded	SVG pasted or loaded
SvgCleared	Canvas cleared

Element types

enum ElementKind {
    Group { name: Option<String> },
    Rectangle { rx: Option<f32>, ry: Option<f32> },
    Circle { cx: f32, cy: f32, r: f32 },
    Ellipse { cx: f32, cy: f32, rx: f32, ry: f32 },
    Path { d: String },
    Text { content: String, font_size: f32 },
    Image { href: String },
    Line { x1: f32, y1: f32, x2: f32, y2: f32 },
}

Architecture

crates/
├── agcanvas/                 # Desktop app
│   └── src/
│       ├── main.rs           # Entry point, window setup
│       ├── app.rs            # Main app state, UI, toolbar, drawing interaction
│       ├── session.rs        # Session/tab state management with history integration
│       ├── history.rs        # Undo tree: branching history with snapshots, checkout, fork
│       ├── persistence.rs    # Workspace save/load (~/.agcanvas/)
│       ├── command_palette.rs # Cmd+K command palette with fuzzy search
│       ├── element_tree.rs   # Structured element representation
│       ├── clipboard.rs      # System clipboard integration
│       ├── mermaid.rs        # Mermaid → SVG rendering (v0.2.0, edge labels)
│       ├── export.rs         # Canvas → PNG export (composites all layers)
│       ├── drawing/
│       │   ├── element.rs    # DrawingElement, Shape, ShapeStyle, hit testing
│       │   ├── boolean.rs    # Boolean shape operations (union, intersection, difference, xor)
│       │   ├── tool.rs       # Tool enum, DragState, ResizeHandle
│       │   └── render.rs     # Shape rendering via egui Painter (incl. Path triangulation)
│       ├── canvas/
│       │   ├── state.rs      # Pan/zoom transformation state
│       │   └── interaction.rs # Mouse/keyboard input handling
│       ├── svg/
│       │   ├── parser.rs     # SVG → ElementTree conversion
│       │   ├── converter.rs  # SVG → DrawingElements (Mermaid conversion)
│       │   └── renderer.rs   # SVG → pixels (resvg/tiny-skia)
│       └── agent/
│           ├── protocol.rs   # JSON message types
│           └── server.rs     # WebSocket server
└── agcanvas-mcp/             # MCP server bridge
    └── src/
        ├── main.rs           # CLI entry point, stdio transport
        ├── bridge.rs         # WebSocket client → agcanvas
        └── tools.rs          # MCP tool definitions

Dependencies

Crate	Purpose
eframe/egui	GUI framework
usvg	SVG parsing
resvg/tiny-skia	SVG rendering
mermaid-rs-renderer	Mermaid → SVG
arboard	Clipboard access
tokio-tungstenite	WebSocket (both server and client)
rmcp	MCP server SDK (Anthropic official)
dirs	Platform data directory paths
i_overlay	Boolean shape operations (union, intersection, difference, xor)
earcutr	Polygon triangulation for Path rendering
serde/serde_json	Serialization

Roadmap

• Multi-frame support (sessions/tabs)
• Shape drawing (rectangle, ellipse, line, arrow, text)
• Selection, move, resize with handles
• Mermaid diagram rendering
• MCP server bridge for AI coding tools
• Agent draw commands (modify canvas from agent)
• Session metadata (creator tracking, descriptions, timestamps, sorting)
• Session persistence (auto-save/restore workspace)
• Command palette (Cmd+K)
• Boolean shape operations (union, intersection, difference, xor)
• Visual undo tree with branching history
• Export to PNG (GUI + MCP tool)
• Pan tool and batch commands
• Mermaid edge labels and native element conversion
• Real code generation (not just stubs)
• Diff view (before/after agent changes)
• Plugin system for code generators
• App icon

License

MIT