agcanvas

A system-level interactive canvas for agent-human collaboration. Draw shapes, paste SVGs from Figma, render Mermaid diagrams, and let AI agents understand your designs via MCP or WebSocket.

What is this?

agcanvas bridges the gap between visual design and code generation. It's a collaboration surface for humans and AI agents — draw your ideas, paste designs, and let agents see exactly what's on the canvas.

┌──────────────────────────────────────────────────────────────┐
│  Figma / Draw / Mermaid         agcanvas                     │
│  ┌─────┐  ┌─────┐  ┌─────┐   ┌──────────────────────────┐   │
│  │ SVG │  │Shapes│  │Merm.│   │  Canvas (pan/zoom)       │   │
│  │Paste│  │ Draw │  │ Diag│   │  ┌────────┐ ┌────────┐  │   │
│  └──┬──┘  └──┬───┘  └──┬──┘   │  │ Parsed │ │ Agent  │  │   │
│     └────────┴─────────┘      │  │  Tree  │ │ Server │  │   │
│              │                │  └────────┘ └───┬────┘  │   │
│              ▼                └──────────────────┼──────┘   │
│         Visual Canvas                           │          │
│                                                 │          │
│  AI Agent ◄──── MCP (stdio) ◄── agcanvas-mcp ──┘          │
│  AI Agent ◄──── WebSocket (JSON) ───────────────┘          │
│  - Sees structure (element tree)                           │
│  - Reads drawing shapes                                    │
│  - Generates code                                          │
└──────────────────────────────────────────────────────────────┘

Features

Canvas & Drawing

• SVG Paste — Copy frames from Figma, paste directly (Cmd+V)
• Shape Drawing — Rectangles, ellipses, lines, arrows, text directly on canvas
• Selection & Editing — Select, move, resize shapes with corner handles
• Mermaid Diagrams — Write Mermaid syntax, render as SVG on canvas
• Sessions/Tabs — Multiple canvases in tabs, each with independent state
• Pan/Zoom — Smooth canvas navigation

AI Agent Integration

• MCP Server — agcanvas-mcp bridge for Claude Code, OpenCode, and Codex
• WebSocket Protocol — Direct JSON API on ws://127.0.0.1:9876
• Structure Parsing — SVG → typed element tree (groups, rects, circles, paths, text, images)
• Semantic Description — Human-readable canvas description for LLMs
• Bidirectional Events — Push notifications to agents when canvas state changes
• Code Generation — Stubs for React, HTML, Tailwind, Svelte, Vue

Installation

From source

git clone https://github.com/yourusername/agcanvas.git
cd agcanvas
cargo build --release

This builds two binaries:

• target/release/agcanvas — The desktop app
• target/release/agcanvas-mcp — The MCP server bridge

Requirements

• Rust 1.70+
• macOS / Linux / Windows

Usage

Basic workflow

1. Open agcanvas

   cargo run --release -p agcanvas
   

2. Draw shapes — Select a tool from the toolbar (or press a shortcut key) and drag on the canvas

3. Paste SVG from Figma — Copy a frame in Figma, then Cmd+V

4. Render Mermaid — Click the Mermaid button in the toolbar, write your diagram, click Render

5. Navigate — Pan (middle-click drag or Cmd+drag), Zoom (scroll wheel), Reset (Cmd+0)

Keyboard shortcuts

Action	Shortcut
Select tool	V
Rectangle tool	R
Ellipse tool	E
Line tool	L
Arrow tool	A
Text tool	T
Delete selected	Delete / Backspace
Cancel / back to Select	Escape
Paste SVG	Cmd+V
New Tab	Cmd+T
Close Tab	Cmd+W
Reset zoom	Cmd+0

MCP Server (AI Agent Integration)

agcanvas-mcp is a standalone MCP server that bridges AI coding tools to the agcanvas desktop app. It communicates with agcanvas over WebSocket and exposes canvas data as MCP tools.

Setup for Claude Code

Add to your project's .mcp.json (or ~/.claude/mcp.json for global):

{
  "mcpServers": {
    "agcanvas": {
      "command": "agcanvas-mcp",
      "args": ["--port", "9876"]
    }
  }
}

Setup for OpenCode

Add to your opencode.json:

{
  "mcpServers": {
    "agcanvas": {
      "command": "agcanvas-mcp",
      "args": ["--port", "9876"]
    }
  }
}

Setup for Codex

Same MCP config format — add the agcanvas entry to your Codex MCP configuration.

Note: Make sure agcanvas-mcp is in your PATH, or use the full path to the binary (e.g., /path/to/target/release/agcanvas-mcp). agcanvas must be running for the MCP tools to work.

See examples/mcp-configs/ for ready-to-copy configuration files.

MCP Tools

Tool	Description
list_sessions	List all open tabs/sessions in agcanvas
get_element_tree	Get the full parsed SVG element tree (structured JSON)
describe_canvas	Get a human-readable description of the canvas
get_element_by_id	Look up a specific element by ID
get_elements_at_point	Find elements at an (x, y) coordinate
get_drawing_elements	Get all user-drawn shapes (rects, ellipses, lines, arrows, text)
generate_code	Generate code stubs (html, react, tailwind, svelte, vue)

All tools accept an optional session_id parameter. If omitted, the active session is used.

WebSocket Protocol

agcanvas also exposes a direct WebSocket server on ws://127.0.0.1:9876 for custom integrations.

Connecting

import websocket
import json

ws = websocket.create_connection("ws://127.0.0.1:9876")

Requests

All requests support an optional session_id parameter. If omitted, the active session is used.

List sessions

{"type": "ListSessions"}

Response:

{
  "type": "Sessions",
  "sessions": [
    {"id": "session-1", "name": "Tab 1", "has_svg": true, "element_count": 15},
    {"id": "session-2", "name": "Tab 2", "has_svg": false, "element_count": null}
  ],
  "active_session": "session-1"
}

Get full element tree

{"type": "GetTree"}
{"type": "GetTree", "session_id": "session-2"}

Get semantic description

{"type": "Describe"}

Get drawing elements (user-drawn shapes)

{"type": "GetDrawingElements"}
{"type": "GetDrawingElements", "session_id": "session-1"}

Get element by ID

{"type": "GetElementById", "id": "button-primary"}

Query elements at point

{"type": "GetElementsAtPoint", "x": 150.0, "y": 200.0}

Generate code

{"type": "GenerateCode", "target": "react", "element_id": null}

Targets: html, react, tailwind, svelte, vue

Ping

{"type": "Ping"}

Events (GUI → Agent)

agcanvas pushes events to all connected agents when state changes:

Event	Trigger
Connected	Agent connects (includes current session state)
SessionCreated	New tab created
SessionClosed	Tab closed
SessionActivated	User switches tabs
SvgLoaded	SVG pasted or loaded
SvgCleared	Canvas cleared

Element types

enum ElementKind {
    Group { name: Option<String> },
    Rectangle { rx: Option<f32>, ry: Option<f32> },
    Circle { cx: f32, cy: f32, r: f32 },
    Ellipse { cx: f32, cy: f32, rx: f32, ry: f32 },
    Path { d: String },
    Text { content: String, font_size: f32 },
    Image { href: String },
    Line { x1: f32, y1: f32, x2: f32, y2: f32 },
}

Architecture

crates/
├── agcanvas/                 # Desktop app
│   └── src/
│       ├── main.rs           # Entry point, window setup
│       ├── app.rs            # Main app state, UI, toolbar, drawing interaction
│       ├── session.rs        # Session/tab state management
│       ├── element_tree.rs   # Structured element representation
│       ├── clipboard.rs      # System clipboard integration
│       ├── mermaid.rs        # Mermaid → SVG rendering
│       ├── drawing/
│       │   ├── element.rs    # DrawingElement, Shape, ShapeStyle, hit testing
│       │   ├── tool.rs       # Tool enum, DragState, ResizeHandle
│       │   └── render.rs     # Shape rendering via egui Painter
│       ├── canvas/
│       │   ├── state.rs      # Pan/zoom transformation state
│       │   └── interaction.rs # Mouse/keyboard input handling
│       ├── svg/
│       │   ├── parser.rs     # SVG → ElementTree conversion
│       │   └── renderer.rs   # SVG → pixels (resvg/tiny-skia)
│       └── agent/
│           ├── protocol.rs   # JSON message types
│           └── server.rs     # WebSocket server
└── agcanvas-mcp/             # MCP server bridge
    └── src/
        ├── main.rs           # CLI entry point, stdio transport
        ├── bridge.rs         # WebSocket client → agcanvas
        └── tools.rs          # MCP tool definitions

Dependencies

Crate	Purpose
eframe/egui	GUI framework
usvg	SVG parsing
resvg/tiny-skia	SVG rendering
mermaid-rs-renderer	Mermaid → SVG
arboard	Clipboard access
tokio-tungstenite	WebSocket (both server and client)
rmcp	MCP server SDK (Anthropic official)
serde/serde_json	Serialization

Roadmap

• Multi-frame support (sessions/tabs)
• Shape drawing (rectangle, ellipse, line, arrow, text)
• Selection, move, resize with handles
• Mermaid diagram rendering
• MCP server bridge for AI coding tools
• Real code generation (not just stubs)
• Agent draw commands (modify canvas from agent)
• Export to file
• Diff view (before/after agent changes)
• Plugin system for code generators
• Undo/redo
• Multi-select and group operations

License

MIT