The Hardware Baseline
To explore how an AI agent works with real hardware, the project first designed and built a 3-axis motion platform as the physical basis for validating the drawing workflow. After the mechanical structure, motion system, and control hardware were assembled and commissioned, early runs relied on conventional image-processing, path-generation, and machine-control software, driving the machine by hand to establish a working baseline for the hardware and the traditional process.
On this manual baseline, every new input image required human judgment at each step about whether the conditions to proceed were met, then calling the corresponding tools one by one to finish processing. Building on this actual process, the workflow design focused on letting the AI agent read user intent, choose a processing route, and organize tool calls, while the image model and deterministic tools each handle the tasks they suit, with explicit checks and confirmation kept at key steps. The result is an end-to-end loop that uses a messaging app as its entry point, is driven by natural-language conversation, and completes a physical plot under staged checks and human confirmation.
Manual judgment · manual tool calls
Language to Plotting
A user sends an image through a messaging app and describes the drawing request. The AI agent reads the input intent, decides the processing route, and calls the corresponding tools, returning checkable intermediate results at each gate until the operator confirms entry into hardware execution.
Reads natural-language requests, decides the processing route, calls tools, and drives the workflow forward.
Converts an image into a form better suited to machine drawing, only when a complex input requires it.
Handles image normalization, complexity checks, path planning, G-code generation, and pre-run checks.
Confirms intermediate results, homes and calibrates the machine, authorizes execution, and makes the final physical-safety call.
-
1Input
- Image + language input
-
2Understand & route
- Agent parses & routes
- A_DIRECTstraight to preprocessing
- B_IMAGE_EDITvia local Diffusion Model
-
3Image processing
- Deterministic preprocessing
- processed image reviewgate
- complexity gategate
-
4Path & code
- Deterministic path planning
- G-code generation & static checks
- Path preview & dry-run
-
5Hardware run
- Operator sign-offgate
- In-session hardware run
- Physical plot output
If any review or check fails, the flow stops and returns to the matching processing stage; it does not continue generating or execute any later hardware action.
A complex photo is first simplified by the image model, then enters deterministic path construction and physical plotting.
A clean graphic keeps its original structure and goes straight into deterministic processing and physical plotting.
A more structurally complex image stays readable enough after binarization to go straight into path construction and physical plotting.
Key Engineering Decisions
Define the responsibility boundaries between AI, deterministic tools, and people
AI handles semantics and uncertain input; deterministic tools generate and check the path and G-code; the operator keeps hardware preparation and final run authority.
Preview first, then run
The image-processing result, the path, the G-code, and the dry-run all need checking; only after the current gate passes does the flow move to the next stage. Problems can be caught before any hardware moves, and the operator always keeps clear points to step in.
Choose the processing route by combining semantic judgment with visual statistics
A multimodal model judges at the level of semantics and visual style, while deterministic scripts assess visual features such as subject separation, background complexity, and binarized readability. The two results together decide whether to process directly or run a visual conversion first; when the judgment is not confident enough, the more conservative route is taken. Both routes still pass the later review and gates.
Technical Contributions
Deterministic path construction for physical plotting
The processed image is vectorized and split into centerline strokes, fill paths, and fill boundaries, then combined into the final drawing path during G-code generation. Fill areas use a continuous serpentine path that reduces repeated pen lifts, pen drops, and idle travel, keeping the hardware motion more continuous.
In-session execution bridge that keeps hardware state continuous
The workflow splits hardware execution into steps such as homing, Z-axis calibration, run confirmation, and G-code sending. To avoid reconnecting to the controller repeatedly, which would reset state or leave positions inconsistent, the execution bridge performs homing, calibration, confirmation, sending, and return-to-position within a single hardware session, while keeping the operator gate at each stage.
Validation
Validation covers three levels: physical output, workflow path, and engineering contracts:
A_DIRECTandB_IMAGE_EDITeach complete a closed evidence chain from input image, to processing result, to path preview, to physical plot.- After the image-processing service moved to a local ComfyUI, a full-chain validation including real hardware execution was completed; G-code sending ran with no stream error, and the physical output was confirmed by the operator.
- Automated tests cover key contracts including route-result verification, the complexity gate, G-code static checks, offline build, and provider configuration and failure handling.
Real-hardware full-chain validation completed after the image-processing service moved to a local ComfyUI.
The current version is positioned as a validated proof-of-concept, with validation focused on the workflow architecture, safety gating, and the real-hardware loop.
- Input images must be convertible into a clear, complexity-controlled black-and-white form; results too dense in detail or too complex in structure are sent back for reprocessing or rejected by the complexity gate.
- The image model's output varies, so every processed image needs a human review.
- The current use case is a supervised semi-automatic drawing flow, where hardware execution always keeps operator confirmation and a physical stop.
What Carries Over
From hardware to agent, strung into one chain
Starting from the mechanical structure, motion system, and control hardware, the agent orchestration, deterministic tool chain, and real-hardware validation are organized into a complete system.
Final run authority still belongs to a person
Based on the different strengths of AI, deterministic tools, and people, responsibilities, interfaces, and gates are divided, and the final hardware-run authority is reserved for the operator.
Done means someone can take over at any time
Through tests, a frozen baseline, documentation, and handoff, the PoC is organized into a project that can be paused, resumed, and iterated further, with full-chain validation redone after the provider migration.