OpenBMB shrunk a multimodal VLM to 0.9B parameters and deployed it on a quadruped robot dog. The robot follows verbal commands ("follow the person in the red jacket"), tracks targets through occlusions, and navigates obstacles. Everything runs on-device with no cloud dependency. 5+ FPS inference on an NVIDIA Jetson Orin NX with approximately 180ms latency.
# The Problem With Robot AI
Robot AI is stuck in a frustrating tradeoff. You either send camera frames to a massive cloud model and accept 200ms+ latency, privacy risks, and network dependency. Or you run a tiny vision-only model on-device and lose all language understanding.
- ✗ 200ms+ latency per frame
- ✗ Needs WiFi/5G always
- ✗ Privacy risk from streaming video
- ✗ Robot stops if connection drops
- ✓ Fast on-device inference
- ✗ No language understanding
- ✗ Cannot parse verbal commands
- ✗ Fixed targets only
- ✓ On-device at 5+ FPS
- ✓ Full VLM with language
- ✓ Verbal commands supported
- ✓ No cloud needed
MiniCPM-Robot sits in the gap. A full vision-language model compressed small enough to run on a robot's onboard GPU. It understands what it sees and what you say.
# The Architecture
The base model is MiniCPM4-0.5B, a compact multimodal VLM from OpenBMB. The architecture splits into three modules, each handling a distinct stage of the perception-to-action pipeline. The RobotTrack variant runs at 0.9B parameters for the tracking task.
Takes raw camera frames at 5+ FPS and converts them into visual embeddings. A fused DINOv3 + SigLIP encoder produces semantically rich representations. The encoder identifies people, objects, and spatial relationships in each frame without needing separate object detection models.
Fuses visual embeddings with natural language instructions. When you say "follow the person in the red jacket", the decoder grounds the phrase "red jacket" to a specific visual region in the encoder's output. This is the core of the system: it connects what the robot sees with what you tell it to do.
Converts the decoder's grounded understanding into velocity commands: forward/backward speed, left/right speed, and angular rotation. These commands feed directly into the robot dog's motor controllers. No intermediate waypoint planning. The model outputs raw motion vectors at every frame.
Camera (5+ FPS) -> DINOv3+SigLIP Encoder (visual embeddings) -> MiniCPM Decoder (+ language command) -> Action Head (velocity: fwd/back, left/right, angular) -> Motor Controllers (quadruped locomotion)
# How It Works On The Robot
Here is a complete walkthrough of what happens when you give MiniCPM-Robot a verbal command. Every step runs locally on the NVIDIA Jetson Orin NX mounted on the robot dog.
User speaks: "Follow the person in the red jacket" Audio transcribed on-device Command tokenized and passed to language decoder Target description extracted: "person in the red jacket"
Camera captures frame at 5+ FPS... Frame -> DINOv3+SigLIP encoder -> visual embeddings 3 people detected in field of view Grounding: "red jacket" matched to Person #2 at (340, 220) Bounding box locked, confidence: 0.94
Action head computes velocity vector... Target offset from center: +45px right, -30px below Distance estimate: 4.2m (within tracking range) Output: fwd=0.8 m/s, lateral=0.3 m/s, angular=0.15 rad/s Commands sent to motor controllers
Loop running at 5+ FPS on Jetson Orin NX... Frame 147: Target visible, tracking normally Frame 148: Target occluded by obstacle Re-identification engaged: searching for "red jacket" Frame 152: Target re-acquired, confidence: 0.89 Tracking resumed. Inference latency: ~180ms
No cloud calls, no network hops, no external APIs. Camera to motor commands in a single forward pass on the Jetson.
Fast enough for smooth tracking of walking humans. The robot adjusts heading and speed approximately every 180ms.
# The Capabilities
MiniCPM-Robot is not a single-trick tracker. The VLM backbone gives it a range of behaviors that pure vision models cannot match. Here is what it can do out of the box.
Lock onto a specific person and follow them continuously. The robot maintains a consistent distance and adjusts speed to match the target's walking pace. Tracks targets up to 15 meters away.
When the target walks behind an obstacle and disappears from view, the model remembers the visual description. Once the target reappears, it re-identifies them reliably (73-90% across EVT-Bench task types) without needing a new command.
Give instructions in plain language. "Follow the person with the blue backpack." "Track the woman near the bench." The language decoder grounds these descriptions to specific visual targets in real time.
The vision encoder sees more than just the target. It detects obstacles in the path and the action head adjusts velocity commands to navigate around them while maintaining the tracking lock.
Issue a new verbal command to switch targets mid-tracking. Say "now follow the tall person on the left" and the robot seamlessly re-grounds to the new target without restarting the pipeline.
# The Numbers
| Metric | Cloud VLM | Vision-Only | MiniCPM-Robot |
|---|---|---|---|
| Parameters | 7B-70B | 50-200M | 0.9B |
| Latency | 200ms+ | ~30ms | ~180ms |
| Language | Yes | No | Yes |
| On-Device | No | Yes | Yes |
| Power | Cloud | ~1W | Edge-class |
# Why This Matters
This is the first end-to-end VLM deployed on consumer-grade edge hardware on a real robot. Not a simulation. Not a demo running on a workstation with a camera feed piped in. A model running on the robot itself, driving its legs in real time.
The dominant assumption in robotics has been that useful AI requires cloud compute. MiniCPM-Robot proves that a sub-1B parameter model can do meaningful perception, language understanding, and motor control on a $400 edge GPU.
The MiniCPM4-0.5B base combined with task-specific heads produces a system small enough for edge hardware but capable enough for real-world tasks. This is not a toy demo. The model handles occlusions, multi-target scenarios, and obstacle-filled environments at production quality.
Previous on-device robot models could see but could not understand words. MiniCPM-Robot takes verbal commands and grounds them visually in real time. This turns a robot from a programmed machine into something you can talk to.
The future of robotics is not bigger models. It is smaller models that fit on the robot. OpenBMB proved that a 0.9B VLM can see, listen, and move on consumer-grade hardware. No cloud. No latency. No excuses. MiniCPM-Robot is what happens when you stop scaling up and start scaling down to where the work actually happens.
Enjoyed this?
New episodes Mon, Wed, Sat.