PawAI / Go2

PawAI in one minute#

The core of PawAI can be summarized in four steps:

1. See: the D435 and perception models recognize a registered user, gestures, poses, and nearby objects.
2. Understand: camera, speech, and system states are normalized into consistent ROS2 events.
3. Decide: PawAI Brain proposes a skill and a response based on context, but never controls the robot directly.
4. Act: Interaction Executive checks safety rules and the current state before allowing speech or a physical action to reach the Go2.

PawAI’s hard rule: the LLM may propose; only the Executive releases production sport motion. Audio and hard-stop remain independent paths, while hard-stop still needs dedicated audit and validation.

Watch the full demo#

This video moves from real-time perception and voice interaction to safety refusal and PawAI Studio’s decision trace. It is a June 2026 system snapshot—not proof that every experimental capability below has reached product reliability.

This demo is narrated in Chinese. The surrounding English text explains each stage and what to watch for.

Full demo: PawAI—autonomous object finding and embodied interaction through multimodal perception fusion.

Why I started building PawAI#

I began with the Go2 ROS2 and WebRTC drivers. Making the robot move, reading a sensor, or calling one predefined action was not the hardest part.

The difficult questions appeared when face, object, pose, speech, navigation, and an LLM entered the same system: who decides what happens next, and who owns the body when two modules want it to move?

If a cloud service fails, perception is wrong, or the LLM invents something, how does the system return to a safe state?

Those questions gradually turned PawAI from a collection of feature prototypes into a systems-engineering project. The focus shifted from “adding one more model” to three more important goals:

• Give every perception module a consistent and traceable event contract.
• Separate decision-making from physical control so the LLM cannot cross the safety boundary.
• Describe capabilities through evidence and fallback paths instead of relying on a successful-looking demo.

This was not designed correctly from day one#

When PawAI began, I was primarily a web and software developer with almost no robot-hardware integration experience.

Our first serious robotics challenge was a Go2 Pro, whose development path is more restricted than the Go2 EDU and relies heavily on WebRTC and community tooling.

My rough personal estimate is more than 500 hours invested in this project.

Nights, weekends, and gaps between classes went into wiring, rerunning tests, reading logs, recording bags, tuning parameters, and discarding solutions that had looked reasonable the day before.

Before the public release, the repository recorded 138 active commit days and more than 1,300 commits authored by me. A later lessons-learned document named over 70 failure modes.

Those numbers do not measure quality or hours, but they preserve the density of the iteration.

On one Sunday in November 2025, I worked from after lunch until 1 a.m. My diary ended with: “I worked until one in the morning, yet it made me feel that engineering might truly be my calling.”

From perception to action: system architecture#

PawAI is one workspace containing ten core ROS2 packages. Perception, decision, action, and shared contracts are separated into layers.

Dependencies point toward common interfaces instead of letting every feature call every other feature directly.

Read the diagram from the bottom up. The Go2 driver and hardware “do the real work.” The Executive decides whether it is allowed. The Brain decides what is appropriate to propose. Perception nodes only turn the world into events.

Between them, go2_interfaces and pawai_contracts define a shared language for events, skill proposals, and execution results.

This separation prevents two failures. A perception module cannot command the robot simply because it detected something. Adding a new model also does not require rewriting Brain, Studio, and the Go2 driver together.

Each ROS2 node can be started, stopped, tested, or replaced independently while the system continues to communicate through stable contracts.

The models actually running inside PawAI#

PawAI does not stop at saying that it “uses AI.” Every perception and speech path has an explicit model, execution location, resource budget, and fallback. This is the main stack used for the June 2026 demo and public release.

The table looks like a clean answer, but model selection reversed three times. Gesture and pose moved from RTMPose saturating the GPU to DWPose TensorRT keypoint drift and JetPack 6 deployment friction.

Only hardware experiments overturned our earlier belief that MediaPipe could not run on Jetson.

The real selection criterion was never single-model accuracy alone. It was: on 8GB of unified memory, can face, pose, gesture, object, and speech still coexist without taking the whole system down?

Why the LLM cannot control the robot directly#

PawAI Brain is not an agent that can simply do whatever it generates. Its output is a proposal that passes through three layers before execution:

1. Safety: intercept dangerous, emergency, or unauthorized requests first.
2. Policy: decide whether a skill is valid in the current state and allowed for this source.
3. Expression: decide how to speak, how to display the decision in Studio, and whether a legal action should be submitted.

This structure is not decoration; the robot can physically move. A model may be wrong, perception may be incomplete, or several events may arrive at once.

Defaulting to no motion, rejecting unauthorized skills, and preserving human intervention matter more than making the demo look intelligent.

The public claim is narrower: the deterministic Safety Gate and skill allowlist exist and have unit-test protection.

That does not mean PawAI never hallucinates, nor that every safety scenario has passed end-to-end hardware validation.

How LangGraph participates in decisions#

PawAI uses LangGraph to orchestrate the Conversation Engine. It does not control motors. The implementation has 12 graph stages; the list below groups them into nine concepts that are easier to read.

The early llm_bridge_node handled the ROS adapter, persona, model fallback, conversation history, environment context, JSON parsing, and TTS output in one place. As it grew, hidden bugs became difficult to locate.

In one case, Chinese replies kept being cut off around a comma. The cause was not the model, but a stale Jetson install/ tree containing an old truncation rule.

That incident led me to extract conversation orchestration into the pawai_brain package.

LangGraph turns one conversation round into a pipeline similar to this:

1. Input normalization: normalize text, speech, and event inputs.
2. Safety gate: handle stop, emergency, and forbidden conditions first.
3. World state: include current face, object, pose, obstacle, and TTS state.
4. Capability context: expose only currently available SkillContracts to the model.
5. Memory / Persona: add short-term dialogue and PawAI’s speaking style.
6. LLM decision: generate reply, an optional skill, and arguments.
7. JSON validation / repair: repair malformed structure; fall back to a rule-based canned response if repair fails.
8. Skill gate: let Brain mark the proposal accepted, trace-only, blocked, or outside the allowlist.
9. Output / trace: emit the reply and proposal while sending every stage to Studio.

The Conversation Engine intentionally returns a small contract:

1
{
2
  "reply": "I am PawAI, a robot dog learning to understand its environment.",
3
  "skill": "self_introduce",
4
  "args": {}
5
}

The reply may become speech; the skill is only a proposal. A low-risk skill such as show_status can pass the gate and execute.

A longer motion sequence may remain trace-only and be triggered from Studio under human control. Navigation and high-risk actions are never unlocked simply because the LLM suggested them.

How Edge, Cloud, and Go2 work together#

PawAI does not send everything to the cloud, nor does it require Jetson to perform every task alone. Deployment is split across four roles:

ASR, LLM, and TTS are designed with layered fallback. The Jetson microphone path can move from SenseVoice cloud to local / Whisper.

The LLM can fall back to another model and RuleBrain, while TTS can fall back to edge-tts / Piper. Safety, the allowlist, and Executive remain local.

That does not mean the fully offline voice loop has passed validation. Studio’s laptop-capture path currently depends on cloud SenseVoice, so it remains a single point of failure.

Offline-first is an architectural direction, not a baseline-passed public capability.

The repository still preserves direct legacy / demo publishers, which production flags must disable. This is technical debt that remains visible; the design constraint is not yet a universal structural fact across the entire repo.

With those publishers disabled, a Brain failure produces no new proposals and an Executive failure closes the regular sport-motion mainline.

Audio and hard-stop stay deliberately independent; the latter still needs dedicated audit and validation.

PawAI Studio: making the robot observable#

After integration, the common problem was not “there is no data.” Data arrived and a module decided, but no action followed.

Terminal logs alone could not quickly reveal where an event had been ignored, degraded, or blocked.

I built PawAI Studio as a conversational console that combines chat, perception panels, and decision traces. The operator can see perception events, Brain proposals, Safety Gate results, and Executive decisions in the same interface.

Studio is not valuable because a polished interface proves a capability works. Its value is the evidence chain: why the robot reacted, why it did not react, and which layer stopped it.

What currently works#

I did not want “supports many features” to replace real validation, so PawAI separates implementation capability, development observations, and trusted measured evidence.

This is not a list of weaknesses. To me, it records PawAI’s most important transition: from demonstrating features to taking responsibility only for what the evidence supports.

This architecture was forced into existence by failure#

The finished diagram can make it look as if we always knew where Brain, Executive, LangGraph, and the Safety Gate belonged.

In reality, many boundaries were written only after hardware moved incorrectly, models misclassified a scene, or the demo pipeline locked up.

1. 2025-11-18 | Video worked; control was dead. We traced SDP, ICE, DTLS, and SCTP before finding an aiortc 1.14 / STUN compatibility problem. Pinning 1.9 and removing STUN cut DataChannel startup from timeout to roughly 0.5 seconds.
2. 2026-03-21 | Model selection reversed three times. RTMPose consumed 91–99% of the Jetson GPU. DWPose suffered TensorRT keypoint drift. Hardware benchmarks showed that MediaPipe CPU inference was the better system-level choice.
3. March–April 2026 | Software met mechanics. Vibration, USB contact, 19V conversion, and current spikes caused eight or more Jetson power losses. Foam, glue, and straps became part of the system.
4. April–May 2026 | Silence did not mean the API was gone. Go2 Megaphone ignored malformed payloads, costing nearly two weeks. Another night’s truncation came from a stale ROS2 install/, not three models.
5. May–June 2026 | “Zero velocity” did not mean stop. Go2 Sport Mode ignored a small Move{x} and kept the previous command alive. Wall impacts forced StopMove(1003), deduplication, heartbeats, and one sport-motion arbiter.
6. June 2026 | Working features still produced a traffic jam. Face, cup, pose, and gesture competed for TTS, Brain state, the robot body, and WebRTC ownership. Pending queues, cooldowns, traces, and fail-closed baselines followed.

These incidents are not trophies. Every collision and false detection meant the safety design was incomplete. What matters is that we kept the failures and converted them into contracts, tests, arbitration rules, and narrower public claims.

More than eight months of engineering evolution#

April 2026: when software developers started opening the robot#

Jetson, D435, RPLIDAR, microphones, and speakers do not attach themselves to a Go2.

We handled voltage, current spikes, USB contact, heat, weight balance, field of view, and LiDAR orientation. I also 3D-printed the custom back cover and brackets before the software had a stable place to run.

2026-04-25 | With the custom 3D-printed back cover installed, the Jetson, D435, and RPLIDAR could finally stand on the Go2 as one complete stack.

The month was unforgiving. One pip install ultralytics broke Torch, NumPy, and every model together. RPLIDAR then brought ARM64 crashes, Cartographer problems, reversed orientation, a shaking mount, and four maps that had to be discarded.

How we developed the system#

By the later stages, PawAI had developed a conservative workflow that fits physical robots well:

1. Isolate one path first: prove face, object, speech, LiDAR, and the Go2 driver independently before entering the full stack.
2. Define contracts before integration: stabilize topics, event schemas, SkillPlan, and SkillResult so modules do not depend directly on one another.
3. Run new systems in shadow first: let LangGraph or a new policy observe the same input without receiving physical control until it is stable enough.
4. Start every motion path with no-motion evidence: verify publishers, ownership, stop gates, and world state before a small HITL movement test.
5. Keep fallback and rollback available: cloud, local, rule-based paths, and feature flags must all support a fast retreat during a live run.
6. Let evidence control the wording: an implemented mechanism, a development observation, and a passed baseline are different levels; the public site describes only the last level’s permitted scope.

This workflow emerged from repeated integration failures. It helped PawAI evolve from a student project with many features into an engineering system that knows what it can currently prove—and when it must stop.

What I worked on#

PawAI was a team capstone. This page focuses on the system integration, Brain, speech, hardware bring-up, testing, and public-release work that I personally owned or deeply contributed to.

My primary work was turning separate hardware, perception models, and AI features into a system that could be operated, observed, and validated:

• Integrating Go2, Jetson, D435, speech devices, and the ROS2 runtime.
• Building the multimodal event and decision flow so perception could enter Brain safely.
• Designing the boundaries among Brain, Executive, and the Safety Gate.
• Handling the degradation chain across ASR, LLM, TTS, and Go2 audio output.
• Building PawAI Studio, traces, CLI tools, demo runbooks, and capability evidence.
• Reworking the architecture and public documentation after navigation, ownership, and live-demo failures.

Three lessons I took away#

What’s next#

PawAI will continue to improve, but the principle stays the same: establish reproducible evidence before expanding public claims.

The next stage includes:

• Extending perception baselines across multiple users, distances, and lighting conditions.
• Improving voice stop, gesture recognition, and full speech end-to-end tests.
• Connecting Studio Evidence directly to trusted baseline data.
• Revalidating navigation under no-motion, emergency-stop, and HITL conditions.
• Turning architecture, speech, perception, navigation incidents, and demo engineering into separate technical articles.

Source code and public documentation: GitHub — roy4222/PawAI