Introduction: Why Spatial Computing Is Transforming AI Assistants
In 2026, the boundary between the digital and physical worlds is rapidly blurring. Spatial computingâ€"which includes augmented reality (AR), virtual reality (VR), 3D visualization, and position-based sensorsâ€"is becoming an essential component of conversational AI assistants. By incorporating spatial context, models can deliver more relevant guidance, automate workflows, and interact with the world more naturally.
This practical guide demonstrates when and how to harness spatial computing to empower modern AI assistants, with concrete examples using the latest models such as Kyutai REKI (a 2.4B VLA model for robotic manipulation), DeepSeek-V4.1-Flash-1017, and the new Google Harness2.
How to design prompts that leverage the spatial dimension
The first step is moving from a flat text prompt to one that encodes coordinates, objects, and actions. A wellâ€'structured prompt should know:
- Where the user or object of interest is located
- What 3D objects are present in the scene
- What the user wants to do (pick, move, label, etc.)
A universal prompt template that works with most VLAâ€'based LLMs is:
Response format:
{
"action": "",
"target": "",
"coordinates": [x, y, z],
"confidence": 0-1
}
Scenario:
User position:
Nearby objects:
Goal:
Notes:
Instruction:
Generate an executable action that accounts for spatial context and reply with the JSON above.
Example 1: Prompt for an AR AI assistant
Imagine building an AR assistant that overlays useful information on a hospital's equipment. The following prompt works with current multimodal models (including DeepSeek-V4.1-Flash-1017) to generate realâ€'time annotations:
Role: AR assistant for medical staff
Context: The user wears an AR headset and looks at an intensive care unit.
Visual data: (RGB-D frames) â€" beds, monitors, IV stands.
Goal: Overlay safe, relevant labels for each device.
Instruction:
For each detected device, emit a JSON with: {"entity": "", "location": [x, y, z], "label": "", "priority": "high/medium/low"}.
Do not include unrelated devices.
Example 2: Prompt for robotic control using a VLA model
The Kyutai REKI model excels at robotic manipulation thanks to its Visionâ€'Languageâ€'Action architecture. To drive a robotic arm toward a target, the following prompt explicitly extracts spatiality:
Model: Kyutai REKI (2.4B VLA)
Scenario: A UR5 robotic arm with a gripper sits on a workbench.
Visual input: (depth camera) â€" 3D scene mapping with target "red screw".
Goal: Grasp the target and place it in a marked destination area.
Instruction:
1. Identify the 3D coordinates of the target.
2. Compute a safe path avoiding other objects.
3. Generate gripper commands: {"joint_angles": [...], "gripper_action": "open/close"}.
Return only the JSON, no additional text.
Tools and frameworks in the 2026 landscape
The technology stack for spatial AI assistants has expanded with new openâ€'source tools and API services.
- Kyutai REKI â€" A 2.4B VLA model optimized for robotic manipulation with an efficient memory that retains spatial details longer.
- DeepSeek-V4.1-Flash-1017 â€" A lightweight, highâ€'performance LLM now equipped with an integrated search API, useful for retrieving dynamic location data.
- Google Harness2 â€" A recursive fineâ€'tuning framework that rewrites an agent’s behavior at evaluation time, ideal for quickly adapting AI assistants to new 3D environments.
- Voxel51 3.0 â€" A unified multimodal data platform for Physical AI that merges search, curation, annotation, and evaluation of sensor data into a single solution for physical AI teams, now featuring Voxel51 Agen...
These tools are often accessible via REST APIs or Hugging Faceâ€'compatible SDKs, making it simple to combine a base model with a custom spatial promptâ€'engineering layer.
Best practices for spatial prompt engineering
- Normalize coordinates. Always convert sensor data into a single reference system (e.g., meters, originâ€'centered).
- Use explicit tokens. Include tokens like
[[position]],[[object_id]]to signal which parts of the scene are important to the