Towards AI Drone Assistants in Extended Reality*
Session Number
3
Advisor(s)
Arthur Caetano, Misha Sra, Department of Computer Science, University of California, Santa Barbara
Location
A117
Discipline
Computer Science
Start Date
15-4-2026 2:15 PM
End Date
15-4-2026 3:00 PM
Abstract
The growing presence of drones enables their use as embodied AI assistants for routine tasks. However, users often lack programming or piloting expertise, necessitating interfaces that facilitate high-level interactions. Although extended reality (XR) has been used to control flight-level drone operation, there is limited understanding of which XR and Agentic AI capabilities are needed for task-level interaction with drone assistants. To identify the necessary drone assistant capabilities, we adopt a research-through-design approach to co-design a low-fidelity prototype of a drone assistant. Our study identifies the expected functionalities of drone assistants, including spatial and procedural memory, intent visualization, and a human-like sense of safety. We demonstrate the technical feasibility of these features by implementing a high-fidelity version and comparing it with the low-fidelity prototypes. Our work advances human-drone interaction through Agentic AI and XR, enabling practical integration of drones into everyday environments for routine task assistance
Towards AI Drone Assistants in Extended Reality*
A117
The growing presence of drones enables their use as embodied AI assistants for routine tasks. However, users often lack programming or piloting expertise, necessitating interfaces that facilitate high-level interactions. Although extended reality (XR) has been used to control flight-level drone operation, there is limited understanding of which XR and Agentic AI capabilities are needed for task-level interaction with drone assistants. To identify the necessary drone assistant capabilities, we adopt a research-through-design approach to co-design a low-fidelity prototype of a drone assistant. Our study identifies the expected functionalities of drone assistants, including spatial and procedural memory, intent visualization, and a human-like sense of safety. We demonstrate the technical feasibility of these features by implementing a high-fidelity version and comparing it with the low-fidelity prototypes. Our work advances human-drone interaction through Agentic AI and XR, enabling practical integration of drones into everyday environments for routine task assistance