Computer Vision and Audio-Driven Control of a Robotic Prosthetic Arm*
Session Number
1
Advisor(s)
Dr. Ashwin Mohan, IMSA
Location
A150
Discipline
Computer Science
Start Date
15-4-2026 10:15 AM
End Date
15-4-2026 11:00 AM
Abstract
Robotic prosthetic arms typically rely on muscle-based control systems, but alternative approaches may enable more flexible interaction with the environment. This project explores the use of artificial intelligence and external sensing to control a robotic arm using environmental inputs rather than direct muscle signals. An ESP32 microcontroller connects the robotic arm to a computer via Bluetooth, allowing commands generated through external processing to control arm movements. A camera and microphone connected to the computer provide visual and auditory input, which are analyzed using pattern recognition algorithms to identify humans and interpret spoken commands. Based on these inputs, the system generates control signals that are transmitted wirelessly to the robotic arm. Offloading computation to an external computer enables the use of more complex algorithms without increasing onboard power consumption or hardware requirements. This approach demonstrates the potential of integrating computer vision and audio recognition with robotic manipulators to enable more adaptive and interactive assistive technologies.
Computer Vision and Audio-Driven Control of a Robotic Prosthetic Arm*
A150
Robotic prosthetic arms typically rely on muscle-based control systems, but alternative approaches may enable more flexible interaction with the environment. This project explores the use of artificial intelligence and external sensing to control a robotic arm using environmental inputs rather than direct muscle signals. An ESP32 microcontroller connects the robotic arm to a computer via Bluetooth, allowing commands generated through external processing to control arm movements. A camera and microphone connected to the computer provide visual and auditory input, which are analyzed using pattern recognition algorithms to identify humans and interpret spoken commands. Based on these inputs, the system generates control signals that are transmitted wirelessly to the robotic arm. Offloading computation to an external computer enables the use of more complex algorithms without increasing onboard power consumption or hardware requirements. This approach demonstrates the potential of integrating computer vision and audio recognition with robotic manipulators to enable more adaptive and interactive assistive technologies.