Virtual Reality Based Multi-Modal Teleoperation Using Mixed Autonomy

Thumbnail Image
Kadavasal, Muthukkumar
Seth, Abhishek
Major Professor
Committee Member
Journal Title
Journal ISSN
Volume Title
Oliver, James
Research Projects
Organizational Units
Organizational Unit
Mechanical Engineering
The Department of Mechanical Engineering at Iowa State University is where innovation thrives and the impossible is made possible. This is where your passion for problem-solving and hands-on learning can make a real difference in our world. Whether you’re helping improve the environment, creating safer automobiles, or advancing medical technologies, and athletic performance, the Department of Mechanical Engineering gives you the tools and talent to blaze your own trail to an amazing career.
Journal Issue
Is Version Of

A multi modal teleoperation interface is introduced featuring an integrated virtual reality based simulation augmented by sensors and image processing capabilities on-board the remotely operated vehicle. The proposed virtual reality interface fuses an existing VR model with live video feed and prediction states, thereby creating a multi modal control interface. Virtual reality addresses the typical limitations of video-based teleoperation caused by signal lag and limited field of view thereby allowing the operator to navigate in a continuous fashion. The vehicle incorporates an on-board computer and a stereo vision system to facilitate obstacle detection. A vehicle adaptation system with a priori risk maps and real state tracking system enables temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle’s teleoperated state. As both the vehicle and the operator share absolute autonomy in stages, the operation is referred to as mixed autonomous. Finally, the system provides real time update of the virtual environment based on anomalies encountered by the vehicle. The system effectively balances the autonomy between human and on board vehicle intelligence. The stereo vision based obstacle avoidance system is initially implemented on video based teleoperation architecture and experimental results are presented. The VR based multi modal teleoperation interface is expected to be more adaptable and intuitive when compared to other interfaces.


This is a conference proceeding from ASME 2008 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference 3 (2008): 1451, doi:10.1115/DETC2008-49144. Posted with permission.

Tue Jan 01 00:00:00 UTC 2008