Overcoming the limitations of commodity augmented reality head mounted displays for use in product assembly

Abstract

<p>Numerous studies have shown the effectiveness of utilizing Augmented Reality (AR) to deliver work instructions for complex assemblies. Traditionally, this research has been performed using hand-held displays, such as smartphones and tablets, or custom-built Head Mounted Displays (HMDs). AR HMDs have been shown to be especially effective for assembly tasks as they allow the user to remain hands-free while receiving work instructions. Furthermore, in recent years a wave of commodity AR HMDs have come to market including the Microsoft HoloLens, Magic Leap One, Meta 2, and DAQRI Smart Glasses. These devices present a unique opportunity for delivering assembly instructions due to their relatively low cost and accessibility compared to custom built AR HMD solutions of the past. Despite these benefits, the technology behind these HMDs still contains many limitations including input, user interface, spatial registration, navigation and occlusion.</p> <p>To accurately deliver work instructions for complex assemblies, the hardware limitations of these commodity AR HMDs must be overcome. For this research, an AR assembly application was developed for the Microsoft HoloLens using methods specifically designed to address the aforementioned issues. Input and user interface methods were implemented and analyzed to maximize the usability of the application. An intuitive navigation system was developed to guide users through a large training environment, leading them to the current point of interest. The native tracking system of the HoloLens was augmented with image target tracking capabilities to stabilize virtual content, enhance accuracy, and account for spatial drift. This fusion of marker-based and marker-less tracking techniques provides a novel approach to display robust AR assembly instructions on a commodity AR HMD. Furthermore, utilizing this novel spatial registration approach, the position of real-world objects was accurately registered to properly occlude virtual work instructions. To render the desired effect, specialized computer graphics methods and custom shaders were developed and implemented for an AR assembly application.</p> <p>After developing novel methods to display work instructions on a commodity AR HMD, it was necessary to validate that these work instructions were being accurately delivered. Utilizing the sensors on the HoloLens, data was collected during the assembly process regarding head position, orientation, assembly step times, and an estimation of spatial drift. With the addition of wearable physiological sensor data, this data was fused together in a visualization application to validate instructions were properly delivered and provide an opportunity for an analysist to examine trends within an assembly session. Additionally, the spatial drift data was then analyzed to gain a better understanding of how spatial drift accumulates over time and ensure that the spatial registration mitigation techniques was effective.</p> <p>Academic research has shown that AR may substantial reduce cost for assembly operations through a reduction in errors, time, and cognitive workload. This research provides novel solutions to overcome the limitations of commodity AR HMDs and validate their use for product assembly. Furthermore, the research provided in this thesis demonstrates the potential of commodity AR HMDs and how their limitations can be mitigated for use in product assembly tasks.</p>