This project explores the incorporation of Dynamic Difficulty Adjustment (DDA) and Conversational User Interfaces (CUIs) within the relatively unexplored domain of head-worn Audio Augmented Reality (AAR). The aim is to provide an engaging AAR experience by combining various interaction modalities such as hand and gaze interaction, spatial sound, and voice input. The challenge lies in seamlessly integrating these interaction modalities to create a captivating experience in AAR. This project presents the design, implementation, and evaluation of an innovative, flow-aspired AAR experience for head-worn AAR, presented as a gamified historical narrative.

The proposed approach integrates user-driven DDA, spatialized auditory feedback, gaze-based interaction, and location-specific musical elements. The DDA component utilizes heuristic distributions based on factors such as round completion time, hint usage, remaining time, and additional time needed. These factors are weighted to calculate an overall score, enabling players of varying skill levels to enjoy a customized difficulty level. Upon completing AAR missions, users can compose their own personalized soundscape.

This AAR experience successfully addresses the challenges associated with personalized AAR interactions. It introduces a voice-assisted system that employs accurate hand and gaze interaction, even in the dynamic setting of a busy museum environment. Through thorough evaluation, it has been demonstrated that the combination of voice, hand, and gaze interaction yields higher usability compared to voice-only and gaze-only interaction methods. The DDA component effectively matches players’ individual styles, ensuring sustained engagement throughout the experience.