PENGEMBANGAN GAME EDUKASI AR DENGAN NPC GURU VIRTUAL BERBASIS MEDIAPIPE DAN AI VOICE INTERACTION

Abstract

The development of interactive learning media has been supported by the integration of Augmented Reality (AR), Artificial Intelligence (AI), and voice interaction technologies. However, learning applications that integrate AR with virtual teacher characters and multiple forms of interaction are still challenging to implement, particularly in terms of system integration and device limitations. This study aims to develop an educational game based on Augmented Reality with a Virtual Teacher NPC that integrates AI, voice interaction, ML Kit Optical Character Recognition (OCR), MediaPipe, chat, and quiz features. The application was developed using Unity and AR Foundation, with Google Gemini used to generate AI responses. Voice interaction was implemented through Speech-to-Text (STT) and Text-to-Speech (TTS), while ML Kit OCR was used to recognize text from scanned learning materials. MediaPipe was implemented to process visual information from the user's face as a supporting component for NPC interaction. The system was evaluated using Black Box Testing, System Usability Scale (SUS), and User Acceptance Testing (UAT). The results of Black Box Testing showed that most of the main functions operated according to the designed scenarios, although several functions were partially successful due to OCR image quality and MediaPipe implementation limitations. Testing with 20 respondents produced an average SUS score of 63.88, while UAT produced an average score of 3.58 out of 5 or 71.61%. These results indicate that the developed application can function as an interactive educational media, although improvements are still required in usability, OCR performance, and MediaPipe implementation.