Text Enrichment and Annotations using Augmented Reality

Theoretical (Analytical):

Practical (Implementation):

Literature Work:

Overview

In this project you will develop an AR Unity application for interactive text annotations and enrichment. The Microsoft HoloLens 2 will be used as AR head-mounted display. The camera stream provided by the device can be used to detect objects (anchors) and to make the visible text accesible to the system (OCR). Based on user interaction, multiple options should be available to work with the text and receive visual feedback, e.g. text highlighting, translation, annotation, and keyword graphs. In a user study, the practicability and usability of the system should be evaluated.

Problem Statement

Manually working with texts comes with disadvantages. While simple tasks like highlighting text can be done without much effort, tasks like translating words, annotating them, or creating abstract knowledge representations from the text like graphs involves more manual work and often further facilities like tablets, notebooks, etc. With this project, we want to exploit the capabilites of modern AR headsets to allow performing all of the explained tasks with only one, highly usable system.

Tasks

  • Work alone or as team of two students.
  • Get familiar with Unity and the technical requirements for developing a AR application and processing the video stream
  • Create a framework for analyzing text, interacting to annotate etc., and displaying visual enrichments
  • Evaluation of the developed techniques by means of a user study. 

Requirements

  • Basic knowledge about visualizations and image/video processing
  • Advanced programming skills in C# or a similar language
  • Good conceptual skills (software architectures)
  • Useful: Git, Unity, AR-Experience, OCR, Python, experience with user studies

Scope/Duration/Start

  • Scope: Bachelor/Master

Contact

References

 

  • Gebhardt, Christoph, Roman Rädle, and Harald Reiterer. "Employing Blended Interaction to Blend the Qualities of Digital and Physical Books/Anwendung von Blended Interaction zur Vermischung der Eigenschaften digitaler und physischer Bücher." i-com 13.3 (2014): 36-42.
  • Qian, Jing, et al. "Dually noted: layout-aware annotations with smartphone augmented reality." Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022.