CHI 2021 Workshop

online

Workshop Day

May 7th, 2021, virtual

Workshop ‘Evaluating User Experiences in Mixed Reality’

Measure user experience in MR (i.e., AR/VR) user studies is essential. Researchers apply a wide range of measuring methods using objective (e.g., biosignals, time logging), behavioral (e.g., gaze direction, movement amplitude), and subjective (e.g., standardized questionnaires) metrics. Many of these measurement instruments were adapted from use-cases outside of MR but have not been validated for usage in MR experiments. However, researchers are faced with various challenges and design alternatives when measuring immersive experiences. These challenges become even more diverse when running out-of-the lab studies. Measurement methods of VR experience recently received much attention. For example, research has started embedding questionnaires in the VE for various applications, allowing users to stay closer to the ongoing experience while filling out the survey. However, there is a diversity in the interaction methods and practices on how the assessment procedure is conducted. This diversity in methods underlines a missing shared agreement of standardized measurement tools for VR experiences. AR research strongly orients on the research methods from VR, e.g., using the same type of subjective questionnaires. However, some crucial technical differences require careful considerations during the evaluation. This workshop at CHI 2021 provides a foundation to exchange expertise and address challenges and opportunities of research methods in MR user studies. By this, our workshop launches a discussion of research methods that should lead to standardizing assessment methods in MR user studies. The outcomes of the workshop will be aggregated into a collective special issue journal article.

For more info, please refer to the workshop proposal.

Call for Participation

To measure MR experiences, researchers apply a range of research methods using objective (e.g., biosignals, logging, behavioural), and subjective (e.g., questionnaires) metrics. However, the assessment methods are heterogeneous and miss consistency among the user studies which impedes transferability of the results.

This one-day virtual CHI2021 workshop will focus on common practices of evaluation methods and their methodological, technical, and design challenges. We invite researchers and practitioners from all subfields of HCI to drive the research agenda of the research practices, technologies, and challenges of MR user studies. This workshop invites submissions of position papers (2-4 pages excluding references according to the (single column) ACM Templates), covering but not limited to the following topics:

  • Measurement methods (behavioural, objective, subjective) for single- or multi-user MR
  • Technical challenges/solutions/artifacts for assessment methods in and outside the lab. E.g., interaction for in-VR measurements, use of biosignals, assessing behavioral measures
  • Experimenter-participant communication (e.g., telepresence, avatarization)

Submissions will be selected by the workshop organizers based on the relevance to the workshop topic and their potential to engender insightful discussions at the workshop. At the workshop, accepted papers will have a 3-4 minutes video presentation. At least one author of the accepted paper must attend the virtual workshop. All participants must register for both the workshop and for at least one day of the conference.

Submission

  • Deadline (Round 1): February 21st, 2021 at 12pm (noon) PT
  • Deadline (Round 2): March 22nd, 2021 at 12pm (noon) PT
  • Format: non anonymized, 2-4 pages excluding references, using ACM Submission Template
  • Submit your paper using CMT

Organizers

  • Dmitry Alexandrovky is a final-year doctoral student at the Digital Media Lab, University of Bremen, Germany. His research interests are immersive interaction, user engagement, and game design research. He works on interface designs for questionnaires in VR and developed an in-VR questionnaire toolkit. His research was awarded with ‘Honorable Mentions’ at CHI PLAY conferences.
  • Susanne Putze is a final-year doctoral student at the Digital Media Lab at the University of Bremen. Her research interests are in HCI, improvement of research workflows, and research communication methods. She works on measuring VR experiences using subjective questionnaires and physiological signals.
  • Valentin Schwind is professor for human-computer interaction at the Frankfurt University of Applied Sciences. His work explores immersive and multimodal user experiences in virtual and augmented reality. He is expert in research of quantifying immersion and presence. Valentin has received multiple awards at CHI and other HCI conferences for his research of avatars and virtual characters.
  • Elisa D. Mekler is an assistant professor at the Aalto University Department of Computer Science. Her research interests include the applications of psychological theories and methods in HCI, as well as the development and validation of UX questionnaires. Elisa’s work has garnered multiple awards at CHI and CHI PLAY.
  • Jan David Smeddinck is an assistant professor at Open Lab and the School of Computing at Newcastle University in the UK. Building on his background in interaction design, serious games, web technologies, human computing, machine learning, and visual effects, his research interests include virtual-, mixed- and augmented-reality with a focus on applications in digital health and education.
  • Denise Kahl is a doctoral student at the German Research Center for Artificial Intelligence (DFKI). In her research she explores the relationship between virtual objects and their physical representations for tangible interaction in optical see-through Augmented Reality. She evaluates AR visualizations by measuring presence using subjective questionnaires.
  • Antonio Krüger is the CEO of the German Research Center for Artificial Intelligence (DFKI) and a professor of computer science at Saarland University heading the Ubiquitous Media Technology Lab (UMTL). He is an internationally renowned expert on human-machine interaction and artificial intelligence. His research focuses on Mobile and Ubiquitous Spatial Assistance Systems, combining the research areas Intelligent User Interfaces, User Modeling, Cognitive Sciences and Ubiquitous Computing.
  • Rainer Malaka is professor for Digital Media at the University of Bremen. He is managing Director of the Center for Computing Technologies (Technologiezentrums Informatik und Informationstechnik, TZI) and Director of the PhD program Empowering Digital Media that is funded by the Klaus Tschira foundation. His research focus is on multimodal interaction in MR, language understanding, entertainment computing, and artificial intelligence. Rainer is councillor of IFIP (International Federation for Information Processing) and chair of IFIP’s technical committee on Entertainment Computing. He has an extensive experience in VR research and evaluation of VR applications from various research projects.

Contact

To get in touch with us, please write to the organizers: evaluating-mr-ws2021@tzi.de