Back to IEEE VR 2008 Panels

User Studies in VR:
What Can We Learn From Them and What Are They Good For?

Silver Legacy Resort Casino
Reno, Nevada

Abstract

Understanding how users interact and perform in virtual and augmented reality has been a research focus in human-computer interaction for over two decades. This research takes the form of usability studies and evaluations to determine, for example, what 3D user interface techniques provide users with the best performance in terms of efficiency and ease of use, how users perceive and understand virtual and augmented worlds, what display hardware is most appropriate for specific applications, and what benefits users receive when working in virtual and augmented reality. In all of these examples, having human subjects in the loop presents many interesting challenges in our ability to gain knowledge from study results and apply that knowledge to further virtual and augmented reality research and development.

In this panel, we will explore a variety of issues and questions that arise when designing, running, interpreting, and applying usability studies in the context of virtual and augmented reality. They include:

• What do user studies in VR/AR really tell us?
• How do user studies in VR/AR differ from those in other fields?
• The isolated variable versus application level study, pros and cons
• Repeatability and user studies in VR/AR
• What statistical analysis techniques are most appropriate for VR/AR user studies?
• Is every study flawed?
• Can we trust studies done with older VR/AR hardware?
• How can we make the user study process in VR/AR better?

Bios

Joseph J. LaViola Jr. is an assistant professor in the School of Electrical Engineering and Computer Science at the University of Central Florida as well as an adjunct assistant research professor in the Computer Science Department at Brown University. His primaryresearch interests include pen-based interactive computing, 3D interaction techniques, predictive motion tracking, multimodal interaction in virtual environments, and user interface evaluation. His work has appeared in journals such as IEEE PAMI, Presence, andIEEE Computer Graphics & Applications, and he has presented research at conferences including ACM SIGGRAPH, the ACM Symposium on Interactive 3D Graphics, IEEE Virtual Reality, and Eurographics Virtual Environments. He has also co-authored "3D User Interfaces:Theory and Practice," the first comprehensive book on 3D user interfaces. Joseph received a Sc.M. in Computer Science in 2000, a Sc.M. in Applied Mathematics in 2001, and a Ph.D. in Computer Science in 2005 from Brown University.

Doug A. Bowman is an Associate Professor of Computer Science at Virginia Tech, where he directs the 3D Interaction Research Group and is a member of the Center for Human-Computer Interaction. His research interests include 3D user interfaces, interaction techniques for virtual environments, the benefits of immersion in VR, and large high-resolution displays. He is a co-author of the book 3D User Interfaces: Theory and Practice, and was awarded a National Science Foundation CAREER grant for his work on domain-specific 3D user interfaces. He and his students have designed, run, and analyzed scores of VR user studies over the past decade. Bowman received his MS and PhD in Computer Science from the Georgia Institute of Technology.

Stephen R. Ellis headed the Advanced Displays and Spatial perception Laboratory at the NASA Ames Research Center between September 1989 and March, 2006 and is current a member of this group. He received a Ph.D. (1974) from McGill University in Psychology after receiving a A.B. in Behavioral Science from U.C. Berkeley. He has had postdoctoral fellowships in Physiological Optics at Brown University and at U.C. Berkeley. He has published on the topic of presentation and user interaction with spatial information in 150 journal publications and formal reports and has been in the forefront of the introduction of perspective and 3D displays into aerospace user interfaces. In particular he has worked recently on kinesthetic techniques to improve cursor and manipulator control under difficult display control coordinate mappings. He has served on the editorial boards of Presence and Human Factors, and has edited a book, Pictorial communication in virtual and real environments, 2nd Ed. concerning the geometric and dynamics aspects of human interface to systems using spatial data.(Taylor and Francis, London 1993). He was awarded a University Medal from Kyushu Sangyo University in Japan in 1992. He recently was awarded the best paper prize at IEEE Virtual Reality 2006 for a publication titled "Demand characteristics of a questionnaire used to assess motion sickness in a virtual environment: or Does the simulator sickness questionnaire make you sick?"

Victoria Interrante is an Associate Professor in the Department of Computer Science, a Faculty Member in the Human Factors and Ergonomics graduate program, and an Associate Member of the Center for Cognitive Sciences at the University of Minnesota. She received her PhD in 1996 from the University of North Carolina at Chapel Hill. Her research in visualization and computer graphics broadly focuses on the application of insights from visual perception to the development of more effective techniques for communicating information. She enjoys interdisciplinary collaborations with colleagues from a variety of departments on campus including Aerospace Engineering, Architecture, and the Institute for Child Development. She is also a PECASE recipient (1999) and an Associate Editor of the ACM Transactions on Applied Perception.

Benjamin C. Lok is an Assistant Professor in the Computer and Information Sciences and Engineering Department at the University of Florida. He is an Adjunct Assistant Professor in the Department of Surgical Oncology at the Medical College of Georgia. His research areas include computer graphics, virtual environments, and human-computer interaction. Professor Lok graduated from the University of Tulsa with a B.S. (1997) degree in Computer Science, and graduated from the University of North Carolina at Chapel Hill with a MS (1999) and Ph.D. (2002). Professor Lok received a NSF Career Award in 2007 and the UF ACM Teacher of the Year Award in 2005-2006. He is a member of the IEEE Computer Society.

Dr. J. Edward Swan II is an Associate Professor of Computer Science and Engineering, and an Adjunct Associate Professor of Psychology, at Mississippi State University. He holds a B.S. (1988) in computer science from Auburn University and M.S. (1992) and Ph.D. (1997) degrees in computer science from Ohio State University, where he studied computer graphics and human-computer interaction. Before joining Mississippi State University in 2004, Dr. Swan spent seven years as a scientist at the Naval Research Laboratory in Washington, D.C. Dr. Swan's research has been broad-based, centering on the topics of virtual and augmented reality, computer graphics, empirical methods, visualization, human-computer interaction, and human factors. Currently, Dr. Swan is studying human-factors aspects of augmented and virtual reality, with a particular focus on the perception of depth and layout, as well as empirical techniques for evaluating and validating visualizations. His research has been funded by the National Science Foundation, the National Aeronautics and Space Administration, the Naval Research Laboratory, and the Office of Naval Research. Dr. Swan is a member of ACM, IEEE, and the IEEE Computer Society.