Impact of the Researcher’s Presence on Remote Usability Testing
Investigate whether researcher’s video settings will affect the result and participant’s performance during remote usability test.
My Role
Researcher, Experiment Designer
Team
Eun-Jeong Kang, Xianyi Li
Timeline
Sep 2022 - Dec 2022, 12 weeks
Background
To answer this question, we controlled the level of visibility of researchers over the camera to answer the research question. Also, we created three hypotheses that need to be validated:
Researcher’s visibility during remote usablity test could affect interface usability result
Researcher’s visibility affects participants’ performance in completing a task
Researcher’s visibility affects participants’ engagement with the researcher
How does researchers’ social presence affect participants’ behaviors in usability testing?
Research Design
Procedure
Participants were recruited through the institutional science research pool (SONA).All are prescreened and students at Cornell University, who are over the age of 18.
32 participants were successfully recruited (n = 32, 11 Males and 21 Females). After their sign-up, we randomly assigned them in to the three conditions so that each condition has the same amount of participants.
Measurements and Findings
During the process of the remote usability test, we recorded several metrics to test our hypotheses
[H1] System Usability Scale (SUS)
We used System Usability Scale (SUS) to measure an interface usability result (h1). We used SUS in our post-survey, SUS composed of a 10-item questionnaire. If the score calculated by the metric is over 70, the interface is assumed that it has ‘good’ usability.
[H2] Task completion time
We measured task completion time when the participants performed each task to understand performance of the participants.
[H3] Questions asked by participants
We collected the number of questions asked by participants when performing each task to examine the interaction of the participants with the researcher.
Discussion
Users found the issue no matter how researchers show themselves
Regarding H1, we found that the usability test result is not highly affected by the researchers’ visibility. Even if the researcher showed themselves as whatever, participants equally evaluated the system and found the usability problems.
Camera settings affect the engagement between users and researchers
The camera setting of the researchers or other elements influencing social presence could be considered (H2, H3). The degree of social presence affects learners’ interaction levels in the context of online learning. Question and dynamic interactions are important in conducting usability tests depending on researchers’ purposes. If the usability test requires think-aloud studies and interviewstudies where participants should be more promoted to share their thoughts, the researcher’s setting with ‘camera-on’ will be more promising to open their minds than a setting without a camera.
Avatar has the potential to be widely used in usability test
Using avatar allows researchers to keep annoymously when desired and might reduce the pressure of participants of being observed. Also, junior researchers could feel less pressure to begin studies if they use an avatar setting. Furthermore, researchers could choose to use a context related figure when they want participants to feel immersed.
Limitation of avatars’ fidelity
Avatars-mediated communication may have limitations in expressing emotions and communication. Avatars could not deliver social cues properly because it express abstractly, and people might misinterpret the expression of avatars. The avatars we used only show simple head movements. Participants may not get the researchers’ intentions during experiments.
Future works
Next Steps
To continue future research in this area, researchers should make some changes to the study design. First, the usability testing task should be more difficult. A reasonable task should involve more steps and should not be easily completed by the participants. At the same time, if available, researchers should personally design web prototypes for usability testing. This effectively eliminates the possibility that the participant has been familiar with the web in the past and eliminates disturbing variables. In the future, we hope to recruit more and more diverse participants and conduct more in-depth experiments with more complete experimental Settings, which can more effectively explore the significance of the results and understand some overlooked factors in this study.
Deliverables
Poster
Presentation of the experiment design and result
ACM Format Paper
Main Takeaways
After conducting this experiment, I found that recruiting and conducting the experiment is an extremely important part of user research. To devlier meaningful insights, researchers need prepare efforts to get enough participants and ensure statistical significance.
Also, I realized that I, as a UX Researcher, should care about how we present ourselves to participants and think about how we can make the remote usability test smooth and stress-free.