PROTOTYPE USABILITY TEST PLAN
Testion Rationale
We have created a wireframe for a virtual machine. This consists of visuals that will explain the structure of the virtual machine as well as how it will work and the scenarios it will be used in. The features we will be testing include the following;
Optimization
How the VM will function on their hardware
Intuition
How functionally similar is the virtual machine to the user's computer?
Does the user have a hard time operating the virtual machine?
Speed
Do users feel the VM operates at an acceptable rate for their needs?
We want to be able to learn how the user feels/responds to our features. Do they agree with the feature? Is it useful? We will be testing and engaging with InfoSci college students who are juniors or at least in the process of taking their technical major courses. For this process, we are hoping to gain at least 5 participants, as the coronavirus makes the situation difficult to contact and engage with others.
Orientation Script
Introduction
Hello! My name is [NAME], and I’m a college student from the University of Maryland working on a project to solve incompatibility issues between Mac and WIndows operating systems for college students. For this session, my team will also be joining us, [ADDRESSING ALL TEAM MEMBERS NAMES]. They will be here to help with conducting the illustration of our prototype concept as well as taking notes and observation.
Thanks for participating and agreeing to help us in solving incompatibility issues among Mac and Windows operating systems. Before we start, I’d like to prepare you on what to expect and how we will be conducting this process.
Disclosures and Informed Consent
For this process, we will be showcasing our prototype concept. You will not have to physically do/conduct anything. We are interested in your honest opinions and feedback, and are open to any of your questions. Your answers will be confidential and your name will not be revealed unless stated otherwise.
We will be recording audio and video as well as taking notes during this discussion. These recordings will only be used for the purpose of this study and will not be released publicly.
This session may last up to 15 minutes.
Guidelines
To help make this session run as smoothly as possible, please:
Speak loudly and clearly
Turn off or set your cell phone to silent during the session.
Speak freely on your opinions towards the prototype
Before we begin, do you have any questions or concerns?
Warm Up
We’re glad to have you here today! What we would like you to do is familiarize yourself with the following wireframes and concept visuals. Don’t worry about analyzing them for right now, just a quick glance at them.
Transition to Test
You will be tasked with examining and analyzing wireframe visuals that will layout the processes and scenarios relating to a virtual machine application. I will be prompting you with a situation that you will be in, in which our wireframes will guide you through a process relating to said situation. We will be here to help guide you and understand the wireframes and how the virtual machine will essentially be “behaving” if it were in a real scenario. Feel free to ask for any kind of clarification during this process if you are unsure.
I would like you to openly share your opinions and thoughts during these tasks. This will allow us to learn and understand if you would want to use this application and if it would provide an actual benefit.
Scenarios and Tasks
Installing/Accessing the Virtual Machine:
You’ve just begun your new semester of classes and are currently in syllabus week. You realize that one of your courses seems to have more programs that work better on Windows so you need to install a virtual machine so you can load a digital virtual Windows OS. That way you can be prepared to use your virtual machine during that course and set up the needed programs for the class.
Your task is to examine and walkthrough a small series of wireframe visual concepts of the installing and setup procedure for a virtual machine. Then, continue to do the same for then using the virtual machine and running programs.
Running/Using the Virtual Machine During a Situation:
You now have your virtual machine setup in which you can boot a virtual Windows OS. At this point, you have also downloaded your required programs for both Mac and Windows OS. You’re in the 3rd week of class and your instructor is showcasing a program in which they expect everyone to follow along on their laptops. You notice that your Mac version of the program is not functioning or getting the same results as the instructors and will need to load up your virtual machine to use the Windows version.
Your task will be to examine and walkthrough another small series of wireframe visual concepts that will give an idea of how the concept will operate in this type of situation.
Qualitative Questions
Are you generally satisfied with this product?
What could we have done to demonstrate this process in a more efficient way?
What questions were you still left with after the demonstration with the system?
Did you see any potential exploits or bugs that could exist?
What features would greatly improve the experience, and what features do we have now that detract from the experience.
Can the user understand the process illustrated in the wireframe? Easily? With difficulty? Not able to?
Can the user install/access the virtual machine? Easily? With difficulty? Not able to?
Can the user find the program? Easily? With difficulty? Not able to?
Can the user smoothly work on the program? Easily? With Difficulty? Not able to?
Closing Discussion
Closing questions:
Would access to a virtual machine help with the user’s work? Yes or no?
Why or why not?
What support would the user need to use such a setup?
What thoughts or concerns does the user have about having a virtual machine resource for class?
Final Thoughts:
Do you have any final thoughts or recommendations about the website or on any of the tasks or questions in this discussion?
Thank you again for your time and participation!
Scoring Instrument
We will conduct a spreadsheet that will allow us to capture the results/responses during the tasks. This can include a scoring rating that the user responds with towards the wireframes processes or positive and negative responses.
ARTIFACTS AND OUTCOMES OF USABILITY TESTING
Process
For our prototyping process, we created wireframes/visuals that would represent the concept of a virtual machine. These visuals would showcase how the prototype should work, what to expect and to give an idea of it working in a hypothetical situation. The prototype is further represented in the form of two tasks for our testers to examine. We walked them through the visuals and continued to incorporate our hypothetical situation as well as explain the processes. The objective of this was to gain the opinions and questions our testers would have as they are examining the concept to see if it would appeal to them and if they would find it useful for their situations.
We were able to recruit some people we had from our previous research and interviews. However, due to scheduling issues, we were only able to acquire 5 people to test our prototype. We were able to have a meeting on Zoom with each of our recruited testers in which we interacted with them on a Miro board.
Each team member switched roles for each test conducted. For our first round, we had Konstantinos and Edward explain and walk the tester through the visual concepts of the prototype. We labeled this role as “Guiders”, in which they elaborate on what is happening in the visuals and what to expect and understand in the situation. While this is happening, we have Brigid, Ziyang and Dylan in the background taking notes of the tester’s inputs, as well as each one of them taking turns asking questions for further elaboration from the tester. They are the “Note takers/Questioners” for this round.
Analyses and Findings
Standard deviations of visuals indicate how measurements are spread out from the mean value of visuals. Outliers were formatted in the different colors in the above chart. We focused on analyzing the visuals that scored 3 points because these were indicators of what needed to be improved immediately and we could modify our prototype according to these outliers. Tester #5 pointed out that the second and forth visuals in Task #1 were too abstract for understanding, Tester #4 pointed out that we should run some Windows programs in the last visual of Task #2, and Tester #3 indicated that our hardware optimization interface should be more friendly. These were all critical suggestions because they did affect our testers' user experience. However, for our qualitative questions, testers gave suggestions similar to above feedback although they scored the visuals highly. Therefore, we must further consider their suggestions to modify our prototype. In general, we have confidence in the concept of our wireframes/visuals because the overall tester appreciation (score) is high enough.
