Usability+Testing+X+Summary+of+Observations

gh
 * Summary of Notes from Usability Testing**
 * Checklist & Observation Notes (let's try to match users - checklist and observations, if there are both)**
 * User #1**
 * the first time through the application seemed to be somewhat difficult for this particular user - although the paths seemed to flow from screen to screen, and no dead-ends were encountered, there still seemed to be confusion - part of the confusion revolved around terminology. For example, this particular user was unsure of the meaning of the following terms: 'Record movie', 'Story board', and 'Local device'.
 * the second time the user went through the application was far smoother - she seemed to have developed a better sense of how one screen connected with another and was able to navigate through a number of paths
 * this user had many questions regarding the purpose of various 'buttons' and sought clarification, while also giving suggestions as to possible functions. For example, with the 'help' button, she suggested that it would tell users 'where to go' (-)
 * this user asked about a 'sign-in screen' for cloud story access and/or saving - good point - we will need to incorporate this into our design
 * suggested that she would like to have the option to notify friends in the 'cloud' that she had posted a new movie; in addition, she also wanted to know if there was an easy way to email movies to her friends from her device (as in Story Robe), or post to youtube from her device.
 * (User #1 has no separate observation notes)**

//**User #2**//

Think/Talk Aloud (JJ) //**Chris, this was the first user you typed up on my laptop, but I don't know which user it actually was as you labeled the write ups 1 and 2. The second one was actually user #5**// //Unknown user number.//

I’m going to touch the button. I’m going to go to get story, get story. I’m going to touch edit story, get story to edit, I’m pressing the button, not working. I think I’m going to edit my story. I would like to add a picture from the clip art gallery. I’m going to click on clip art. I’m going to the back button, take a picture. Use a picture from my browser, save and add a picture. Add a picture. I’ve already added a picture and look at all the pictures I have so far. Now I’m going to choose a story name, choose a story icon, just go to home. Ok so now I’ve found a picture and going to edit my story. I’m going to go to edit my story button, add text. Oooh cool. I’m going to drag my storyboard and add sound. I’m going to record sound, that was fun, I’m saving it. Oh I’m going to go back, I’m not going back. Cloud, oh nice. I’m going to go back home. New story, choose a story icon from my pictures, I’m going to use the same pictures. I selected my clip its from my storyboard and saving my story. I’m done I think?

We ask the user to read a story from the cloud:

Story click on cloud. Share cloud stories, my cloud stories. I’m going to read my story from the cloud. I’m done the story. I’m going to look at the shared cloud stories. I’m done I think.


 * User #3: Checklist and Observation notes combined (JJ)**
 * on first run through for editing a story, user found most things quite easy to understand, although often hesitated and clarified before using...easily remembered for second and third run throughs
 * was unsure about the differences between picture selection choices i.e. icons, my pictures, take picture. Needed explanation
 * said that the titles versus actions on each screen were a bit confusing i.e. screen titles at the top, sometimes confused with repeat of title for an actual action button
 * much confusion over Back button action and process. Wondered why she could only return to a main choice screen by returning step by step though use of the back button repeatedly
 * Suggested that we needed 'Go to home' 'Go to edit' and 'Previous'
 * Asked whether each screen saved automatically as you left it, or whether a pop up would check your choice of save, not save, delete etc.
 * If a user accidentally clicked on 'delete' will there be a pop up that asks, "Are you sure?"
 * Asked whether we intended each user to have an account # and/or password. In addition, therefore, would we provide a "remember your password' function.
 * thought a tutorial would be a good addition for first time user.
 * also suggested that a reminder screen to the user about setting up account and password would be helpful
 * check list shows completion of all tasks with relative ease. Dead ends only occured due to our built in limitations

> · The story board could be a larger. > · The user preferred to write a title at the same time while viewing the picture. > · The user wanted the features to be more colourful ( Due to the limitations of the prototype we were not able to do so, moreover the product is being created mainly for children with Aspergers-our target audience, hence we ensured it is not too flashy….
 * Observation Notes**
 * Not sure of user #, but this is the second round with AC.UC when I partnered with Vanita (Vanita, will you please check your sheet and add user number and then add your notes too? Thanks)**
 * this user quickly navigated through the screens and seemed very content to click and explore
 * suggested that 'search' and 'browse' functions were the same - perhaps functioning buttons would be better able to demonstrate the unique nature of these functions
 * said "It's good to share stories"
 * as she was looking at the screen she said, "I will do as a sequence" - which was 'bang on' - on the 'create a story' screen the 'buttons' are laid out so that users will first name their story, second choose an icon, then third, create the story.
 * asked about using an existing picture from her computer/device
 * suggested that when she is entering text for a picture she should see the picture on the same screen, same thing for add sound
 * said "this function is good" when looking at the 'record movie' screen
 * asked, "In this function, can you share with someone else?"
 * wondered if she could save from the 'edit' page
 * wondered about seeing the story on the edit story screen ... how would this work?
 * suggested that the "interface is too simple to arouse the interest of children"
 * thought we should have some kind of acknowledgment of completion - should be more encouraging to users to encourage children to tell their stories - suggested a 'congratulations' at the end (motivational)
 * we explained that it is a mock-up that did not allow the addition of images
 * also we clarified that the application is more intended for adults to use in assisting children develop social stories than for children to use themselves.
 * user said that she had misunderstood //who// was going to be using the application - thought it was the children
 * There should be a difference between the main tab(title) and the tabs that provide options.
 * · One of the user also mentioned to provide an option for emailing a particular story/video/pictures instead of using the cloud.
 * · The user also provided feedback that once the video/story is completed it should tell the user about the success of the task. For eg. “Congratulations
 * Not sure of user # Checklist Notes (Vanita)**
 * Not sure of user # Checklist Notes (Vanita)**


 * User #5: Checklist (JJ)**
 * this user seemed less comfortable with 'think aloud' although was able to navigate through screens 'okay'
 * on second run through found navigation easy and recognized previously visited screens
 * was more comfortable with directed activities (See think/talk notes below)
 * on each screen, stopped and examined, and then nodded understanding
 * made several suggestions which are documented in the notes below.(See think/talk notes below)


 * User #5: Think/Talk Notes (JJ)**

I try to move the mouse to edit stories try to click it but I didn’t see little hands. I try other parts, no hyperlinks //(Jeanette says click on edit story).// Local device, can I press this? I press edit story, ahhh. I’ll choose add text first. Click and drag to change order. Should I type something. Press back. Now add some pictures. What is clip art? //(Jeanette explains)// Take a picture, my pictures. //(What do you think is my pictures?)// My pictures on my hard drive. Back, Ok add sound. (//If you were going to add sound there, what would you do?)// I prefer to choose sounds in my harddrive. //(add sound, would you? You would prefer us to have a storage of sounds somewhere?)// Yes. Back. Add movie, ok. I’m going to record movie. Record movie using the camera, not a high resolution I guess. I want to add some videos from youtube, not just my own. Or maybe the length of the movie had some influence on the upload speed. Storyboard. I’ve done all this. //(What do you think the storyboard is about?)// A preview of the story I’ve created. What exactly is split? //(Jeanette explains)//. I think it contains all the functions I need. What about the voice over? //(Jeanette says it’s the add sound.// //There’s where you could record your sound. (Jeanette instructs him to create a story. How would you do this?)// Choose a story name, enter text. I prefer to add text first. //(That would be your title)// Ahhh. Choose a story icon. Clip art or my pictures. It depends on the story content? I can pick pictures from the internet? (//Yes)// Create your story, edit stories the same page? (//That makes sense?)// Yep. //(Go to the welcome page and get story)// Pick cloud, my cloud story. //(Jeanette explains that’s where your story would be. Go back and tell me what the difference between browse and search)// For search type key words but browse I can see the existing names I don’t type anything I just choose one. Shared cloud stories online, stories I want to share with others. I can share. //(Those are pretend stories you’ve written. What are your options here?)// I’m not sure, maybe record a video? //(No not here, you can have a look at your story. Is there anything you would like here?)// A full screen option. //(Try get story in the cloud, try searching. We lost our tagged stories)//

Questionnaire Notes:

1. What did you like about Virticue?

- I can create my story, personalize it - I have many choices when I create stories. - Easy to navigate - Has a number of options either create your own story from the beginning or save and then come back later and edit your story - Sharing options is great - Easy navigation - Clean and clear design - Clear navigation - Precise names on all the buttons - Create my own story, that’s the point! I like creation! - Multiple elements to add in the story - It is a resourceful library, containing valuable stories that guide children to be socialized

2. What did you not like about Virticue?

- Sometimes there are many layers under each button, sometimes I am confused by it. - Back button, it is better if there is one to main “home” page rather going back to each steps that you already did to get to your main menu - Beyond your control - Can’t navigate and add pics - Looking forward to the ‘real’ design - It’s not colourful enough (since it’s just a prototype maybe) - Nothing else really, I think it’s great - Not easy access to reach the function you want. Lots of clicks

3. What did you find confusing, complex, or difficult to use in Virticue?

- The same as 2 - Everything seems to be easy to navigate and understand - I thought it was very easy to navigate through - The links were a good size and the words were clear and to the point - Files with large volume are slow to upload I guess - Overall it’s not at all confusing or complex - The interface of “add text” and “add sound” use the same. Going straight is better

4. How would you suggest improving Virticue?

- I can share it and submit it online - Putting video tutorial help user to understand to use the website - Does this come in other languages? - Other font types? - Define your target audience because I thought it was young kids when I just saw the application. Everyone has his/her stories. Maybe more options about resources for text, picture, music, video - The interface is too simple, and it should be colourful

5. Are there any tasks that we have not considered that you think we should include?

- Almost covered my idea - Back button and video tutorial - No I think you’ve covered all bases! I love that you can add sound as well - It was simple to navigate through! I was itching to add a story and pictures! - The format of the story created - Users should have different applications to run the story they created. The format should be compressed enough especially for long stories with high resolution videos - Add a link to the “new story” “choose a story type” - What is the story for, you can create a category to classify those different types of stories

6. Was it easy, or not, to find your way around? Why or why not?

- easy, I just click on the buttons and then navigate it clearly - easy to navigate - I found it easy to use because it has clear navigations and it runs in accordance with most people’s thinking process when they want to create a story - Just ok .............................

//Hi, // //Here is the introduction (if we would like to use for our paper)to our user testing along with the observation, // // Thanks // // Vg.... // //User testing:- // As Preece, Rogers &Sharp(2002) say, “Observing users interacting with software, even casual observing, can tell you an enormous amount about what they do, the context in which they do, how well technology supports them, and what other support is needed.”(p. 359). Hence we decided to observe our users, interacting with our prototype prior to initiating the development stage. We followed the procedure using a framework specified by Preece, Rogers &Sharp(p. 369). We decided to record the events by taking active notes, preparing a checklist and a short questionnaire to include subjective responses. Prior to conducting our ‘observation’ we ensured that the equipment was working, and we had suitable laptop, notebook, and pens. Since we have spent a lot of time working together in a class, trust and acceptance was pre-established yet we ensured our participants were comfortable.We also ensured we recorded our observations within 24 hours as pointed by Preece, Rogers, & Sharp that human memory can be unreliable. “Be prepared to go through your notes and other records as soon as possible after each evaluation session to flesh out detail and check ambiguities with other observers or with the people being observed. This should be done routinely because human memory is unreliable. A basic rule is to do it within 24 hours, sooner is better!” (p. 369). Our observation has been that we were able to flesh out details while taking ‘active notes’ as the users were interacting and doing the task and not by using the ‘checklist’. The checklist we prepared didn’t really work well. We observed that our checklist lacked the questions which corresponded to the observed responses. Moreover it was difficult looking at a question, timing a task and observing the response. However since one of our group member was also taking active notes it was easy to compare and make good interpretation. “Consider checking your notes with an informant or members of the group to ensure that you are understanding what is happening and that you are making good interpretations.” (Preece, Rogers & Sharp, 2002, p. 370). Our collective observation has been that all of our users were able to navigate and complete the task with ease and in a reasonable amount of time. “A criterion for assessing whether a system is easy to learn is to apply the “ten-minute” rule”(Nelson, 1980). Since our users were able to navigate and complete the task within 10 minutes, it suggested our system had worked and did not fail. However some of the implications for further improvement that we found were: · There should be a distinction between the main tab(title) and the tabs that provide options. · The story board could be a larger. · The user preferred to write a title at the same time while viewing the picture. · The user wanted the features to be more colourful ( Due to the limitations of the prototype we were not able to do so, moreover the product is being created mainly for children with Aspergers-our target audience, hence we ensured it is not too flashy…. <span style="color: #e46c0a; font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 200%; margin: 0cm 0cm 0pt 36pt;">This is however our personal opinion, hence we highlighted and separated it from what happens. “ As you review your notes, try to highlight and separate personal opinion from what happens. Also clearly note anything you want to go back to.”(Preece, Rogers, & Sharp, 2002. 369).  · <span style="color: #e46c0a; font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 200%;">One of the user also mentioned to provide an option for emailing a particular story/video/pictures instead of using the cloud.    · <span style="color: #e46c0a; font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: 200%;">The user also provided feedback that once the video/story is completed it should tell the user about the success of the task. For eg. “Congratulations!”