Experiment

Introduction and Hypothesis

The experiment tested three different help system designs for an application called PhotoFinder (Kiosk version) that was designed and developed by the HCIL (Human Computer Interaction Laboratory) staff at the University of Maryland, College Park. PhotoFinder was designed with the goal to develop an understanding of user needs, appropriate tasks, and innovative designs for the users of digital photos.  Currently PhotoFinder provides dynamic queries, query previews, Boolean user interface, multiple thumbnail display, and drag and drop capabilities to the user.

Null Hypothesis: The mean of the three treatments will not differ and therefore we will conclude that there is no statistically reliable difference between text only, animated images only, and integrated interactive help system's.

Hypothesis: The average performance in the integrated interactive group will be statistically greater than the text only and animated images only groups.

Independent Variables: The PhotoFinder application with a database of UMCP-related pictures and labels. The experiment consisted of three treatments:

• Text only help system
• Animated Images only help system
• Integrated Interactive help system

Dependent Variables: Task performance time, and subjective ratings.

Currently, there are many different kinds of help systems, with two of the most common being online and offline printed materials.   Our experiment compared three different kinds of online help systems: Text only, Animated images only, and an Integrated interactive help system. The text only help system would appear on the computer screen before the application fully loaded and thus was the first thing the user saw. The Animated images help facility allowed the user to visually see how to complete a task.  And the integrated interactive help was a series of numbered frames that would pop up inside the applications user interface.  The different frames were designed as a series of steps for the user to follow in order to complete a given task.  No matter what help system a subject tested, once the application fully loaded the help system they were testing was accessible by clicking on the help button located in the tool bar at the bottom of the application screen. 

Snapshots of Help Systems Used (click on image to enlarge)

Text Only	Animated Images Only	Integrated Interactive

Each subject was given a two-minute training period at which time the user interface and help system was explained.  Following the training period the subject was asked to complete the following five tasks:  search for a picture of Dan Mote the current President of the University of Maryland College Park (UMCP), annotate a picture of Testudo from the Mascot collection with the name Testudo, delete the annotation for the Testudo picture in the Testudo collection, e-mail the picture of Testudo you annotated to yourself, and create a slide show of the photo’s in the Buildings collection.   When the subject completed all five tasks they were asked to complete a short subjective questionnaire.

Six pilot subjects were used in the initial experiment design.  Two different pilot subjects tested each treatment. The subjects were given two minutes of training during which PhotoFinder's user interface was explained.  However, neither the location nor the how to’s of using the help facility were explained.  Before the subjects began the first task they were asked to time how long it took them to complete each task and write the time down on the task sheet themselves.  Each subject was asked to complete four tasks and fill out a short two-page questionnaire.  We found the need to make a few changes to the help menus and experiment design were found to be needed:

• During experiment testing, all subjects will be shown where the help button is located as well as how to use it.
• Inconsistencies in the terminology of the different help systems were fixed before subject testing began.
• The task to delete the Testudo annotation was added to the experiment.
• During experiment testing the tester will time how long it takes the subject to complete each task.
• Due to subject complaints, changes were made to the questionnaire.

The initial requirements were that subjects should be either UMCP psychology or photography majors with no prior experience with the PhotoFinder application.  The only other subject requirement was that the subjects had to be computer literate; at least able to maneuver within a Windows program.  The focus was on psychology majors because they could relate to the experiment process and photography majors because they could benefit from using the PhotoFinder application. Having no experience with or knowledge of PhotoFinder made the help facilities more likely to be used by the subjects.

We were able to recruit 30 subjects from upper level psychology and photography classes to participate in the experiment, but only 14 showed up for experiment testing.  Due to time constraints, we ended up recruiting 16 UMCP Computer Science students with no prior experience or knowledge of PhotoFinder. As a result, we recruited a total of 30 subjects to actually participate in the experiment.  There were 10 subjects for each treatment and each subject tested one treatment.

Subject Demographics:

• Computer Experience :
  • 4 beginner computer users
  • 17 intermediate computer users
  • 7 advance computer users
  • 2 individuals refused to indicate their computer experience.
• Gender :
  • 14 Females
  • 16 Males
• Age Groups:
  • We tested 18 individuals between the ages of 18 - 25.
  • 8 individuals between the ages of 26 - 33.
  • 2 individuals between the ages of 42- 49.
  • 2 individuals refused to state their age group.

Materials

The PhotoFinder application itself and the integrated interactive help system were written in Visual Basic.  The text only and Animated images only help systems were HTML-based applications.  A SQL server-based database was used to house the collections of photographs used in the experiment.  The photographs used were of the UMCP campus, mascot, and University President Dan Mote.  All experimental testing was conducted in the HCIL laboratory on Windows2000 computers using the same types of mouse, keyboards, and monitor. 

For recruiting purposes, a sign-up sheet was designed with fields for subjects’ name, phone number, e-mail address, and available day and times for testing.  For testing purposes, a consent form, task sheet, data spreadsheet, and a questionnaire were designed (see Appendices also). 

Before the training period began, each subject was asked to read and sign a consent form, officially agreeing to be a participant in the study.  After the subject had been trained they were handed a two-page task sheet consisting of five tasks  Upon completion of all five tasks each subject was asked to fill out a two-page questionnaire. 

The procedures of the experiment were simple.  Once the subject arrived at the HCIL laboratory for testing, they were asked to read and sign a consent form.  The tester would start PhotoFinder's server then client.   When the subject was seated in front of the computer the tester would give a two-minute training session, making sure to explain the interface and help system.  Finally the subject was instructed to complete each of the five tasks one at a time.  The subject was asked to inform the tester when they began and finished each task so the tester could note the amount of time it took the subject to complete the task.  When the subject completed the last task they were asked to fill out the questionnaire.

For the most part experimental testing went well, however, a few problems arose.  The first problem was that the majority of the subjects tested did not look at the help menu for the first task, i.e., locating the picture of Dan Mote.  And the subjects who were provided with a help menu before the application fully loaded never bothered to look at it, they just closed the help menu and began the first task.  A second problem was associated with the interpretation of the second task. The second task was to annotate the picture of Testudo in the Mascot collection.  A lot of subjects neglected to read the next sentence on the task sheet that told them what annotating meant.  As a result the subjects had the misconception that annotation meant creating a caption.   One subject was so frustrated over this fact that he insisted he had completed the task and eventually got up from the computer and walked away.  Overall, the problems that came up were solved by PhotoFinder's help facility.