Discussion

Task Performance

ANOVA shows no statistical significance among the three groups’ mean Total Task Time (TTT).  The t-test was used to analyze three pairs of mean TTT, i.e., Text only and Animated images only, Animated images only and Integrated interactive, and Text only and Integrated interactive.  No statistical significance was found from this method of measurement either.  Therefore, the null hypothesis could not be rejected.

The mean of each task of the three treatments was analyzed using ANOVA. The results show no statistically significant difference on Tasks 1, 2, 3 and 4. Only ANOVA on the mean of Task 5 shows a statistically significant difference. This supports the conclusion that the hypothesis can be accepted; Integrated Interactive online help has the best Task 5 performance time.

In discussions among the team members before the experiment started, it was observed that most of the time, as users, we ignored using any form of help until it was necessary.  We found some online help not helpful, e.g., the University of Maryland’s UNIX system online help.  After PhotoFinder’s demonstration from Mr. Kang, we all agreed that PhotoFinder is a reasonably simple program, well written and self-explanatory. 

All subjects were informed that the tester could not answer any question while the subjects performed the tasks.   During the experiment, only one subject read the whole online help document (Text only version) before he began to perform the tasks.  Many of the subjects guessed their way around, avoiding using the online help.   

Task 1 was designed to be the easiest task. The task is to search for a given photograph within the database collections.  Only a few of the subjects searched systematically, while the rest just randomly searched through the collections.  The least time any subject spent on this task was five seconds.  The statistics show that the subjects assigned to Text only online help spent the least mean time on this task compared to the other two groups. (See Figure1.) We cannot come up with any psychological reason why the subjects searched for the photograph the way they did. It could possibly be that some subjects glanced through the collections and gambled with their choices, since there was no harm done if they guessed wrong.

Task 2 was the most time consuming on average. The Text only group spent more time, 44.93 second on average, on this task than the two other groups. Many subjects did not use help until after they were frustrated by not being able to complete this task. We meant the design of the fifth task to be the hardest, but the second task turned out to be the hardest, based on the time consumed. It is agreed that it is a nice feature of PhotoFinder, but the terminology, "Annotate the picture," used in PhotoFinder and the tasks sheet is confusing even to native English speaking people. This was confusing to us also when we were revising the Text only online help. According to The Compact Oxford English Dictionary, Second Edition, 1991, Clarendon Press, Oxford,

annotate (verb) means, "to add notes to, furnish with notes (a literary work or author)", or "to add or to make notes.

Annotated (adjective) means "furnished with notes (by an editor)."

Label (noun) means, "a narrow band or strip of linen, cloth, etc.," or "a small strip of paper or parchment attached to a document by way of supplement to the matter contained therein; hence, a supplementary note, comment, or clause, a codicil", or "a narrow strip of material attached to a document to carry the seal", or "a slip of paper, cardboard, metal, etc., attached or intended to be attached to an object and bearing its name, description, or destination," etc.

Label (verb) means, "to affix a label to, mark with a label."

We suggest the word "label" would be more appropriate for this task metaphor.

The most interesting observation we found was during the performance of Task 3, "Delete the annotation from the Testudo picture in the Mascot collection." Many subjects did not use the delete button at the bottom of PhotoFinder's screen. During their trial and error effort, they still did not use the available online help. Many used the delete key on the keyboard instead.

Tasks 4 and 5 were performed smoothly. Statistics show Task 5 consumed the least time on average. (See Figure 1, Appendices.) As stated above, ANOVA on the three groups' Task 5's means shows statistically significant differences. Hence, we can accept the hypothesis; Integrated Interactive online help has the best performance time on Task 5.

“When we collect data in an experiment, there are two general factors that influence the value of the observations, treatment effects and random effects,”[8]. Nonetheless, our subjects’ performance could have been influenced by many factors.  For example, the experiment conductors were present next to the subjects for timing the subjects’ task performance and the anxiety of trying to be the best (shortest time consuming in this case).

Also, due to the competitive nature of some people, some subjects probably avoided spending time on help in order to be the fastest among all subjects. One particular subject asked to see at the end whether his time was the shortest. Other anxiety could come from the subjects' embarrassment with using a computer (computer illiterate). One subject felt that she was "just plain not smart enough" to do the experiment. Peer pressure could also be one of the influencing factors. Two subjects came together and sat down to the experiment at about the same time. One finished first, while the other, who was still performing tasks, showed signs of giving up. Stereotype could be another influencing factor. The testers and the subjects were of both genders. On many occasions, the tester and the subject were of opposite genders. For example, we observed that when the tester was female and the subject was male, the subject seemed to want to do "short cuts" or find the tricky way to perform the task. One subject asked whether if he did something, would it crash the program. The female tester assured him that he could do what ever he thought appropriate to accomplish the task's assignment and also reminded him that the program had an online help feature that could help him do the tasks. This subject still continued to try his trick and it worked. This could possibly be that males assume that they are more computer literate then females and want to show off their knowledge and abilities with technology. Another example, when driving, most males do not stop to ask for directions even when they already lost. Could this be similar to when males use computers, i.e., that they don't want to use help? Is it a guy-thing?

Subjective Comments

ANOVA also shows no statistical significance among the subjective scoring groups.  The scores were designed to be from 1 to 9 for question 5 to 13.  (See Subjective Questionnaire.)  A subject from the Text only group gave the lowest score of 1 to question 6, “Location of help.”  The Text only online help was presented to the subjects at the beginning of the PhotoFinder program, which may explain this low score.  The sum of average scores reveals the highest rating by the subjects of the Animated images only group.  This may possibly be due to that treatment showing how to do tasks through animation instead of requiring reading and following instructions as under the two other help systems.  If the subjects were exposed to all three treatments, they could have compared these three versions of online help systems (Within-group experiment) and chosen the one they most preferred.  Since this experiment was not designed to be "Within-group” the scores were rated by the subjects from what they saw with no other helps versions for comparison.  The subjects’ comments are worth noting.  They are listed below:

What were the problems during the use of PhotoFinder?

Text only online help

• I had a bit of trouble annotating.
• Didn’t know how to do task#2.       
• Lots.  The slide show was the worst. The instruction was very confusing on help.
• Only problem was that the help should be the most concise and easy to follow wording, so even a beginning computer user could comprehend it.
• Deleting label.
• The program shut down from a run time error.

Animated images only online help

• Couldn’t delete label already there.
• Help didn’t help solve problems.
• Help did not help solve problems
• In task 3, no mention was made of not being able to remove items you did not annotate.  Also the help screen was very small and difficult to see.  I would have preferred instruction written with the option to display visually.
• Unclear directions.
• Didn’t know how to do particular tasks.
• Had to use help to find how to annotate picture of Testudo.  Everything else came naturally.

Integrated interactive online help

• No problems during the use of PhotoFinder.
• Couldn’t delete annotation on Testudo.  Help did not solve the problem.  It didn’t do what it said it would do.
• Was not very clear on how to do stuff.
• I wasn’t sure about “annotate” vs. “caption”; got confused and used the wrong one.
• No problems.
• Annotating person’s name on the photo.
• Some items (like the search mechanism) were unintuitive and slide show via the dragging method.
• Delete: I was expecting a button not to press backspace.     

Did the help system tested help you complete the task?

Text only online help

• 3 subjects said the help facility helped them answer all 5 tasks.
• 3 subjects said the help facility helped them answer some of the tasks.
• 1 subject said the help system was unhelpful.
• 3 subjects did not answer this question.

Animated images only online help

• 4 subjects said the help facility was helpful for all 5 tasks.
• 1 subject said the help facility was helpful for some of the tasks.
• 1 subject said the help system was not helpful.
• 4 subjects did not answer this question.

Integrated interactive online help

• 3 subjects said the help facility was helpful for all 5 tasks.
• 2 subjects said the help facility was helpful for some of the tasks.
• 1 subject said the help facility was not helpful.
• 4 subjects did not answer this question.

If you answered, “Yes, some” or “No” to the previous question, what are the problems that help did not help?

Text only online help

• Confusion on initial annotation.
• Help didn’t help solve problems.  Help says nothing about the problems that I had.  My test’s blegel [sic] info was more helpful.  It’s very brief; says nothing on how to add photos to the database.
•   PhotoFinder didn’t describe terminology's used in the application.  The help window was too large, poorly position at the bottom of the screen, better to list steps instead of current paragraph form.

Animated images only online help

• Couldn’t delete label.
• 1, 2, 4 and 5 were intuitive so I did not use help.  3 required help, but it did not address the problem.

Integrated interactive online help

• Just wasn’t sure of what I was doing irregardless [sic] of program.  I’m just plain inexperienced.       
• Delete was confusing.

What were the problems with the use of help?

Text only online help

• None really
• Confusing in some instruction.
• But I think you should put the toolbar in a more common place, like up top, where people are most familiar with or on the side.

Animate only online help

• Help moved quickly, hard to see.
• In task 3, the annotation was made by a previous user.  I had to log in as that user to delete his/her annotation.  This was not discussed in help.
• It’s in separate window.  So I had to switch between them.  The animation was too quick so it was hard to see how to do a task in some occasions.
• None

Integrated interactive online help

• No problem with the use of help.
• Wasn’t clear, lots of boxes pointing all over the screen (should display one box at a time to make it more clear).
• None.
• N/A>
• Determining which help went first (even though numbered you have to search around the screen as box 1, sometime is on the left, right, top, bottom, ….)
• Hard to identify the order on help boxes.

Other comments and/or suggestions.

Text only online help

• Nice program, interesting.
• I think you should have a box like “Add caption”, where you have “Name,” because it is somewhat confusing to have to find the name and drag it for first time users.  But it actually may be okay for those that have gone through the program.
• The menu should be at the top like other program.  The layout is not aesthetically pleasing; I’m still not clear on how to add a caption that I can see during the slide show.  Annotate and caption: are they the same or different for the purposes of the help menu?  The terminology should be consistent.  The pictures in UMD file are very pretty.
• Good pictures, like the capability of the application like the slide show feature, legends are placed in too brightly colored background.  Like the send mail feature and how most tasks can be accomplished in single steps.       

Animated images only online help

• Help was easy to use.  I didn’t have to read anything.
• Pretty good program.
• Fun.
• I like the animation, it really helped.  The wording of question 2 is kind of confusing.  I would say, “Add caption to the photo.”  The appearance of PhotoFinder, there might be other color combinations you can use to make this program look better.

Integrated interactive online help

• The blocks in help was [sic] a little confusing to understand, I thought.
• Fun
• N/A
• Nope
• None

In summary, subjects using the Text only online help complained about the terminology, wording and unclear instructions of the help. The help context should be listed step-by-step instead of as a paragraph of instructions. They also recommended that the help button be placed on top, as it is found in most programs. Some Animated Images only online help's subjects recommended slowing the animation. Some subjects in this group recommended having both text and images in the help. Some commented that the help is clear and easy to follow. The subjects using Integrated Interactive online help thought the instruction boxes were too big and blocked the screen. They felt it was hard to identify the order of the instruction boxes. The common comment from the subjects in the three groups was that the terminology used in Task 2 (annotate and caption) was confusing. This comment was not surprising to us, since we felt the same. From the testers' observations and note taking, 29 of 30 subjects used online help some time during the experiment, as stated by [11], training materials are often necessary. The most common part for which they looked for help was to accomplish Task 2. Overall, the majority of subjects liked the program and its features.