Experiment

Introduction

With all of the advantages and disadvantages of each instruction format, it is unclear which format is best to use for online instructions. This study was designed to investigate the relative effectiveness between the use of text-only instructions, graphics-only instructions and a combination of the two. These instructions were all on a web page, and the subjects viewed them with Netscape's web browser.

Speed and accuracy of the subjects in building the models were the measurements that we used to determine the relative effectiveness of each format of the online instructions. The online instructions of this study presented the step-by-step tasks for completing three different building block models with varying difficulty levels.

It is important to note that the instructions provided in this experiment were only the steps necessary to accomplish the task of building a model. Most of the empirical studies that we have referenced test the learning and retention of the subjects after reading instructions or viewing demonstrations. This experiment is measuring the user's ability to follow the provided instructions, not their ability to learn or retain any information.

Our hypothesis is that the graphics-only instructions will yield the fastest times to completion with the fewest errors. We believe that the addition of text to the graphical instructions will distract the subjects and increase their assembly times. We also think that the text-only instructions will produce the longest times with the most errors.

For our pilot study, we used three sets of instructions for the same model. The primary goal of this was to get a sense of whether our format for running the experiment was going to work or not. We tested 4 subjects, all students at the University of Maryland, to give us a feel for what needed to be changed in our final implementation of the project.

One thing that we noticed early on, was that the images for the graphics-only and text with graphics instructions needed to be precached into the web browser. We had anticipated this even before the pilot; however, after the pilot study, we saw that this was essential to maintain a clean, consistent testing environment. Some of the pictures took a while to load, and this would have skewed our data.

Another key learning experience from the pilot testing was that it was very difficult for us to count the number of errors after the completion of a model. We considered an error to be misplaced piece on the model, or a component that was never attached. Missing or misplacing a piece to these simple models, lead to many more errors. So we decided to categorize the error counts to none, few and many.

Time was another factor. Although not critically important for the data itself, the amount of time it took to complete the three trials for each subject may have an effect on our collection of the data. Originally, we did not think it would take very long to get a subject in, have them run through all three instruction types, and fill out the survey. We then found out that we would need to allot approximately a half an hour per subject to complete all the tasks. This helped when it came time to get subjects to participate in our experiment. We could give them a more accurate approximation of how long it would take to complete the experiment.

The actual text instructions needed to be revised. This was primarily to fix some minor grammatical and spelling errors, but also to make them more understandable. This helped to cut down on the overall time for each subject to complete the tasks, and it provided more accurate, and less varied data for the text-only and text with graphics instructions.

Finally, after submitting our proposed survey, we found that we needed to revise it as well. At first, it had more questions in it. This was our intent, as we figured that we would be editing out the less necessary and relevant questions later. Also, there were minor fixes, such as adjusting the subjective scales from 0=BAD - 4=GOOD, to a wider spectrum of 0=BAD - 9=GOOD. The majority of the adjustments to the survey was to clear up small mistakes, and clarify the questions for the subjects.

24 students and staff of the University of Maryland, and engineers from Hughes Network Systems served as participants. Half of the subjects were male and half were female. Every subject had a college degree or was working on one.

This study is a 1 x 3 design. It uses a single independent variable: Online instructions. The 3 treatments are text-only, graphics-only and a combination of text and graphics.

Step-by-step instructions were created for the assembly of the three building block models: a UFO, a car and a helicopter. Each instruction set is comprised of 8 to 10 web pages, which provide incremental building steps towards the completion of the model. The sequence of building steps was determined by the instructions that come with each model and were followed when developing all of the web based online instructions. Each step involves the assembly of 1 to 3 pieces or segments. The pages for a particular set of instructions are linked with "next" and "back" links.

The types of instructions are as follows:

1. Graphics-only: These are the instructions that were included with the purchase of each model.
2. Text-only: These instructions were created by translating the provided graphics-only instructions into words. These instructions also included a physical description of each piece.
3. Graphics with text: These instructions were a hybridization of the pictures from the graphics-only instructions and the text from the text-only instructions.

There was no training required for the subjects.

Every subject assembled all three models, using a different type of instruction set for each. For example: subject #1 assembled the UFO using text-only, the car using graphics-only and the helicopter using the text with graphics instructions. This ensured that the user becoming familiar with the model did not influence the times associated with the completion of each model. The order of assemblies was assigned at random.

The instructions were pre-cached, so that web traffic and load times would not be a factor in the subject's time to completion. The subject assembled one model at a time while the experimenter observed and timed them. After each model was complete, the experimenter examined it to categorize the number of errors. They errors were classified as none, few and many. The subjects used a mouse to navigate the pages of the instructions, and the keyboard was removed from the working area.

After the subject completed all three models, they filled out a survey, which included both objective data and subjective points of view.

Subjects seemed to rush through the assembly of the models. Many of them became frustrated with the text instructions and even vocalized the words while they read. Few of them tried to correct their errors, during the assembly, by taking advantage of the “back” link to reference previous pages. The effects of these behaviors will be examined in the discussion of the results.

A background survey conducted after the experiment showed that the subjects seemed to fall into one of two categories in regard to their building block experience. Either they had 3 or less years of experience, or they had 15 or more years of experience with the building blocks.

The surveys showed that the years of web experience the subjects had varied between 2 and 10 years. While this may not have been a crucial statistic to the experiment (due to the simplicity of the web interface involved), we feel that it was good to have such a variety of experience in our subjects.

When asked to rate their preference of learning style (i.e.: textual vs. graphical), the majority of our subjects claimed that they preferred a more visual learning method. We are hoping that this either reflects the current trend in instructional methods, or will be an influence on them in the future.

When asked which of the presented instruction styles the subjects preferred, roughly 50% of them indicated that they liked the text with graphics style and the other 50% indicated that they liked the graphics-only style. Comments written by the subjects implied the former group found the pictures to be complementary to the text, while the latter group just found the text to be unnecessary and distracting.

Department of Computer Sciences Direct questions and comments to the student editorial team