Experiment

Description
The Software Feedback experiment centers around two independent variables which represent the level of feedback shown to the user. These independent variables are feedback in the form of text and a graphical progress bar. Treatments for both variables are; present, and not present, thus there are a total of four different treatments. The textual feedback, as shown in Figure 1, is limited to the phrase, "Please Wait, the system is processing your request." and the progress bar, as shown in Figure 2, is an incremental blue bar supplied in Microsoft Visual Basic® 6.0. The combined text and progress bar feedback window is shown in Figure 3. Both the textual feedback and the graphical progress bar are shown to the user in a centered window of smaller dimensions than the main program. The feedback window dimensions are kept constant between different treatments which leads to differences in text spacing and progress bar height between the treatments. The dependent variable is the number of menu command calls.

Treatment #2 (Figure 1.)


Treatment #3 (Figure 2.)


Treatment #4 (Figure 3.)

Each subject is asked to complete a simple multi-part task on four different versions of the same email client software. In this way every subject is exposed to each treatment of the independent variables, making the study a repeated-measures within subject experiment. The total number of subjects is set at 24 to accommodate every order which the subjects might encounter the independent variables. Every task within each treatment is delayed by 15 seconds and the appropriate feedback -or lack thereof- is displayed at the beginning while the task isn't executed until the end of the delay.

After affirming that the subject is willing to devote approximately 15 minutes to helping with a controlled study, the subject is asked to sign the Experimental Consent form. Then, The subject undergoes training on the email client interface by performing the requested tasks inside the Demo -which is a zero-delay/zero-feedback version of the email client. Afterwards, the subject performs the experiment on the four different treatments, and fills out the questionnaire. The experiment proctor as well as the printed task list are available for reference to the subjects at all times during the training and experiment.
All 24 trials were performed on PCs running Microsoft Windows NT® Workstation 4.0.

Hypothesis
The null hypothesis is that no statistically significant difference in command calls will be seen for different treatments of the independent variables.

Our hypothesis is that the greater the level of feedback, the more patient the user will be. Thus, we are expecting our subjects to issue the largest number of erroneous commands for the software version which has no feedback, and the smallest number of erroneous commands for the software version which supplies a progress bar with textual explanation of the actions.

This was quickly seen as overkill, and the current (albeit still too long) task list with seven delayed tasks was chosen. Also, the email client was changed so that users' excessive command calls weren't executed (only the first command call is executed until the delay time runs-out). The initial version also compounded the delay time based upon the total number of command calls. This was changed so that each command call would complete in the prescribed time period (15 seconds).

Subjects
There were 24 subjects used for the study (15 male, 9 female). Three subjects were taken from the Office of Student Financial Aid, the remaining were recruited from the campus WAM labs. Of the subjects, 11 reported using a computer over 16 hours per week, four reported using a computer between 10 and 15 hours per week, six reported using a computer between five and nine hours per week, and three reported using a computer less than four hours per week.

Eight of the subjects rated their "computer literacy" as excellent, 12 rated their computer literacy as adequate, and the remaining four rated their computer literacy as poor.

Materials
The following link points to a zip file containing the files which are on the experiment disk: exp.zip.
These include a compiled Visual Basic program named "mailer.exe" and two .DAT files.