Hypothesis:
Students that have training will perform better than students that
do not, however the latter will incur the most incidental learning. Moreover,
those students that solely utilize the browse feature will complete the
specified tasks faster and more accurately than those who only use the
search feature or those who can utilize both features.
Subjects:
Eighteen participants volunteered from the undergraduate and graduate
programs at the University of Maryland.
There were some history majors and seven females and eleven males. To equally
distribute the groups, the researchers took a pre-survey of the participants.
They significantly weighed factors such as success with searches on the
internet and gender. Everyone had experience with the web, in general and
no one had experience with the LOC. The researchers concluded that these
were the non-significant factors not taken into account while setting up
the groups.
Variables:
For this experiment the researchers had two classes, “guided” training
and “self” training , totaling six groups. These two classes were divided
into three groups and the experimental structure of the groups were :
Class I - “Guided” Training
Class II - “Self” Training
Class I, “guided” training, had guided instruction on how to use the AM homepage along with a task demonstrated to the participants. Class II, “self” training, had five minutes of free exploration of the AM homepage with no instruction. The dependent variables that the researchers were testing for were speed, accuracy, and incidental learning.
Materials:
Materials that were generated for this experiment consisted of a pre-survey, five task questions, a record time sheet, and post-survey. The pre-survey, as mentioned before, was to determine the basis and to equally distribute the prejudices among the six groups (Shown in Appendix 2). The task questions were developed by the researchers and these tasks were rated on a scale of easy to difficult (Shown in Appendix 1). The difficulty of the task was measured by the anaytical skills and cognitive ability required to find the answer to the question. For example, interpreting the meaning of a document, ability to scroll up and down the page to find answers, or just finding a date needed a variety of different skills and abilities. Some of the task questions were developed to be easy to find using the browse feature but medium or hard by using the search feature. The record time sheet was to keep track of the time it took for the users to find the answer and to write down any query formulations or observations. For example, the researchers observed the query formulations the users did during a search or browse and also noticed the mental state of the users, whether they were getting frustrated or how they were responding. The post-survey was given when the tasks were completed to capture subjective feedback and note incidental learning that happened during the use of the AM homepage. (Shown in Appendices 3 and 4).
Pilot Study:
The researchers had done a pilot study to adjust any tasks that seemed impossible, such as gauging the time it took to find the answers to the task questions. The time limit for each task question in the pilot study had been three minutes. However, they determined the time was not long enough to find the answers to some of the task questions so for the “real” experiment the time limit was set to five minutes per task question. Other problems that the researchers realized were:
In fine tuning the procedures, the researchers made sure the following things were kept consistent when they ran the “real” experiment:
After adjusting and fine tuning, the researchers had the procedures developed and ready for the real participants.
Procedures:
Overall Instructions
All users were instructed that they had five minutes to find the answer
to each task question, write it down on the slip of paper, and to raise
their hand when finished. The users were informed to return to the begining
of the AM homepage and to sit quietly until the five minutes was up. Also
the users were told that all questions that they had about the instructions
that were given should be asked before the experiment began. Participants
were also requetsed to use only the information that they saw on their
screens and disregard any prior knowledge that they had about the subject
matter. The observers recorded the completed time when the users wrote
down the answer and raised their hand. The observer also would log what
query formualtions were used and what other collections were used by the
user beside the Civil War collection, to find the answer to the task question.
The accuracy of the answer was checked later. The speed at which the user’s
performed the task was recorded with a stop watch to the nearest second.
The observer’s monitored the behavior of the user closely during the experiment,
i.e. whether talking to themselves, getting frustrated, or just wiggling
around in their chair. Then after all the task questions had been completed,
the post-survey was handed out to get feedback from the users. Finally,
the users were debriefed and were told the answers to the questions if
they didn'’t get correctly.
“Guided” Training Instructions
In Class I, these three groups had guided five minute tour of how to
use the features (i.e. search feature, browse feature, or both search and
browse features) of the AM homepage. The same task, which was to find a
photograph of Thomas Ewing with a difficulty rating of easy in search and
in browse was demonstrated to all three groups, using the subject index
and timeline in browse or key words in search. The first guided group used
the browse feature only. This group was shown how the homepage was organized
by historical collections, and that these collections are organized by
type, i.e. Photographs, Documents, Motion Pictures, and Sound Recordings.
Key features of browse were the special presentation of the “Time
Line” which has events and photographs from 1861 to 1865 of the Civil
War period and the subject
index feature, where the user could find people and places organized
alphabetically. This group was not allow to use any of the search capabilities.
The second guided group used only the search feature. This group also got
an explaination of how the AM homepage is organized. Key features of search
were explained such as searching all the collections, part of the collections,
searching for key words, or even exact key words. This group was not allow
to use any of the browse capablities. The third group used both search
and browse features and received both search and browse demostrations.
“Self” Training Instructions
In Class II, these three groups had no guided tour of the AM homepage
but had five minutes to explore the site themselves. These users had to
figure out how to use the key features by themselves.