2.3 Pilot Study Results
The pilot study was conducted in order
to assure that the following guidelines regarding the
experimental tasks were followed:
- The subject of the presentation was
neutral to the subjects in such a way that any previous
knowledge would not be relevant to the subjects’
performance.
- The subject of the presentation was
interesting, and hence the subjects could perform just
like in a real situation, where they are really searching
for information.
- The information was organized in a
hierarchical and neat format, making easy to subjects to
perceive the document inner organization.
- The presentation was organized in a
nice and appealing way in order to be attractive to the
subjects to avoid any external distraction for the
subjects.
- The time-measuring capability was
embedded inside the tool and/or the presentation. This
way the subjects would not be distracted by doing any
task outside their domain (which is "to find the
answer").
The content of our presentation is
summarized in Appendix
A, which is from Encarta’95,
the general purpose encyclopedia from Microsoft.
During the pilot study, our four
subjects and our evaluation turned out to point several flaws in
the overall design that worked against our own established
guidelines. Basically, the improvements required can be
summarized as the following:
- The presentation should be cleaner
with a consistent transition between layers of
information, less textual information (i.e. only an
sketch of key facts), less contents, a consistent
organization and color schema, and a common navigation
command set.
- A training phase to get the
subjects used with the navigation commands, as well as
the environment of the experiment, should be provided to
diminish the anxiety and to make the subjects more
comfortable.
- A smoother transition from the
training phase to the experiment in order to make easier
for the proctoring person to handle the experiment
environment.
- A reorganization of the data
collection process, with a reordering of question, and
the addition of a new variable (the number of hyperlinks
traversed), to make the overall experiment potentially
more relevant regarding the results generated.
All the improvements were implemented as
required. In addition to these improvements, a set of operational
guidelines was set in order to assure that all the subjects would
take the experiment in similar conditions:
- The geometry of windows (Pad++ and
Mpad), and the positioning should be constant. The Pad++
window was set to be 900 x 645 for all subjects.
- The experiment should be taken in
one of the two machines prepared with similar environment
at the HCIL.
- The user workplace should have only
Pad++ and Mpad windows opened.
