Results

The dependent variables measured were:
1. Accuracy of the answer
2. Time to complete a given task (i.e. answer the question).

Raw data from the experiments is presented in Appendix 5. It was noted that accuracy essentially became a non-factor, not influencing the statistical analysis. The participant’s level of accuracy remained constant accross variable groups. What this means is that most of the time (66%), when users thought they found an answer within the given 5 minutes, it was correct.

Although it was originally planned to capture numerical data on incidental learning and subjective satisfaction from the post-survey and subject it to statistical analysis, this essentially became infeasible partly due to the lack of motivation on the participants end and partly due to the frustration built up among the users by the end of the experiment (as discussed later). In essence, even after conducting the post-survey, enough numerical data was not gathered to mathematically notice any trend in incidental learning and subjective satisfaction. However, as mentioned in the experiment section, the experimenters maintained a record sheet to notice/ observe participant reactions and behaviour under various situations and this helped us get the subjective results as described later in this section.

Statistical analysis of the time variable (specifically the single factor ANOVA and t-tests) is presented in Appendices 6 - 10. A brief synopsis of the same is first presented in this section followed by a listing of the subjective results that were noted.

Browse with Training v/s Search with Training

Subjects doing the tasks on browse with training performed better (required less time) than subjects working on search with training in all questions except question 3. (Table 1, Figure 1). Even though the results of an anova and t-test comparing browse with training and search with training do not give statistically significant results at .05% confidence interval (Appendix 6), it can be seen from Table 1 that the mean times on questions 1, 2, 4 and 5 for browse with training are less than those for search with training. However, the standard deviations in the case of browse with training are significantly higher than those of search with training implying that search with training consistently takes a relatively longer time.

Table 1: Browse with Training v/s Search with Training

Q #	Mean Br w/tr	Std. Dev. Br w/tr	Mean Se w/tr	Std. Dev. Se w/tr
1	140.33	94.11	231.67	118.36
2	213.00	103.18	300.00	0.00
3	234.00	114.32	88.33	26.16
4	186.33	108.87	300.00	0.00
5	231.67	103.69	268.67	50.00

Figure 1: Browse with Training v/s Search with Training

Browse without Training v/s Search without Training

Table 2 compares the mean times and standard deviations for browse without training and search without training. Questions 2 and 4 took a significantly longer time on the search than on the browse while question 5 took a longer time on the browse. Questions 1 and 3 had similar mean times with the search slightly lower than the browse (Figure 2). In this case again, the results of an anova and t-test comparing browse without training and search without training do not give statistically significant results at .05% confidence interval in favor of either method (Appendix 7). Once again, the browse had lower standard deviations than search.

Table 2:Browse without Training v/s Search without Training

Q #	Mean Br w/o tr	Std. Dev. Br w/o Tr	Mean Se w/o tr	Std. Dev Se w/o tr.
1	273.67	45.61	268.00	55.43
2	226.67	127.02	294.67	9.24
3	286.00	24.25	277.33	32.58
4	221.00	136.83	300.00	0.00
5	239.33	105.08	168.33	117.07

Figure 2:Browse without Training v/s Search without Training

Browse with Training v/s Browse without Training

Comparison of the mean times of browse with training and browse without training (Table 3, Figure 3) shows that browse with training consistently required lower times than browse without training. Again, the statistical analyses of anova and t-tests do not show a significant difference between browse with training and browse without training at .05% confidence interval (Appendix 8). However, the standard deviations in both the cases are consistently high.

Table 3: Browse with Training v/s Browse without Training

Q #	Mean Br w/tr	Std. Dev. Br w/tr	Mean Br w/o tr	Std. Dev. Br w/o tr
1	140.33	94.11	273.67	45.61
2	213.00	103.18	226.67	127.02
3	234.00	114.32	286.00	24.25
4	186.33	108.87	221.00	136.83
5	231.67	103.69	239.33	105.08

Figure 3: Browse with Training v/s Browse without Training

Search with Training v/s Search without Training

Table 4 compares the mean times and standard deviations for search with training and search without training. Questions 1and 3 took a significantly longer time on the search without training while question 5 took a longer time on the search with training. Questions 2 and 4 had similar mean times. (Figure 4) In this case again, the results of an anova and t-test comparing search with training and search without training do not give statistically significant results at .05% confidence interval in favor of either method (Appendix 9).

Table 4: Search with Training v/s Search without Training

Q #	Mean Se w/tr	Std. Dev. Se w/tr	Mean Se w/o tr	Std. Dev. Se w/o tr
1	231.67	118.36	268.00	55.43
2	300.00	0.00	294.67	9.24
3	88.33	26.16	277.33	32.58
4	300.00	0.00	300.00	0.00
5	268.67	50.00	168.33	117.07

Figure 4: Search with Training v/s Search without Training

Browse or Search with Training v/s Browse or Search without Training

In the Search or Browse Category, questions 1, 2, 3 and 4 gave better results (needed lesser time) without training than with training. Only question 5 required lesser time with training. (Table 5, Figure 5). The standard deviations in both the cases were similar. However, the results of an anova and t-test comparing the two do not give statistically significant results at .05% confidence interval in favor of either method (Appendix 10).

Table 5: Browse or Search with Training v/s Browse orSearch without Training

Q #	Mean SB w/tr	Std. Dev. SB w/tr	Mean SB w/o tr	Std. Dev. SB w/o tr
1	179.00	105.66	175.67	50.81
2	300.00	0.00	246.00	93.53
3	210.33	63.41	180.00	105.00
4	270.67	50.81	239.33	73.92
5	225.67	52.97	276.00	41.57

Figure 5: Browse or Search with Training v/s Browse orSearch without Training

Summary

Summarizing the statistical results, although no statistically significant effects were found, we can conclude from noticing the graphs that overall, doing the tasks with training is better than doing them without training. Another important factor noted was the large number of times subjects could not complete the tasks within the alloted 5 minutes (50% of the tasks were timed out).

Subjective Results

Some other subjective items that were noted were the following:

While some of the users in the free exploration groups used their time to look at search tips and carefully investigate the website, others used that time to browse through matters of their interest like the baseball collection or the first ladies’ portraits etc.
A lot of users on the search tasks exhibited confusion and frustration on seeing the search results table. On seeing zero results in the first box of the search table, on participant sadly exclaimed, “..not doing too good, am I?” Another participant referred the search results to being “....too damn vague!”
When the search results came up on the screen, participants tended to select the first item on the list and frustration was high when the first few items on the list did not bring out the desired answer.
There was a tremendous lack of motivation noticed among participants that could bias the results. One participant was noted saying, “...I’m writing this answer even though I know this must be wrong....” Another participant asked, “....What if I’m wrong?”
Another point exhibiting lack of motivation was the fact that the participants did not pay much attention during training and and did not care to use it while carrying out the tasks, justifying that by saying, “I like to go my own way!”
Participants often lost themselves in the hierarchical structure while searching or browsing leading to the conclusion that the search and browse modes are not cross-referenced well. A participant infact mentioned, “...Isn’t it ironic? This is where I wanted to be....by chance!”
Some amount of frustration was also noticed among users who were forced to use browse instead of search, since searching the net was what most of our participants were familiar with. One participant exclaimed in frustration, “Why can’t I type in a keyword?” Another noted, “If I can’t use search, there’s no way, I’ll find it”
From our post survey it was noted that a large number of participants acknowledged the fact that the American Memory site had a large and comprehensive database and many of them (72%) felt that they would use the site in future for history research. However there were a few who were frustrated with system response times and inability to get the answers. One such participant, when asked what one could learn from this experience stated, “...one thing I learnt is to get a good history book!”

Continue

Return to the title page