In conducting this study, we hoped to establish quantifiable evidence supporting the need for training and the superiority of browse as a navigational tool versus search for novice users. Unfortunately though, we did not capture statistically significant evidence to sustain our theories perhaps due to the limited number of participants. We did however observe several interesting behaviors that are noteworthy.
Search & Browse Engines
Overall, both the search and browse engines proved effective in completing most of the assigned tasks. However, some of the novice users, limited to using Browse, seem to have performed marginally better on certain tasks than those using Search or both navigational tools. This outcome may be attributed to the fact that Browse is very motivating to users since it provides rapid feedback and that Browse requires a low cognitive load. Search on the other hand, demands more mental effort was because one must be able to construct appropriate search queries when given a problem. Interestingly though, most of the participants said that they preferred the Search engine perhaps because all of the subjects were familiar with WWW (per their answers on the presurvey). In addition, subjects usually stopped searching and/or browsing when there were no more compelling triggers for further exploration or when they felt that they had done an appropriate amount of exploration for the task. Also, participants started their initial information retrieval by Browsing (maybe to refine their understanding of the options available) and ended up Searching which was enhanced by the knowledge gained through the browsing.
Additionally, from our observations, Browse seemed useful in two situations: (1) when users recognized that the question provided them with the date of the event (2) when users could recognize the event but didn’t quite know how to describe it (lack of keywords). Therefore, when effectively executed, the Browse feature lends itself to successful information retrieval. But Browsing can also lead to serendipitous findings. First, browse invited unexpected discoveries not related to the task at hand. For example, a user began browsing through the Civil War Photograph Collection looking for the purpose of the Emancipation Proclamation and ended up in a section entitled “Military Facilities”. Second, browse forced the user to scroll down the screen in order to utilize some its key features like timeline. Due to this layout, users without training overlooked some of the key features of Browse which may have caused them to time-out on questions weighted as “Easy”. Next, the titles of Collections were limiting and misleading. Even though we stressed that all answers could be found in the Civil War Photograph Collection, users hesitated to utilize this collection because the question required them to retrieve a “document” not a photograph which the title infers.
Alternatively, the Search engine enabled users to narrow their search to specific issues and directly obtain relevant information quickly if they had a good understanding of the task domain. This is evident on question #3. After reading the question most of the users immediately formed a mental model which led them to all enter “Fort Sumter” in the Search field which gave a mean response of 1 minute. Another highlight of the Search engine is that keywords indicated in the query search are boldly highlighted within the retrieved documents. This feature seemed to help users find answers in the retrieved document.
The Search engine also has several pitfalls. First, after receiving query results, most users did not know what do next or how relevant the hits were to their query. Second, some users chose the first item on the list regardless of its relevance. To assist them, we recommend a query results window that appears in a corner that does not obscure any of the displayed documents with an associated star schema that indicates how relevant the selected document is for their query. Also, to reach a variety of users, a help facility that explains the search engines capabilities and features and how to refine/enhance searches (i.e. what characters are allowed, punctuation ,etc.) would be useful.
Another pitfall was that Search results were unpredictable and uncontrollable. This was noted when users entered one query as “Gen. Robert Lee” and another query as “General Robert E. Lee” and they received different results which prevented them from finding the correct answer. Finally, the labels for links to documents where not specific, that is, the title did not reflect the content of the document. For instance, all the users, who utilized the Search engine to find out the purpose of the Emancipation Proclamation, timed-out because the answer to this question was located in a document entitled “I saw the stars” which does not correlate to the content within the document.
Overall, the coupling effect of both engines with proper training, appears to offer novice users the optimal balance for retrieving information from the American Memory homepage. This is supported by the fact that the users completed task 3, which required retrieval of specific documents, faster using the Search engine and task 2, which specified a specific time period, faster using the Browse engine.
Information Retrieval Strategies
Poor domain knowledge and unsuitable mental models seemed to present problems for users when formulating precise and effective queries. This highlights the important roles that vocabulary and cognitive skill play in determining search strategy. Most subjects were able to extract key terms from the question to enhance their search but yet there were others that knew the answer and could not articulate the terms or abstraction used by the search engine. An example was noted when a user knew that Appomattox Courthouse was the answer to question #4 but could not formulate the appropriate query and ended up timing out.
Moreover, some subjects assumed that the search engine had sophisticated capabilities to handle most information entered. They entered full length questions, phrases and sentences enclosed with quotes such as “Where did General Lee Surrender?” and “Who were the conspirators for Lincoln assassination?” which yielded no hits or hits with unrelated information which stunned most users. Thus, natural language queries also play an important role in information seeking strategy.
In general, subjects appeared to use the brief system feedback to modify and refine their query formulation, thereby started the process of adapting their mental models. For example, users scrolled through some of the documents and used terms located in the documents to better refine their query selection. This “improvement” attitude was also reflected when users utilize a specific feature or query formulation and was successful, they tried to re-use the same procedures on other questions.
Training Effects
Most of the participants that had guided training and that were limited to using Browse seemed to have performed better than those that freely explored the AM web site. A possible explanation is that most users were familiar with Search engines which was indicated in the Pre-Survey. Another explanation is that the browse training was more in-depth than that of the search engine. That is, the training for the browse engine was more comprehensive which required an overview of the many key whereas the search engine training was limited to the explanation of a basic string search but a possible improvement could be for us to give a more in-depth training with Boolean logic and refinement features. Finally, training provides the user with the necessary experiences to formulate a more suitable mental model for the given task [4].
Futhermore, during the exploratory training, most of the subjects did not fully navigate through the system to determine its full capabilities, and instead they opted to explore collections of more interest to them such as photographs in the “Portraits of Presidents” Collection. This could have accounted for the poor performance on tasks rated “easy”. For example, question #1 gave the date of General Lee’s surrender and they where asked to find the location of the surrender. When the users overlooked the Time-Line feature supported by Browse, they normally timed-out on this question. Furthermore, users, with exploratory training, who could use either the Search or Browse engine took longer to complete task than users who were restricted to one or the other engine. An explanation for this performance is that users had more to learn and explore. This could have taken them more time to gather their information-seeking strategies or plan of attack.
Other Effects
Intellectual fatigue is a likely factor that contributed to the low performance rate of most users. Most users were not motivated and this possibly led to poor selection of appropriate navigation strategies which could have biased the accuracy results. The slow response time was another factor that may have contributed to low performance because it was very disruptive and users became distracted and bored while waiting for their results. Several users mentioned that they were not able to fully enjoy the web site due to slow system response time.
It also appeared that users became “lost” and tried to rely on the hierarchy of system to re-orient themselves but the Search & Browse modes were not cross-referenced well and the hierarchy was not clear to the users, as a result user became idle or kept performing the same recursive moves which led them in circles. To assist users in this area, a schematic representation of the hierarchical structure which displays where users are currently located within the system and the possible paths to information would be nice.
Even though incidental learning was not indicated for the majority of the participants, users expressed positive feedback toward the American Memory homepage. They were very impressed with its: visual presentation, vast amount of information, comprehensive database and existence. Users also mentioned that they would consider visiting the homepage in the future. However, a few users find books easier to read and more accessible then hypermedia systems like the American Memory.