The Library of Congress (LOC) is a research library, spanning a vast and varied collection, that operates to serve those prepared to work in a complex and scholarly environment, particularly Congress and its staff [1]. With the advent of the LOC’s National Digital Library (NDL), the service mission of the LOC has expanded to address a much broader user community, the public at large [1]. As a result, the design and structure of the NDL must support a host of individual characteristics, preferences, and experiences that are not orthogonal including: personal attributes, experience in the domain knowledge related to information need, and experience in using library systems and research techniques [1].
A user needs assessment of the NDL interface was conducted by several researchers from the Human Computer Interaction Lab (HCIL) at the University of Maryland in late 1995 and early 1996 to help the LOC design an interface that it is powerful and easy to use by a variety of users, ranging from school children (K-12) to scholars. The team identified several system needs including: support for a variety of information-seeking strategies; on-line tutorials or tours and last resort communications; graphics; clear delineations within the NDL as well as among the collections of the physical library, and the Internet; rapid display; the identification of primary-source and secondary-source materials; and an ability to search across collections [1].
Furthermore in the first half of 1996, the same research group concluded that to make browsing and searching easy and effective, the NDL interface should incorporate descriptive data about the collections and underlying objects; minimize the need to navigate through several levels before previewing a collection or in other words, surface the hidden treasures; consistently catalogue items in the collection; facilitate rapid response time; avoid lengthy hit lists; and allow users to quickly scan materials [2].
The LOC seems to have heeded the researchers’ advice and successfully implemented the NDL. According to the New York Times, the web site attracts more than 20 million hits a month, many of those initiated by schoolchildren [3]. The article though does not address how easily the visitors, particularly novice users, can find specific information nor does it acknowledge how satisfied visitors are with their experience.
A team enrolled in the Human Computer Interaction course offered at the University of Maryland wanted to explore the answers to these questions and to assess the effectiveness of the two information seeking features, search and browse, designed and supported by NDL for retrieving historical data within the American Memory (AM), Civil War Collection on the World Wide Web (WWW).
The intent of the study is to answer the following research questions with respect to the AM collection:
The results of the study should provide the LOC with feedback regarding the NDL design to improve their information service now available to the "bigger audience" including Internet surfers, students (K-16), scholars, and Congress.
Mental models represent an individual’s picture of a problem, derived by past observations or interactions with the environment and augmented or discarded by new experiences [4]. They are also affected by intelligence and motivation [5]. The models enable a user to both understand problem situations and predict consequences of actions contemplated for solving the problems [5]. Therefore, they may impact the way a user attacks a problem and revises the problem-solving strategy as feedback is given.
In terms of electronic searches, some users may apply print models (i.e., encyclopedias), not taking full advantage of full-text searching or hypertext capabilities [6] whereas other users may apply WWW models to find information. These existing mental models serve as basis for satisfactory performance, but the users need to develop unique mental models for interactive systems to attain optimal performance [6].
The interface can provide the user with relevant information to develop a better model of the problem, thereby enhancing performance and increasing satisfaction [4]. Training also provides the user with the necessary experiences to formulate a more suitable mental model for a given task [4]. Furthermore, high expectations about a system or novelty effects may actually be stimuli to the development of new mental models [6].
The information seeking strategy, which can be modified, is formulated according to the interaction among the following sub-processes: problem recognition and acceptance, problem definition, search system selection, query formulation, query execution, examination or evaluation of results, information extraction, reflection and if necessary, iteration [5 and 7], all of which are impacted by mental models. It is inherently a complex task primarily because it involves the articulation of an information need, often ambiguous, into precise words and relationships that match the structure of the system being searched [8].
Cognitive processes that identify key concepts and relationships lead to a definition of the problem [7]. The searcher’s understanding of that problem or task and the system is then combined to formulate a query [7]. Queries of novice searchers are especially revealing because each may represent a best attempt to compact an information problem into a single access point; expert searchers by contrast are likely to use a more systematic approach in which a series of queries is explicitly planned [6]. Moreover, novice searchers typically do not formulate queries that narrow a search to the context they have in mind [9] as evidenced by starting with too general a topic or with peripheral topics [6]. In general, searchers spend proportionally more time with their first queries than with subsequent queries [6]. Also, if a searcher expects a certain outcome after the task is completed, the query and ensuing action may be biased [7].
Once the results of the initial query are returned, the searcher must judge the relevance of the information contained in the response to assess if further query formulations and executions are required. The quantity, type, and format of the response affect the individual’s judgment [7]. Furthermore, deciding when and how to iterate requires an assessment of the information seeking process itself, how that process relates to the acceptance of the problem and the expected effort, and how well the extracted information maps onto the task [7]. To the extent that users cannot extract information given appropriate text, and whether this inability to extract information once its context was retrieved is due to subjects losing sight of the goal because they were focused on the system, the situation, or reading bears future investigation [5]. Searchers who begin with no hits and who subsequently adjust the query to find hits though would seem to have a good understanding of the system and be making progress toward success [5].
Information Seeking Features: Search and Browse
The search feature enables a system user to express an information need accurately. On the other hand, the browser must rely on a designer’s articulation of the information need, typically generic in nature. As a result, one can surmise that the search method is superior to browse because a user can narrow a search to a specific issue and obtain relevant information quickly.
However, formulating search queries for answering complex questions requires a very high cognitive load [4]. Search also demands short-term and working memory, thereby increasing the potential for more errors, negating the performance benefits of the information seeking feature. Interestingly though, several search experts said they typically search beyond the first answer and try to provide verification and follow-up information which slows the performance yet improves the error rate [7].
Moreover, the search strategy requires considerable time and incentive to learn [4]. In fact, Boolean logic, which provides a system user with the ability to search on every word and to combine concepts, appears to be one of the most difficult aspects of information retrieval and requires substantial modification of mental models, particularly for encyclopedia researchers [6 and 8]. As a result, most users do not take advantage of the more sophisticated capabilities of the system like Boolean logic, truncation, proximity, and word order features and instead, perform simple searches [8]. Overall, users have a poor understanding of search strategy and lack suitable mental models [1] as evidenced by the users who enter actual sentences to query a system [5].
Alternatively, the browse feature demands a relatively low cognitive load to formulate a query and relies on the user’s ability to identify a single key concept [4]. In effect, browse is a title search, analogous to looking in the print index volume for a term; users only need to learn the mechanics of using the system (i.e., hypertext links), not new information-seeking strategies [6]. The goal with browse is to place the user somewhere in the ball park from where the user will scan the output for information of interest [4]. As a result, it is especially appropriate for ill-defined problems and for exploring new task domains and ineffective for fact retrieval [10]. Additionally, browse may negatively affect a user’s experience with the system and task performance since the user’s must spend "extra" time filtering out irrelevant topics from the list of items retrieved.
Typically, browse is rather inefficient early in the process [7]. The effectiveness of a browsing strategy depends largely on the system’s ability to facilitate the searcher’s filtering activity [4] which is clearly aided by problem understanding and domain knowledge [7]. Also, the browse method is facilitated by rapid response times; alternative views or templates that users may apply to traverse databases or resultant subsets; and display technology and techniques that allow large chunks of the database to be viewed easily and dynamically [7]. Furthermore, helpful instructions or tutorials and descriptive titles or labels should be made available to counteract the disadvantages of browse. Moreover, the subjects should be interested in the subject matter since fatigue, both intellectual (boredom) and physical may limit browsing; subjects may feel that the reward is not sufficient to overcome the boredom associated with scanning large lists of files [4].
Browse though is motivating since it provides rapid feedback about a query [7]. As a result, most users prefer browse over search [4]. Humans seek the path of least cognitive load [5] and will trade performance efficiency for it [6]. Even expert searchers favor the browse feature in some instances, particularly when they have long term commitments to an area of research and benefit from extraneous information in that area [10].
Overall, system designers must strike a balance between the two information seeking features, search and browse because flexibility inevitably leads to complexity [10]. Also, for users to be most successful in utilizing information systems, training on both the search and the browse features should be provided so that users can adapt their information seeking strategies to their individual abilities, styles, and information needs [4].
The system interface may assert a powerful effect on the user [4]. The display should therefore be user-centered with the intention of focusing on user characteristics, both global and individual, and the information seeking task, simple or complex [1]. In fact, the system should adapt to various user abilities and styles [6]. In other words, it should support functionality that gives the user the ability to customize or tailor the front end interface [11] especially because experts in the task domain welcome great power and control whereas novices benefit from limited menus and less control [10]. Overall, the interface must be easy to learn and use as well as support a range of search strategies so that both novices and experts are well served and can complete their information seeking tasks without the benefit of a human intermediary [1].
The interactive features of electronic systems are extremely important since they foster the development or modification of existing mental models to those more appropriate for the system [6]. (Interestingly, users do not tend to utilize the features unless forced [8].) Furthermore, the impact of a user’s actions must be immediately visible to reduce errors and improve performance [10]. Additionally, Boolean operators should be made available to assist experts, and tutorials or helpful hints regarding query formulation, refining, and result filtering should be provided to novices [6].
Relevance feedback should also be considered since it helps users adapt their mental models and construct appropriate queries. The Institute for Learning Technologies is a good example of how it relevance feedback is used on the WWW. They utilize color (red for high confidence and black for low confidence) and numeric scores of confidence as well as list brief summaries of retrieved documents to improve user performance and perhaps satisfaction with their electronic library. Relevance feedback has also been adopted by UK Social Science Research Council who established a web-enabled collection of historical material covering the lives and events of an English village. The latter organization also supports full-text queries and provides both browse and search information seeking methods.
Moreover, designers should adhere to the following data display guidelines: group items in a way that is natural and comprehensible to users, emphasizing compact design; reduce or minimize response time where able to meet expectations; and limit query result sets to 100; ensure that whatever data a user needs will be available in the proper sequence; maintain a consistent format from one display to another; use short, simple sentences; adopt some logical principle by which to order lists; ensure that labels can be associated with their data fields; label each page to show its relation to others; and begin every display with a title or header [2 and 12].
Additionally, designers should favor breadth over depth with respect to menus, forms, or pages, limited to three to four levels [12], particularly because the user will be more successful and satisfied with the system if a minimal number of moves is required to access data [5]. Also, when an article is displayed, terms from the query should be highlighted on the screen to facilitate browsing via scanning of the text for relevance [5]. Interestingly though, highlighting of the query term(s) in the text of an article may cause relevant information to be missed, since only the exact terms are bolded [6].
Other research studies have highlighted the importance of organizational markers such as section headings and paragraph indentations for processing long texts to avoid potential menu-floundering; the use of outlines particularly for textual material; the need for navigation buttons to be positioned at the top of the page, in the region of the Browser buttons (if created locally), especially to minimize scrolling, the user’s least favorite operation; the need for a Home link since users when lost, typically return to the home page to reorient; and the use of Table of Contents, Maps, and Indices to avoid recalling options from memory [6 and 13].
In order for users to develop suitable mental models to use a system effectively, training and experience are necessary [10]. As a result, training has a definite, positive effect on performance [4]. People who receive training also report higher rates of satisfaction. However, users are not inclined to seek training or even read available documentation [8]. Interestingly too, their perception of how much time it takes to learn a system versus the amount of time actually required is extremely underestimated [8].
In any case, training materials should be made available to all users in a variety of formats to address needs like initial training for novices, refresher training for infrequent users, and assistance throughout the search process for all users [8]. Supplemental thesauri, online help, and online tutorials or modules are just a few examples of training materials. Designers can also facilitate the learning process by applying metaphors (judiciously), explicitly highlighting unique features not appropriate for the metaphor [6], and by creating an easily navigable interface. Moreover, users should be encouraged to use the browse feature which enables searchers to make connections between various articles, thus improving their understanding of the material [14].
In general, people prefer active learning, solving real tasks which are highly motivating [15]. Furthermore, practice exercises assure that the learner is exposed to the software’s most important functionality [15]. Those students who solely rely on guided exploration may not cover the capabilities and difficulties of the software and thus perform worse than the hands-on learners [15]. Additionally, in a previous study, those trained conceptually versus procedurally performed better on complex searching tasks [8]. However, an incomplete and simple conceptual model used to present the system to new users may limit their understanding of the system and their ability to apply it to future problems [10].
Motivation is a function of natural needs for control, competence, and belonging [16]. Students in general are not as motivated when asked to learn something that does not interest them, have little or no control over choice, and lack the personal skills or resources needed to be successful [16]. In order to motivate one to learn, one must design a task that stimulates or interests the student [16].
Exploration training may encourage learners to create problems that go beyond the training materials, leading to discovery of new concepts or novel combinations of simple concepts [15]. Moreover, systems that invite browsing will encourage more explorations and excursions of the content related and unrelated to the task [7] particularly because there is a tendency for searchers to examine articles out of curiosity [6]. In effect, browsing leads to direct acquisition of additional, albeit unrelated knowledge [4].
Interestingly too, subjects that use browse are likely to assimilate information that is not relevant to the task but that subjects are unwilling to discard; this is not the case for those who had search training [4]. It may be that subjects are not able to discriminate between relevant and irrelevant information [4].
Overall, performance appears to be determined by a number of controllable and uncontrollable factors. The following list incorporates ideas from several empirical studies including [1, 6, 8, and 17]:
Controllable
Uncontrollable
Other factors that may potentially affect performance include: academic discipline, indirectly suggesting personality characteristics, technical aptitudes, and reasoning ability; spatial ability, found to be the best predictor of the time taken to locate target texts in a retrieval system; and reasoning ability [18].
Overall, the most important variable determining success appears to be system experience or frequency of use [8].