Experiment

2.1 Introduction 

The project deals with the structure and organization of links on the E-Maryland page. Currently the home page of the site uses dynamic menus to link to pages within the site. While dynamic menus offer an interactive method of displaying links, it requires eye and hand coordination, which compromises the navigation efficiency. The dynamic menu also impedes the visibility of other categories on the page and limits the number of options visible at a given time for the user, thus potentially increasing the error rate of users. The experiment will help determine whether a page with a static menu design would be more efficient for navigating through the site. 

Another related issue for site navigation of E-Maryland, is the hierarchical site structure, which is essentially an intuitive method of organization. However, the site contains only one level per page, and thus the users must wait for several pages to load before reaching their desired link. Although this allows the user to focus on his/her selections, it is difficult at times for the user to determine the correct path of links. Decreasing the number of levels would not only increase navigation efficiency but also clarify the breakdown of categories, so that users would more likely select the shortest path to their link. 

Three versions of the E-Maryland page are presented to the subjects. There is an original version, where the home page has dynamic, interactive menus, and a deep hierarchical web site structure. An alteration of the home page is made for the second version. This version has a static menu, where the links of the four main categories are displayed in fixed position. The rest of the site remains the same as the original. The last version has a static home page, and a broad 2-level site structure. The first level contains the main categories and links, while the second page contains all the links within that category. Since the page becomes fairly long and requires scrolling, the links are sorted into their respective groups. The link the user clicks on the first page is also used to target the specific location of the group in the site. 

The experiment compares the optimal method of menu presentation and organization on a page by having our subjects complete a list of tasks, such as links to search for, and measuring their time of accurate completion, error rate, and subjective preferences.   Since short-term memory is involved in site navigation, the recall error rate helps to determine another aspect of efficiency.

To test the navigational efficiency of the three versions, we have a 1x3 within-subjects experiment design. The main metric used will be the time of completion rounded to the nearest second. 

Independent Variable 

Online menu structures with the following treatments: 
 

• Dynamic Menu with narrow, deep web structure 
• Static Menu with narrow, deep web structure 
• Static Menu with broad, shallow web structure 

Dependent Variables 

• Time for correct completion of a list of links to search for 
• Subjective preference of each treatment 
• Short-term Memory Error Rate 

Hypothesis 

Subjects using the static menu with the broad, shallow structure will complete the tasks the fastest, but subjectively users will like the interactive nature of the dynamic menu design. Users will have the hardest time learning how to navigate the original menu, because of the novel design and because the design can interfere with seeing the other categories. The original dynamic menu will probably have the least number of error rates in memorizing the four categories since the categories are more explicit, and the site is not crowded with other options in the menu. Expert users who prefer functionality over novelty will prefer the static menu with broad, shallow structure, while others will prefer the original interactive design. 
 

2.2 Pilot Study 

Pilot tests were conducted on 5 subjects, who were University of Maryland students. Each type of E-Maryland portal had 1-2 subjects try out the 3 different sets of links. The results indicated a wide range of times for completion. The static, shallow menu structure usually yielded faster times, and most users preferred its ease of use. However, the original version was more visually appealing for most of the pilot subjects. For 2 of the subjects, we also tried to count the errors (i.e. each time the user had to return to a level above his/her current level). But since it was difficult to catch each time the user clicks the back button, we had to throw the results out and decided not to count the error rates in the final experiment. 

An attempt to clarify ambiguities in the survey questions and in the instructions was made. Since we assume the subjects will have knowledge of Internet use, no training will be necessary to conduct the experiment. We also found some bugs in our pages, which we updated. 

In addition, a short-term memory test was added as another measure of web design effectiveness. In order to cover all 4 categories for each set of tasks, the tasks were also changed from 3 links to 4 links per set. Each link in a set is from a different main category (Lifelines, Citizen Services, Maryland Scene, and Business Services). Each link was also determined to be about 4 levels deep in the hierarchy in order to ensure a constant difficulty level. 
 

2.3 Subjects 

Twenty-four subjects participated in the experiment. The subjects were from University of Maryland or were residents of Maryland, and therefore potential users of the E-Maryland web site. Some subjects were solicited in the WAM lab, while others were friends of the group members and students from Dr. Michelle Hugue’s data structures class. Since the website is designed for average daily users, no background requirements were necessary. 

Fourteen males and 10 females participated in the experiment.  The subjects were fairly proficient with the Internet.  Of the 24 subjects, 21 rated themselves a 7 or above in proficiency level and in the use of the Internet as a primary source of information.  Along the personality lines, 17 of the subjects selected 7 or above when asked whether they think things through thoroughly before they acted.

2.4 Materials 

Testing was conducted mainly in the AVW Microsoft NT Lab. The pages that were used for testing can be found at: 

• Original - Dynamic Deep
• Static Deep
• Static Shallow

Subjects were given a consent form to read over and sign. They were then asked to complete a background survey to determine anonymous general information and their web proficiency and assigned a specific ordering of sets to complete. The survey included a brief overview and instructions to prepare them for the test. 

A verbal explanation and tour of the 3 different sites were given to show the differences of each of the sets and warned that they were prohibited to use the Search function. Since there are three versions of the E-Maryland site, there are 6 orderings. Each subject will be given a specific ordering to balance the results so that each ordering will have 4 subjects. The task list was given and the test began once the subject was prepared. A subjective survey was also given after the user completed the tasks to determine their preferences of each version of E-Maryland and allowed them to add comments or suggestions for site improvement. 
 

2.5 Procedures 

Before the users begin the task, they are given an overview of the experiment and encouraged to find the links as quickly as possible. Users are presented with the version they are asked to use for the first set. The users are asked to find the four links per set. Subjects are timed for locating individual links. The group member conducting the test instructs aloud the link the subject must find, and points on task list the name of the hyperlink. After the subject clicks on the correct link (the site does not have to load) the time was stopped and recorded. If the subject does not find the link within a 3-minute time limit, the user will be asked to proceed with the next task. 

The subjects are asked to return to the home page in preparation to find the next link. After the first set the user is asked to name the 4 main categories on the home page of the E-Maryland site. The subject then continues to find the links of the next 2 sets. While the experiment took place, some general observations were made based upon the subjects’ verbal exclamations during the test and the actions they took in searching for the correct link. 
 

2.6 Problems 

The experiments proceeded fairly smoothly. We had a few problems with using consistent timing devices, since not all members had a stopwatch. Because of this, all times are rounded to the nearest second. We were also unable to conduct all experiments in the same lab, and therefore about 5 tests were conducted using outside computers, which may affect some of the final results. A few links in the task list also seemed to appear especially difficult to find since their name may have been ambiguous to the subjects. For instance the link “Travel Advisory”, many users perceived the term “travel” as in touring rather than commuting. 

Besides ambiguous terms, some links in the task list seemed to have been placed in questionable categories. For instance, “Online Take-Out and Delivery” appeared under “Culinary Events” whereas it seems more intuitive for the users to be under Dining or Restaurants. Many subjects failed to view “Disease Fact Sheets” as under “Emergency and Critical Services”. The links that were difficult to find may have affected the results since the subjects timed out. 

Several problems were encountered while creating the sites and refining the experiment. The original pages were dynamically created using java servlets which used links from a database. In order to create a controlled experiment, we had to recreate the pages, which turned out to be a tedious task given the large number of links in the site. Also, this means that the pages do not take into account the access delays for database queries, since the links are all static in the html code.