Website Structural Navigation

Experiment

Introduction and Hypothesis

The heart of the experiment is the timed test for three websites. Two different versions of each site was created—one with navigation bars and one without. Users are asked to travel from one page to another within a website. The three sites chosen were selected sections from Yahoo!, AltaVista and C|Net. After the instructions are explained, the time begins and the subject navigates within that website to their destination. For purposes of this experiment, all links that take the user in the wrong direction were changed to a dummy page that told them to go back. This was necessary because the pages were copied to this server (to allow for the nav bar changes) and random links off this server would get users lost. The search engines were also disabled.

The base (placebo) users would be anyone who tested a site with nav bars. The experiment will compare those times to the users who tested a site without navbars.

The second part of the experiment is the subjective survey which asks the testers whether and why they believe the navigation bars are useful or not.

Based on intuition and asking people, we are hypothesizing that there would be very little time difference between the sites that have a navigation bar and those that do not. Most people probably will just go straight to the site's homepage and ignore the bar if it exists.

Variables

The independent variable for this experiment was the inclusion or exclusion of a nav bar in each test site. A random number generator in Perl decided which site to choose for every task.

There are five dependent variables from the experiment:

1. The time the subject took to complete each task. This was done in Perl by subtracting the start time (recorded by a cookie) from the end time.
2. The possibility that the subject 'gave up' on a certain task. The time for these is not calculated into the completion time graph.
3. Internet connection speed. Most users had a direct internet connection, others had 56.6K modems that would not significantly decrease the test time at all.
4. Experience on the web. It can be assumed that advanced users will be more familiar with navigation and work faster.
5. Browser (type and version), operating system, processor speed, memory load, monitor size, and any other computer-specific options. Only the browser and OS were recorded. The timing differences between the other specifications should average out, so they were not taken into account as error rates.
   
   

Pilot Study Results

The pilot study consisted of four people, all semi-proficient with computers and the web, to try the test site. All of the Perl programming and the tasks worked fine, but a few changes were made according to their input.

The bottom frame for each of the tasks originally explained its instructions and contained a link displaying "I'm There!" that users were supposed to click if they reached their destination. We found that the subjects were pressing that link prematurely or not at all when they reached their destination. Also some people were lost and just could not complete a task. A change was therefore made so that the link to the actual destination page performed the same function as the former "I'm There!" button. That link on the bottom frame was then changed to "I Give Up". to allow someone to simply skip the current test.

The instruction pages were made less confusing by spreading them out from one long paragraph to an indented list. Some more instructions were added to help the user know more about what they will be doing. Also, the phrase "(Campus Ethernet)" was added to the description for their connection information question because many people did not know what a T1 line was.

One final change was based on navigation difficulties. Without the nav bars, some of the pilot study subjects did not know where to go at all. That is because the links to each of the three homepages were not clearly marked (consider it bad design on the original sites). This problem came up especially on the C|Net experiment. The word "Homepage" or "Home" was therefore added to the start page of all three sites. This way it makes it a little more obvious where the main homepage link is.

Subjects

For the bulk of the subjects, an e-mail was sent to the "party list" for the Maryland Marching Band (which Noah is a member of) to ask for help. Other messages were sent to various friends and family as well. We asked them to run the tasks on their own computer.

The resulting subjects, 67 in all, consisted of almost entirely college students. Only three adults older than 50 years old completed the tasks. Users were asked how experienced they are with the World Wide Web. The results are:

Almost everyone had a direct (Ethernet) connection to the Internet. There were only five subjects using Macintosh computers and just one person used Linux. Everyone else was using a flavor of Windows (95, 98, NT4 or NT5). Since there is little difference in parsing times between browsers and operating systems, no load-time error rates were taken into effect.

Materials

Most timed tests on website usage are conducted in fixed laboratory environments where every test subject is given an identical view of the pages. However, in the real world, no two web browser displays are identical. Web pages can be designed for any specific type of computer, browser and resolution, but are open to any level of interpretation by the other types. This project allowed users to test the sample websites on ANY computer screen. Therefore, the actual end-user material is a computer with an Internet connection and web browser.

The opening questionnaire is consisted of asking how they are connected to the Internet and how familiar they are with the web. It can be found here. The subjective survey at the end of the experiment can be seen here.

All of the internal programming was done in HTML and Perl. Every page sent an e-mail to Noah when processed. There was one e-mail for every page, so five messages per test were sent (intro questions/consent, three experiments and subjective survey). The result e-mails can be seen in the appendices.

Procedures

The procedure for Task #1 is to travel from the web page index of Antarctica to that of Australia's Military in AltaVista. Without navigation bars, the user must go to Altavista's homepage, then Regions, then Australia, then Military. With navigation bars, the 'Regional' link in the bar can be clicked to save two steps.

Task #2's procedure is to navigate from the High School Cross Country page on Yahoo! to the section on Trout Fishing. This can be done by going to the site's homepage, then to Recreation (or Sports), then Fishing, then Trout Fishing. Navigation bars allow the subject to go straight to Sports.

The procedure for Test #3 is to go from the ICQ 2000a Software Review on C|Net to the Eudora 4.3 Review. This is accomplished by going to the homepage, then Internet, then Email, then Eudora 4.3.

Problems

After the experiment was completed, some people came forward that they cheated. Since all links were changed to "dummy.cgi" except the real one, a very few subjects just scrolled their mouse cursor over the links and looked in the status bar for a non-dummy one. One person even went to the HTML source code to find the real link.

Another problem with the experiment was the use of C|Net as a test site. The navigation for this website is very difficult to understand, especially when the nav bar was disabled. Even after the link changes mentioned above in the Pilot Results, the site managed to confuse many people.

Other than that, there were no major problems to deal with during the experiment.