One from Column A and one from Column B: Simultaneous vs. Sequential Menus

Discussion

Task completion time data seems to indicate that the relative performance of sequential and simultaneous menus is dependent upon the type of task. Simultaneous menus provided significantly superior performance for questions that did not require multiple selections from any given menu (Type 1). For tasks that involved progressively greater amounts of backtracking, performance with simultaneous menus improved relative to that of sequential menus, leading to a statistically significant difference for Type 3 questions.

While these results provide reason to believe that simultaneous menus may be appropriate in certain contexts, we were surprised by the superior performance of sequential menus for type 1 and type 2 tasks. Artifacts of our experimental design may have influenced these results:

Screen Layout : For the sequential menus, each menu appeared in the screen in the space occupied by the previous menu. This layout minimized the mouse movement required: all tasks could be completed in the upper left quadrant of the screen. On the other hand, the simultaneous menus were displayed in a vertical column of three frames, occupying the entire left half of the screen. The extra movement required may have degraded performance. While simultaneous menus have the inherent problem of occupying greater screen space and thus requiring more mouse movement; alternative menu arrangements may reduce this impact. Experimentation with varying simultaneous menu layouts may provide interesting insights into the impact of screen layout. Alternatively, our data might be correlated with screen layouts in order to generate a model that accounts for the distance between targets.

Presentation of Task Questions : For both types of menus, subjects read task questions from a sheet of paper. This seemed to require significant effort, particularly for more complicated questions, as some participants shifted their gaze between screen and paper several times. Repetition of this experiment with a reduced cognitive load - possibly by placing task questions on the screen - might reveal greater differences between the menu types.

Familiarity of Menu Presentation Style : Since most of the participants were undergraduate and graduate students with substantial experience using web browsers, it seems reasonable to conclude that the sequential menu format was quite familiar and well understood. As simultaneous menus are far from common, there may be a learning effect involved in the results. In order to understand this effect, each of the authors performed the experimental tasks several times with each menu layout. For each combination of menu type and question, the fastest performance times were extracted to form a set of ``best'' times. Analysis of these times shows a clear benefit for simultaneous menus for Type 2 and Type 3 tasks. These results support the possibility of a training effect. Further research into the interaction between learning and menu type might provide clearer guidelines as to when simultaneous menus would be preferred over sequential menus.

Menu Item Ordering : The orderings of menu items is the second menu was not optimal. Instead of the alphabetic and numeric ordering used for the county and year menus, items in the industry menu were ordered according to the essentially random ordering used in original data tables taken from the Census Bureau Web Site . Although this ordering was used for both treatments, it may have caused the data to be disproportionately influenced by the increased difficulty of the second menu.

Task Choice : All three task types in this experiment involved closed-end questions with known answers. However, performance with simultaneous menus improved relative to sequential menus increased with the amount of backtracking required. Repetition of this experiment with tasks that involved more backtracking may lead to results that are more favorable towards simultaneous menus.

The results for the type 2 and type 3 questions indicate that some refinement of the model presented above is necessary. Specifically, it would seem that the choice time required for a menu in a simultaneous layout is greater than the time required for the same menu in a sequential layout: E_seq(n_i) < E_sim(n_i). Intuitively, this difference might be explained by the contents of the menu screens, as discussed above. Additional discrepancies might be accounted for by a model of menu choice time that accounts for learning time. For any given menu, the reading time should decrease with each subsequent return to the menu. Further investigation into these effects would be necessary to gain a deeper understanding of these factors. While some of this work may be possible with the data collected for this experiment, a separate experiment might be needed to generate accurate predictive models.

Simultaneous menus faired reasonably well on the post-test subjective questionnaire: ratings for the two types of menus were roughly comparable on all of the subjective questions. Since each subject used only one of the two menu presentation styles, a true preference comparison between the two styles is not possible. Further study involving within-subjects comparison of the two menu styles might clarify issues related to user preference while providing additional data for performance comparisons.