Programming Assignment 2

CSC 323 - Data Analysis and Statistical Software

Due: Section 602: 5/29/2002 & Section 603: 5/28/2002

 

You are a recent hire at a company that develops calibration software for the high-end home theater industry. You have been assigned to the software development team that has just completed a product that competes with the leading product on the market but augments it with a DVD product similar to that produced by Ovation Software. You will be involved in testing the user interface of the DVD product.

Initial feedback from an independent evaluation group has been critical of the interface. In particular, the evaluation group complains that the menu is too complicated and is error prone. They report that, on average, users make twenty five errors in completing a standard task suite. Furthermore, they recommend a radically new interface.

The CEO has asked your team to respond to these findings. Your boss argues that the evaluation group did not take into account the target audience of the product and so the average error count should be much lower than reported. The CEO has directed your boss to support his comments with empirical evidence from a properly designed experiment. The CEO also suggests that if the average error count claimed by the evaluation group is reasonable then the interface must be redesigned before releasing the product to market.

You have been asked to assist with a controlled experiment involving forty-five users who were observed working on the standard task suite used by the evaluation group. The task suite consists of three tasks, each of which involves navigation through several menu levels. The number of errors committed for each task was recorded.
Note: See the Usability Metrics essay from Jakob Nielsens useit.com site for additional details on usability. Optionally, see the essay by Donald Norman entitled DVD Menu Design: The Failures of Web Design Recreated Yet Again for comments on DVD menu design.

You have been presented with the data collected for this experiment. Each observation in the file consists of the following values:

Notice that the error counts provided are for individual tasks in the suite. You are interested in total error count. Remember that you are only interested in users that completed the task suite (i.e. Status C). If necessary, see "DATA step statements", points 6 and 8, SAS Review.

Note: Do not edit the data to remove observations. Unwanted observations must be bypassed by using appropriate SAS statements.

Conduct a thorough analysis of these data. You will need to conduct a test of hypotheses and submit a report summarizing your findings. See additional details below.

  1. Write a SAS program to analyze this dataset. Your program should do the following:
    1. Read your data from an external file.
    2. Compute total error count.
    3. Ignore users that did not complete the suite.
    4. Execute the PRINT procedure.
    5. Use the appropriate SAS procedures to produce the statistics needed to conduct your hypothesis test.

    Note: For PROC PRINT, be sure to use labels for column headings rather than variable names. Use names for data sets and variables that are meaningful. You should generate an appropriate title for the output of these procedures.

  2. Write a short analysis (no more than two pages) of the output of your SAS program. Remember that your analysis is a test of hypotheses and so must address the following:
    1. State the primary hypotheses. That is, the NULL and ALTERNATIVE hypotheses for the experiment described above.
    2. Address the normality issue. That is, do you need to establish normality for your primary hypotheses. Justify your answer.
      Note: If you think normality must be established, do not assume normality. Instead, state and address the normality hypotheses. By so doing, you will know if normality is reasonable.
    3. Determine the p-value for your primary hypotheses (i.e. compute the p-value as outlined in Step 3 of the Hypothesis Testing lecture notes). Remember to discuss the significance of the p-value obtained for the primary hypotheses.
    4. Given your findings, briefly explain why you should (or should not) provide a point estimate for the parameter in question. If you argue that a point estimate for the parameter is appropriate you must provide it as well as a 90% confidence interval for the parameter (with an interpretation).