Programming Assignment 2

CSC 323 - Data Analysis and Statistical Software

Due: 3/5/2002

 

You are a recent hire at a company that develops next-generation technologies for the consumer electronics industry. You have been assigned to the embedded/firmware software development team and will be involved in the final testing phase of a new product.

You have discovered that the product is the new flagship Surround Processor for a high-end manufacturer of audio/video products. You have also discovered that the firmware for this product contains 35,000 lines of C++ code. In addition, the user interface for the product is a standard on screen display (OSD) branching/tree menu system.
Note: Here is a branching/tree menu system used in a popular high-end processor.

Initial feedback from an independent evaluation group has been critical of the OSD interface. In particular, the evaluation group complains that the menu is too complicated and is error prone. They report that, on average, users make thirteen errors in completing a standard task suite. Furthermore, they recommend a radically new interface.

The CEO has asked your team to respond to these findings. Your boss argues that the evaluation group did not take into account the target audience of the product and so the average error count should be much lower than reported. The CEO has directed your boss to support his comments with empirical evidence from a properly designed experiment. The CEO also suggests that if the average error count claimed by the technology group is reasonable then the interface must be redesigned before releasing the product to market.

You have been asked to assist with a controlled experiment involving forty-five users who were observed working on the standard task suite mentioned by the evaluation group. The task suite consists of five tasks, each of which involves navigation through at least three menu levels. The number of errors committed for each task was recorded.
Note: See the Usability Metrics essay from Jakob Nielsens useit.com site for additional details on usability. Optionally, see the essay by Donald Norman entitled The Perils of Home Theater for general comments on standards and usability in a Home Theater setting.

You have been presented with the data collected for this experiment. Each observation in the file consists of the following values:

Notice that the error counts provided are for individual tasks in the suite. You are interested in total error count. Remember that you are only interested in users that completed the task suite (i.e. Status C). If necessary, see "DATA step statements", points 6 and 8, SAS Review.

Note: Do not edit the data to remove observations. Unwanted observations must be bypassed by using appropriate SAS statements.

Conduct a thorough analysis of these data. You will need to conduct a test of hypotheses and submit a report summarizing your findings. See additional details below.

  1. Write a SAS program to analyze this dataset. Your program should do the following:
    1. Read your data from an external file.
    2. Compute total error count.
    3. Ignore users that did not complete the suite.
    4. Execute the PRINT procedure.
    5. Use the appropriate SAS procedures to produce the statistics needed to conduct your hypothesis test.

    Note: For PROC PRINT, be sure to use labels for column headings rather than variable names. Use names for data sets and variables that are meaningful. You should generate an appropriate title for the output of these procedures.

  2. Write a short analysis (no more than two pages) of the output of your SAS program. Remember that your analysis is a test of hypotheses and so should at least address the following:
    1. State the primary hypotheses. That is, the NULL and ALTERNATIVE hypotheses for the experiment described above.
    2. Address the normality issue. That is, is do you need to establish normality for this analysis. Justify your answer.
      Note: If you think normality must be established, state and address the normality hypotheses.
    3. Determine the p-value for your primary hypotheses. Remember to discuss the significance of the p-value obtained for the primary hypotheses.
    4. Given your findings, briefly explain why you should (or should not) provide a point estimate for the parameter in question. If you argue that a point estimate for the parameter is appropriate you must provide it as well as a 90% confidence interval for the parameter (with an interpretation).