Programming Assignment 2

CSC 323 - Data Analysis and Statistical Software

Due: 7/17/2002

 

You are a recent hire at a company that develops next-generation technologies for the consumer electronics industry. You have been assigned to the embedded/firmware software development team and will be involved in the final testing phase of a new product.

You have discovered that the product is a Surround Processor/Controller from a high-end manufacturer of audiophile products. The processor will replace an entry level product that was targeted at the competitive mid-fi market (i.e. MSRP: $1500 - $3500). The hope is that this new product will have a better reception than the previous product and will capture a significant segment of the market.

After several meetings with the DSP engineers you discover that the firmware for this product contains 50,000 lines of C code. In addition, the user interface for the product is a new design that replaces the on screen display (OSD) branching/tree menu system of the previous product. The expectation is that the new OSD will dramatically improve usability of the new product compared to the old.
Note: Here is a branching/tree menu system used in a popular high-end processor.

Initial feedback from an independent evaluation group has been critical of the new OSD interface. In particular, the evaluation group complains that the menu is error prone. They report that, on average, users make twelve errors in completing a standard task suite. Furthermore, they recommend a thorough redesign of the interface.

The CEO has asked your team to respond to these findings. Your boss argues that the evaluation group did not take into account the target audience of the product and so the average error count should be much lower than reported. The CEO has directed your boss to support his comments with empirical evidence from a properly designed experiment. The CEO also suggests that if the average error count claimed by the technology group is reasonable then the interface must be redesigned before releasing the product to market. The CEO indicates that this will have an adverse impact on the profitability of the company.

You have been asked to assist with a controlled experiment involving thirty eight users who were observed working on the standard task suite used by the evaluation group. The task suite consists of four tasks, each of which involves navigation through at least three menu levels. The number of errors committed for each task was recorded.
Note: See the Usability Metrics essay from Jakob Nielsens useit.com site for additional details on usability. Optionally, see the essay by Donald Norman entitled The Perils of Home Theater for general comments on standards and usability in a Home Theater setting.

You have been presented with the data collected for this experiment. Each observation in the file consists of the following values:

Notice that the error counts provided are for individual tasks in the suite. You are interested in total error count. You are also only interested in New users (i.e. User Type N). Expert users were included in the experiment for control purposes only. If necessary, see "DATA step statements", points 6 and 8, SAS Review.

Note: Do not edit the data to remove observations. Unwanted observations must be bypassed by using appropriate SAS statements.

Conduct a thorough analysis of these data. You will need to conduct a test of hypotheses and submit a report summarizing your findings. See additional details below.

  1. Write a SAS program to analyze this dataset. (40%)

    Your program should do the following:

    1. Read your data from an external file.
    2. Compute total error count.
    3. Ignore expert users.
    4. Execute the PRINT procedure.
    5. Use the appropriate SAS procedures to produce the statistics needed to conduct your hypothesis test.

    Note: For PROC PRINT, be sure to use labels for column headings rather than variable names. Use names for data sets and variables that are meaningful. You should generate an appropriate title for the output of these procedures.

  2. Write a short analysis (no more than two pages) of the output of your SAS program. (60%)

    Remember that your analysis is a test of hypotheses and so should at least address the following:

    1. State the primary hypotheses. That is, the NULL and ALTERNATIVE hypotheses for the experiment described above.
    2. Address the normality issue. That is, do you need to establish normality in order to address your primary hypotheses? Justify your answer.
      Note: If you think normality must be established, do not assume normality. Instead, state and address the normality hypotheses. By so doing, you will know if normality is reasonable.
    3. Determine the p-value for your primary hypotheses (i.e. compute the p-value as outlined in Step 3 of the Hypothesis Testing lecture notes). Remember to discuss the significance of the p-value obtained for the primary hypotheses.
    4. Given your findings, briefly explain why you should (or should not) provide a point estimate for the parameter in question. If you argue that a point estimate for the parameter is appropriate you must provide it as well as a 90% confidence interval for the parameter (with an interpretation).