Programming Assignment 2

CSC 323 - Data Analysis and Statistical Software

Due: Section 101: 11/4/2002 & Section 401: 11/5/2002

 

You are a recent hire at a company that develops next-generation technologies for the consumer electronics industry. You have been assigned to the embedded/firmware software development team and will be working with DSP engineers on the final testing phase of a new product.
Note: See the What is Embedded Computing article (IEEE Computer, Jan 2002, vol 35, 1). Also, see the DSP FAQ if you are interested in knowing more about DSP's.

You have discovered that the product is a Surround Processor/Controller from a high-end manufacturer of audiophile products. The processor will replace an entry level product that was targeted at the competitive mid-fi market (i.e. MSRP: $1500 - $3500). The hope is that this new product will have a better reception than the previous product and will capture a significant segment of the market.

After several meetings with the DSP engineers you discover that the firmware for this product contains 50,000 lines of C code. In addition, the user interface for the product is a new design that replaces the on screen display (OSD) branching/tree menu system of the previous product. The expectation is that the new OSD will dramatically improve usability of the new product compared to the old.
Note: Here is a branching/tree menu system used in a popular high-end processor.

Initial feedback from an independent evaluation group has been critical of the new OSD interface. In particular, the evaluation group complains that the menu is error prone. They report that, on average, users make sixteen errors in completing a standard task suite. Furthermore, they recommend a thorough redesign of the interface.

The CEO has asked your team to respond to these findings. Your boss argues that the evaluation group did not take into account the target audience of the product and so the average error count should be much lower than reported. The CEO has directed your boss to support his comments with empirical evidence from a properly designed experiment. The CEO also suggests that if the average error count claimed by the evaluation group is reasonable then the interface must be redesigned before releasing the product to market. The CEO indicates that this will have an adverse impact on the profitability of the company.

You have been asked to assist with a controlled experiment. A randomly selected sample of users were observed working on the standard task suite used by the evaluation group. The task suite consists of six tasks, each of which involves navigation through at least three menu levels. The number of errors committed for each task was recorded.
Note: See the Usability Metrics essay and the Success Rate essay from Jakob Nielsens useit.com site for additional details on usability. Also, see the essay by Donald Norman entitled The Perils of Home Theater for general comments on standards and usability in a Home Theater setting.

You have been presented with the data collected for this experiment. Each observation in the file consists of the following values:

Notice that the error counts provided are for individual tasks in the suite. You are interested in total error count.

Note: Do not edit the data in any way. You must code your SAS program to read each observation as defined above.

Conduct a thorough analysis of these data. You will need to conduct a test of hypotheses and submit a report summarizing your findings. See additional details below.

  1. Write a SAS program to analyze this dataset. (40%)

    Your program should do the following:

    1. Read your data from an external file.
    2. Compute total error count.
    3. Execute the PRINT procedure.
    4. Use the appropriate SAS procedures to produce the statistics needed to conduct your hypothesis test.

    Note: For PROC PRINT, be sure to use labels for column headings rather than variable names. Use names for data sets and variables that are meaningful. You should generate an appropriate title for the output of these procedures.

  2. Write a short analysis (no more than two pages) of the output of your SAS program. (60%)

    Remember that your analysis is a test of hypotheses and so should at least address the following:

    1. State the primary hypotheses. That is, the NULL and ALTERNATIVE hypotheses for the experiment described above.
    2. Address the normality issue. That is, do you need to establish normality in order to address your primary hypotheses? Justify your answer.
      Note: If you think normality must be established, do not assume normality. Instead, state and address the normality hypotheses. By so doing, you will know if normality is reasonable.
    3. Determine the p-value for your primary hypotheses (i.e. compute the p-value as outlined in Step 3 of the Hypothesis Testing lecture notes). Remember to discuss the significance of the p-value obtained for the primary hypotheses.
    4. Given your findings, briefly explain why you should (or should not) provide a point estimate for the parameter in question. If you argue that a point estimate for the parameter is appropriate you must provide it as well as a 90% confidence interval for the parameter (with an interpretation).