Programming Assignment 2

CSC 323 - Data Analysis and Statistical Software

Due: 11/5/2003

 

You are a recent hire at a company that develops next-generation technologies for the consumer electronics industry. You have been assigned to the embedded/firmware software development team and will be working with DSP software engineers on the final testing phase of firmware for a new product.
Note: See the What is Embedded Computing article (IEEE Computer, Jan 2002, vol 35, 1) and the DSP FAQ if you are interested in knowing more about embedded computing and DSP's. Also, the paper The Rise of Embedded Media Processing (Analog Devices) presents an interesting, and informative, perspective on Embedded Computing.

You have discovered that the product is the new flagship Surround Processor for a high-end manufacturer of audio/video products. You have also discovered that the firmware for this product contains 35,000 lines of C++ code. In addition, the user interface for the product is a standard on screen display (OSD) branching/tree menu system.
Note: Here is a branching/tree menu system used in a popular high-end processor.

Initial feedback from an independent evaluation group has been critical of the OSD interface. In particular, the evaluation group complains that the menu is too complicated and is error prone. They report that, on average, users make 11.5 errors in completing a standard task suite. Furthermore, they recommend a radically new interface.

The CEO has asked your team to respond to these findings. Your boss argues that the evaluation group did not take into account the target audience of the product and so the average error count should be much lower than reported. The CEO has directed your boss to support his comments with empirical evidence from a properly designed experiment. The CEO also suggests that if the average error count claimed by the technology group is reasonable then the interface must be redesigned before releasing the product to market.

You have been asked to assist with a controlled experiment involving several users who were observed working on the standard task suite mentioned by the evaluation group. The task suite consists of five tasks, each of which involves navigation through at least three menu levels. The number of errors committed for each task was recorded.
Note: See the Usability Metrics essay from Jakob Nielsens useit.com site for additional details on usability. Optionally, see the essay by Donald Norman entitled The Perils of Home Theater for general comments on standards and usability in a Home Theater setting.

You have been presented with the data collected for this experiment. Each observation in the file consists of the following values:

Notice that the error counts provided are for individual tasks in the suite. You are interested in total error count. Remember that you are only interested in users that completed the task suite (i.e. Status C). If necessary, see "DATA step statements", points 6 and 8, SAS Review.

Note: Do not edit the data to remove observations. Unwanted observations must be bypassed by using appropriate SAS statements.

Conduct a thorough analysis of these data. You will need to conduct a test of hypotheses and submit a report summarizing your findings. See additional details below.

  1. Write a SAS program to analyze this dataset. (50%)

    Your program should do the following:

    1. Read your data from an external file.
    2. Compute total error count.
      Note: You will need an assignment statement in your data step.
    3. Ignore users that did not complete the suite.
    4. Execute the PRINT procedure.
    5. Use the appropriate SAS procedures to produce the statistics needed to conduct your hypothesis test.

    Note: For PROC PRINT, be sure to use your defined labels as column headings rather than variable names. Use names for data sets and variables that are meaningful. You should generate an appropriate title for the output of these procedures.

  2. Write a short analysis (no more than two pages) of the output of your SAS program. (50%)

    Remember that your analysis is a test of hypotheses and so should at least address the following:

    1. State the primary hypotheses. That is, the NULL and ALTERNATIVE hypotheses for the experiment described above.
    2. Given your hypotheses, complete the following:
      1. Address the normality issue. That is, is do you need to establish normality in order to address your primary hypotheses? Justify your answer.
        Extra Credit: If you think normality must be established, show how you may use SAS to do this without assuming normality. That is, state and address the normality hypotheses and then use SAS to determine the normality p-value.
      2. Determine the p-value for your primary hypotheses (i.e. compute the p-value as outlined in Step 3 of the Hypothesis Testing lecture notes). Remember to discuss the significance of the p-value obtained for the primary hypotheses.
    3. Given your findings, briefly explain why you should (or should not) provide a point estimate for the parameter in question. If you argue that a point estimate for the parameter is appropriate you must provide it as well as a 90% confidence interval for the parameter (with an interpretation).