Statistics Used with Surveys
There are two basic goals that we try to accomplish when analyzing data from survey research.
Goals:
1 Describe the results obtained.
When doing survey research one simple goal is to describe the results we obtained.
In other words, we simply want to tell people what we found out.
2 Explore possible relationships among the variables measured.
Not only do we want to describe how people measured on each variable, but we also want to compare how variables are related to each other. In other words, we want to know if there is some relationship between one variable and another variable.
For example:
Are people's political beliefs related to their family values?
Are people's political beliefs related to their beliefs about deception?
Are students' preferences for the number of exams given related to how well they think they are doing in the class?
Describing Results of Surveys
For right now, let's only focus on the first goal. Simply describing the results obtained.
The goal is simply to describe to others how people filled out the survey. Simply trying to describe the responses they gave.
How could you do this? Well, one way would be to provide a copy of everyone's response so that everyone could see how everyone filled out their questionnaire. However, this is often impractical and somewhat meaningless. It is rather hard to get a sense of what was going on by looking at how every individual filled out their survey.
So, what we do is report the results obtained using various descriptive statistics that we have covered before.
In other words, we describe people's responses in terms of the mode, median, and mean.
1. Determine if using a single item or multiple item questionnaire.
Single item questionnaires: One question or item was used to measure a variable.
Multiple item questionnaires: Two or more questions were used to measure a variable.
a. Single Item Questionnaire
1. Always report how many people filled out the survey.
(n = number of people)
2. Report the appropriate measure of central tendency for each variable.
If nominal measure - provide the mode - also helpful to provide frequency or percentage of people placed into each category.
If ordinal measure - provide the median
If interval measure - provide the mean and standard deviation*
If ratio measure - provide the mean, standard deviation, and range*
* note: If distributions of scores are bimodal then report the modes. If distributions contain extreme scores (very high or very low scores - not near other scores) then report the median.
b. Multiple item questionnaire.
With a single item questionnaire, it was very easy to describe the results. But with a multiple item questionnaire, we run into some problems.
How do you report the results of a multiple item questionnaire?
1. Always report how many people filled out questionnaire.
2. Create a score for each person by adding together their responses on the questions or items measuring a given variable.
So, I could create a score for each person in terms of their political beliefs by adding their scores on all of the items that measure political beliefs.
However, how do I know that these multiple items are really measuring the same thing. That is, what if one of the items isn't written very well, or actually measures something other than what I intended, well then I am going to be in trouble.
Two solutions are used to determine how well items measure a variable when using multiple item questionnaires.
3. Must first determine which items should be added together.
Can't just assume that since we wrote the question to measure a person's -- say political beliefs that the question actually does that. It may be measuring something completely different. So, we need to find out which items should be added together, and which items should be dropped. Need to find the good and the bad questions, somehow.
One way to do this is through a statistical analysis called:
a. Item analysis
The purpose of item analysis is to identify the items that are measuring the same variable and eliminate the items that are not.
In other words, before summing together results, let's make sure that each item is measuring the same thing.
An item analysis simply identifies how well a set of items are measuring the same construct.
In other words, it identifies the items that are measuring the same variable, and also points out the items that are somewhat off target.
Item analysis tells you two things.
1. Identifies which items to keep and which items to drop.
2. Describes how consistent the kept items are with each other.
Cronbach's alpha or coefficient alpha. Measure of how consistent or reliable the items are with each other.
Ranges from 0 to +1
0 means that the group of items are not measuring the same variable.
+1 means that the group of items are 100 percent consistent with each other. They are measuring the exact same thing.
However, it is almost impossible to write questions that are measuring exactly the same thing. So, we do not set 100 Percent consistency as our goal.
In general, a cronbach's alpha of .70 or better is considered acceptable.
If a group of items as a cronbach's alpha of .70 or better, then it is acceptable to add up the score on all of the items to form a single measure of that variable.
b. Factor analysis
Another way of determining which items should be added together is to conduct another statistical analysis, called a factor analysis.
Factor analysis is very different from item analysis, in that in an item analysis, we see how well our set of items measured a given variable.
In other words, we said how well do these 10 items measure the same thing (Item analysis).
However, with factor analysis we simply say, there are 37 items. How many different variables or factors are these items really measuring.
In other words, we use factor analysis to determine how many variables a set of items are measuring.
The basic idea behind factor analysis, is to identify groups of items that measure the same construct.
So, if we did a item analysis on these 37 items, we hope that the item analysis would tell us that we were measuring five different variables.
However, a factor analysis, might simply tell us that we were measuring two variables, three variables, or a dozen different variables.
Factor analysis tells you several things.
1. Identifies the number of variables actually being measured.
2. Identifies which items should be grouped together.
So, if we did a factor analysis on these items. We might find that ten of the items measured one variable, ten of the items measured another variable, and ten of the items measured another variable and the remaining items really don't belong together.
In other words, maybe our questionnaire really only measure three different variables.
However, factor analysis does not tell you what the variables are conceptually. Factor analysis, will say that these five items belong together as a group, and they measure the same thing, but the researcher must interpret what they are measuring.
3. Interpretation of variables being measured is a judgment call.
So, when you do a factor analysis, it might tell you that these ten items belong together, but then you have to explore the items and try to come up with a common theme underlying them. You must decide what it is they are actually measuring.
Once we know which items group together or measure the same concept, then we can simply add the responses on those items together for each person.