EDF600 – Research Methods
Lola Aagaard’s Notes
Chapter 7 (8th) / Chapter 11 (7th)
Correlational Designs
I. Correlational studies – include both correlation and prediction studies.
A. Correlation = how two or more quantitative variables are related to each other. Every person in your study MUST have data for each variable. Are changes in one variable associated with changes in the other(s)? (Do they vary together?)
For example, is verbal ability correlated with grades in high school English? You might expect that kids with higher verbal ability would get higher English grades, right? And kids with lower verbal ability would get lower grades. If that is the case, then there is a correlation between verbal ability and English grade – the two things vary (go up or down) together.
IMPORTANT POINT: CORRELATION DOES NOT INDICATE CAUSATION! Just because two things are correlated does not mean that one caused the other. There might be a third variable that you don’t know about that is causing both things to vary. There was a study in the last several years on the use of antibiotics and breast cancer, for instance – more antibiotic use was linked to more breast cancer. What else might explain that relationship? Maybe people who got sick a lot (and needed antibiotics) had weaker immune systems and thus were also more susceptible to cancer.
The number of children born in the United States is related to the number of roads built in Europe. So, if their roads are in need of more work, should we have more kids over here? Or if we have a population explosion will it result in the paving of all of Europe? No, of course not. The world economy is probably behind both things – in good economic times people have more babies and there is money to build roads.
The only way you can ever conclude that one thing caused something else is if you did an experimental study (we’ll talk about those in the next chapters). So if you find a high correlation in one study, you would have to follow-up with experimental research to be able to say one thing caused the other. You can speculate in your discussion of correlation results about why the two things are related, but suggestions for intervention or changes in instruction based on correlations are not appropriate. (That doesn’t mean that people don’t jump to conclusions all the time based on correlations – they do! But they shouldn’t!)
1. Positive (or direct) correlation = two variables both go up at the same time or they both go down at the same time. As long as they go the same direction at the same time, it is a positive or direct correlation. Verbal ability and English grade is an example. Hours of studying and your grade in EDF600 (ha!). Babies born in the U.S. and number of roads built in Europe. The length of your foot and your shoe size.
2. Negative (or inverse) correlation – If the two variables go in opposite directions (one goes up when the other goes down), then the correlation is negative or inverse. Example: Amount of time spent watching T.V. the night before midterms and midterm exam score. Probably the more T.V. you watch the night before your midterm, the lower your midterm score will be (because you didn’t study). And vice versa, if you didn’t watch T.V. your score will probably be higher. The number of years of age past 50 and your ability to remember things. (Old age is when you spend more time thinking about the here-after. You walk into the other room and think, “Now, what was I here after?”) The number of hours of sleep the child got the night before and the number of times they get distracted from their schoolwork the next day (less sleep is related to more distractions – in fact, the symptoms of sleep deprivation can mirror those of ADD).
3. No correlation – change in one variable is not consistently accompanied by change in the other variable. Your age and your phone number. Your shoe size and your IQ score. Your waist measurement and your score on the EDF600 midterm. There is no relationship between these things.
B. Correlation coefficient – when correlation is calculated, the resulting statistic is called a correlation coefficient.
1. Range of value -- It ranges from –1.0 through 0 and on up to +1.0. A perfect positive correlation would be +1.0, while a perfect negative would be –1.0. No correlation = 0. So the farther away the coefficient is from zero in EITHER direction, the stronger the correlation (the more related the variables are to each other). A correlation of –0.7 is a stronger relationship than a correlation of 0.65 because 0.7 is farther from zero than 0.65.
2. What is considered high or low? – It depends on the specific circumstance, but a rule of thumb is:
0-.35 (either positive or negative) – low or no correlation
.35-.65 – moderate
.65 or more – strong
But it really makes a difference what you are looking at. Another way to interpret correlation is through the coefficient of determination.
3. Coefficient of determination – this is simply the square of the correlation coefficient (the correlation times itself). It gives you the percent of common or shared variance in the variables – how much they overlap each other. Or if you know one variable, how much of the other would you be able to predict?
So a correlation of .30 squared (.30 x .30 = 0.09 – convert that to a percent by multiplying by 100) equals only 9% shared variance – not very much. That is why .3 is considered a low correlation. A negative .30 would have the same overlap between variables, because a negative times a negative equals a positive (-.3 x -.3 = 0.09 also).
A correlation of .90 (or -.90) indicated 81% shared variance – a high amount of overlap between the variables.
If you know what Venn diagrams are (overlapping circles), you can draw them to illustrate this. I don’t know how to do it in Word….
4. Types of correlation coefficients – there are two common types for linear correlations, plus several others less common.
a. Pearson’s r (product-moment coefficient) – this is the most commonly used one and sometimes a research article won’t even specify its use, but just say that the variables were correlated. It is appropriate when both variables are interval or ratio level data. (Remember the levels of measurement from a previous chapter?) So correlating your GRE score with your graduate school GPA would use a Pearson’s.
b. Spearman’s rho -- The Spearman is used when at least one of the variables is ordinal level (ranked) rather than interval/ratio data. If you were going to correlate ACT scores with class standing (ranking), then it would be appropriate to use a Spearman rather than a Pearson.
c. Point biserial -- allows you to correlate an interval/ratio variable with one that only has two values (dichotomous) – for instance, you could correlate gender with ACT that way (assign numeric values to gender – 0=male; 1=female -- and run the correlation).
C. Sample size and significance – remember that a large sample size is like looking through a microscope – it makes small things seem huge and significant. If you have 1000 people, even correlations below .10 would be statistically significant. That just means you have confidence that there are real (albeit very small!) correlations between the variables. But that doesn’t mean you should care! No matter what the table (A-2 in the back of your text) tells me about significance, a correlation of 0.19 means the two variables overlap less than 4%, so I’m generally not interested.
The reverse is true, also. With a small sample size (looking through the wrong end of the binoculars), even a strong correlation is not seen as statistically significant – with only 5 people in your sample you can’t claim a correlation of 0.750 as significant at the .05 level. (We’ll get into what “.05 level” means in a later chapter. Don’t worry about it for now.)
D. Plotting correlation data – you can plot your data and get a visual eye estimate of the correlation (it has “inter-ocular impact” – it hits you between the eyes). Put one variable on the X-axis and the other on the Y. With positive correlations, the dots rise from the lower left to the upper right. Negative correlations fall from the upper left to the lower right. Weakly correlated data look like a big mess.
You may be able to spot outliers in plots of moderately to strongly correlated data – just some points that don’t seem to be following the general trend.
Most correlational techniques assume that any relationship between variables is going to be linear (straight line). If you have two variables that are related in a curvilinear way, a regular correlation coefficient will give you too low an estimate of the relationship. Plotting will help you determine the shape.
Curvilinear examples: dollar value of car and age of car – value starts high and drops as soon as you leave the lot, continues dropping until car is so old it could be considered antique, then value starts climbing again. Anxiety and achievement – zero anxiety is not very conducive to learning, a little anxiety is associated with increases in achievement, but at some point it becomes counter-productive and achievement starts dropping. There are specific types of correlation that will detect curvilinear relationships (eta ratio).
E. Shot-gun approach, fishing expedition, treasure hunt -- these are terms that refer to the lazy and non-theoretical approach to correlational studies where you measure your participants on 20 variables and correlate all of them to see what turns up. This approach is not recommended because it is not based on hypotheses and often the correlations that do show up can’t be confirmed in another sample (because they were based on error and chance). Repeated studies of variables that appear correlated are recommended in order to have confidence in the conclusions.
F. Need for variability -- It is best if the variables actually have quite a bit of variability to them! If everyone has nearly the same value for one variable, you won’t get much of a correlation. Take the example of ACT and grades. If you look at that relationship the freshman year in high school, you get a stronger correlation than if you look at it in graduate school. Why? Because of the restriction of range of the data – the lower ACT students didn’t go on to graduate school, so the variability of the data has reduced, thus lowering the correlation. (Imagine a scatter plot of 9th grade data, then erase the lower ACT scores to show the remaining points. You go from a decent positive correlation to a blob showing little or no correlation.)
G. Measurement validity and reliability is EVERYTHING when doing correlational studies. If you have no confidence in your measures, then the conclusions based on the correlations of those measures will be worthless.
Instruments with low
reliabilities result in lower correlations. Invalid instruments mean your
results are meaningless.
H. Regression (prediction) studies – these studies are done when you want to predict one variable based on the value of another one with which it is correlated. When you plot your data you can see that there is a pattern to it (assuming a fairly strong relationship). It is that pattern that allows prediction. So if you knew a participant’s verbal ability score (the predictor variable) you might be able to predict their English grade (the criterion variable) based on the pattern of previous scores.
The statistical technique of regression essentially draws a line through the center of the plot to represent an average (“best fit”) relationship between the variables. The regression line is a prediction line – if you wanted to predict the value of the second variable given the first, you would use the equation for the regression line, because it is the “line of best fit” for your data.
The more highly correlated your variables, the better your prediction will be (as you could see from the shared variance estimates). If you want to be even more accurate, add more predictor variables (called multiple regression).
It is quite common in educational research to try and predict a school’s average achievement scores. All kinds of predictors are put in those equations, from the percentage of kids on free lunch, to teachers’ years of experience, to the value of land in the county. (That would be a multiple regression because there are more predictors than just one.)
EDF600 – Research Methods
Lola Aagaard’s Notes
Chapter 8 (8th) / Chapter 12 (7th)
Causal Comparative Designs
I. Causal-comparative (also called ex post facto – after the fact)
A. Uses existing groups –
1. the researcher does not decide who is in which group (no random assignment to groups); but you can randomly select participants from the groups for the study sample.
2. the researcher does not decide which group gets a treatment – there isn’t any treatment to give because this is essentially descriptive research, also. The groups already differ on the independent variable – the “treatment” has already happened (this is called “no manipulation of the independent variable”). The researcher just takes advantage of the naturally occurring groups.
Causal comparative studies deal with independent variables that
a. cannot be manipulated – comparing groups based on gender,
age, SES, grade in school
b. should not be manipulated for ethical reasons – children’s reaction to various hyperactivity drugs (you don’t want to take kids who need the drug off of it if they get assigned to the “no drug” group, nor do you want to give the drugs needlessly to kids who don’t need them but get assigned to the experimental group; fitness of people who run marathons vs. being couch potatoes (could you start running marathons tomorrow without dying of a heart attack if assigned to the experimental group?),
c. are difficult to manipulate for various reasons – sometimes political considerations come into play. Everyone wants their kid in the classroom with the new type of instruction or the computers or whatever. When parents see the informed consent form, they start calling, wanting their kid in the group that gets the new stuff. It makes it hard on the school to resist the pressure and hard for the researcher who wanted to randomly assign kids to groups when the pressure prevails.
Example 1: A researcher looks at teachers with different kinds of certification (you can’t manipulate that or randomly assign to it – they already have the certification or they don’t), and how their students score on the CATS test.
Example 2: Someone conducts a study of the attitudes toward school of children from families where parents are highly involved in their children’s education compared to children from families where parents are minimally involved, if at all. You could randomly assign families to be more or less involved, but probably shouldn’t. What if a highly involved family got the “no involvement” assignment and their child lost ground that year? It really isn’t ethical to destroy what is already going well for a student! Plus it would be difficult to get an uninvolved family to get involved just because they were assigned to that group of the study.
B. Cannot establish cause-effect – only experimental designs can claim to do this. The results of causal-comparative studies should be interpreted like those from correlational studies, because some other variable might be behind the apparent connection you see in your study.
For instance, say you looked at type of high school teacher training (standard undergraduate Teacher Education Program [TEP] vs. the alternative Masters of Art in Teaching [MAT]) and student achievement. Your data indicated that students of teachers educated one of these ways vastly outscored the students of teachers educated the other way. You still could NOT conclude that it was the training of the teachers that caused the difference in student scores, because this was NOT an experimental design – you did NOT randomly assign people to be trained one way or the other.
You could say that the data showed the two groups of students had significantly different achievement levels. You could say that teacher training was related to student achievement and you could speculate about why that might be, but you would have to be very careful not to make causal statements.
C. Difference from correlational design – causal-comparative studies are still looking at relationship in a way, but a different way than with correlational studies. The problem statements for causal comparative studies will talk about finding differences between groups rather than relationships between variables. But the many of the same topics can be studied either way.
For example, the certification study in Example 1 on the previous page could be done in a correlational manner. Assign ordinal numeric labels to the type of certification – say, 0 for emergency, 1 for provisional, 2 for fully certified. Then use a Spearman correlation to look for relationships between certification and the average student test score for each teacher.
Or a correlational study between years on free lunch and student test scores could be turned into a causal-comparative study by dividing the students into groups based on one of the variables. For instance, make 2 groups of students (0-2 years of free lunch and 3 + years) and then see if the groups have different test scores.
D. How to make the best of it – The biggest weakness in causal-comparative designs is the inability to randomly assign to groups (to ensure the groups are as similar as possible aside from the independent variable) – random assignment limits the danger of confounding variables. But you can help assure that in other ways. Collecting information on many different variables, whether demographic, academic, experiential, etc. can help you determine whether your groups are similar on extraneous (confounding) variables. The more similar they are, the stronger the study. Here are some other ways to do that:
1. Matching – pair up participants, one from each group, based on their similarity on a confounding variable (or variables) of interest. If years of experience is a possible confounding variable in the teacher certification study, then see if you can match up the groups on that variable. Of course, there might not be very many highly experienced teachers in the provisional certification group, and thus highly experienced teachers might have to be eliminated from the study altogether. This is a problem! For other types of things, matching may work very well.
For instance, in a study of family involvement and student achievement, you might want to match the “involved” and “uninvolved” groups by the grade in school of their child. Parents tend to be more involved at the elementary level than at the high school level, so it would confuse things if all the involved were elementary families and all the uninvolved were high school families.
2. Make subgroups and control by adding that variable to the design -- If you think level of parental involvement might be a confounding variable in the study of years on free lunch and student test scores, collect information on it and divide all the families up with respect to level of parental involvement. It is like having a second independent variable, but it is called a control variable. That would give you a two-dimensional analysis – little or lots of years on free lunch crossed with high, medium, and low involvement.
3. Analysis of covariance – a purely statistical way of controlling extraneous variables. It allows you to statistically equate the groups on the variable of interest, so when you analyze your free lunch data it is as if they all had the same value for parent involvement.