SAT statistical claims: what a study can conclude
Two questions decide what a study can claim: who was chosen, and how they were assigned. Random selection widens whom a result describes; random assignment supports cause.
These are the concepts this skill rests on, the same pages as in the lesson.
Two Kinds of Study
Observational Study or Experiment
In an observational study, the investigators just record what people (or animals, or things) already do or already have. In an experiment, the investigators impose a condition: they decide which participants get which treatment.
A researcher who compares the test scores of students who already take music lessons with those of students who don't is running an observational study. A researcher who decides which students take music lessons, and then compares their scores, is running an experiment.
The group that gets the treatment is the treatment group. A group that doesn't get it is a comparison group, also called a control group.
Who Decided the Condition?
One question tells you the type of study: who decided which group each participant is in? If the investigators decided, it's an experiment. If the participants chose, or the groups already existed, it's an observational study.
Three things do not make a study an experiment:
- Size, length or care. Recording what 20,000 people do for ten years is still an observational study.
- Options the investigators offered. If a school offers a printed textbook and a digital one and each student picks, the students decided. That makes it observational.
- Random selection. Choosing participants at random decides who is studied, not who gets which condition.
Words like "trial" or "test group" don't settle it either. Just ask who decided.
What a Comparison Group Is For
A comparison group shows you what happens without the treatment, over the same stretch of time and under the same conditions. If the groups were alike at the start (random assignment, later in this lesson, makes them alike) and were handled the same way except for the treatment, then the gap between them at the end is the treatment's doing.
Say a school randomly assigns half of its ninth graders to after-school math tutoring for a semester, and the other half get no tutoring. Both groups' scores rise, because every student is still taking math class. In the graph, the comparison group climbs from 60 to 76 with no tutoring at all. The tutored group climbs to 84, but only the 8-point gap between the groups belongs to the tutoring.
A comparison group isn't there to make the study bigger or to be fair to the participants, and it doesn't widen whom the result describes. Its one job is to be the baseline.
Before and After Is Not a Comparison
When every participant gets the treatment and the outcome is measured before and after, the change mixes the treatment with everything else that changed over that time: the season, practice, growing older, the weather. Nothing in the study can pull them apart.
Say a town puts new streetlights on every street, and the next year there are fewer car break-ins than the year before. Maybe break-ins would have dropped anyway. Streets without the new lights, measured over the same year, would tell you, but the town has none.
Last year's records, or an earlier group's results, don't count as a comparison group either. They come from a different time, with everything else that changed since then mixed in.
Random Selection: Whom a Result Describes
Population and Sample
A population is the whole group a question is about. A sample is the part of it that is actually studied. A sample is selected at random when chance alone decides who is chosen, and every member of the population could be chosen.
Say a school has 1,500 students: that's the population. If a computer picks 100 names from the school's complete enrollment list, those 100 students are a random sample of the school's students. The population sampled is the group that could have been chosen: here, every student on that list.
A Result Reaches Only the Population Sampled
A result from a random sample can be generalized to the population the sample was selected from: it holds for that whole group, but for no bigger one.
Suppose 200 students are selected at random from all the high school students in one district. The result describes the district's high school students. It doesn't describe all high school students in the state, because students outside the district had no chance to be chosen. When a question asks for the largest population, "students at one of the district's high schools" is too narrow: the result reaches the whole district.
So the first move is always the same: find the group the sample was selected from. That group is how far the result reaches.
Volunteers and Convenience Samples
Two common kinds of sample are not random. A volunteer sample is made of people who chose to take part, like an online poll or a call-in vote. A convenience sample is whoever was easy to reach, like shoppers at one store or students in one class. Both are likely to be biased: some parts of the population show up too much, and others are left out.
Ask shoppers at a farmers' market how often they eat fresh vegetables, and you will likely overstate how often the town's residents do: people who shop at a farmers' market probably buy more vegetables than most. A sample like that describes the people in it, and that's all.
A Bigger Sample Does Not Fix the Method
If a method leaves out part of the population, or lets people pick themselves, a bigger sample just repeats the problem. Five thousand online votes lean the same way as fifty.
A random sample of a few hundred can represent a population that a biased sample of many thousands can't. Size does matter for a random sample: it affects how close its estimate is likely to be (Skill 15). But size never makes a biased method fair.
One more trap: if many of the people selected at random don't respond, the ones who do respond picked themselves, and the sample can be biased all over again.
An Estimate, Not an Exact Value
A random sample's result is an estimate of the population's value. The population's value is probably close to it, but it doesn't have to equal it, and another random sample would usually give a slightly different result.
Suppose 61% of a random sample of 400 of a town's voters support a plan. You can say "61% of the 400 voters in the sample support the plan" (a fact about the sample) and "about 61% of the town's voters support the plan" (an estimate). You can't say "exactly 61% of the town's voters support the plan" or "another random sample of 400 voters would also show 61%." Just how close "about" is comes up in Skill 15: Inference from Sample Statistics and Margin of Error.
Random Assignment: Whether It Is a Cause
Random Assignment Builds Comparable Groups
In an experiment, random assignment means chance alone decides which participants get which treatment. Then every other difference among the participants (fitness, habits, age, anything, whether anyone measured it or not) tends to be spread about evenly across the groups.
So if the groups differ at the end, and they were handled the same way except for the treatment, the treatment is the explanation left standing. That's why random assignment lets a study show a cause, for the participants who were in it.
Confounding Variables
Without random assignment, the groups you are comparing may differ in other ways too. A confounding variable is a third variable (not itself caused by the thing being studied) that differs between the groups and also affects the outcome. It can create an association even when the thing being studied does nothing at all.
In months when more ice cream is sold, more people get sunburned. Ice cream doesn't cause sunburn: hot, sunny weather raises both.
To test a possible confounder, ask two questions. Does it differ between the groups? Could it affect the outcome? Only a variable that passes both can explain the association. The outcome itself, the thing being studied, and anything that is the same in every group all fail.
Observational Studies Show Association
In an observational study the groups formed themselves, so a confounding variable can pile up in one group. That's why an observational study can show an association ("is associated with," "tends to," "is linked to") but can't, on its own, show a cause ("causes," "reduces," "improves," "leads to").
Adults who run regularly have lower resting heart rates, on average, than adults who don't. But runners chose to run, and people who were already fitter are more likely to choose running. In the figure, 8 of the 10 runners were fitter at the start, but only 3 of the 10 non-runners. Fitness at the start differs between the groups and affects heart rate, so it's a confounding variable.
More people, more years or more careful measuring don't change this. A huge observational study measures an association very precisely, but it's still just an association.
Two Questions, Four Designs
The Two Questions
Every claim about a study rests on two separate questions, and each randomization answers only its own:
- Were the participants selected at random from a population? If so, the result reaches that population. If not, it describes only those studied.
- Was the treatment assigned at random? If so, a difference between the groups can be credited to the treatment, a cause, for those studied. If not, the result shows only an association.
A study can do both, just one, or neither. The two mixed cases are the ones that catch most people. Random selection without random assignment gives an association that reaches the population, but no cause. Random assignment among volunteers gives a cause for the volunteers, but no reach beyond them.
Kind, Strength, Scope
Test a stated conclusion three ways:
- Kind: does it claim a cause? Only random assignment earns one.
- Strength: does it say "every," "always," "proves" or "eliminates" where the study showed a difference on average? A result also holds for the period measured, not beyond it: a three-month study says nothing about the years after.
- Scope: which group does it describe? It may reach the population sampled at random, or, without random selection, only those studied.
A conclusion is supported only if it passes all three. A sentence that names no group, like "This app lowers stress," sounds like a claim about everyone, so it fails scope unless the study selected at random from everyone it seems to describe.
Claim What Was Earned, No More and No Less
Wrong answers can miss in either direction. Claiming more than the design earned is the usual mistake. But claiming less is a mistake too when what it says is false: if the treatment was assigned at random, "the study can't show a cause" is simply wrong.
For volunteers who were assigned at random, the earned claim is a cause, for the volunteers. "It causes it for everyone" claims too much. "It is linked, but no cause was shown" and "nothing can be concluded" claim too little, and both are false here. A weaker statement that's still true, like "the two groups differed," isn't wrong, which is why questions ask for the conclusion that is best supported.
Design Backwards From the Claim
To pick a design, start from the claim someone wants to make, then give it exactly what it needs:
- A cause: impose the treatment and assign it at random.
- Reach to a population: select the participants at random from that population, the exact group the claim names.
- Both: select at random from the population, then assign the treatment at random.
A district wants to claim that daily reading logs raise reading scores for its fourth graders. That is a cause and a population, so the design must do both: select fourth graders at random from the whole district, then randomly assign them to keep a log or not. The design must also measure the outcome the claim names: here, reading scores, not the number of books the students check out. If a design is missing a randomization, making it bigger, longer or more careful doesn't add it.
One Change That Fixes a Design
Many design questions describe a study that is almost right and ask for the one change that would let it support a claim. Find the part of the claim the design hasn't earned, add that randomization, and leave the rest alone.
Not every flaw costs the same. A flaw that limits the reach, such as using only volunteers or only one school, leaves a cause standing for those studied, if the treatment was assigned at random. A flaw that spoils the comparison puts something besides the treatment into the difference between the groups: no comparison group at all, or groups handled differently after they were assigned, such as a coach who gives extra attention to the group she hopes will improve. Random assignment makes the groups alike at the start, but it can't fix groups that are treated differently afterward.
Keep going
Build the basics first
Foundations has free readiness checks and lessons in the English and math the SAT expects.
Check Your Skills, the strategy for each kind of question, worked examples and practice from Easy to Hard are in the Quaero SAT Course.
More Problem-Solving and Data Analysis
Ratios, rates and units
Set up ratios and proportions, work with rates, convert units by cancelling them, and recognize proportional relationships.
Read the concepts →SAT essentials · MathPercentages
Learn what a percent means, find shares of a total, and apply one percent change or several in a row.
Read the concepts →SAT essentials · MathOne-variable data
Find the mean and median, read dot plots, histograms and box plots, compare spread, and see how one value changes a data set.
Read the concepts →SAT essentials · MathTwo-variable data
Read two-variable data, predict with a line of best fit, understand residuals, interpret slope and intercept in context, and choose a model.
Read the concepts →SAT essentials · MathProbability
Find a probability as a part over a whole, combine events, read two-way tables, and handle conditional probability and independent events.
Read the concepts →SAT essentials · MathSample statistics and margin of error
Tell a sample from a population, build an interval from an estimate and its margin of error, see why sample size matters, and scale results up.
Read the concepts →