Skip to content
SAT essentials · Math

SAT statistical claims: what a study can conclude

Two questions decide what a study can claim: who was chosen, and how they were assigned. Random selection widens whom a result describes; random assignment supports cause.

These are the concepts this skill rests on, the same pages as in the lesson.

Two Kinds of Study

Observational Study or Experiment

In an observational study, the investigators just record what people (or animals, or things) already do or already have. In an experiment, the investigators impose a condition: they decide which participants get which treatment.

Two rows, each sorting the same six people into an exercise group and a no-exercise group. Top row, an observational study: each person chose a group, and the investigators only record it. Bottom row, an experiment: the investigators decided who goes in which group. Observational study: each person chose people → exercise no exercise Experiment: the investigators decided people → exercise no exercise

A researcher who compares the test scores of students who already take music lessons with those of students who don't is running an observational study. A researcher who decides which students take music lessons, and then compares their scores, is running an experiment.

The group that gets the treatment is the treatment group. A group that doesn't get it is a comparison group, also called a control group.

Who Decided the Condition?

One question tells you the type of study: who decided which group each participant is in? If the investigators decided, it's an experiment. If the participants chose, or the groups already existed, it's an observational study.

A decision diagram. The question "Who decided which group each one is in?" branches two ways. If the investigators decided, the study is an experiment. If the participants or circumstances decided, it is an observational study. A note below lists things that do not decide the type: how large, long or careful the study is, random selection, and words such as trial or test. Who decided which group each one is in? the investigators the participants, or circumstances experiment observational study These do not decide the type: how large, long, or careful the study is; random selection; the words 'trial' or 'test'

Three things do not make a study an experiment:

  • Size, length or care. Recording what 20,000 people do for ten years is still an observational study.
  • Options the investigators offered. If a school offers a printed textbook and a digital one and each student picks, the students decided. That makes it observational.
  • Random selection. Choosing participants at random decides who is studied, not who gets which condition.

Words like "trial" or "test group" don't settle it either. Just ask who decided.

What a Comparison Group Is For

A comparison group shows you what happens without the treatment, over the same stretch of time and under the same conditions. If the groups were alike at the start (random assignment, later in this lesson, makes them alike) and were handled the same way except for the treatment, then the gap between them at the end is the treatment's doing.

Say a school randomly assigns half of its ninth graders to after-school math tutoring for a semester, and the other half get no tutoring. Both groups' scores rise, because every student is still taking math class. In the graph, the comparison group climbs from 60 to 76 with no tutoring at all. The tutored group climbs to 84, but only the 8-point gap between the groups belongs to the tutoring.

A line graph of average test score against weeks into the semester, from week 0 to week 16. Both groups start at 60. The comparison group, with no tutoring, rises to 76: that 16-point rise is everything else, such as math class. The tutored group rises to 84. The 8-point gap between the groups at week 16 is what the tutoring did. 0 4 8 12 16 60 64 68 72 76 80 84 88 weeks into the semester average score tutored group comparison group 16 points: both groups rise this much anyway 8 points: what the tutoring did

A comparison group isn't there to make the study bigger or to be fair to the participants, and it doesn't widen whom the result describes. Its one job is to be the baseline.

Before and After Is Not a Comparison

When every participant gets the treatment and the outcome is measured before and after, the change mixes the treatment with everything else that changed over that time: the season, practice, growing older, the weather. Nothing in the study can pull them apart.

One group, every member treated, measured before and after. The after bar is taller than the before bar. The extra part is labeled: the change could be the treatment, or the season, practice, and anything else over that time. One group, every member treated before after the change: the treatment, or the season, practice, and anything else?

Say a town puts new streetlights on every street, and the next year there are fewer car break-ins than the year before. Maybe break-ins would have dropped anyway. Streets without the new lights, measured over the same year, would tell you, but the town has none.

Last year's records, or an earlier group's results, don't count as a comparison group either. They come from a different time, with everything else that changed since then mixed in.

Random Selection: Whom a Result Describes

Population and Sample

A population is the whole group a question is about. A sample is the part of it that is actually studied. A sample is selected at random when chance alone decides who is chosen, and every member of the population could be chosen.

A large oval labeled population, full of dots. Eight dots scattered across the whole oval are green. An arrow points to a box labeled sample that holds the eight green dots, with the note selected at random. population → sample selected at random

Say a school has 1,500 students: that's the population. If a computer picks 100 names from the school's complete enrollment list, those 100 students are a random sample of the school's students. The population sampled is the group that could have been chosen: here, every student on that list.

A Result Reaches Only the Population Sampled

A result from a random sample can be generalized to the population the sample was selected from: it holds for that whole group, but for no bigger one.

Three nested rectangles. The outer one is all high school students in the state, marked too far. The middle one, in green, is high school students in the district, the group sampled from, marked: the result reaches here. Inside it, a smaller rectangle is one high school in the district, marked too narrow. Green dots, the sample, are spread across the whole middle rectangle. all high school students in the state: too far high school students in the district sampled from: the result reaches here one high school in the district: too narrow dots: the sample

Suppose 200 students are selected at random from all the high school students in one district. The result describes the district's high school students. It doesn't describe all high school students in the state, because students outside the district had no chance to be chosen. When a question asks for the largest population, "students at one of the district's high schools" is too narrow: the result reaches the whole district.

So the first move is always the same: find the group the sample was selected from. That group is how far the result reaches.

Volunteers and Convenience Samples

Two common kinds of sample are not random. A volunteer sample is made of people who chose to take part, like an online poll or a call-in vote. A convenience sample is whoever was easy to reach, like shoppers at one store or students in one class. Both are likely to be biased: some parts of the population show up too much, and others are left out.

A rectangle labeled population. Its upper left corner is shaded orange and labeled volunteers. Every dot in the sample box at the right is orange: the whole sample came from that corner. The rest of the population is labeled: no chance to be in the sample. population volunteers no chance to be in the sample → sample

Ask shoppers at a farmers' market how often they eat fresh vegetables, and you will likely overstate how often the town's residents do: people who shop at a farmers' market probably buy more vegetables than most. A sample like that describes the people in it, and that's all.

A Bigger Sample Does Not Fix the Method

If a method leaves out part of the population, or lets people pick themselves, a bigger sample just repeats the problem. Five thousand online votes lean the same way as fifty.

Three squares, each a population. In the first, a small sample comes from one shaded corner: biased. In the second, a large sample comes from the same corner: still biased. In the third, a random sample is spread over the whole square: representative. biased small sample from one corner still biased large sample from one corner representative random sample from everywhere

A random sample of a few hundred can represent a population that a biased sample of many thousands can't. Size does matter for a random sample: it affects how close its estimate is likely to be (Skill 15). But size never makes a biased method fair.

One more trap: if many of the people selected at random don't respond, the ones who do respond picked themselves, and the sample can be biased all over again.

An Estimate, Not an Exact Value

A random sample's result is an estimate of the population's value. The population's value is probably close to it, but it doesn't have to equal it, and another random sample would usually give a slightly different result.

A dot plot titled percent in each of 10 random samples. The scale runs from 56 to 66. The dots are at 58, 59, 60, 60, 61, 61, 61, 62, 63 and 64: each random sample gives a slightly different percent. 56 58 60 62 64 66 percent in each of 10 random samples each dot is one sample of the same size

Suppose 61% of a random sample of 400 of a town's voters support a plan. You can say "61% of the 400 voters in the sample support the plan" (a fact about the sample) and "about 61% of the town's voters support the plan" (an estimate). You can't say "exactly 61% of the town's voters support the plan" or "another random sample of 400 voters would also show 61%." Just how close "about" is comes up in Skill 15: Inference from Sample Statistics and Margin of Error.

Random Assignment: Whether It Is a Cause

Random Assignment Builds Comparable Groups

In an experiment, random assignment means chance alone decides which participants get which treatment. Then every other difference among the participants (fitness, habits, age, anything, whether anyone measured it or not) tends to be spread about evenly across the groups.

Twenty participants, ten fitter at the start (filled dots) and ten less fit (open rings), are assigned at random to a treatment group and a comparison group. Each group ends up with five of each kind. participants assigned at random treatment group comparison group fitter at the start less fit at the start

So if the groups differ at the end, and they were handled the same way except for the treatment, the treatment is the explanation left standing. That's why random assignment lets a study show a cause, for the participants who were in it.

Confounding Variables

Without random assignment, the groups you are comparing may differ in other ways too. A confounding variable is a third variable (not itself caused by the thing being studied) that differs between the groups and also affects the outcome. It can create an association even when the thing being studied does nothing at all.

A diagram of a confounding variable. Hot, sunny weather, at the top, raises both ice cream sales and sunburns, at the bottom. Ice cream sales and sunburns are joined by a dashed line labeled linked, but neither causes the other. confounding variable hot, sunny weather ice cream sales sunburns raises raises linked but neither causes the other

In months when more ice cream is sold, more people get sunburned. Ice cream doesn't cause sunburn: hot, sunny weather raises both.

To test a possible confounder, ask two questions. Does it differ between the groups? Could it affect the outcome? Only a variable that passes both can explain the association. The outcome itself, the thing being studied, and anything that is the same in every group all fail.

Observational Studies Show Association

In an observational study the groups formed themselves, so a confounding variable can pile up in one group. That's why an observational study can show an association ("is associated with," "tends to," "is linked to") but can't, on its own, show a cause ("causes," "reduces," "improves," "leads to").

Adults who run regularly have lower resting heart rates, on average, than adults who don't. But runners chose to run, and people who were already fitter are more likely to choose running. In the figure, 8 of the 10 runners were fitter at the start, but only 3 of the 10 non-runners. Fitness at the start differs between the groups and affects heart rate, so it's a confounding variable.

Two panels comparing 10 runners with 10 non-runners. Top, the people chose for themselves: 8 of the 10 runners were fitter at the start, but only 3 of the 10 non-runners, so fitness piles up in one group. Bottom, the same 20 people assigned at random: 6 fitter runners and 5 fitter non-runners, so fitness is spread about evenly. fitter at the start less fit at the start chose for themselves: fitness piles up in one group runners non-runners assigned at random: fitness is spread about evenly runners non-runners

More people, more years or more careful measuring don't change this. A huge observational study measures an association very precisely, but it's still just an association.

Two Questions, Four Designs

The Two Questions

Every claim about a study rests on two separate questions, and each randomization answers only its own:

  1. Were the participants selected at random from a population? If so, the result reaches that population. If not, it describes only those studied.
  2. Was the treatment assigned at random? If so, a difference between the groups can be credited to the treatment, a cause, for those studied. If not, the result shows only an association.
A two-by-two grid. Columns: selected at random, and not selected at random. Rows: assigned at random, and not assigned at random. Assigned and selected: a cause that reaches the population sampled. Assigned, not selected: a cause for those studied only. Selected, not assigned: an association only, reaching the population sampled. Neither: an association only, for those studied only. selected at random not selected at random assigned at random not assigned at random cause reaches the population sampled cause for those studied only association only reaches the population sampled association only for those studied only Each randomization answers only its own question.

A study can do both, just one, or neither. The two mixed cases are the ones that catch most people. Random selection without random assignment gives an association that reaches the population, but no cause. Random assignment among volunteers gives a cause for the volunteers, but no reach beyond them.

Kind, Strength, Scope

Test a stated conclusion three ways:

  • Kind: does it claim a cause? Only random assignment earns one.
  • Strength: does it say "every," "always," "proves" or "eliminates" where the study showed a difference on average? A result also holds for the period measured, not beyond it: a three-month study says nothing about the years after.
  • Scope: which group does it describe? It may reach the population sampled at random, or, without random selection, only those studied.
The claim "This app lowers stress for every adult" split into three parts, each with its test. Kind, the word lowers: a cause is claimed, so was the app assigned at random? Strength, the word every: was the difference on average, or for every one? Scope, the word adult: who could have been selected? Claim: "This app lowers stress for every adult." kind "lowers" cause claimed: assigned at random? strength "every" on average, or every single one? scope "adult" who could have been selected?

A conclusion is supported only if it passes all three. A sentence that names no group, like "This app lowers stress," sounds like a claim about everyone, so it fails scope unless the study selected at random from everyone it seems to describe.

Claim What Was Earned, No More and No Less

Wrong answers can miss in either direction. Claiming more than the design earned is the usual mistake. But claiming less is a mistake too when what it says is false: if the treatment was assigned at random, "the study can't show a cause" is simply wrong.

Four claims about a study of volunteers who were assigned at random, from strongest to weakest. Causes it, for everyone: too much. Causes it, for the volunteers: earned. Linked, but no cause shown: too little. Nothing can be concluded: too little. Both too-little claims are false here. Volunteers, assigned at random causes it, for everyone too much causes it, for the volunteers earned linked, but no cause shown too little nothing can be concluded too little Both too-little claims are false here.

For volunteers who were assigned at random, the earned claim is a cause, for the volunteers. "It causes it for everyone" claims too much. "It is linked, but no cause was shown" and "nothing can be concluded" claim too little, and both are false here. A weaker statement that's still true, like "the two groups differed," isn't wrong, which is why questions ask for the conclusion that is best supported.

Design Backwards From the Claim

To pick a design, start from the claim someone wants to make, then give it exactly what it needs:

  • A cause: impose the treatment and assign it at random.
  • Reach to a population: select the participants at random from that population, the exact group the claim names.
  • Both: select at random from the population, then assign the treatment at random.
Three rows matching a claim to what the design needs. A cause: assign the treatment at random. Reach to a population: select at random from that exact population. Both: select at random, then assign at random. A note: a bigger or longer study is not on this list. the claim what the design needs cause → assign the treatment at random reach to population → select at random from that exact population both → select at random, then assign at random A bigger or longer study is not on this list.

A district wants to claim that daily reading logs raise reading scores for its fourth graders. That is a cause and a population, so the design must do both: select fourth graders at random from the whole district, then randomly assign them to keep a log or not. The design must also measure the outcome the claim names: here, reading scores, not the number of books the students check out. If a design is missing a randomization, making it bigger, longer or more careful doesn't add it.

One Change That Fixes a Design

Many design questions describe a study that is almost right and ask for the one change that would let it support a claim. Find the part of the claim the design hasn't earned, add that randomization, and leave the rest alone.

Two kinds of flaw. Left, in green, limits the reach: volunteers only, or one school or one city; a cause can still hold for those studied. Right, in red, spoils the comparison: no comparison group, or groups treated differently afterward; no cause is shown. limits the reach volunteers only one school or one city cause can still hold for those studied spoils the comparison no comparison group groups treated differently afterward no cause is shown, even if assigned at random

Not every flaw costs the same. A flaw that limits the reach, such as using only volunteers or only one school, leaves a cause standing for those studied, if the treatment was assigned at random. A flaw that spoils the comparison puts something besides the treatment into the difference between the groups: no comparison group at all, or groups handled differently after they were assigned, such as a coach who gives extra attention to the group she hopes will improve. Random assignment makes the groups alike at the start, but it can't fix groups that are treated differently afterward.

Keep going

Check Your Skills, the strategy for each kind of question, worked examples and practice from Easy to Hard are in the Quaero SAT Course.

More Problem-Solving and Data Analysis