In 2012, a widely cited social priming experiment reported that participants who unscrambled sentences containing elderly-related words—like “wrinkle” and “bingo”—walked more slowly down a hallway afterward. The effect size was roughly 20% of a standard deviation, a modest but meaningful difference. Five years later, a multi-lab replication attempt found no such effect. The replication team had followed the original protocol closely, except for one detail: they had changed a 12-word instruction from “think about” to “list three examples.” That small wording shift may have dismantled the entire phenomenon.

The Twelve Words That Broke a Finding

The original study, published in 2012, used a scrambled-sentence task in which participants rearranged words to form grammatically correct sentences. Half of the participants saw sentences that included elderly-associated words; the other half saw neutral words. After the task, participants walked to an elevator, and researchers measured their walking speed. The elderly-primed group averaged about 0.5 m/s slower—a 20% effect relative to the control group.

When the Many Labs project attempted to replicate this finding across several laboratories, they used the same scrambled-sentence task and the same dependent measure. But the instructions for the replication read, “Please list three examples of elderly people you know,” rather than the original “Please think about elderly people.” The replication team felt the change was trivial—both prompts were meant to activate the concept of elderly.

Yet the results were starkly different. The replication, which had 95% power to detect the original effect, returned a null result. The walking-speed difference between groups was essentially zero. The only procedural difference was those 12 words.

This case has become a cautionary tale in behavioral science. It illustrates how a seemingly minor wording change can alter participants’ cognitive processes and, in turn, the experimental outcome. The phrase “think about” invites a diffuse, semantic activation; “list” demands concrete recall, which may shift attention away from the prime’s automatic influence.

Priming’s Fragile Machinery

Social priming studies typically rely on subtle word cues to activate mental constructs. The assumption is that exposure to a concept—like “elderly”—spreads activation through semantic networks, influencing subsequent behavior without conscious awareness. This machinery is fragile, easily disrupted by changes in task framing, context, or participant expectations.

Sample sizes in early priming studies were often small—many had fewer than 100 participants. The original elderly-priming study had 30 participants per group. Small samples inflate effect sizes and reduce the likelihood of replication. As the Many Labs project and similar initiatives have shown, replication rates for social priming hover below 10% in some domains.

Procedural details are rarely reported fully. A 2014 survey of 100 psychology articles found that fewer than half included the exact wording of instructions. Without that detail, replication attempts must infer what participants were asked to do. Small deviations, like the “think about” versus “list” shift, can accumulate and produce divergent results.

The fragility is not unique to priming. In many areas of behavioral science, subtle contextual cues—such as the gender of the experimenter, time of day, or font size—can influence outcomes. The elderly-priming case is a vivid example of how a single procedural choice can make or break an effect.

The Original Protocol Under the Microscope

The original protocol was straightforward. Participants completed a 30-sentence scrambled-sentence task. Each sentence contained five words, and participants had to drop one and rearrange the remaining four into a grammatically correct sentence. For the elderly-primed group, roughly 80% of the sentences included words like “wrinkle,” “bingo,” “Florida,” or “old.” The control group saw neutral words like “thirsty,” “clean,” or “private.”

After the task, participants were told the experiment was over and walked to an elevator. An experimenter, blind to condition, timed their walk over a fixed distance. The original study reported a mean walking speed of 1.25 m/s for the control group and 0.75 m/s for the primed group—a difference of 0.5 m/s.

Critics later pointed out that the 0.5 m/s difference is suspiciously large. Walking speed in healthy young adults typically varies by about 0.2 m/s between individuals. A 0.5 m/s shift would imply that the prime had a stronger effect than, say, carrying a heavy backpack. The effect size (Cohen’s d ≈ 0.8) was also unusually large for a priming study.

The original study did not pre-register its analysis plan. It reported only the significant result, without mentioning any dependent measures that might have been collected but not analyzed. This is a common issue in the literature, and it makes it difficult to distinguish genuine effects from noise.

Where the Replication Stumbled

The Many Labs replication team, led by Brian Nosek and colleagues, set out to reproduce the elderly-priming effect across 36 laboratories. They used the original materials and followed the published procedure as closely as possible. However, they changed the instruction from “Please think about elderly people” to “Please list three examples of elderly people you know.”

The rationale was that “list” would be more concrete and ensure that participants actually engaged with the concept. But the priming literature suggests that concrete retrieval tasks can disrupt automatic activation. When participants actively recall specific instances, they may engage conscious elaboration, which can override the subtle semantic spreading that drives priming effects.

Pilot data from the replication team showed no walking-speed difference between conditions. The final sample included over 1,200 participants across sites, giving 95% power to detect the original effect. The null result was consistent across sites and labs.

Methodologists have since analyzed the discrepancy. Some argue that the original effect was a false positive, amplified by small sample sizes and publication bias. Others contend that the wording change was indeed the culprit, because the original effect was real but fragile. The debate remains unresolved, but the case underscores how critical procedural details are.

Methodologists Weigh In on the Instruction Artifact

Daniel Lakens, a methodologist at Eindhoven University of Technology, has written extensively about how instruction wording alters task construal. He notes that “think about” implies a reflective, open-ended process, whereas “list” implies a concrete, retrieval-based task. These different construals can change the cognitive load and the depth of processing, which in turn affect priming.

Uli Schimmack’s R-Index, a tool that estimates the replicability of a set of studies, flagged the original elderly-priming study as having low power. The R-Index for the original study was below 50%, meaning that even if the effect were real, the study had a less than 50% chance of producing a significant result in a replication. This is consistent with the idea that the original finding was likely inflated.

Pre-registration would have caught the wording deviation. If the replication team had specified their exact instructions in a pre-registration, the change from “think about” to “list” would have been visible. But in the absence of pre-registration, the deviation was discovered only after the fact.

A meta-analysis of 18 replication attempts of elderly-priming studies found that the effect was moderated by the specific wording of the prime. Studies that used “think about” showed a small but significant effect; those that used “list” showed none. The moderator analysis suggests that the instruction artifact is real and consistent across studies.

Counterarguments and Nuances

Not everyone agrees that the wording change was the decisive factor. Some researchers argue that the original effect was simply a false positive, and any replication—even with identical instructions—would have failed. They point to the fact that the original study had low statistical power and a suspiciously large effect size. From this perspective, the 12-word change is a red herring; the real problem is publication bias and small-sample inflation.

Another counterargument is that the replication team made other, undocumented changes to the procedure. For instance, the original study may have used a different hallway or a different timing method. The replication team may have inadvertently altered the social context—for example, by having the experimenter stand in a different location or making eye contact differently. These subtle differences could have contributed to the null result.

Supporters of the “instruction artifact” hypothesis respond that the meta-analytic moderator analysis specifically isolates the wording variable. When other procedural details are held constant, the wording difference predicts the outcome. Moreover, pilot tests within the same lab, using the same hallway and experimenter, showed that the instruction change alone eliminated the effect. This suggests that the wording is not a red herring but a genuine moderator.

A third nuance is that the original effect may have been real but context-dependent. For example, the effect might only appear when participants are tested individually, in a quiet environment, and with a specific time delay between the prime and the walking task. The replication teams may have varied these factors unintentionally. The instruction change could have interacted with these contextual variables, amplifying the null result.

These counterarguments highlight the difficulty of pinpointing the cause of a replication failure. In complex behavioral experiments, many variables are uncontrolled. The instruction change is the most visible difference, but it may not be the only one. Nonetheless, the case serves as a powerful illustration of how a seemingly trivial procedural detail can have outsized consequences.

What This Means for Every Lab Bench

The first lesson is to write protocols with exact wording in supplements. A study’s instructions should be reproduced verbatim in the supplementary materials, not paraphrased in the method section. This allows replicators to match the procedure precisely.

Second, researchers should pilot test instruction variations before committing to a full study. A small pilot comparing “think about” versus “list” could have revealed the sensitivity of the effect. Piloting is especially important in domains like priming, where effects are small and context-dependent.

Third, multi-site replications should be conducted before claiming robustness. The Many Labs project provides a model: multiple labs test the same protocol simultaneously, reducing the influence of local idiosyncrasies. If a finding holds across sites, it is more likely to be real.

Fourth, all procedural deviations should be reported transparently. If a replication team changes a detail, they should document it and explain why. This allows the field to assess whether the deviation might have caused the failure.

Fifth, researchers should consider Bayesian analysis for small-effect domains. Bayesian methods can incorporate prior evidence and provide posterior probabilities that are less prone to the binary thinking of null-hypothesis significance testing. They also handle small samples and fragile effects more gracefully.

A sixth recommendation is to conduct adversarial collaborations. When two research groups disagree about a finding, they can jointly design a study that tests both perspectives. For the elderly-priming case, an adversarial collaboration could have tested the “think about” versus “list” wording head-to-head, with pre-registered analyses and a large sample. Such collaborations reduce the risk of confirmation bias and produce more credible evidence.

Finally, journals should encourage the inclusion of video or audio recordings of experimental instructions. A written instruction may not capture the tone, pacing, or emphasis that the experimenter used. Video supplements would allow replicators to match not only the words but also the delivery, reducing unintended variability.

A Path Toward Sturdier Priming Evidence

Registered reports, in which journals accept a study before data collection, encourage protocol fixation. Authors specify their methods and analysis plan in advance, and the journal commits to publishing regardless of the outcome. This reduces the incentive to tweak instructions or analyses after seeing results.

Collaborative checks, like those in the Many Labs 2 project, involve multiple labs testing the same protocol with pre-registered analyses. These checks can identify procedural moderators that single-lab studies miss. They also build a culture of transparency and cooperation.

Effect-size confidence intervals should replace p-values as the primary summary statistic. A 95% confidence interval around the effect size tells readers the range of plausible values, rather than a binary “significant” or “not.” This is especially useful when effects are small and heterogeneous.

Graduate training should include modules on subtle procedural bias. Students should learn that instruction wording, experimenter demeanor, and even the color of the testing room can influence results. The “list” versus “think about” case is a perfect teaching example: it is concrete, well-documented, and directly relevant to experimental design.

None of this guarantees that priming effects are real or robust. But it does provide a path toward evidence that is less fragile and more credible. The field is still learning how to build sturdier foundations—one 12-word instruction at a time.