Back to problems
solvedProblem hub

The headline finding flips when the sample is weighted

Created Nov 6, 2026, 8:30 AM · Updated Nov 10, 2026, 5:10 PM

The messy version

Unweighted, the effect is a clean 6.2 points in favour. Applying the demographic weights the funder asked for, it goes to -1.4 and the confidence interval crosses zero. Same respondents, same question wording.

Where it came apart:
  under-25 respondents: 11% of the sample, 24% of the population
  weight applied to that cell: 2.2
  and that cell is the only one where the effect runs the other way

So the whole result rests on about forty people who are being counted twice over. That is not a finding, it is a sampling problem wearing a finding's clothes. Need to say plainly in the write-up which estimate is reported and why, rather than quietly picking the flattering one.

When you're ready

Take it for a run.

Start a thinking session and come back with one concrete next move.

History

Thinking sessions

2 total

On Nov 8, 2026, 7:45 AM, you took this problem for a Manual session. 20 minutes.

Earlier note: Recomputed with the funder's weights and without. Decided to report both, with the caveat stated up front.

On Nov 6, 2026, 12:00 PM, you took this problem for a Manual session. 28 minutes.

Earlier note: Checked which cell the weights were doing the work in. It was the under-25s, on their own.