Nonzero Lab Research · Statistics · 2026-08-28 · 6 min read

How Many Trials Do You Need? Sample Size Behind a Win Rate

18 of 30 correct (60%) means a 42–76% confidence interval; separating 55% from a coin flip takes about 780 trials. Sample size decides belief.

The illusion of seven in a row

What does it feel like to be right seven times in a row? For most people, it feels like "I've figured something out." The feeling arrives fast — sometimes five is enough, and three can already move you.

Yet by pure luck, seven straight wins is not a rare event. Do the arithmetic: if each call were a coin flip, seven in a row is a 1-in-128 outcome (assuming independence and even odds). That sounds small until you put it on a calendar — someone running five blind sessions a day passes through over a thousand "seven in a row" windows per year, and pure luck will hand out a seven-streak on schedule.

Feeling and evidence part ways right here: the feeling has concluded by trial seven; the evidence hasn't said a word.

We are built to trust small samples

Tversky (1971) named this tendency the "law of small numbers": people systematically overestimate how representative small samples are, believing that a dozen or a few dozen observations reveal the underlying rate. The bias holds even for statistically trained researchers — when designing experiments, they too place too much faith in small samples.²

Kahneman (2011) later made this finding a central case study in how statistical intuition fails.³ For a trader the implication is direct: your trust in "how it's been going lately" is a cognitive bias that was named more than fifty years ago, not a form of insight.

There is a subtlety worth naming: the bias survives knowing about it. People who can recite the law of small numbers still feel the pull of a seven-win streak. The point of quantifying the noise band is not to win an argument with yourself in the moment — it is to have a rule installed in advance, so the feeling never gets a vote.

How wide is the interval behind a win rate?

Make it concrete. Confidence intervals for a proportion can be computed with the method of Wilson (1927).¹ Two common training scenarios:

Eighteen out of thirty looks like "winning 60% of the time," but the interval runs from 42% to 76% — covering both "worse than a coin flip" and "genuinely strong." An interval that wide means thirty trials have almost no verdict power over how good you are.

The correct reading of a confidence interval is also worth stating plainly. It is not the comforting "there is a 95% chance your true rate sits inside." It is "if the same experiment were repeated many times, intervals computed this way would cover the true value 95% of the time." For a practitioner the useful reading is simpler: the wider the interval, the less seriously you should take the point estimate.

Even at one hundred trials with sixty correct, the lower bound has barely cleared 50%. Whether your true rate is 51% or 68% remains undecidable at that sample size.

How many trials does it take?

Turn the question around: to tell a true win rate of 55% apart from the 50% of a coin flip, how many trials do you need? At 5% significance and 80% power:

This table deserves a pause. If your edge is slight (55%), detecting it takes nearly eight hundred recorded decisions — at five blind sessions a day, most of a year. The larger the edge, the fewer trials required, but even a robust 65% needs about eighty to separate from luck.

The table also reads backwards: it tells you which differences are invisible at small samples. Two people each run fifty sessions; one wins 28, the other 24 — an eight-point gap in win rate, entirely inside the noise band, saying nothing about whose method is better. Leaderboards, screenshots, weekly "how I'm doing" posts — none of them mean anything at this scale.

Any conclusion of the form "I tried it twenty times and won 70%" is, next to this table, just a reading from inside the noise.

Why can't a win rate stand alone?

There is a second dimension this article only touches: payoff ratio. A 45% win rate with a 2:1 payoff and the same rate with a 1:1 payoff are entirely different methods. The sample-size argument applies to payoff ratio as well — a few outsized wins or losses dominate the average, and it takes a larger sample to wash them out.

The two numbers also trade against each other psychologically: methods that win often tend to win small, methods that win big tend to win rarely. Whichever profile yours has, the honest unit of account is the combination — how the two multiply across a sample large enough to trust.

Win rate and payoff ratio must be read from the same sample, and the sample must be accumulated under one fixed rule. That is why the unit of review should be the week and the month, never the single trade.

Tally weekly, not per trade

The numbers above compress into one discipline: sample size decides what you are allowed to believe. Twenty trials make a story, a hundred make a clue, a few hundred make evidence. The discipline is unglamorous, but it saves more money than any indicator — it stops you from promoting luck into skill, and from converting "it felt right" into position size.

It also removes a whole category of emotional churn. If no single session can change the verdict, there is nothing to celebrate or mourn mid-week. The work is the accumulation; the verdict arrives on its own schedule.

Research on deliberate practice likewise stresses feedback spread over many repetitions (see Can Trading Intuition Be Trained?); whether the environment supports learning at all is covered in Why Your Chart Review Lies to You.