The right length for an A/B test is however long it takes to collect the sample size your statistics require, not a fixed number of days. That sample size is set before launch by four inputs: your baseline conversion rate, the minimum effect you want to detect, your significance level, and your statistical power. Divide the required sample by your daily traffic, and you get your duration. Everything else in this guide explains those inputs and the traps that break them.
- Duration = required sample size รท daily eligible traffic, then round up to whole weeks.
- Sample size is driven by four levers: baseline conversion rate, minimum detectable effect (MDE), significance level (ฮฑ, usually 5%), and power (usually 80%).
- Smaller effects need far more traffic. Detecting a 1% lift can take 10โ20x the sample of a 20% lift.
- Never stop early because it “looks significant.” Peeking inflates your false-positive rate dramatically.
- Run at least one to two full weeks so weekday/weekend behavior is representedโeven if the math says fewer days.
- An inconclusive test is not a failed test. It often means the experiment was underpowered, not that the idea didn’t work.
What is A/B testing?
A/B testing (also called split or bucket testing) is a controlled experiment comparing two versions of something:
- Variation A: the control (no change)
- Variation B: the variation (one change)
You isolate a single change so you know exactly what caused the difference. As Shopify puts it, this replaces “someone has a good feeling about it” with data on real customer behavior.
Common goals for A/B testing
Shopify groups the usual goals into five buckets:
- Increase product sales– e.g. benefit-led vs. discount-led copy
- Improve add-to-cart rates – e.g. “only 3 left” urgency tags
- Reduce cart abandonment – e.g. free shipping vs. easy returns messaging
- Grow email or lead sign-ups – e.g. “10% off” vs. “keep in touch” CTAs
- Lift click-through rates – e.g. button color, size, and copy
Whatever the goal, tie it to one measurable metric before you build anything.
The stats that decide your test length
You can’t answer “how long?” without four numbers. Skip them, and you’re guessing.
1. Statistical significance
What it is: whether the result is real or just luck.
Shopify frames it bluntly – did this version truly win, or did we get lucky? Customer behavior swings day to day, so small differences happen by chance.
You typically declare significance when:
- You’ve gathered a meaningful sample size
- The test ran long enough to cover normal variability
- Your tool reports 95%+ confidence
The flip side is the significance level (ฮฑ). Evan Miller’s calculator defines ฮฑ as the % of the time you’d detect a difference assuming none exists, your false-positive rate. The standard ฮฑ of 0.05 means a ~1-in-20 chance of calling a winner when nothing changed.
2. Sample size
What is sample size?
Sample size refers to the minimum number of unique users or events required per variation to detect a statistically significant change if a real effect exists.
Determining the appropriate sample size upfront guards against two major experimentation pitfalls:
Overpowered tests: Running tests longer than necessary, wasting valuable traffic and delaying rollout spee
Underpowered tests: Failing to identify real improvements because the user count was too low.
It’s the biggest driver of duration. Your daily traffic is roughly fixed, so a bigger required sample simply means more days.
- Practical floor: at least 1,000 visitors per variation (Shopify)
- Common reasons a test never reaches significance: sample too small, change too subtle, or audience behavior too variable
3. Minimum detectable effect (MDE)
What it is: the smallest improvement you actually care about catching.
Set it by what would change your decision, not the lift you’re hoping for.
- Larger MDE โ needs less traffic
- Smaller MDE โ needs dramatically more
Convert’s example: a 30% lift shows up fast on modest traffic; a 0.5% lift hides inside daily noise and needs far more data.
โ ๏ธ Watch the MDE type – mixing these up throws your sample size off by a wide margin.
| MDE type | Meaning | On a 5% baseline |
| Relative | % of the baseline | 10% relative = 5% โ 5.5% |
| Absolute | Flat percentage-point change | 10% absolute = 5% โ 15% |
4. Statistical power
What is significance in an A/B test
Statistical significance measures the confidence that the observed performance difference between $A$ and $B$ is caused by the change introduced rather than random sampling noise.
- The Null Hypothesis (H0): Assumes there is no real difference between the Control and Treatment. Any observed variance is pure noise.
- SurveyMonkey
- The Alternative Hypothesis (H1): Assumes the variation causes a genuine shift in user behavior.
- The p-value: The probability of observing a difference at least as extreme as the actual test result, assuming H0 is true.
- Significance Level (ฮฑ): The threshold for Type I errors (false positives). The industry standard is ฮฑ = 0.05 (yielding 95% statistical confidence). If p < a, the result is declared statistically significant. Statsig + 1
What it is: the chance your test detects a real effect when one exists.
- Formula: Power = 1 โ ฮฒ
- Standard target: 80% โ a true winner is caught 80% of the time, missed 20%
- So about 1 in 5 well-designed tests still come back non-significant by chance
Power โ significance โ they guard opposite errors:
| Question it asks | Protects against | |
| Significance (ฮฑ) | How much evidence before we call a winner? | False positives |
| Power (1 โ ฮฒ) | If a real effect exists, will we detect it? | False negatives |
Four things move power (Convert): sample size, MDE, ฮฑ, and data variability (wildly ranging order values = more noise = bigger sample needed).
How do A/B testing tools detect significance?
Tools compare A vs. B against the visitor count each version received. More traffic = a more reliable comparison. Two engines are common:
- Frequentist: computes a p-value and confidence interval; flags a winner when p crosses ฮฑ (e.g., p < 0.05). This is the 95%-confidence convention.
- Bayesian: reports probabilities like “chance to beat baseline” and expected loss, instead of a fixed threshold.
Pro tip – run an A/A test first. Test two identical versions to learn your normal noise floor. Shopify’s example: identical pages might land at 3.2% vs. 3.4% from pure randomness – now you know what a “real” future lift has to clear.
The sample size formula in practice
You don’t need to hand-calculate it โ every good tool has a calculator. You just need to read what it produces. Three sourced examples show the range:
| Scenario | Inputs | Result |
| Atticus | 3% baseline, 10% rel. MDE, 95% conf., 80% power | ~38,000 / variation (76k total) |
| Convert | 5% baseline, 10% rel. MDE, 99% conf., 90% power | 59,182 / variation โ 24 days at 5k/day |
| Evan Miller | 10.2% baseline โ 13.2% target | 2,545 / variation (big baseline + big effect = tiny sample) |
The 6-step power analysis (Convert):
- Establish your baseline from your own recent data
- Set your MDE by what would change your decision
- Choose ฮฑ (default 5%)
- Set power (default 80%)
- Calculate required sample size
- Convert that sample into a duration using your traffic
So how long should you run it?
Once you have the required sample, duration is just arithmetic:
Duration = total required sample รท eligible visitors per day
- Convert: 40,000 needed รท 2,000/day โ 20 days
- Atticus: 76,000 needed รท 5,000/day โ 15 days โ “not three days, not ‘until we see green.'”
And if traffic is huge? If you need 10,000 per variation and get 50,000 per day, you might hit the number in a few days โ but still don’t stop that fast. Read the two-week rule next.
The two-week minimum (+ full-week rule)
Even when traffic lets you hit the sample in 3โ4 days, run at least 1โ2 full weeks.
- Why (Convert): a short window can catch one unusually busy period instead of a normal cycle.
- Full-week rule (Atticus): always round up to complete weeks. Monday shoppers โ Saturday browsers. A 15-day requirement โ round to 21 days (3 full weeks).
- Don’t stop on a Friday โ you’ll overweight weekday behavior.
External factors that stretch duration
Your math assumes a stable environment. Three forces break that (Atticus):
- Seasonality โ Black Friday behaves nothing like February
- Marketing campaigns โ a paid-traffic flood changes who is in your sample
- Product/site changes โ anything shipping mid-test is a confounding variable
A practical framework
Before every test, document three things and share them with the team (Atticus):
- Required sample size โ from your four variables
- Expected end date โ sample size รท daily traffic
- Early-stopping criteria โ safety only (e.g. a severe conversion drop)
Now “is the test done yet?” has only two honest answers: “we haven’t reached sample size” or “yes, on Tuesday.” That kills both peeking and the pressure to call winners early.
Bonus: factor duration into prioritization. An 8-week test has a higher opportunity cost than a 2-week test of similar expected impact; run the faster one first.
The “magic number” myth
There is no universal minimum. Anyone saying “always run two weeks” or “always get 1,000 conversions” is handing you a heuristic that may not fit.
- High-traffic e-commerce testing a big checkout change โ maybe 4 days
- B2B SaaS testing a pricing-page headline โ maybe 8 weeks
Calculate it every time โ then apply the two-week floor and full-week rounding on top.
The 3 traps that ruin A/B tests
Most bad decisions come from these, not from bad tools.
Trap 1 – Ending the test too soon (peeking)
Significance isn’t stable; it fluctuates as data arrives.
Check a running test ~20 times, and you have a ~ 50% chance of seeing a “significant” result at some point โ even with no real difference. (Atticus)
Each look is basically a fresh test, so repeated checks compound your false-positive rate.
- Fix: decide the end date before launch and hold to it.
- If you must monitor (safety): use sequential testing, which adjusts the threshold at each peek โ but note it usually needs a larger sample.
Trap 2 – Regression to the mean
Extreme early results drift toward the true average as data piles up. Memorize this example (Atticus):
| Day | Observed lift |
| Wednesday | 25% (p < 0.01) ๐ |
| Friday | 12% |
| End of week 2 | 4% |
Nothing “declined.” The early number came from a small, unrepresentative sample. Only adequate sample size protects you โ ignore day-three lifts and day-three drops.
Trap 3 – Misreading inconclusive tests
An inconclusive result is not a zero. It often means the test lacked the power to see a real effect.
Convert’s cost example:
- SaaS with $1M/month revenue, 5% baseline
- Test sized for a 10% MDE; real checkout lift is a modest 2% (5% โ 5.1%)
- Underpowered test reports “not significant” โ team discards it
- Cost: $20,000/month = $240,000/year in unrealized growth
Underpowered testing also poisons your winners:
| Power level | True positives | False positives | False-discovery rate |
| 10% power | ~5 | ~5 | 50% |
| 80% power | ~40 | ~5 | ~11% |
(100 experiments, 50 real / 50 null, ฮฑ = 10%.) This is why analysts “call Twyman’s Law” on a giant win from a tiny sample โ it’s probably noise until it survives scrutiny (the “winner’s curse”).
Special situations
Low-traffic pages
If the math says six months, you have options (Atticus + Convert):
- Increase the MDE – hunt a 20% lift, not 5%. Forces bolder changes anyway.
- Aggregate traffic – test one CTA change across all product pages, not one at a time.
- Variance reduction (CUPED) – can cut required sample by 50%+.
- Run in a high-traffic segment (e.g. mobile only) – but interpret the win only for that segment.
Running multiple tests at once
Fine to do – but mind the multiple-comparisons problem.
Four variations vs. one control = four comparisons, each with its own 5% false-positive chance. Combined, the odds climb well above 5% without a correction. (Convert)
- Apply an ฮฑ correction (e.g., Bonferroni-style) for many variations
- Coordinate overlapping tests so one doesn’t secretly confound another
What is Multi-armed bandit Testing?
While standard A/B tests maintain a fixed 50/50 split across the entire experiment lifecycle, a Multi-Arm Bandit (MAB) algorithm dynamically updates traffic allocation in real time based on incoming performance data.
A multi-armed bandit dynamically shifts more traffic to the winning variation while the test is live – instead of a fixed 50/50 split. The name comes from slot machines (“one-armed bandits”). It balances:
- Explore: keep gathering data on each option
- Exploit: send traffic to the current leader
(via algorithms like epsilon-greedy or Thompson sampling.)
| Best for | Trade-off | |
| Bandit | Short campaigns, promos, many creatives | Optimizes earnings, not a clean effect-size estimate |
| Classic A/B | Permanent decisions needing a trustworthy result | Slower; “wastes” traffic on the loser |
Rule of thumb: want a reliable how much better? โ classic A/B. Want max conversions in a limited window โ bandit.
How to run an A/B test (5 steps)
- Set a goal + hypothesis – one primary metric; a measurable prediction (“a 20%-off banner will raise add-to-cart rate”).
- Build the variations – A = control, B = one change. More than one change = multivariate testing.
- Run it – split 50/50; run for your calculated duration (โฅ 2 weeks, โฅ ~1,000 visitors/variation).
- Analyze – check significance vs. your goal. Inconclusive? Qualitative feedback can still guide you.
- Implement + iterate – ship a significant winner to everyone; otherwise refine and re-run. It’s a loop.
If your sample size calculation says you need 10,000 visitors per variation and you get 50,000 per day, you might reach statistical requirements in a few days.
While high-traffic sites reach the 10,000-visitor mark in hours or days, stopping the test early remains unsafe. Ending a test after 2 days introduces day-of-week bias and fails to account for delayed conversions or long sales cycles.
Inconclusive tests
An inconclusive result occurs when a test fails to achieve statistical significance ($\alpha > 0.05$) within the precalculated sample size window.
- Why tests end up inconclusive:
- The actual impact was smaller than your Minimum Detectable Effect (MDE).
- High variance/noise in the target metric.
- The hypothesis was fundamentally incorrect.
- Actionable Next Steps: Accept, revert to the Control, re-evaluate qualitative user research, and pivot to testing higher-impact variables.
The ideal A/B test duration
The ideal experiment duration satisfies three conditions:
- It achieves the required statistical sample size.
- It spans at least 14 to 28 full calendar days (capturing 2-4 weekly cycles).
- It does not exceed 6 weeks (to prevent cookie churn and sample pollution).
Estimated Days = Required Sample Size per Variation x Number of Variations / Average Daily Visitors to Test Page
Frequently asked questions
How long should an A/B test run, in one sentence?
Long enough to collect the sample size your statistics requireโderived from your baseline rate, MDE, significance level, and powerโthen rounded up to full weeks, with a floor of one to two weeks regardless of how fast you hit the number.
How much traffic do I need for an A/B test?
It depends entirely on your baseline and the effect you want to detect. A large baseline detecting a large effect might need only a couple thousand visitors per variation; a small baseline detecting a small lift can need tens of thousands. As a practical floor, aim for at least 1,000 visitors per variation before trusting anything.
Can I stop my A/B test as soon as it hits significance?
No. Stopping the moment a result looks significant (“peeking”) inflates your false-positive rateโchecking a test ~20 times gives roughly a 50% chance of a spurious “significant” reading even when nothing is truly different. Decide the sample size and end date up front and hold to them, or use a sequential-testing method built for monitoring.
What does it mean if my A/B test is inconclusive?
Usually that the test lacked the power to detect the true effectโnot that the change had no effect. An underpowered test can miss a real, profitable lift. Note the power level, treat the result as “not visible at this sample,” and re-run with more traffic or a larger MDE if the decision matters.
Why do early A/B test results change so much?
Regression to the mean. Early numbers come from small, unrepresentative samples, so a dramatic day-three lift often shrinks toward the true effect as data accumulates. A 25% early lift can settle at 4%. Adequate sample size is the only reliable protection.
What’s the difference between statistical significance and statistical power?
Significance (ฮฑ) controls false positives- how much evidence you require before declaring a winner. Power (1 โ ฮฒ) controls false negatives- how likely you are to detect a real effect if one exists. Standard defaults are 95% confidence (ฮฑ = 5%) and 80% power.
Knowing how long to run an A/B test isn’t about picking several days. It’s about collecting enough data to make a reliable decision. Calculate your required A/B test sample size before launch using your baseline conversion rate, minimum detectable effect (MDE), statistical significance, and statistical power, and then estimate the duration based on your daily eligible traffic.
Resist the temptation to stop early when results look promising. Let the test run through at least one to two full weeks, avoid peeking, and treat inconclusive results as signals to review your sample size rather than proof that an idea failed. A disciplined testing process consistently produces better decisions than chasing quick wins.
Ready to run statistically sound A/B tests?
Launching an experiment is easy. Designing one that produces trustworthy, actionable results is much harder. At BrillMark, we help businesses plan experiments with the right sample size, statistical framework, and implementation strategy, then develop, QA, and launch tests that generate reliable insights and measurable revenue growth.
Whether you need A/B test development, Shopify experimentation, CRO support, or help building a scalable experimentation program, our team has delivered thousands of successful tests for global brands and agencies over the past 10+ years.
๐ Ready to stop guessing and start making data-backed decisions?
Contact BrillMark today to discuss your next A/B test and turn experimentation into a sustainable growth engine.