What Is It and How Does It Happen? Identification, Root Causes, Bayesian Vulnerability, and Remediation
In digital experimentation and online controlled experiments (A/B testing), data integrity is the primary prerequisite for valid decision-making. Leading product engineering organizations rely on A/B testing to measure the causal impact of new features, algorithmic adjustments, and user interface updates.
However, when the underlying data collection or assignment mechanism breaks, the experiment produces biased metrics that can lead teams to ship harmful change.
What is a “Sampleโ?
In online controlled experimentation, the term sample describes the group of unique experimental units allocated to and evaluated within each variant of a test.
A fundamental distinction must be drawn between the unit of randomization and the unit of interaction:
- Unit of Randomization (The True Sample): The unique identifier assigned to an experimental arm, such as a persistent User ID, account UUID, or browser cookie. This unique unit defines the sample size.
- Unit of Interaction (Sessions/Visits): The cumulative actions, visits, or event telemetry generated by assigned units over time.
Before launching an experiment, engineers configure an intentional traffic allocation ratio. Standard split configurations include:
- Equal Splits: 50/50 for a two-arm test; 33.3/33.3/33.3 for a three-arm test.
- Unequal Splits: 90/10 or 80/20 splits, frequently used during risk-mitigated feature rollouts or canary deployments.
What is a Sample Ratio Mismatch (SRM)?
A Sample Ratio Mismatch occurs when the ratio of observed unique units assigned to each variant diverges from the pre-configured expected ratio by a magnitude that is statistically near-impossible under uncorrupted random assignment.
To understand this conceptually, consider flipping a fair coin:
- If you flip a coin 100 times and observe 52 heads and 48 tails, the minor deviation is expected random variation ($p = 0.688$).
- If you flip a coin 100 times and observe 80 heads and 20 tails, the probability of seeing such an extreme split by chance is approximately $0.000000055$ ($p < 0.0001$).
In an A/B test configured for a 50/50 split, observing a 60/40 count split across 100,000 users is the statistical equivalent of a loaded coin. It proves that the randomization mechanism or data logging pipeline is broken.
Why Sample Ratio Mismatch is a Critical Problem in A/B Testing
An SRM is not merely a cosmetic tracking glitch; it is a critical threat to data integrity that invalidates experimental conclusions.
Severe Selection Bias and Loss of Unconfoundedness
The primary objective of random assignment is to ensure unconfoundedness. Randomization balances both observed and unobserved confounding variables, such as user intent, device power, network speed, and geographic location, identically across control and treatment groups.
When an SRM occurs, random assignment has failed. This introduces selection bias, meaning specific sub-populations of users are systematically over-represented or under-represented in one group. Any observed metric difference between Control and Treatment may be caused entirely by baseline demographic differences rather than the feature being tested.
Explosion of False Positives (Type I Errors)
Ignoring an SRM drastically increases the rate of false discoveries. Industry empirical benchmarks show that experiments suffering from an unaddressed SRM generate up to double the rate of statistically significant results compared to clean tests. Product teams end up declaring “winning” variations that are actually neutral or harmful to the user experience.
Reduction of Statistical Power and MDE Distortion
When sample ratios drift away from planned allocations, the effective sample size decreases relative to the optimal split. This reduces statistical power, widening confidence intervals and increasing the Minimum Detectable Effect (MDE). Genuine improvements made by product teams become harder to detect.
Structural Reasons Why Sample Ratio Mismatch Occurs
An SRM acts as a non-specific symptom for a wide variety of engineering bugs across the software lifecycle. Microsoft ExP and DoorDash research categorize SRM root causes into three distinct operational stages:
Stage 1: Assignment Stage Failures
Assignment failures occur when the hashing or bucketing engine fails to allocate traffic according to the target probabilities.
Inconsistent User Identifiers: If the system switches between anonymous browser cookies, device UUIDs, and logged-in account numbers, users are counted multiple times or dropped unevenly across variants
Asymmetric Feature Ramp-Ups: Ramping Treatment from 10% to 50% while leaving Control at 50% without resetting or properly segmenting cumulative user reporting creates an artificial SRM in cumulative scorecards.
Stage 2: Execution Stage Failures
Execution failures happen when assignment logic works correctly, but client-side runtime errors or latency prevent assigned users from rendering or logging the experience.
Client-Side JavaScript Crashes: If a treatment variant contains a JS error that crashes older mobile browsers, affected users fail to fire exposure logs. Control logs the user, but Treatment loses them, creating a sample imbalance.
Redirect Latency and Bounce Rates: In redirect tests (where Control is site.com/page and Treatment redirects to site.com/page-v2), redirect latency causes users on slow connections to bounce before the treatment page loads. This filters low-bandwidth users out of Treatment, introducing severe selection bias
The “Bugfix Bias“: If engineers fix a mid-experiment treatment bug and update the dashboard start date without re-randomizing users, an SRM occurs. Low-intent users who encountered the bug may have already churned, leaving an artificially high-intent population in Treatment.
Stage 3: Log Processing and Telemetry Failures
Log processing failures occur within data transport pipelines, ETL jobs, and warehouse ingestion layers.
- Missing Counterfactual Trigger Logging: In triggered A/B tests (evaluating users who reach a specific feature), exposure logs must fire symmetrically. If Treatment logs an event upon rendering a new element, but Control fails to log an equivalent “counterfactual” event for viewing the baseline location, Treatment will show an artificially inflated sample.
- Asymmetric Bot Filtering: Automated scrapers interact differently with variants if DOM structures or response times change. Upstream bot-detection rules may aggressively strip session logs from one arm, triggering an SRM downstream.
Can SRM Still Affect Tests Using Bayesian A/B Testing?
A common misconception is that switching from Frequentist to Bayesian A/B testing eliminates or bypasses the need for Sample Ratio Mismatch checks. This is false.
The Fallacy of Bayesian Immunity
Statistical frameworks (Frequentist or Bayesian) are mathematical mechanisms for drawing inferences from data. Both frameworks operate on the fundamental assumption that the data-generating process is unconfounded and properly randomized.
In Bayesian A/B testing, the posterior distribution P(ฮธ | D) of the conversion rate ฮธ given the observed data D is derived using Bayes’ theorem:
P(ฮธ | D) = [P(D | ฮธ) ร P(ฮธ)] / P(D)
Where:
- P(ฮธ) is the prior distribution.
- P(D | ฮธ) is the likelihood function.
- P(D) is the marginal likelihood (also called the evidence).
When a Sample Ratio Mismatch (SRM) occurs, the observed data D becomes corrupted by selection bias. The likelihood function P(D | ฮธ) assumes that observations are exchangeable and that users are randomly assigned to each variation.
When sample selection is biased, this assumption no longer holds. As a result, the likelihood function mathematically models the combined effect of the feature change and the sample selection bias, rather than isolating the true treatment effect. Consequently, the posterior distribution P(ฮธ | D) becomes biased, leading to misleading probability estimates and potentially incorrect business decisions.
Garbage In, Garbage Out
No choice of prior distribution P(ฮธ) can correct a structurally biased likelihood function.
If selection-biased data enters a Bayesian model:
- The posterior probability distribution P(ฮธ | D) concentrates around a biased estimate.
- Outputs such as Probability of Superiority (P(Treatment > Control)) or Expected Loss may declare high-confidence winners based on corrupted input data.
- Prior-predictive checks and credible intervals accurately reflect the uncertainty of the biased sample, not the true underlying population.
In other words, Bayesian inference is only as reliable as the data it receives. If the observed data is systematically biased because of an SRM, the posterior distribution will faithfully quantify that biased evidence rather than recover the true treatment effect.
Sample Ratio Mismatch Identification in A/B Testing
Pearson’s Chi-Squared (ฯยฒ) Goodness-of-Fit Test
The standard mathematical method for detecting a Sample Ratio Mismatch (SRM) is Pearson’s Chi-Squared Goodness-of-Fit Test. The test evaluates the null hypothesis (Hโ) that the observed user counts across variants follow the planned allocation probabilities.
For an experiment with k variants:
ฯยฒ = ฮฃ [(Oแตข โ Eแตข)ยฒ / Eแตข], for i = 1 to k
Where:
- Oแตข is the observed unique user count for variant i.
- Eแตข = N ร pแตข is the expected unique user count.
- N = ฮฃOแตข is the total observed sample size.
- pแตข is the planned traffic allocation for variant i.
The test statistic follows a chi-squared (ฯยฒ) distribution with k โ 1 degrees of freedom (df = k โ 1).
Practical Calculation Example
Consider a two-armed A/B test (k = 2) with a planned 50/50 split:
- pโ = 0.50
- pแตฆ = 0.50
Suppose the experiment records 1,000,000 total unique users.
Expected counts
- Eโ = 500,000
- Eแตฆ = 500,000
Observed counts
- Oโ = 503,000 (50.3%)
- Oแตฆ = 497,000 (49.7%)
Applying the chi-squared formula:
ฯยฒ = (503,000 โ 500,000)ยฒ / 500,000 + (497,000 โ 500,000)ยฒ / 500,000
= 9,000,000 / 500,000 + 9,000,000 / 500,000
= 18 + 18
= 36
For df = 1, a ฯยฒ value of 36 corresponds to a p-value โ 0.0000000019 (p < 0.0001).
Although the observed allocation of 50.3% vs. 49.7% appears nearly balanced at first glance, the chi-squared test decisively rejects the null hypothesis (Hโ), demonstrating that the deviation is far too large to be explained by random chance alone. This provides strong statistical evidence that a Sample Ratio Mismatch (SRM) is present.
Decision Thresholds (p < 0.01 vs. p < 0.001)
While many hypothesis tests use a significance level of ฮฑ = 0.05, experimentation platforms such as Statsig, SiteSpect, Datadog, and Microsoft ExP typically use much stricter thresholds for Sample Ratio Mismatch (SRM) detection, such as p < 0.01 or p < 0.001.
Online experiments often include hundreds of thousands or even millions of users. At these sample sizes, the chi-squared test becomes extremely sensitive, meaning even trivial random fluctuations can produce statistically significant results.
Using stricter thresholds such as p < 0.01 or p < 0.001 reduces false alarms while still detecting genuine randomization failures and structural issues.
Sequential SRM Detection
Applying a standard chi-squared test repeatedly to streaming experiment data creates a peeking problem, where continuously checking the data increases the probability of false-positive SRM alerts.
To address this, modern experimentation platforms use Sequential SRM Testing, which enables continuous monitoring while controlling the overall Type I error rate.
Group Sequential Pearson Tests
One approach is to analyse data in scheduled batches (for example, daily) using alpha-spending functions, where the cumulative significance level satisfies:
ฮฃ ฮฑโ โค ฮฑ, for k = 1 to โ
This ensures that the overall probability of a false-positive SRM detection remains below the desired significance level.
Bayesian Supermartingale Bounds
Another approach monitors a non-negative supermartingale O(n). Under the null hypothesis (Hโ), Ville’s inequality guarantees:
P(supโโฅโ O(n) โฅ 1/ฮฑ) โค ฮฑ
This framework allows experimentation platforms to monitor SRM continuously in real time without inflating the false-positive rate.
The Golden Rule: Users vs. Sessions
SRM should always be evaluated using unique assigned users (Uโidโ), not raw session or visit counts.
Suppose a treatment successfully increases engagement. Users in the treatment group may return to the website or app three times more frequently than users in the control group. If SRM is evaluated using sessions, the treatment will appear heavily overrepresented, even though the original user assignment was perfectly random.
A session imbalance in this situation reflects higher engagement, not a randomization failure.
As long as the allocation of unique users satisfies the SRM test (p โฅ 0.01, or the platform’s chosen threshold), the experiment’s randomization remains valid.
Metric Inversion and Simpson’s Paradox
An SRM can distort metrics at both aggregate and sub-segment levels. For example, an SRM that over-indexes high-spending iOS users in Treatment can make aggregate revenue look positive. However, evaluating performance within iOS and Android sub-segments individually may show negative conversion lifts across every cohort. This structural artifact is known as Simpson’s Paradox.
Triggered vs. Untriggered Scorecard Discrepancies
Experimentation platforms typically generate two types of scorecards:
- Untriggered Scorecards: Include all assigned users, regardless of whether they interacted with the tested feature.
- Triggered Scorecards: Include only users who were actually exposed to or interacted with the tested feature.
If the Untriggered Scorecard passes the SRM check (p โฅ 0.01) but the Triggered Scorecard fails (p < 0.001), this usually indicates that the randomization engine is functioning correctly. Instead, the problem lies in the event-triggering logic, exposure tracking, or counterfactual logging for the treatment variant.
In other words, users are being assigned correctly, but the system responsible for recording feature exposure or trigger events is introducing bias into the triggered population.
Impact of SRM on Experiment Scorecards
| Scorecard Element | Status in a Clean Experiment | Status Under Active SRM | Operational Impact |
| Primary Metric (OEC) | Measures the true treatment effect (ฮฯ). | Measures the combined effect of the treatment and population bias. | High risk of shipping harmful or ineffective features. |
| Guardrail Metrics | Correctly monitors application performance and user experience. | Can be distorted by differential user dropouts or selection bias. | Performance regressions may go undetected. |
| Segment Scorecards | Shows consistent trends across user cohorts. | Becomes susceptible to Simpson’s Paradox, where aggregated and segmented results conflict. | Can lead to contradictory conclusions across segments. |
| Triggered Scorecards | Accurately measures users who interacted with the feature. | Often exhibits severe allocation imbalances when exposure logging is faulty. | Helps identify localized instrumentation or event-logging bugs. |
How to Fix SRM: Troubleshooting and Remediation Protocols
When a Sample Ratio Mismatch (SRM) alert is detected, experimentation teams should follow a structured troubleshooting process to identify the root cause before trusting or interpreting any experiment results.
Step 1: Verify the Identifier Unit
Confirm that the SRM calculation is based on unique users, not sessions or visits.
Using session counts can produce false SRM alerts because treatment users may naturally generate more sessions if the feature increases engagement.
โ
Step 2: Check the Experiment Configuration
Review the experiment setup to ensure:
- The traffic allocation ratio was not changed after the experiment started.
- Variants were assigned according to the intended split.
- No configuration or targeting rules were modified during the test.
โ
Step 3: Perform a Temporal Onset Audit
Determine when the SRM first appeared.
- Day 1: Usually indicates a problem with random assignment or traffic allocation.
- Mid-test: Often points to a deployment, bug fix, instrumentation change, or infrastructure issue introduced after launch.
โ
Step 4: Perform Segment Decomposition
Calculate SRM separately across important user segments, such as:
- Operating system
- Browser
- Geographic region
- Device type
If the SRM appears only within a specific segment, the problem is likely localized rather than affecting the entire experiment.
โ
Step 5: Audit Trigger Logs
Compare the Untriggered Scorecard with the Triggered Scorecard.
- If both fail the SRM test, investigate the assignment or randomization process.
- If only the Triggered Scorecard fails, investigate feature exposure tracking, event instrumentation, or counterfactual logging.
โ
Step 6: Patch and Re-Salt
After identifying the root cause:
- Fix the underlying code or configuration issue.
- Discard all corrupted experiment data.
- Restart the experiment using a new salt string (or randomization seed) to ensure completely fresh user assignment.
Only after these steps are completed should a new experiment be launched and its results considered trustworthy.
Step-by-Step Diagnostic Workflow
1. Verify the Metric Unit
Ensure the chi-squared test is calculated using unique users (Uโidโ) rather than session or visit counts.
Using sessions instead of users can generate false SRM alerts because successful treatments often increase user engagement and therefore the number of sessions.
2. Check the Configuration History
Review the experiment configuration to determine whether the traffic allocation percentages were changed after the experiment began.
If allocation ratios were modified during the experiment, perform separate chi-squared tests for each period in which the allocation remained constant. Combining data from different allocation windows can produce misleading SRM results.
3. Isolate the Temporal Onset
Plot daily user assignment or exposure counts throughout the experiment.
- An SRM present from Day 1 typically indicates problems with the assignment engine, hash collisions, or targeting logic.
- An SRM that appears mid-experiment usually points to a secondary deployment, schema change, telemetry issue, or bug fix introduced after launch.
Identifying when the imbalance first appeared significantly narrows the search for the root cause.
4. Perform Segment Grid Decomposition
Analyse user allocation separately across important dimensions, including:
- Device type
- Operating system
- Browser
- Geographic region
If a significant SRM (p < 0.001) occurs only within a specific segment, such as a single browser version, the issue is likely caused by a client-side script, browser compatibility problem, or platform-specific implementation bug.
5. Audit Trigger Logic
Compare the SRM results for the Untriggered Scorecard and the Triggered Scorecard.
A mismatch that appears only in the triggered population often indicates missing exposure events, faulty trigger conditions, or incomplete counterfactual logging rather than a failure of the randomization process.
The Re-Salting Protocol
Once the root cause such as a JavaScript error, redirect failure, or telemetry loss has been identified and corrected, do not resume the existing experiment or reuse its data.
Any data collected while an SRM was present is fundamentally biased and cannot be repaired through statistical adjustments.
Instead, engineers should update the experiment’s randomization salt:
SaltโEโ โ SaltโE_v2โ
Changing the salt generates an entirely new random assignment for users, ensuring that the restarted experiment is independent of the corrupted run. This prevents carryover effects, eliminates retention bias, and restores the validity of the experiment’s randomization.
Best Practices for Preventing Sample Ratio Mismatch (SRM)
Automate Real-Time Guardrails
Implement automated SRM monitoring using Pearson’s Chi-Squared Test or Sequential SRM Tests with a detection threshold of p < 0.01. Perform all calculations using unique users (Uโidโ) rather than sessions, and automatically pause or terminate experiments when a statistically significant SRM is detected.
Ensure Symmetrical Telemetry and Execution
Verify that client-side experiment code executes consistently across all supported browsers, operating systems, and devices. Ensure that exposure events, telemetry, and counterfactual trigger logs are recorded identically for both the control and treatment groups so that neither variant is systematically underreported.
Maintain Allocation Integrity
Prevent hash or salt collisions across concurrent experiments, minimise redirect latency during user assignment, and generate a new randomization salt whenever a bug fix or experiment configuration change requires the test to be restarted. Re-salting ensures that the new experiment begins with an independent and unbiased user allocation.
Contact Us for A/B Testing Support
Detecting Sample Ratio Mismatch is only the beginning.
The real value comes from building experiments with accurate traffic allocation, validated analytics, and rigorous QA so you can trust every result and make confident optimization decisions.
Brillmark helps businesses move beyond basic experimentation with analytics implementation, conversion optimization, A/B testing development, QA validation, and full-service experimentation programs. Our team has supported 200+ agencies and global brands in building reliable experimentation programs that drive measurable business growth.
Whether you need help diagnosing Sample Ratio Mismatch, validating your tracking implementation, improving experiment quality, or building a structured experimentation program, Brillmark can help.