ML Model A/B Test Checklist for Recommendations and Pricing

An A/B test of an ML model is an experiment in which a random subset of users gets the new model’s decisions while the rest get the current one, and the company compares a business metric for the two groups. Before launch, you need to define the hypothesis, primary metric, sample size, and stopping rule in advance, check that the split is balanced, and agree on what decision will be made for each outcome. Below are 36 checklist items across eight stages, with owners and “done” criteria.

Checklist prepared by AIrassvet October 21, 2026 for product owners, e-commerce leaders, and analysts who are bringing recommendation, search ranking, dynamic pricing, or forecasting models into production. It is based on publications by researchers from Microsoft, Stanford, and Criteo experiment platforms. We did not run our own experiment for this article: the numbers in the examples are taken from primary sources or are clearly marked as calculations.

In brief
Who this checklist is for and what it does not cover
Why offline metrics are not enough
What you need before you start
Stage-by-stage checklist
How to calculate sample size: example
Copyable experiment spec template
Common mistakes
When A/B testing is not a fit
Frequently asked questions
Primary sources and methodology

In brief

Offline accuracy does not guarantee revenue growth. The decision to roll out a model is made based on an online experiment with real users.
Sample size is calculated before launch. Using the Kohavi et al. (2009) formula, a 5% conversion rate and 5% sensitivity require about 122,000 users in each group.
Unequal split is the main sign of a flawed test. In a Microsoft study (KDD 2019), sample ratio mismatch (SRM) was found in about 6% of experiments.
Peeking breaks the statistics. If you stop the test as soon as the p-value drops below 0.05, the share of false wins increases.
The decision is fixed in advance: what you do when the metric goes up, stays flat, or drops.

Who this checklist is for and what it does not cover

This checklist is for teams comparing two versions of an ML model or a model against the current rule on live traffic from a website, app, or catalog: recommendations, search, pricing, discounts, and forecasts with a direct impact on the user. We do not cover offline model validation, multi-armed bandits, Bayesian analysis, or how experimentation platforms are built. We described how to launch recommendations end to end in the article “Recommendation System for an E-Commerce Store”.

Terms. *Control (A)* — the group with the current model. *Test (B)* — the group with the new model. *Primary metric (OEC)* — one preselected metric used to make the decision. *MDE* — the minimum effect the test should be able to detect. *Power* — the probability of detecting an effect if one exists. *SRM (sample ratio mismatch)* — a mismatch between the actual and planned group proportions. *Guardrail metrics* — protective metrics that should not deteriorate.

Why offline metrics are not enough

A model is usually first evaluated on historical data: prediction accuracy, NDCG, recall. But historical data reflects users’ reactions to old recommendations, not new ones. Criteo researchers (Gilotte et al., 2018) explicitly describe offline evaluation as a preliminary step before an online test and show that traditional counterfactual estimates on their data do not deliver a satisfactory bias-variance tradeoff.

That is why offline metrics help filter out weak models, while the go-live decision is made by an A/B test. You can speed up early hypothesis screening with simulation, but it does not replace an experiment — see the detailed breakdown of A/B test simulation with AI agents.

What you need before you start

Two ready versions: the current model or rule (control) and the new model, both running on the same request stream.
A split mechanism: an experimentation platform or an in-house service that consistently assigns each user to one group.
Logs: impression, click, purchase, and rejection events with the user ID, group, and model version.
Historical data for several weeks to estimate metric variance and seasonality.
People and roles: product owner, analyst, ML engineer, developer; for pricing and discounts, a lawyer or risk specialist.

Stage-by-stage checklist

Check off an item only when the “done” criterion is met. Roles are flexible: in a small team, one person may cover several roles.

Stage 1. Hypothesis and decision

☐ The hypothesis is formulated as: “the new model will change metric X for users Y because Z” — *product owner*.
☐ The business decision that depends on the result is named: roll out, iterate, or reject — *product owner*.
☐ It is written in advance what you will do if the primary metric rises, shows no effect, or declines — *product owner, analyst*.
☐ The offline evaluation of the new model is no worse than the current one and is documented — *ML engineer*.

Done when: the hypothesis and decision rule are recorded in the experiment spec before launch.

Stage 2. Metrics

☐ One primary metric (OEC) tied to revenue or retention is selected: revenue per user, conversion rate, share of successful searches — *product owner, analyst*.
☐ Two to five guardrail metrics are defined: response time, error rate, returns, unsubscribes, margin — *analyst*.
☐ For each metric, the formula, unit of analysis, and data source are recorded — *analyst*.
☐ It is verified that the metric is calculated the same way in both groups and from the same logs — *analyst, developer*.
☐ It is decided how outliers will be handled: for example, capping extreme orders — *analyst*.

Done when: the metric formulas are agreed upon and calculated on historical data.

Stage 3. Sample size and duration

☐ The variance of the primary metric is estimated from historical data — *analyst*.
☐ An MDE is set that makes business sense, not just “any increase” — *product owner*.
☐ Sample size per group is calculated at a 5% significance level and at least 80% power — *analyst*.
☐ The duration covers at least one full weekly cycle and is long enough to reach the calculated sample size — *analyst*.
☐ If traffic is insufficient, a metric with lower variance or variance reduction using pre-experiment data (CUPED) is considered — *analyst*.

Done when: sample size, duration, and end date are recorded and do not change during the test.

Stage 4. Randomization and split

☐ The randomization unit has been selected: user, session, store, region — and it matches the analysis unit — *analyst, developer*.
☐ The user is consistently assigned to the same group across repeat visits and on different devices, as much as possible — *developer*.
☐ Bots, employees, and test accounts are excluded in the same way in both groups — *developer, analyst*.
☐ An A/A test or split check has been run: unchanged groups do not differ on the metrics — *analyst*.
☐ Overlap with other experiments on the same pages has been checked — *product owner*.

Done when: The A/A check did not show significant differences, and group shares match the plan.

Stage 5. Model and Infrastructure

☐ Each event log includes the model version and experiment group — *ML engineer, developer*.
☐ The response speed of the new model under load is not worse than the agreed threshold — *ML engineer*.
☐ There is a fallback option: if the model errors out, the user receives an answer from the current version — *developer*.
☐ Model features are computed in real time the same way as during training — *ML engineer*.
☐ There is an experiment kill switch that sends all traffic back to control without a release — *developer*.

Done when: The load test has been completed, and the kill switch has been manually verified.

Stage 6. Launch and Monitoring

☐ During the first few hours, the test runs on a small share of traffic to check for errors — *developer*.
☐ SRM is checked daily: the actual group shares are compared with the plan using a statistical test — *analyst*.
☐ Guardrail metrics are monitored with emergency stop thresholds — *analyst, ML engineer*.
☐ No winner decision is made before the planned end date unless a sequential analysis method is being used — *analyst*.

Done when: The test reached the planned date without SRM and without triggering any guardrail thresholds.

Stage 7. Analysis

☐ If SRM is found, the results are not interpreted until the cause is identified — *analyst*.
☐ The effect is reported with a confidence interval, not just a p-value — *analyst*.
☐ The day-by-day effect trend has been checked for novelty effect: an early lift that later disappears — *analyst*.
☐ Segment breakdowns are defined in advance, and those found after the test are labeled as hypotheses for the next experiment — *analyst*.

Done when: The report includes the effect, interval, guardrail metrics, SRM check, and limitations.

Stage 8. Decision and Rollout

☐ The decision was made according to the rule from Stage 1, not based on the most favorable breakdown — *product owner*.
☐ The rollout is phased, and a small control group (holdout) remains to track long-term impact — *product owner, developer*.
☐ The experiment record, report, and model version are saved in the shared experiment database — *analyst*.
☐ A date has been set for the next model check for data drift — *ML engineer*.

Done when: The new version is on all traffic, the report is available to the team, and the model owner has been assigned.

How to Calculate Sample Size: Example

Kohavi et al. (2009) provide an approximate formula for a 5% significance level and 80% power with two equally sized groups:

n = 16 · σ² / Δ², where *n* is the number of users in each group, *σ²* is the variance of the main metric, and *Δ* is the absolute effect you want to detect. For 90% power, the coefficient 16 is replaced with 21.

Main metric	Source data from the Kohavi et al. article	Calculation	Users in each group
Purchase conversion rate	conversion rate of 5%, need to detect a 5% change relative to it	16 · 0.05 · 0.95 / (0.05 · 0.05)²	≈ 121,600
Revenue per user	average $3.75, standard deviation $30, change of 5%	16 · 30² / (3.75 · 0.05)²	≈ 409,600

The calculation was performed by AIrassvet using the formula and source data from the article. The comparison shows the authors’ practical conclusion: a metric with lower variance, such as conversion instead of revenue, reduces the required sample size — in their estimate, by 3.3 times in this example. A second approach is CUPED: at Bing search, the method reduced variance by about 50%, meaning the same power with half the users or half the duration (Deng et al., 2013).

Copyable Experiment Charter Template

Copy the template into your team knowledge base and fill it out before launching the test.

A/B TEST CHARTER FOR AN ML MODEL
Experiment name and ID:
Product owner / analyst / ML engineer:
Hypothesis: the new model ___ will change ___ for ___ because ___
Control (A): model/rule, version ___
Test (B): model, version ___, training date ___
Main metric (OEC), formula, analysis unit:
Guardrail metrics and emergency stop thresholds:
Randomization unit and group shares (for example, 50/50):
Traffic exclusions (bots, employees, test accounts):
Metric variance (evaluation period):
MDE, significance level, power:
Sample size per group:
Start date / planned end date:
Stop rule (fixed duration or sequential method):
Decision if it goes up / stays flat / goes down:
Predefined segments for analysis:
Result: effect, confidence interval, SRM check, limitations:
Decision made and date:

Common Mistakes

Stopping at the first sign of significance. Johari et al. (2015) show that standard p-values and confidence intervals become unreliable if a test is monitored continuously and stopped based on the result; sequential methods are needed for this.
Ignoring SRM. Fabijan et al. (KDD 2019) describe SRM as a common sign of data issues: causes include group assignment errors, lost logs, bot filtering, and latency in one of the variants.
Novelty effect and habituation. Kohavi et al. (2009) warn that experienced users may initially react worse to a navigation change, while interest in something new can temporarily inflate the result.
Too many metrics without a primary one. If a report contains twenty metrics, at least one will randomly appear “significant”; decisions should be made based on one preselected metric.
Different logs across groups. If the new model logs events differently, metric comparison is meaningless — verify this in an A/A test.

When an A/B Test Is Not Suitable

A user-level A/B test is not suitable or requires a different design in the following cases:

Low traffic. If reaching the planned sample size would take months, first compare the models offline, choose a metric with lower variance, or test on a more frequent event.
Users influence each other. In marketplaces, taxi, and delivery, a new model in one group may change the offer for the other group; in that case, randomize by regions or by time intervals.
Prices and discounts. Different prices for different customers require legal review and attention to reputational risk; these are often tested by store or region. For more on the risks, see the article on dynamic pricing with AI.
Rare decisions with a high cost of error. Credit or medical decisions require separate validation procedures that this checklist does not replace.

Frequently Asked Questions

How long should an A/B test of an ML model run?

Long enough to reach the required sample size, but no shorter than a full weekly cycle. You determine the duration before launch based on sample size and traffic: behavior differs on weekdays and weekends, and a short test can capture a novelty effect. Kohavi et al. (2009) note that the number of visitors increases roughly linearly with experiment time. You cannot change the test duration midway because of interim results.

Can you compare more than two models in one test?

Yes, but each additional group reduces traffic per variant and increases the risk of an accidental “winner.” Kohavi et al. (2009) give a more conservative sample size formula for multiple variants: n = (4rσ/Δ)², where r is the number of variants. In practice, candidates are first screened offline, and the one or two best models are moved into the online test.

What should you do if the group split does not match the plan?

Stop interpreting the results and look for the cause. Fabijan et al. link sample ratio mismatch (SRM) to assignment errors, lost or delayed logs, bot filtering, and redirects. Until the cause is found, group metrics cannot be compared: even a large effect may be explained by different users being assigned to the groups. After the issue is fixed, the test is usually restarted.

Do you need an experimentation platform, or is your own code enough?

For the first tests, your own split service and clean logging are enough if the team checks for SRM and runs an A/A test. A platform is needed when the number of experiments grows: it maintains the registry, tracks overlaps, and automatically calculates metrics. In both cases, discipline in the experiment brief matters more than the tool you choose.

How do you test a catalog search model?

The same way as recommendations, but the main metric is usually the share of searches that lead to a purchase or add-to-cart action, while guardrail metrics include the share of empty results and response time. The randomization unit should be the user, so one person does not see different result sets. We described how search itself works in the article about smart search for an online store.

Primary sources and methodology

Kohavi, Longbotham, Sommerfield, Henne. Controlled experiments on the web: survey and practical guide, 2009 — sample size formula, example with conversion and revenue, novelty effects.
Fabijan et al. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD 2019 — frequency and causes of SRM.
Deng, Xu, Kohavi, Walker. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED), WSDM 2013 — variance reduction.
Johari, Pekelis, Walsh. Always Valid Inference: Bringing Sequential Analysis to A/B Testing, 2015 — the problem of continuous monitoring.
Gilotte et al. Offline A/B testing for Recommender Systems, 2018 — limitations of offline evaluation.
Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments, Cambridge University Press — a reference book on experiments.

How the checklist was created. We broke down recommendations from the primary sources into eight stages of the experiment lifecycle and, for each item, specified the role and the criterion to verify. Items about ML model infrastructure (log versioning, fallback, feature computation) are engineering best practices, not research findings. Limitations: we did not validate the checklist in our own experiment, and the figures from the papers apply to the authors’ conditions.

How AIrassvet helps validate an ML model in an A/B test

A good model only creates value if its impact is measured honestly. On the AIrassvet website you can find the areas that address this task:

ML forecasting and predictive analytics: dynamic pricing, churn prediction, and inventory optimization models that can be compared with the current rule in an experiment;
process audit and data preparation: selecting the primary metric and checking logs and historical data to calculate sample size;
integration, testing, and deployment: embedding the model into the product with logged versions, a fallback option, and staged rollout.

The first step is to choose one process where the model should make the decision and document the current outcome, data sources, constraints, and acceptance criteria.

Discuss the project

Bottom line: the test decides, not the model

An A/B test of an ML model is a way to turn offline accuracy into a verified business outcome. Most mistakes are made before launch: a vague hypothesis, multiple primary metrics, an unaccounted sample size, and a split without an A/A check.

That is why you should complete the experiment brief, go through all 36 checklist items by stage, and agree in advance on the decision for each possible outcome. Then the test result becomes a trigger for action, not a debate about how to read the numbers.