Holdout Test
Withholds a marketing intervention from a randomly-selected group to measure incremental lift.
Last updated
Definition
The simplest causal experiment: split your audience randomly, show the ad to one group, suppress it from the other, measure the conversion-rate gap. The gap IS the incremental lift. Cleaner than DiD because randomization eliminates confounders. Limited by platforms — Meta and Google offer programmatic holdouts (Brand Lift studies, Conversion Lift) that ship the math for you.
How it applies in India
On WhatsApp in India a holdout costs little to run: a reminder that is not sent is a template message that is not charged.
An example
A store sends a cart-recovery message to shoppers who left items behind. Recovered carts look like proof that it works, but some of those shoppers would have come back anyway. Holding back one shopper in ten, chosen at random, gives the answer: the difference in return rate between the two groups is what the message added.
How WatEase uses it
WatEase builds holdouts into the places where it sends on its own. Cart and browse recovery drip campaigns hold back 10% of shoppers by default (anything from 5% to 50% is allowed), chosen by a stable hash so the same shopper stays in the same group, and the Carts page compares return rates once each side has 200 baskets. Broadcasts launched by an AI agent can hold back up to half the audience and measure orders and replies over 14 days. Website offers, the store agent and cart suggestions keep holdouts too, and ads get a quarterly town holdout. Holdouts are what keep WatEase’s agentic marketing honest: agents act, and the holdout shows what the action was worth.
Frequently asked questions
How big is the holdout on WatEase’s recovery messages?
Ten per cent of shoppers by default, adjustable between 5% and 50%. The comparison waits until each side has 200 baskets, so an early streak of luck does not decide it.
Related terms
- IncrementalityThe lift caused BY a marketing intervention, vs. what would have happened without it.
- Causal InferenceStatistical methods that estimate causal effects from observational or experimental data.
- Difference-in-Differences (DiD)A causal method that compares the change in outcome over time between a treated group and a control group.
Try WatEase
Run your WhatsApp commerce on the platform built for India — Cloud API, GST invoices, UPI checkout, opt-in tracking, and 0% markup on WhatsApp messages (competitors add 20–25%). Pricing is on request — quoted by the WatEase team.
Get Started Today