Marketing Release (Geo-testing)
Overview
Marketing Release measures the impact of a change launched in some markets but not others, such as a campaign, a price change or a feature rolled out market by market.
It compares the markets that got the change (the tested markets) with a synthetic control. The synthetic control is a weighted mix of the markets that didn't get the change (the donors), built so that it tracked the tested markets closely before the launch. After the launch, the gap between the tested markets and their synthetic control is the effect.
A "market" can be any segment value: a country, a region or a store type.
Release Impact or Marketing Release?
|
|
Release Impact | Marketing Release |
|---|---|---|
| Released to | The whole population | Specific markets |
| Compares against | The same users' trend before the release | Markets that didn't get the change |
| Use it for | Product changes released to everyone | Campaigns and changes launched in some markets only |
| Strength | User-level data, so it can detect smaller effects | Absorbs events that hit all markets (holidays, competitors) |
| Main risk | Other events after the release | Events that hit only the tested markets |
Release Impact removes the effect of changes in user mix. A campaign mostly works by changing the user mix, so measure campaigns with Marketing Release.
Analysis assumptions
- The donors weren't affected by the change. None of them got it, and it didn't spill over to them.
- Before the launch, the tested markets and the donors moved together.
- Nothing else happened only in the tested markets during the analysis window. Events that hit all markets, such as holidays, are fine: the donors absorb them.
Best practices
- Pre-release period: 8–13 weeks. The model needs this history to learn how the tested markets move with the donors. Keep it at least 45 days longer than the post-release period, so the null days test can run. Avoid windows that include a tracking change.
- Post-release period: whole weeks, long enough for the effect to settle. Campaigns often need 6–12 weeks. Don't go longer than needed, because uncertainty grows with each day.
- Donors: every market that isn't tested is a donor by default. Use the population filter to exclude a market that had its own campaign or change during the window. At least 5 donors are recommended.
- Several tested markets: they are combined for the main result, and each one also gets its own result.
Setting up the analysis
| Field | What it does |
|---|---|
| Tested population | The segment value(s) that got the change, e.g. Country is GB, all other values will be used as potential donors. You can exclude donors by using the population filter field (see below). |
| Release date | The day the change went live |
| Pre-release / post-release period | The windows before and after the release date |
| Primary KPI / secondary KPIs | The KPIs to measure. The primary KPI decides the headline result. |
| Population filter (optional) | Limits the data, including which markets can be donors |
| Impact estimation | Average daily effect, or the cumulative effect over the post-release period |
| Confidence threshold | The confidence needed to call a winner (default 90%) |
Reading the results
- Summary table: each KPI in the tested markets vs. its synthetic control, with the change and its confidence. A trophy marks a significant improvement. Each tested market has its own row.
- vs. pre-release KPI: a toggle that shows the naive before/after comparison, for reference only. Use the synthetic-control result for decisions.
- KPI over time: the tested markets' actual KPI, the synthetic control and its uncertainty band.
- Donor weights: how much each donor contributes to the synthetic control.
- Validation level (High / Low): click it to see which checks passed.
Validation checks
Each KPI gets a validation level. High confidence means the result passed the checks, or failed only minor ones. Low means it failed a serious check or several smaller ones. Treat a Low confidence result with caution and look at which checks failed.
| Check | Fails when | What to do |
|---|---|---|
| Ran Null Days Test | The test couldn't run: the pre-release period is too short | Use a longer pre-release period |
| Null Days Test | The effect isn't larger than the "effects" found on 30 earlier dates with no release | Look for other events before the release |
| Change in prediction accuracy | The model's error didn't grow after the release more than after those earlier dates | Same as above |
| Ran Geo Null Test | Too few donors to run the test | Add donors |
| Geo Null Test | The effect isn't larger than the "effects" found when each donor is treated as if it got the change | Make sure no donor got a similar change |
| Prediction Error Ratio Test | The model's error grew about as much in the donors as in the tested markets | Same as above |
| Pre-existing trend | The tested markets were already drifting away from the donors before the release | Treat the effect as an upper bound |
| Unstable confidence | The confidence only just crossed the threshold | Wait a few more days |
| Missing dates | Many days are missing around the release date | Check the data on those dates |
| Sparse KPI | The KPI is zero on more than 15% of days | Use a broader population or KPI |
| Enough post-release days | Fewer than 5 days of post-release data | Wait for more data |
Further reading
- Brodersen, K. H. et al. (2015). Inferring causal impact using Bayesian structural time-series models. Annals of Applied Statistics, 9(1), 247–274.
- Abadie, A., Diamond, A. & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies. Journal of the American Statistical Association, 105(490), 493–505.
- Help Center: Release Impact