Marketing Release (Geo-testing)

Overview

Marketing Release measures the impact of a change launched in some markets but not others, such as a campaign, a price change or a feature rolled out market by market.

It compares the markets that got the change (the tested markets) with a synthetic control. The synthetic control is a weighted mix of the markets that didn't get the change (the donors), built so that it tracked the tested markets closely before the launch. After the launch, the gap between the tested markets and their synthetic control is the effect.

A "market" can be any segment value: a country, a region or a store type.


Release Impact or Marketing Release?


Release Impact Marketing Release
Released to The whole population Specific markets
Compares against The same users' trend before the release Markets that didn't get the change
Use it for Product changes released to everyone Campaigns and changes launched in some markets only
Strength User-level data, so it can detect smaller effects Absorbs events that hit all markets (holidays, competitors)
Main risk Other events after the release Events that hit only the tested markets

Release Impact removes the effect of changes in user mix. A campaign mostly works by changing the user mix, so measure campaigns with Marketing Release.

Analysis assumptions

  1. The donors weren't affected by the change. None of them got it, and it didn't spill over to them.
  2. Before the launch, the tested markets and the donors moved together.
  3. Nothing else happened only in the tested markets during the analysis window. Events that hit all markets, such as holidays, are fine: the donors absorb them.

Best practices

  • Pre-release period: 8–13 weeks. The model needs this history to learn how the tested markets move with the donors. Keep it at least 45 days longer than the post-release period, so the null days test can run. Avoid windows that include a tracking change.
  • Post-release period: whole weeks, long enough for the effect to settle. Campaigns often need 6–12 weeks. Don't go longer than needed, because uncertainty grows with each day.
  • Donors: every market that isn't tested is a donor by default. Use the population filter to exclude a market that had its own campaign or change during the window. At least 5 donors are recommended.
  • Several tested markets: they are combined for the main result, and each one also gets its own result.

Setting up the analysis

Field What it does
Tested population The segment value(s) that got the change, e.g. Country is GB, all other values will be used as potential donors. You can exclude donors by using the population filter field (see below).
Release date The day the change went live
Pre-release / post-release period The windows before and after the release date
Primary KPI / secondary KPIs The KPIs to measure. The primary KPI decides the headline result.
Population filter (optional) Limits the data, including which markets can be donors
Impact estimation Average daily effect, or the cumulative effect over the post-release period
Confidence threshold The confidence needed to call a winner (default 90%)

Reading the results

  • Summary table: each KPI in the tested markets vs. its synthetic control, with the change and its confidence. A trophy marks a significant improvement. Each tested market has its own row.
  • vs. pre-release KPI: a toggle that shows the naive before/after comparison, for reference only. Use the synthetic-control result for decisions.
  • KPI over time: the tested markets' actual KPI, the synthetic control and its uncertainty band.
  • Donor weights: how much each donor contributes to the synthetic control.
  • Validation level (High / Low): click it to see which checks passed.

Validation checks

Each KPI gets a validation level. High confidence means the result passed the checks, or failed only minor ones. Low means it failed a serious check or several smaller ones. Treat a Low confidence result with caution and look at which checks failed.

Check Fails when What to do
Ran Null Days Test The test couldn't run: the pre-release period is too short Use a longer pre-release period
Null Days Test The effect isn't larger than the "effects" found on 30 earlier dates with no release Look for other events before the release
Change in prediction accuracy The model's error didn't grow after the release more than after those earlier dates Same as above
Ran Geo Null Test Too few donors to run the test Add donors
Geo Null Test The effect isn't larger than the "effects" found when each donor is treated as if it got the change Make sure no donor got a similar change
Prediction Error Ratio Test The model's error grew about as much in the donors as in the tested markets Same as above
Pre-existing trend The tested markets were already drifting away from the donors before the release Treat the effect as an upper bound
Unstable confidence The confidence only just crossed the threshold Wait a few more days
Missing dates Many days are missing around the release date Check the data on those dates
Sparse KPI The KPI is zero on more than 15% of days Use a broader population or KPI
Enough post-release days Fewer than 5 days of post-release data Wait for more data

Further reading

  • Brodersen, K. H. et al. (2015). Inferring causal impact using Bayesian structural time-series models. Annals of Applied Statistics, 9(1), 247–274.
  • Abadie, A., Diamond, A. & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies. Journal of the American Statistical Association, 105(490), 493–505.
  • Help Center: Release Impact

Still need help? Contact Us Contact Us