Release Impact
Overview
Release Impact is an analysis that tests how product changes—typically new versions or feature releases—affect selected KPIs (including conversion, engagement metrics, and revenue). While it serves the same purpose as an A/B test, this analysis doesn't randomly allocate users to variants and control groups. Instead, it assumes the entire population experiences the change. The analysis creates a synthetic control group to compare with the post-release period, estimating impact through a proprietary Loops algorithm. This algorithm uses data from before and after the release, accounting for user mix changes and complex trend patterns. The methodology has been extensively tested against A/B test benchmarks and proven highly accurate.
While this method effectively mimics A/B test estimations, it involves more complex statistics and requires specific assumptions. Though we employ various procedures to test these assumptions, best practice dictates verifying them before running the analysis.
Analysis assumptions
The following assumptions are necessary to estimate the release effect:
- The effect must be constant over time (appearing as a step function). Version releases typically follow this pattern, but brief changes like KPI "spikes" aren't suitable. Gradual releases—where some users experience the change before others—are supported but require clear tracking of which users were treated at each release stage.
- No other step-like events (such as holidays or major events) should occur on the release date. While major marketing strategy changes can be partially controlled for through user mix changes, a release during a significant holiday makes it impossible to separate the holiday's effect from the release's impact.
- Any other step-like events (such as other releases) during the analyzed period—typically one month before and after the release date—must be specified in the analysis parameters. This allows us to model these events appropriately and exclude them from tests that assume no other changes occurred.
- The user segments mix is not affected by the release. In other words, we assume all segments are determined prior to the KPI’s reference date (typically the join date), such as demographic / first-device data. We assume this because we use these changes to separate the product changes impact from marketing actions impact that often goes along new releases and affects the segment mix. For example, behavioral segments that rely on user action taken after the KPI’s reference date can bias the data.
Best practices
Release Impact is designed to estimate the impact of product changes. When A/B testing is viable (i.e., when true randomization and allocation mechanisms exist), it's preferable since it's simpler and more robust, requiring fewer assumptions. However, Release Impact is a good alternative when A/B testing is impractical, costly, or impossible.
Release Impact is not suitable for measuring marketing campaigns. It is designed to remove the effect of changes in user mix, and a campaign mostly works by changing the user mix. To measure a campaign, or any change launched in some markets but not others, use Marketing Release. It compares the tested markets with a synthetic control built from markets that didn't get the change.
Due to its assumptions and limitations, avoid running Release Impact analyses during weeks with holidays, major events, or other releases that affect a large portion of your user population. If such events occur within the analysis range but not during the same week, you may proceed with the analysis by indicating these events in the setup (Advanced parameters → Other important dates).
Gradual releases. The most common case is a mobile app release: a new version rolls out through the app stores, often as a phased release, and users update at their own pace. Weeks later, part of the user base is still on the previous version, so no single date separates "before" from "after". The same applies to any change where exposure is staggered. In these cases use Gradual Release (Measurement Timeline → Gradual Release): select the version property, then the previous and new versions as they appear in your data (some platforms report build numbers rather than version names).
Instead of splitting users at a single release date, each user is assigned to the treated or control group according to the version they were exposed to during their KPI window. Both groups are observed concurrently, so time-varying confounders such as seasonality and campaigns largely cancel out. Differences in user composition are adjusted with the same propensity stratification used in a regular Release Impact.
Finally, the analysis works best with complete data (no missing dates), dense data (few zero-value dates), enough history (typically a month before and at least 1 week after the release), and rich segment data.
How many users do I need? There's no fixed minimum. It depends on how noisy the KPI is and how small an effect you want to detect. As a rough guide, typical product effects start to become detectable at around 1,000 users per day. Smaller products can still detect large effects. Check the validation level of each result.
How long should the pre- and post-release periods be?
Pre-release: 28 days (the default). The model uses this period to separate the release from the KPI's normal trend. With too little history, ordinary drift is easily mistaken for a release effect. In our testing, shorter windows produced substantially more false positives, while 28 days kept them close to the intended level. Going much beyond 28 days rarely helps.
The analysis also uses about 2 more months of earlier data for the null days test, so make sure about 3 months of clean data exist before the release. You don't need to set this; the pre-release period stays 28 days.
Post-release: at least one full week of matured KPI data, in whole weeks. Two to four weeks is a good default. Extend it only if you expect the effect to build up over time.
Interface
The analysis consists of three types of pages:
- Overview - presents a summary the main results over all analyzed KPIs.
- Tested KPIs - a deep dive into each KPI result
- Appendix - additional result that one might find interesting
Overview
The overview section shows 2 main modules:
- The first module is a summary table displays each KPI's value and its synthetic control calculated by our algorithm. A trophy icon indicates the winning "variant" when its KPI value is higher (for "higher is better" KPIs) and shows statistical significance (confidence > 90%). We run multiple validation tests to verify the previously mentioned assumptions and data integrity. The results appear for each analyzed KPI, and clicking the validation label opens a pop-up window with detailed test information. Above the summary table, a toggle lets you switch between two comparisons: the post-release period versus our calculated synthetic control (the causal result) or versus the pre-release KPI (the naive result). We don't recommend using the naive result as it can be significantly biased. This toggle exists to demonstrate the difference between causal and naive calculations.
- The second module displays two graphs. The first graph shows the confidence level (the statistical significance of the causal result) over time, starting from day 7 after the release. While this graph defaults to showing the main KPI's confidence trend, you can view any secondary KPI by selecting it from the tabs above. The second graph displays both the actual KPI and its synthetic control across the same time period, with confidence bands for each.
Tested KPIs
Each KPI has its own page with three modules:
Summary: the summary table, the actual KPI against its simulated control, and the number of users before and after the release.
Segment changes: how the KPI changed in each segment between the pre- and post-release periods. These are naive comparisons (post vs. pre average), not estimates of the effect within each segment. They do show which segments moved during the release, and by how much.
Model breakdown: how the causal result is built, step by step. The graphs show how the simulated control is calculated and how each step changes the result:
- Naive effect: the pre vs. post average KPI.
- Trend and seasonality only: ignores segments. A significant result means the KPI shows a step at the release date.
- Segment mix only: ignores trend and seasonality. A significant result means a change in user mix explains much of the pre vs. post difference. This step controls for changes unrelated to the release, such as marketing campaigns, assuming the release itself doesn't change the user mix.
- Full model: combines steps 2 and 3 into the causal estimate.
Appendix
The appendix displays additional metrics around the release date, including:
- Changes in feature adoption rates across all KPIs' relevant time ranges.
- Changes in the median time for entities to reach each KPI.
These metrics, combined with the main results and segment changes, provide a comprehensive view of all changes surrounding the release date.
Methodology
The Release Impact Analysis uses advanced statistical methods to estimate how product changes affect specific KPIs. It overcomes the major biases found in simple pre-versus-post comparisons (naive estimations) by accounting for temporal patterns (like complex trends and seasonality) and external factors such as marketing campaigns that coincide with the release.
Why Naive approaches fail
The simplest way to measure a product change's effect on a KPI would be to compare average values between users who joined after the release ("post-period") and those who joined before ("pre-period"). However, this approach—while straightforward—is strongly discouraged because it leads to misleading results due to significant biases. The main issues are:
- A change in user mix of the business: The mix of users after a release can differ significantly from those before it. This may happen when releases coincide with marketing campaigns or due to natural variations in user acquisition. Consequently, a product release that had no real impact might appear to have a significant effect simply because of changes in the user population.
- The dynamics of time: Simple before-after comparisons ignore important time-based patterns like trends and seasonality. For example, if a release occurs during an upward trend, the post-release average will appear higher than the pre-release average—suggesting a positive impact even when the raw data clearly shows a negative effect.
To address these biases, our methodology combines two statistical approaches:
- Treatment Propensity Stratification: to account for changes in user mix.
- Regression discontinuity analysis with smooth terms (generalized additive models) and seasonality: to account for temporal trends around the release date.
Step 1: Addressing changes in user mix with propensity stratification
The problem
Users in the pre- and post-release periods can differ significantly, whether due to marketing efforts around the release or natural variations. These differences manifest in changing segment distributions—for example, shifts in the proportion of Android users, users from the US, CA, and UK, or users from web sources. Since KPI levels often vary substantially between segments, these distributional shifts can explain changes in the overall KPI. We assume segment assignments are independent of the release since they are determined before the KPI's reference date (typically the join date). Therefore, we need to eliminate KPI changes caused by shifting user mix to isolate the true causal impact of the release. Without accounting for these shifts (also called confounders), any observed difference in KPI between adopters and non adopters may reflect these underlying differences rather than a true causal relationship.
How Propensity Stratification Works
For a release analysis, treatment simply means whether a user is in the post-release or pre-release period. We use a decision tree to group segments based on their likelihood of being in the post period, then analyze each group separately.
We group users into segments with similar treatment propensities using a decision tree. These segments are defined based on both:
- Demographics (e.g., country, device, marketing source).
- Behavioral Features (feature adoptions that occurred before the reference date, if applicable).
Within each segment, pre-release and post-release users become more comparable, which reduces confounding effects. We use statistical methods throughout this process to prevent overfitting and optimize the tree structure.
Step 2: Addressing time dynamics
The problem
As demonstrated in the graph below, trends in the data can severely bias the release estimation using a simple pre vs post analysis. Moreover, KPIs often exhibit strong weekly seasonality. If this is not taken into account this can further increase the noise and bias in the estimation. To account for that we use a form of Regression discontinuity analysis.

How Regression Discontinuity Works
Within each segment detected in step 1 we use a generalized additive model which models:
- A smooth trend, which is general curve that fits the data trends, taking into account the noise and number of users to optimize the curve construction and avoid overfitting.
- Weekly seasonality, modeled by adding a day-of-week fixed effects (i.e., separate intercepts).
- Tenure - as an important predictor of the KPI, we use it in the model as well (as a smooth term so the modelled effect is more flexible).
- Release effect - this models a step function of the impact of the release after taking into account the trends and seasonality (remember - the segment mix changes are taken into account by the first step).
- Number of new users joining on the same day as the user - this helps accounting for marketing campaigns etc
Using a BIC score, we select or remove features from those listed above to prevent overfitting. The model fits across all pre- and post-release data. For each analyzed segment, the impact estimation comes from the model's release effect feature. The model determines the optimal fit by incorporating trend, seasonality, release impact step, and tenure effect. The synthetic control is then created by predicting the same data with one key difference: the release feature is set to "pre-release" mode for the post-release period.
To visualize this, consider the following real example. The graph below shows the actual KPI (black line) around the release date (dashed green line). The red line shows the fitted model, which detected a downward trend, weekly seasonality, and a significant step change at release. The dashed blue line represents the synthetic control, which continues the pre-release trend and seasonality pattern. The estimated impact of the release is shown by the difference between the fitted line (red) and the synthetic control (dashed blue) in the post-release period.

Many people mistakenly assume that the synthetic control in Release Impact analysis is created by training a model solely on pre-release data to forecast post-release outcomes (such as with the Prophet algorithm). However, this approach would be less effective at detecting changes because forecasting is inherently more complex than model fitting. It would require much more data and stricter assumptions to achieve the same statistical power and avoid bias.
Validation checks
While Release impact applies advanced statistical methods to mimic A/B-tests it is inherently more susceptible to bias and requires more assumptions to work properly. To mitigate this risk, we apply extensive validation checks that are designed to provide greater confidence in the release result. These validations include 14 different checks which eventually feed a confidence scoring mechanism which determines the level of confidence in the analysis assumptions.
KPI Data validations
We perform four types of validations on the KPI data before running it through the model.
- Percent missing dates: Problem: dates with missing KPI data, especially around the release date, can bias the result. Test: We check whether there are too many missing dates in the week before and after the release, and if there are any - we indicate the specific dates in which the data are missing. Possible resolution: Users are encouraged to check for any data validity issues around the indicated dates.
-
Sparse KPI:
Problem: A combination of small daily samples with a KPI which is very sparse (e.g., a conversion rate of 0.1%) can result in a considerable number of days where the KPI value is 0. This situation is not ideal for the model. Test: We check the percentage of 0-value KPI dates in the analysis period and warn in case where it is unusually high (>15% of dates). Possible resolution: We encourage users to either remove any population limitations (to increase sample size) or select a different KPI which would be less sparse.
-
Enough post-release dates:
Problem: The trend model works best if there is at least 1 week of data after the release date. Note that if the KPI has a long bound (e.g. conversion within 2 weeks) then this means at least 3 weeks should pass from the release before there is full KPI data for the first week. Test: Check whether there is less than a full week of fully matured KPI data. Possible resolution: Either wait more time for the data to collect and mature, or select a shorter bound for the KPI so it matures earlier.
-
KPI level break:
Problem: A sudden, lasting jump in the KPI before the release usually means a tracking or definition change, such as a new event, a changed filter or a data migration, rather than real user behaviour. The model would read part of that jump as trend, which distorts the release estimate. Test: We check whether the KPI's 7-day average jumps by 40% or more compared with its trend over the previous 7 days, and whether the jump is far larger than normal day-to-day noise. If so, we list the dates, and the validation level is set to Low whatever the other checks show. Possible resolution: Check what changed in the data on the listed dates. Then either start the analysis after that date, or fix the KPI definition so it is consistent across the change.
Null days validation tests
To validate that our detected effect represents a true signal rather than random noise, we compare it to earlier periods before the analysis window. Even if the analysis detects statistically significant effects in earlier dates, we check that the effect in the release date is more significant than the effect found in at least 90% of the other dates. An important assumption for the validity of this test is that there are no other releases or major influential events in the pre-release period (if there are such events, the user should clearly define these in the analysis settings).
The null days test also feeds into the confidence. The confidence shown for each KPI combines two independent sources of evidence: the model's parametric significance, and an empirical, placebo-based significance, which ranks the release-date effect against the effects estimated on about 30 earlier dates with no release. The reported confidence takes the more conservative of the two, so a result is significant only when both support it. If too few placebo dates can be estimated (usually because there isn't enough history), only the model's significance is used.
-
Ran Null days validation test:
Problem: The null days validation test sometimes cannot run, for example when there isn't enough historical data before the release. This reduces our confidence in the results since this validation is fairly important. Test: Verify that the null days validation test executed successfully. Possible resolution: Ensure KPI and segment data extends several months before the release date.
-
Percentile in null test:
Problem: When the effect on the release date is not at least in the 90th percentile of effects of pre-release null dates, this reduces our confidence in the result. Test: Check the percentile of effect compared to pre-release null dates (i.e. dates which were not indicated as other release or major event dates). Possible resolution: Check whether there are other release dates or major influential events in the pre-release period, and indicate them in the analysis parameters.
-
Outlier null dates:
Problem: Null days test assumes the comparison periods were free from major events or other releases. Specific null dates with an unusually large effect suggests these assumptions might not hold, which could invalidate our other test’s results. Test: Test for outliers in the null dates effects. Possible resolution: Check whether there are other release dates or major influential events in the pre-release period, and indicate them in the analysis parameters.
Segments data validations
We run three more analyses specifically focused on segment data richness and its weekly changes. These tests are crucial for properly handling external confounding effects (such as marketing campaigns—see step 1 of methodology) and for identifying other potential problems.
-
Detected segments with moderately-high changes around release:
Problem: In the best practices section, we encourage users to avoid including segment groups that are affected by the release, such as usage of a feature introduced by the release. Since such segment groups can severely bias the result, we apply a procedure that automatically detects and removes highly suspicious segments (those showing extreme changes around the release). When these changes are clearly extreme, we have high confidence in this process. However, things become trickier with segments showing moderately high changes. In these cases, while we still remove the segment from the analysis, we have less confidence in the result since there's a higher chance we've removed an informative segment. Test: We keep notes of all the segments groups we automatically excluded due to too-big differences between pre and post release periods, and measure the degree of differences. Out of all the excluded segment groups, we examine the one with the least amount of change. If the level of change of that group fall in a certain range (of large enough change to exclude but low enough to raise doubts) the test fails. Possible resolution: Manually exclude segment groups which are not suitable for the analysis (as they might be causally affected by the release).
-
Number of informative segments:
Problem: The model needs rich segment data to account for user mix changes around the release (as explained in step one of the methodology). Without enough segment data, we may fail to detect and account for potential biases.
Test: We check if there are at least 5 segment groups that passed the segment validity checks (meaning they didn't show extreme changes between pre- and post-release periods).
Possible resolution: Add more segments to the analysis.
-
Outlier weeks in segment distribution changes:
Problem: As mentioned before, the Null days test assumes the comparison periods were free from major events or other releases. Specific weeks in which unusually large segment changes are observed suggests these assumptions might not hold, which could invalidate our other test’s results. Test: We quantify weekly changes in the mix of every segment group and then test whether any group exhibits certain weeks with unusually high level of change in segment mix. If we find any, we indicate in which segment group and which specific weeks these changes were observed. Possible resolution: make sure no additional releases or holidays happened in the specified weeks, and if so - indicate these in the analysis parameters.
Model results validations
Finally, we also run some validation tests on different stages of the model results looking for red flags.
-
Causal tree splits:
Problem: The basis of stage 1 of the model (propensity stratification) is a tree algorithm that splits the population to different segments. In some cases, no statistically significant splits are detected, which might indicate not enough data to debias the results. Test: Check if the tree model found any splits for the population. Possible resolution: Add more meaningful segment groups to the data.
-
High variability in effect between group:
Problem: In stage 2 of the model, we estimate the effect separately for each tree-segment found in stage 1. Sometimes, the effects in these segments show opposite directions. When a significant portion of the population shows contradictory effects, this raises a red flag (even though it’s technically possible due to heterogeneous treatment effect) Test: Check which portion of the population resides in segments with statistically significant contradicting effects. Possible resolution: Get support from Loops team.
-
Causal equal naive:
Problem: When not enough data is available for the analysis the causal effect will equal the naive effect of post vs pre release. This mean that there are both no causal tree splits and also no trend and seasonality model, which implies a low signal to noise ratio. Test: Check whether the causal effect equals the naive one. Possible resolution: Collect more data.
-
Unstable confidence:
Problem: While the estimated effect using the most recent data could appear statistically significant, this could be due to random noise in post release dates. An effect which is not only significant but also stable across time is stronger evidence of an actual impact. Test: The release effect is estimated on all available lengths of post release period from 7 onward (i.e., 7 days, 8 days, 9 days etc.). We run a test to see whether the effect converged to a high enough confidence, which means it is both high and stable. Possible resolution: Keep collecting data for a few more days.
Each validation test carries a different weight in our general confidence score. This scoring system helps us assess our confidence level in each result and provides clear steps for improvement when validation tests fail. We developed this process based on numerous client escalations and analyst feedback to replicate our internal debugging procedures. The scoring mechanism has been refined through extensive training on cases that our domain experts manually evaluated for confidence levels.
Results
Since Loops includes an A/B testing platform, we can benchmark our methodology against these gold-standard tests and improve our algorithms to match their results. Using dozens of A/B tests across different KPIs from various clients, our current algorithm achieves 90% accuracy—meaning the release impact results (statistically significant positive/negative, or non-significant) match the A/B test results 90% of the time. In comparison, naive pre-post calculations only achieve 50% accuracy against the same A/B test data.
Summary
Release Impact Analysis is a specialized methodology that measures how product changes—particularly new versions or feature launches—affect key performance indicators (KPIs) such as conversion, engagement, and revenue. Like an A/B test, this method compares performance before and after changes, but with one key difference: it analyzes the entire user population instead of splitting users into control and test groups. Release Impact creates a synthetic control group, maintaining the scientific rigor of randomized experiments while avoiding their complexity and cost.
The approach requires specific conditions to work effectively. It needs a clear, sustained impact (a "step function" effect) rather than temporary spikes. It also requires the absence of other major events—such as holidays, marketing campaigns, or additional releases—during the analysis period. When such events occur, users must specify them so the algorithm can account for their influence and isolate the true release impact. Segment data is crucial: the method assumes that user characteristics and behaviors used for segmentation were established before the product change, preventing confusion between release effects and marketing-driven changes. Through statistical techniques, we adjust for changes in user mix and temporal patterns, significantly improving accuracy (from roughly 50% to over 90% when compared to A/B test benchmarks).
Finally, comprehensive validations ensure reliability at every step, identifying potential issues like sparse data, unstable trends, or unaccounted events. By uniting methodological rigor with practical safeguards, Release Impact serves as a robust alternative when A/B testing isn't feasible.