How We Determine Explainability


Overview

At Loops, our Root Cause Analysis (RCA) identifies the root causes of changes in a KPI. These changes could be spikes or drops, and the goal is to uncover actionable insights. One key approach RCA uses is identifying segments that significantly contribute to the change. To achieve this, we use an explainability score that balances a segment's impact with its size and contribution to the KPI.

This method ensures we focus on segments that are not just large contributors to the change in the KPI but also have a disproportionate impact compared to their size, helping you quickly pinpoint meaningful drivers behind changes.

Explainability Score: A High-Level View

The explainability score quantifies how much a segment contributes to a KPI change relative to its size and expected behavior. Segments with higher scores are considered "explainers" for a KPI change. Here's how we define and calculate it:

  1. Contribution to Change (CTC): Measures how much of the KPI change is attributable to a segment, factoring in both changes in the KPI behavior within the segment and changes in the segment’s share.
  2. Share of KPI Contribution: refers to the portion of the overall KPI value that can be attributed to a specific segment, taking into account both its size and its KPI-related impact. Instead of merely counting how many entities (e.g., users) belong to a segment, we weight the segment by the actual value of the KPI within that segment. For instance, if the KPI is revenue, the share of KPI contribution is the fraction of the total revenue generated by that segment. If the KPI measures conversions or retention, it’s the portion of total conversions or retained users attributed to that segment, rather than just its share of the total user base.
  3. Expected values: for both the KPI and share of entities (e.g users) we calculate their expected values based on historical data and an algorithm we developed for that purpose. Expected values can refer to either (1) predictions by our model, or (2) specific periods of comparison, such as the previous week.
  4. Explainability Score: Combines all the above metrics to highlight segments that contribute disproportionately to the change in the KPI across the population, relative to their general contribution to the value (e.g., a segment with a small share that accounts for most of the change).

Key Concepts and Metrics


Contribution to Change (CTC)

CTC measures how much a segment contributes to a KPI change in the population. It considers two key factors:

  • The change in the segment's KPI value relative to the population .
  • The change in the segment's size (share of the population or KPI).

We calculate two versions of CTC:

  • Absolute CTC: The absolute contribution of the segment to the KPI change. The sum of all these is the KPI change in the population. This contribution appears in the UI.
  • Percentage CTC: The contribution to change of the segment compared to the change in the population. The sum of all these is 100%. While this version of CTC does not show in the UI, it is that version that is used for explainability calculations.

The formula for percentage CTC is:


For more detailed calculations, visit our Contribution to Change documentation.

Explainability Score Formula

The explainability score balances a segment’s percentage CTC with its expected share of the KPI (share of KPI contribution rather than the share of users/entities):

Where share of KPI contribution defined by:

With this formulation, segments with higher explainability scores highlight changes that are significant and disproportionate to the segment’s size, making them critical drivers of KPI trends.

Thresholds for Explainability

To make results actionable, we classify segments into three levels based on their explainability score:

  • High Explainer: Score > 0.35
  • Medium Explainer: Score between 0.15 and 0.35
  • Low Explainer: Score between 0 and 0.15

Segments with scores below 0 are considered "non-explainers" and excluded from actionable insights.

Ensuring Accuracy: Guardrails

We apply several safeguards to ensure explainability results are robust and actionable:

  1. Statistical Significance Tests: Segments must pass tests comparing observed vs. expected KPI values and shares. These tests include:
    • Value vs. Expected Value: Does the segment’s observed KPI differ significantly from its expected KPI? Expected KPI refers to model predictions or the compared period, such as the last week
    • Share vs. Expected Share: Is the segment’s observed share significantly different from its expected share?
    • Value in Segment vs. Population: Is the KPI value within the segment significantly different from the population-level KPI?
    • Multiple Comparisons Correction: Adjustments to reduce false positives when analyzing numerous segments.
  2. Business Relevance Rules: Filters exclude segments that are too small, too large, or uninformative. Examples include:
    • Small Segment Size: Excludes segments with fewer than 20 entities.
    • Small KPI Share: Excludes segments contributing less than 1% to the KPI.
    • Overwhelming Contribution: Excludes segments contributing more than 95% to the KPI.
  3. Exclusion of Negative Explainers: Segments with percentage CTC< 0 (opposite to the population trend) are excluded.
  4. Composite Segment Filters: Stricter thresholds for combined segments (e.g., "Country + Device") to ensure relevance.
  5. Inflation Protection: Patterns of inflated explainers are detected and excluded to avoid noise.
  6. Neutralized Segments: Segments are excluded if they are newly introduced after the reference period, are “Missing,” “Other,” or “0” labels with unusual behavior, or belong to a group where segments (e.g., “converted” vs. “not converted”) show sharply differing KPI values.

Why This Method Matters

This approach has been validated across diverse clients and real-world cases, ensuring it identifies meaningful root causes while filtering out noise. By prioritizing high-explainer segments, Loops empowers you to make data-driven decisions with confidence.

Examples

Here are examples of how explainability scores classify segments:


By understanding how explainability is measured, you can trust that the insights provided by Loops’ RCA will help you focus on the most impactful drivers of change, saving time and guiding effective actions.

Still need help? Contact Us Contact Us