The churn you never see: Why prevention is so hard to measure

Short answer

Why is it so hard to measure churn that Customer Success has prevented?

Because you only see what actually happened. A customer who stayed after an intervention might have stayed anyway – and the churn that was prevented does not appear in any report. The effect cannot be observed, only estimated, by comparing with a group that did not receive the intervention. The most reliable way is a randomly selected control group.

A Customer Success Manager contacts a customer who has stopped using the product. They find the problem, fix it, and three months later the customer renews. The report says one thing: a renewal. It looks exactly like all the other renewals that quarter.

The question nobody can answer with certainty is whether the customer would have left without the conversation. Maybe. Maybe not. This is what makes prevention so hard to value: when it works, nothing happens – and what does not happen cannot be counted.

This article is about why the problem arises, which fallacies it leads to, and how the effect can be estimated anyway.

Why what didn't happen can't be observed#

Saying that an intervention “saved” a customer is really comparing two versions of the same customer relationship: what happened, and what would have happened without the intervention. The second version is called the counterfactual outcome – what would have happened if something had been different.

The problem is that only one version ever exists. The customer either had the conversation or did not. You can never observe both outcomes for the same customer at the same time.

In statistics this is well known. In 1986 the statistician Paul W. Holland called it the fundamental problem of causal inference: you cannot observe the effect of a treatment on a single unit, because you only see the outcome with or without the treatment – never both.¹ Customer Success has exactly the same problem. It has just rarely been named.

Two versions of the same customer relationship

What happened

1. The customer shows risk signals
2. The CSM gets in touch
3. The problem is solved
4. The customer renews

Visible: one renewal, like all the others.

↕ Only one of the stories can be observed. The effect of the intervention is the difference between them.

What would have happened

1. The customer shows risk signals
2. Nobody gets in touch
3. The problem grows
4. The customer cancels

Invisible: the churn that never became churn.

The consequence is that the effect of preventive work is always an estimate. That does not mean it cannot be measured. It means it has to be measured indirectly, by comparing with customers who did not receive the intervention – and the quality of the estimate depends entirely on how comparable those customers are.

Three fallacies it leads to#

When the counterfactual outcome is not visible, the gap is often filled with assumptions. Three of them are particularly common.

Everyone who stayed was saved#

Many Customer Success teams keep a list of customers they have “saved”: customers who were at risk, received follow-up and stayed. The problem is that some of these customers would have stayed anyway. Such a list counts everyone who stayed, not those who stayed because of the intervention.

Add such lists up over a year, and the result may be that the team has “saved” more customers than it is reasonable to believe. The list is not worthless – it documents work and provides examples – but it is not a measure of effect.

At-risk customers get better on their own – partly#

Customers are often flagged as at risk at a bad moment: a month of low usage, a complaint, a contact person on holiday. Many such measurements move back towards normal by themselves, without anyone doing anything. In statistics this is called regression to the mean.

This means that a group of at-risk customers who receive follow-up will almost always look better afterwards. The question is how much better they look than they would have without follow-up.

Hypothetical example

60% → 15 pp

Without a comparison group, the intervention appears to work for 60% of the customers. With a comparison group, the estimated contribution is 15 percentage points.

No churn means no need#

The third fallacy is the most costly. If prevention works, churn goes down. It may then look as if the problem was smaller than assumed – and that the resources can be used elsewhere.

If Customer Success work is then reduced, churn often returns, but usually with a delay. If customers have agreements with an end date, they do not leave the next day. They leave at the next renewal, which may be many months away. Where customers pay for usage and can leave at any time, there is no such date: the churn comes silently, as usage that gradually declines. In both cases it is hard to connect the increase to the decision taken long before. Read more about the distinction in What is Customer Success?

Why the rescue gets the credit#

The counterfactual problem also affects which work is seen. A dramatic rescue has a story: a customer on the way out, a meeting, a solution, a renewal. It can be told in a management meeting.

Good preventive work has no such story. Better onboarding means fewer customers end up at risk in the first place. There is no individual customer to point to, no crisis averted – just a curve that is a little lower than it would otherwise have been.

The result can be that the organisation rewards the visible work and deprioritises what works earlier in the customer relationship. This is a professional judgement, not a research finding, but the mechanism is easy to recognise: what can be counted gets attention.

Better onboarding has no rescue story. It just has fewer customers who needed rescuing.

How the effect can still be estimated#

Even though the counterfactual outcome can never be observed directly, it can be estimated. The methods below are ordered from strongest to weakest. They all rest on the same idea: finding a group of customers who resemble those who received the intervention, but who did not receive it.

Methods for estimating prevented churn

Stronger basisWeaker basis

1. Random control group

   Holdout

   A randomly selected share of customers does not receive the intervention.

2. Staggered rollout

   Everyone gets it – in turn

   Compare those who have received it with those still waiting.

3. Natural comparisons

   Capacity, thresholds

   Customers who happened not to be reached.

4. Forecast versus actual

   Risk model

   Expected churn compared with actual churn.

5. Before and after

   Time series

   Simple, but affected by everything else that changed.

6. Rescue log

   Examples

   Documents work, not effect.

Random control group#

The most reliable method is to select a random share of customers who do not receive the intervention for a period, and compare the results. Because the selection is random, the groups are alike from the outset – including on things that have not been measured. The difference in churn between the groups is then the best estimate of the effect.

The objection is that it can feel wrong to withhold help. That is a real consideration, but it can often be resolved:

• Withhold the new, not the basics. The control group still gets normal Customer Support and ordinary follow-up – just not the new intervention being tested.
• Start with automated interventions. For emails, in-product messages and digital customer journeys, control groups are common practice in marketing and rarely controversial.
• Keep the control group small and time-limited. It does not need to last forever – just long enough for the difference to be measured.

Staggered rollout#

If all customers are to receive an intervention but cannot all receive it at the same time, the order can be used as a comparison. The customers who get the intervention first are compared with those who have not yet received it. If the order is random, the method is almost as strong as a control group – and no customers are permanently left out.

Natural comparisons#

Sometimes comparison groups arise by themselves. A team does not have the capacity to contact all at-risk customers. One customer ends up just below the threshold for personal follow-up, another just above. Such situations can provide useful comparisons.

The weakness is that the groups are rarely completely random. The customers the team had time to contact may have been chosen because they seemed easiest to save. The comparison then partly measures the selection, not the intervention.

Forecast versus actual#

If the company has a risk model or health score, you can compare how many customers the model expected to leave with how many actually left. The difference is a rough estimate of the effect.

The method assumes the model was accurate before the intervention was introduced. Otherwise it is impossible to separate the effect of the intervention from errors in the model. It is also vulnerable to regression to the mean: a model that flags customers at a bad moment will overestimate how many of them would have left.

Before and after, and the rescue log#

Comparing churn before and after an intervention is simple but weak, because price, product, competition and economic conditions may have changed in the same period. A rescue log is useful for understanding what works in individual cases, but it says nothing about how many of the customers would have stayed anyway. Both can be used as support – neither as proof.

A worked example with a control group#

The example below shows how a control group makes the invisible visible – and why the estimate must be interpreted with caution.

Share of at-risk customers who left within six months

Show the numbers

Group	Share who left
Control group	25%
With intervention	15%

Without the control group, the company would only have seen that 136 of 160 at-risk customers stayed. It is easy to present that as 136 saved customers. The control group shows that most of them – an estimated 120 – would probably have stayed anyway.

The estimate is still uncertain. With 40 customers in the control group, each customer accounts for 2.5 percentage points. Had 8 rather than 10 left, the difference would have been 5 percentage points, and the estimate halved. Small groups produce large swings. That argues for repeating the measurement over several periods, and for reporting the estimate as a range rather than a precise figure.

How to report prevention honestly#

The counterfactual problem does not go away, but it can be handled openly. A few principles make reporting both more honest and more convincing:

1. Say what the comparison is. “Churn was reduced by 10 percentage points compared with a random control group” is a completely different statement from “we saved 136 customers”.
2. Report uncertainty. A range is more credible than a precise figure, especially with small groups.
3. Talk about contribution, not cause. Retention is affected by much more than Customer Success. See How do you measure the value of Customer Success? for a more thorough discussion.
4. Show leading indicators. When the effect on churn cannot yet be measured, you can show that the intervention affects indicators that, in your own data, are associated with retention – for example completed onboarding or time to value.
5. Follow the cohorts over time. Prevention often only shows once a whole cohort has passed renewal. See How to measure churn in a fast-growing company.

What you can do now#

The most important thing is to decide how the effect will be measured before an intervention is introduced, not afterwards. Once the intervention has been rolled out to everyone, the chance of a good comparison is often gone.

The churn that never became churn will never appear in a report. But with a good comparison it can be estimated – and that is enough to make better decisions about what actually works.