How Whisperr measures lift

The Impact page compares two groups of your users: users Whisperr chose to message, and a holdout that Whisperr keeps silent on purpose. The difference between the groups is the lift. This page explains how the groups are made and how each number is computed, so you can decide how much to trust it.

The holdout

By default, 10% of your users are in the holdout. Whisperr assigns each user to a group from a hash of your app id and the user id. The assignment is random across users but fixed for each user: while the holdout share stays the same, a user in the holdout stays in the holdout on every decision.

For a holdout user, Whisperr still scores the user and decides which message it would send. It then writes and sends nothing. The holdout follows the same cooldowns and limits as everyone else, so the two groups differ only in whether Whisperr’s message went out.

The holdout applies in every mode except Drafts only. In Drafts only, nobody gets a message, so there is nothing to compare. If the holdout is set to 0%, the Impact page says the holdout is off.

Who is in each group

• Reached: users Whisperr chose to message.
• Holdout: users Whisperr chose for a message and then kept silent.

A user counts in the group they were assigned to, whether or not the message was delivered or opened. This includes a message that a manager rejected when messages need approval. Each user counts once, with the outcome of the first time Whisperr chose them. A user with many messages does not count more than a user with one.

When your Whisperr plan or trial limit stops a message, Whisperr holds it for every user. For came back and unsubscribed, those users are in neither group, so the holdout contains only users the reached group could also reach.

What counts as coming back

A user came back when Whisperr receives app activity from them in the 7 days after Whisperr chose them. Seven days is the default measurement window. Whisperr waits one more day for late events, for example events that a mobile app queued offline, and then records the outcome once. It does not change later.

Some events count and some do not:

• An event counts only if it happened inside the 7 days and Whisperr received it after the decision. An old event that arrives late does not count.
• Events that only a message can cause do not count: push_opened, and any event that carries a Whisperr message id. A holdout user cannot send them, so counting them would favor the reached group. The app_opened event that a push tap causes still counts.

The Impact page also compares the share of users who unsubscribed. This includes an unsubscribe or a spam complaint on a Whisperr email, and push permission that went from allowed to denied, in the same 7 days.

Why each group needs 200 users

A difference between two small groups is mostly chance. Whisperr shows a result only when both groups have at least 200 users with a known outcome. Until then, the Impact page says it is too early to tell and shows how many users each group has. The holdout is the smaller group, so it usually decides when the first result appears.

The same minimum applies to each intervention and to each revenue number.

What the range means

Each result has a 95% confidence range. The range shows how far the true difference could plausibly be from the number shown, given how many users are in each group. More users give a narrower range.

• A range above zero: reached users did better than the holdout.
• A range below zero: reached users did worse than the holdout.
• A range that crosses zero: there is no clear difference yet.

For came back and unsubscribed, the number is the reached share minus the holdout share, in percentage points. The range is a standard two-proportion interval for that difference.

Revenue kept

When you connect a billing source, for example RevenueCat, Superwall or Adapty, the Impact page also compares money. It uses the same holdout assignment, but counts every user Whisperr considered under the holdout, including users whose message was held by a plan or trial limit. It reads them 30, 60 or 90 days after each user’s first such decision.

• Paying users kept: the share of reached users with paid access on that day, minus the share of holdout users, times the number of reached users. A trial is not paid access.
• Revenue kept: revenue per reached user minus revenue per holdout user, times the number of reached users. Revenue is what your billing source reported in USD in that period. Refunds count as negative.

Users without billing data stay in their group and count as not paying. Payments without a USD price add nothing, so revenue kept can be lower than the true amount. Without any billing data, the revenue numbers stay empty.

Common questions

Why does the Impact page say it is too early to tell?

Each group, reached and holdout, needs 200 users with a known outcome. A user's outcome is known about 8 days after Whisperr first chose them: the 7-day window plus one day for late events. Until both groups have 200 users, the page shows how many each group has.

Does a push open count as coming back?

No. A push_opened event, or any event that carries a Whisperr message id, exists only because Whisperr sent a message, and a holdout user cannot send one. The app_opened event that the tap causes still counts, because a holdout user sends the same event when they open the app.

Do holdout users still get messages from my own tools?

Yes. The holdout only stops the messages that Whisperr sends. It does not change messages that you send with other tools.