Testing and Learning

Part 3 of the Blueprint closes the loop. Measurement told you what is happening; testing tells you whether your explanation for it is right, and whether your proposed fix actually works.

A/B Testing Framework

A simple, context-driven framework

Experimentation and A/B testing approaches can become very complex, and complexity has its merits in the right context.

Here, though, I’ll boil it down to its basic principles and show that within the Blueprint framework you can come up with solid hypotheses that are straightforward to test and that show results.

Context: “what is happening?”

The context comes from the work done in Measuring Performance, particularly when visualising and acting on insights.

Once you have context, such as understanding that the completion rate for a CTA is decreasing while the number of opportunities is increasing, make sure the broader picture is clear. That clarity is what lets you make an evidence-based hypothesis that asymmetrically favours the change you’re testing.

Prediction: “why is it happening?”

Correlation does not equal causation. That’s why we form a hypothesis and test it.

Based on the context above, your prediction is going to be very well targeted, because you already know which aspect of engagement you’re trying to improve.

Word the “why” hypothesis as follows:

We believe X because Y. If we change A, then metric M will improve.

• What is happening (X): the observed metrics and trend from the context established in Measuring Performance
• Why it’s happening (Y): the cause you infer from data and insight, based on your business knowledge and understanding of what drives engagement
• What you’ll change (A): the lever you can pull to try to improve M, based on insights from Visualising and Acting on Insights. This is the change trialled on users during the A/B test
• How you’ll judge success (M): the decision metric: the rate or count of successes or opportunities from X that you’re trying to improve

A note on the null hypothesis

The null hypothesis is the inverse of the prediction. We take the baseline assumption that there is no effect between the two groups and any observed difference is due to random chance.

The aim of testing is to either reject this assumption, or be unable to reject it.

Testing an improvement

To test the improvement, define which users are eligible and create two versions of the platform experience:

• Group A: a control with the current version
• Group B: a variant with the proposed change

Then randomly split eligible users between them. The goal is to see whether the change truly causes the desired improvement in metric M.

A note on eligibility

Make sure the audience is relevant to the improvement in question. If you’re testing an improvement during onboarding, it makes sense to assign only new users into group A or B. This can be done through the A/B testing tool or analytics SDK you’re using.

A note on guard rails

Sometimes a change has an undesired effect. During the testing phase, monitor all metrics and revert to the control version if certain parameters are met.

For example: if you’re looking to increase the completion rate of onboarding, you might decide that if the rate drops below x% you will revert to control, to minimise the negative impact of the change. Similarly, if related metrics like the number of opportunities change drastically, consider reverting, since an unexpected secondary effect has impacted other areas of the process or app.

Analysis and decision making

Statistical significance tells you whether the observed difference between A and B is likely real, and not just random noise.

Generally your platform’s testing or analytics SDK will have tools in place to ensure the correct calculations are being used. Typically the following are involved:

• Z-score: a way of standardising the observed difference in effect between the conversion rates of groups A and B. It shows how many standard errors (how much your estimate would typically vary from sample to sample due to random chance) the effect is from 0, the null hypothesis value. A higher z-score means further from “no effect”
• P-value: the probability you’d see this z-score or larger if the null hypothesis were true, i.e. if the difference between groups A and B were entirely due to chance. We typically want a p-value below 5% to consider the effect real
• 95% confidence interval (CI): a range of plausible true effects (the difference between groups A and B). If you repeated the experiment many times and calculated a 95% CI for each, the true value of the difference would be contained within 95% of those intervals. If the p-value is above 5%, the 95% CI includes 0%, meaning the difference between the groups is consistent with random chance

Decision time

If the outcome is statistically significant, you can confidently go ahead with the change and make version B your new standard.

Then continue to observe the effects through your dashboards and identify further opportunities for improvement. That observation feeds the next round of context, and the cycle continues.