3 Alternatives to A/B Testing

A/B testing is commonly used by growth and product teams to experiment with changes in features, design and copy before rolling out permanent changes. But more often than not it is a complete waste of time and used by people to defend job security.

Do not run an A/B test under these scenarios:

1. You do not have enough users
2. You are making minor changes
3. You want to know why
4. You are not talking to your customers

Here are three alternatives to A/B testing:

1. Conduct a user interview
2. Prototype to learn
3. Find what’s broken and fix it

When should I not be doing A/B testing?

1. You do not have enough users

If you have less than 1,000 daily active users engaging with a particular feature or webpage it is not worth running an A/B test.

For the A/B test to be useful you will likely need to run more than one variation (multi-variate test) which will make the audience smaller and require you running it for at least a few months. You and your team will be wasting time observing the results on a daily basis. Time you could better spend fixing problems.

If you have more than 1,000 daily active users, consider whether it’s even worth the risk of making any changes if your users aren’t complaining. If there are problems you should have heard of it by now.

2. You are making minor changes

Unless you have the audience size and resources of Google or Twitter, testing minor changes is a waste of time. Good enough is usually enough.

Instead, just ship your change and compare the before and after results. If the results are good, pat yourself on the back and move on. If the results are bad you can always roll back. Most likely the results will be unchanged and you will have wasted everyone’s time.

3. You want to know why

The biggest problem with A/B tests is they don’t explain why one result is better. Knowing that a customer will click on a subscribe button with a darker shade of green doesn’t help explain why they wanted to subscribe in the first place. Knowing the why is crucial to designing better features and messaging.

4. You are not talking to your customers

Engineers love running A/B tests because it means they don’t have to talk to customers. If your team that is proposing to run an A/B test is not talking to your customers, tell them to come back after they have spoken to at least five customers on the problem they are solving.

What are the alternatives to A/B testing?

1. Conduct a customer interview

Here are three tips on how to conduct faster and better customer interviews:

• Identify the problem: you are trying to solve. For example, increasing demo requests, adding comment, sharing photos, inviting users
• Find your customers: who have tried to make progress in the problem you are solving. Reach out to them via cold email, calls or directly via their account manager. Aim to secure at least 3 interviews
• Focus on the problem not the solution: let them talk about the struggle and progress they were trying to make. Here are five example questions you can ask to uncover the problem (article)

2. Prototype to learn

Use prototypes during interviews to uncover the trade-offs people are willing to make in using your solution.

Innovator of the Jobs To Be Done (JTBD) theory, Bob Moesta advocates using divergent prototypes as a way of “helping people eliminate what they don’t want and helping us build the criteria of what they do want”.

Imagine you are developing a feature to allow users to share their progress with internal stakeholders. Divergent prototyping would include presenting a user four completely different methods of sharing to gauge the trade-offs they make. For example, 1) Share via emailing a public link; 2) Download PDF; 3) Add user to account; 4) Mention a user by adding a comment.

Do not get feedback on multiple prototypes of the same version, for example different colour, visual design or copy. Use contrasting prototyping and allow users to eliminate options. This helps uncover the trade-offs of the solution they are really after.

3. Find what’s broken and fix it

Before you begin A/B testing, identify as many unmet basic expectations you currently have. Speak with your customer facing team, ask them what customers complain about most. Fix what causes your customers the greatest frustration. Use the Kano model as a framework to prioritise the problems you should solve (article).

References

• The 5 Skills of an Innovator (Video) Bob Moesta 2019

• Building a Winning Strategy Using the Kano Model (Video) - Jared Spool