Cognitive Reflection Test - Examples of Reacting vs Checking

The Cognitive Reflection Test (CRT) is a simple three question test that measures people’s tendency to answer questions with the first idea that comes to their mind without checking it.

The first and most popular question is the “bat and ball problem” which more than 50% of students at Harvard, MIT and Princeton get wrong.

Sections

1. The Bat and Ball Problem
2. How to Improve Your CRT Score
3. Examples of Reacting vs Checking

1. The Bat and Ball Problem

Problem:

• A bat and a ball cost $1.10.
• The bat costs $1.00 more than the ball.
• How much does the ball cost?

First Reaction:

• 10 cents

Nearly all people have a first reaction of 10 cents which is wrong. The correct answer is 5 cents.

In a study of 3,428 university students, 50% of participants across Harvard, MIT and Princeton gave the incorrect intuitive response and did not actively check their response. A failure to check is instructive because it requires such minimal mental effort.

Anyone who would pause and check would recognise that if the ball is 10 cents the bat would need to cost $1.10 and the sum would be $1.20, which is above the target $1.10.

Daniel Kahneman describes this cognitive bias as the distinction between a person’s two processes of thinking:

• System 1: executed quickly with little conscious deliberation; and
• System 2: executed slowly and more reflective

System 1 processes occur instantly and effortlessly for example knowing the answer to 5 x 5 or indicating left when turning the car left.

System 2 processes require more effort, concentration and calculation for example the answer to 25 x 7 or correcting for car slides on icy roads.

2. How to Improve Your CRT Score

Here are three ways you can improve your CRT score and mitigate against biases of System 1 thinking:

1. Acknowledge: that you will always have an initial “gut” reaction to an event or problem and jump to solution mode. This is unavoidable and it is ok. Note your initial reaction and solution;
2. Pause and check: your first idea before sharing your feedback. Check who the decision would impact and how much effort the solution would require; and
3. Ask questions: to find out who really has the problem and where the problem comes from. Check again whether your first idea would answer these questions.

3. Examples of Reacting vs Checking

Here are three examples of how product, customer success and marketing teams may react with their first idea and how they could pause and check.

Example	React First Idea	Pause & Check
Product Watching an end-user struggle to upload a file	Propose feature solution of drag and drop file upload	Before solution mode: check funnel data of what % of users fail upload are those not uploading files blocking the flow of other users are there non-UX factors causing this (e.g. file size, internet speed)
Customer Success Unhappy customer feedback that solution was not a fit with their internal system and they may now churn	Blame sales team for selling to clients with poor system fits and create a new process to screen out incompatible customers	Before solution mode: check how many existing customers are at risk of incompatibility. Are they using workarounds could the cause of the incompatibility be known in advance of the sale does the customer still see value in the solution
Marketing Prospect complains angrily they are getting too many marketing emails	Disable this week’s new marketing campaigns	Before solution mode: confirm how many other recent complaints have been received check if prospect is an existing paying customer or opportunity if not, check if prospect fits the ideal sales qualified target

To learn more about how to pause and check, read about the 4 simple questions you can ask to help identify and solve the right problems before jumping to solution mode (How to solve the right problems).

References

• Shane Frederick (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives.
• Daniel Kahneman (2011). Thinking Fast and Slow. Chapter 3, The Lazy Controller