Making reliable forecasts the outcomes of a project is important for project planning. A typical example for a researcher is planning a collaborative new research paper. You have to coordinate a team and resources to pull it off. You will need to estimate how long different tasks will take, so you can coordinate the team. You will also need to forecast the likely outcome.
You would also typically make forecasts about the benefits on completion, such as the likely journal or other benefits such as policy decisions it will influence.
Below is a set of questions to test how you think about forecasting the outcomes of projects. First take the quiz. Then below is some discussion of how this self-evaluation can help you improve your decision making when planning projects.
The test is based on the Foxes and Hedgehogs dichotomy proposed by Philip Tetlock, a psychologist who specialises in how humans make forecasts.
A nice write up of the framework is in Nate Silver’s book Signal and the Noise.
Test
Think about a project you planned and have been working on recently, such a research paper you are leading. Now scores yourself on each one of the below questions. Each question is in two parts. Score yourself 1-5, where 1 is a strong yes to the first question in each pair and 5 is a strong yes to the second question in each pair.
Try to be as honest with yourself as possible, don’t score yourself for what you think you should have done, score yourself for what you did (even better, get someone who knows you to score for you). The evaluation will be more useful that way.
Q1
- I sought multiple perspectives and opinions on my project plan
- I sought the perspective of one expert or relied on my own expert judgement
Q2
- I sought perspectives from people from different disciplines, with different backgrounds, or different viewpoints
- I sought perspectives only from experts in a relevant field
Q3
- Once the project began, I adapted the plan as new information came to light
- I stuck to my plan
Q4
- I acknowledged mistakes in the project planning when something didn’t got to plan
- When something didn’t got to plan it was typically the result of bad luck
Q5
- I planned for multiple possible contingencies in project outcomes
- I made a detailed plan for a single forecasted outcome.
Q6
- I used data from past project outcomes to forecast the likely outcomes of this project
- I used my expert judgement to forecast the likely outcomes
Q7
- I estimated the chances that different contingencies would eventuate.
- I only considered one outcome or, if I considered multiple outcomes I didn’t compare their chances of success.
Q8
- I used data from past projects to estimate how long this project would take
- I estimated how long the project would take based on my expert judgement
Your score
Now add up your scores. If you scored less than 20 you are more like a ‘fox’ in your forecasts. If you scored more than 20 you are more like a hedgehog.
Fox vs hedgehogs in project planning
Foxes tend to be better at forecasting than hedgehogs. Foxes take into account diverse viewpoints, consider the complexity of the world, hold multiple theories for how the world operates at the same time, use data to inform forecasts, estimate the chances of different outcomes, adapt to changes in outcomes and acknowledge their mistakes.
Hedgehogs based their forecasts on a single theory, have simple theories for how the world works, use theories or their ‘gut’ to forecast outcomes, don’t draw on different perspectives, stick to a single plan or theory and tend to attribute forecasting mistakes to bad luck.
If you want to know more read the book or watch the video above.
It is not that hard to think more like a fox when you are planning and implementing your next research project. Just scan the questions above and try to action the first in each pair (e.g. seek multiple perspectives on your plan, use data from past projects to estimate how long this one will take).
- The test is not at all scientific, I’m sure it could be designed by a real psychologist to be much less biased.