As we consider making predictions in the illegal wildlife trade, I think we need to be very considerate of who we include as an expert. We should be clear that an expert isn’t just someone with a certain degree level or status. It can be anyone with substantial knowledge on a topic who has developed intuitive judgement on the matter (Flyvbjerg 2001). And we also need to make sure that we know how to properly elicit information from experts.
And so in this blog I introduce to you the IDEA protocol for expert elicitation.
First, what do we mean by “expert elicitation”? In short, say we want to predict an emerging trend in the illegal wildlife trade. Or the impact a conservation proposal will have on a certain trade, for instance the displacement from one illegal market to another. When we want to predict future events or forecast consequences, there is value in using an expert in an uncertain future. At times even more so from a group of experts (Kaplan et al. 1949). But experts are not immune to psychological biases from social pressures and influence. So when we ask for their judgements, we must be careful to ask the right questions, in the right way. With this thinking in mind, the Delphi process was originally developed.
The Delphi process can be described as the first forecasting tool used with a group of experts. It has four features to limit psychological bias and better the accuracy of our predictions (Linstone and Turoff 1975, Rowe and Wright 1999):
Anonymity – by making responses anonymous within the group
Iteration – by giving experts a chance to alter or stick to their original responses
Controlled feedback – by summarising the initial responses to the group
Statistical group response – by averaging the group response
So in practice, the Delphi method would involve asking several experts to answer the same questions in at least two rounds. After the first round, an anonymised, statistical summary of results would be shared among the group. This would allow experts to revise their answers in the second round based on the collective results from the first. And so in theory a better prediction could be made. But this process is still fairly limited, as it doesn’t provide a platform to discuss summarised or extreme responses, nor does it control for biased judgements. In general it seeks group consensus, which isn’t necessarily a practical approach for the complexities of the illegal wildlife trade. In response to these shortcomings, the IDEA protocol was developed (Burgman 2015) to improve on the expert elicitation process.
The IDEA protocol adapts the process of expert elicitation to additionally control for biases from ambiguous language, limited knowledge and natural unpredictable change. Most notably, Burgman (2015) highlights linguistic misunderstandings from ambiguous, vague, under specific, context-lacking, conflicting or over-complicated questions. This is to say that biased perceptions can arise from:
Framing effects – being influenced by the context or scale of the question
Anchoring – being influenced by initial numerical estimates
Availability bias – predicting a higher probability for recent or well-known events
Motivational bias – being influenced by personal feelings, emotions and circumstances
Risk aversion – having a preference for a safer bet
Insensitivity – lacking consideration for small numbers
Overconfidence – being more confident, or wanting to appear confident, but not necessarily in correlation to higher accuracy.
Essentially, the IDEA protocol lays out useful steps to take along our expert elicitation process and how we can Investigate, Discuss, Estimate and Aggregate predictions. In this way it guides us in the planning, elicitation and concluding stages of the process. You can follow these steps below (Hemming et al. 2017):
As such the IDEA protocol can guide our study of wildlife crime (link). It is better adapted to the complexities of wildlife crimes as it doesn't overlook obscure or 'unpopular' forecasts. By following this protocol, we can ask questions in the right way to better predict future events. These events can include emerging trends of the illegal wildlife trade or predictions of effective conservation interventions. In this way, we can utilise experts more efficiently in tackling wildlife crime problems.
Usefully, Burgman has provided additional tips to consider when eliciting information from experts using the IDEA protocol. I leave you with this list here as a takeaway thought for future working (p.141):
Be clear about what you want from experts: estimates of simple facts, predictions of the outcomes of events, or advice on a best course of action.
Be clear about the domains of expertise that will help, and choose people whose skills, training or verified experience (where it exists) are squarely in those domains. This may include people who, on paper, don’t fit the typical ‘expert’ mould. Where the opportunity exists, offer appropriate training.
Choose as many experts as possible, don’t be concerned about their age, number of publications, peer status, technical qualifications or apparent impartiality.
If the matter at hand is politically sensitive or socially or emotionally charged, ensure that the experts have diverse relevant opinions or positions. Work to diversify the culture, gender, context and perspectives of the participants. Try to include people who are less self-assured and assertive, and who integrate information from diverse sources.
Compose questions to avoid arbitrary linguistic misunderstandings and psychological trip wires such as framing, anchoring, availability bias and so on.
Use structured question formats to counter tendencies towards overconfidence, and oblige participants to consider counter-factuals and alternative theories.
Use structured, facilitated group interactions to counter dominance effects, anchoring and other factors that lead to groupthink.
- Burgman, M. A. (2015). Trusting judgements: How to get the best out of experts. Cambridge University Press, Cambridge. Amazon Link.
- Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and how it can succeed again. (1st ed.). Cambridge, Cambridge University Press. Amazon Link.
- Hemming, V., Burgman, M. A., Hanea, A. M. & Wintle, B. C. (2017). A practical guide to structured expert elicitation using the IDEA Protocol. Methods in Ecology and Evolution, 9: 169-181. Open-Access Link.
Header Photo Credit: Pixabay. Link.