Decision-based evidence making
I wanted to update my earlier post, with some working definitions of: ‘decision quality’, ‘evidential quality’ and ‘option quality’, as these core components of decision science are fundamental to our belief (assertion…) that ‘development is a decision science.’ Certainly, as discussion rages about whether pharma employs eminence-based decision making, or on which evidence it relies in ‘evidence-based decision making’, the emerging role of decision science will change structures for the better.
Here’s the earlier article:
All decisions are not equal. Qualitatively, quantitatively, they vary in many ways, not least in terms of import, impact and information.
Decisions also have 'quality': decision quality is something to gauge before you make a decision, not something that can be wound back in the presence of new information. The question: 'was it a good decision, at the time?' is critical, not 'does it now seem a good decision?'
So, given that pretty much everything we do in pharma is decision-based, one might imagine the emphasis would be on improving decision quality. One, however, would be very wrong.
In order to do 'evidence-based decision making', which is the mantra in most places, you need evidence upon which to make that decision. If all you have collected is narrow and focused, your decision quality is inevitably impacted: simply, you have too few alternatives to consider when you make your decision.
The decision to 'find out' is not the same as the decision to 'do'. As Annie Duke, author of Thinking In Bets, points out, the early 'bets' in poker are not trying to win, they're trying to learn - small, meaningful investments in a future decision. Likewise, in many chess metaphors, you'll hear 'you can't win the game in the early rounds, but you can lose it.' If early moves are investment-based, and exploratory, in games of decision-making, why not in pharma?
Decision quality begets the need for evidence quality. Even if you set aside how low validity most translational models are (let's not pretend that AI has changed that by more than a decimal behind the 0), the evidence behind early stage decisions is both weak and suffers from 'absence of evidence' problems more than 'evidence of absence' problems.
'Was it a good decision, at the time?' then becomes a function of how you chose to spend your time pre-decision. Spend time, as Annie Duke, David Epstein (and every military strategist, and every philosopher ever), would say, in planning to make a decision in the future, and you change your mindset.
Evidence-based decision making needs decision-focused evidence making. Knowing that you have a decision to make, you can generate an evidence plan (a Plan To Learn as we call it). It should, if you plan to optimise Decision Quality, have wide range, alternatives and a healthy reading of import and impact, as well as depth and quality of evidence.
That, in a paragraph, is why we're so focused on early phase becoming exploratory, rather than confirmatory - because it certainly is rather narrow and focused now. Picking and running into phase I is like the Barnes move in chess, apparently the 'worst possible opening move' (I loved this para from Wikipedia):
The move does exert influence over the central square e4, but the same or more ambitious goals can be achieved with almost any other first move. The move 1.f3 does not develop a piece, opens no lines for pieces, and actually hinders the development of White's king knight by denying it its most natural square, f3. It also weakens White's kingside pawn structure, opens the e1–h4 diagonal against White's uncastled king, and opens the g1–a7 diagonal against White's potential kingside castling position.[4]
Decision quality stands or falls on 'things you do know, and things you do not.' Pretending that 'things you do not' will be revealed by a narrow and focused phase I is a key reason for low phase II decision quality (and guess what, for phase III and beyond, as a consequence...).
We recognise this is not a small change from where things are. But it is logically, rationally, critical.
So, what do I mean? Here are some basic ways to frame the three elements of decision science.
What are the components of decision quality?
Decision quality refers to the degree to which a decision meets the criteria for being considered good or optimal. The components of decision quality can vary depending on the context and the goals of the decision-making process, but some common components include:
Relevance: The decision should be relevant to the problem or opportunity being addressed and aligned with the goals and objectives of the decision-maker.
Accuracy: The decision should be based on accurate, up-to-date, and relevant information.
Objectivity: The decision-making process should be objective and unbiased, free from personal or organizational biases.
Timeliness: The decision should be made in a timely manner, considering the urgency and importance of the problem or opportunity being addressed.
Feasibility: The decision should be feasible to implement, considering the resources and constraints available.
Flexibility: The decision should be flexible, allowing for adjustments and changes if necessary.
Transparency: The decision-making process should be transparent, with clear and well-documented reasons for the decision.
Responsibility: The decision-maker should be accountable and responsible for the consequences of the decision.
With that in mind, what are the components of evidential quality?
Evidential quality refers to the degree of reliability and credibility of the evidence used to support a decision. The components of evidential quality include:
Validity: The evidence should be valid, meaning it accurately reflects the phenomenon being studied.
Reliability: The evidence should be reliable, meaning it is consistent and reproducible.
Relevance: The evidence should be relevant to the decision being made, and directly address the problem or opportunity being addressed.
Representativeness: The evidence should be representative of the population being studied, and not biased or skewed towards a particular group.
Completeness: The evidence should be comprehensive, covering all relevant aspects of the issue being studied.
Objectivity: The evidence should be objective and free from personal or organizational biases.
Credibility: The evidence should be credible, with a strong track record of reliability and accuracy, and from a credible source.
With both of those in mind, what are the components of option quality?
Option quality refers to the degree of suitability and effectiveness of different options for solving a problem or taking advantage of an opportunity. The components of option quality include:
Relevance: The options should be relevant to the problem or opportunity being addressed, and aligned with the goals and objectives of the decision-maker.
Effectiveness: The options should be effective in solving the problem or taking advantage of the opportunity, and provide the desired outcomes.
Feasibility: The options should be feasible to implement, considering the resources and constraints available.
Resource requirements: The options should have appropriate resource requirements, such as time, money, and personnel, that are realistic and achievable.
Risk: The options should be evaluated in terms of the associated risks, including financial, legal, and ethical considerations.
Flexibility: The options should be flexible, allowing for adjustments and changes if necessary.
Compatibility: The options should be compatible with existing systems, processes, and technologies.
Sustainability: The options should have a long-term outlook, considering the sustainability of the desired outcomes over time.