When Oncology KOLs Disagree, Whose Judgment Is Right?

Transforming Conflict into Strategic Foresight

Pharma uses/ overuses advisory panels. A question I’ve mulled since my first: if you have a room of 10 people, and one, only one, disagrees with the majority, who’s to say they’re wrong (apart from the other 9)? Most ad panel processes tend to look for consensus (Delphi, for example), but consensus is often at odds with insight. Although we often construct ad panels with a mind to Tetlock’s principles*, we spend too little time using his tenets post-hoc.

I’ve seen teams recording insights that they ‘like’, despite lots of dissent, so even techniques like running the audio recording through a tool like Notebook LLM can be useful for taking the emotion out. It can be attractive to look for averages, but even the usual breakdown of median, mean or mode can be misleading, if the gem lies in the opinions of that one standout.

I’ve often been the ‘one’ in a room, so I’ve tried to break down the most scientific approach to ad panels.

---

Envision a strategic convening: a virtual advisory board comprising 10 preeminent oncology key opinion leaders (KOLs) - specialists in immuno-oncology, hematologic malignancies, or precision therapeutics. The focus: anticipating the pivotal evolution in cancer care, perhaps the broader integration of antibody-drug conjugates (ADCs) across solid tumors or the scalability of CAR-T therapies beyond blood cancers. Their analyses are rooted in frontline trial data and clinical acumen, yet consensus eludes them. One forecasts transformative adoption of next-generation KRAS inhibitors for pancreatic cancer; another tempers expectations citing resistance risks and reimbursement constraints. In oncology, where projections influence multi-billion-dollar pipelines and expedite access to life-extending innovations, harmonizing these divergent views is imperative. Informed by forecasting methodologies from Philip Tetlock’s research* and oncology’s dynamic landscape, this guide provides a disciplined framework to extract actionable foresight from discord.

Understanding Divergence Among Oncology KOLs

Oncology’s evidentiary foundation - bolstered by guidelines from the National Comprehensive Cancer Network (NCCN) and European Society for Medical Oncology (ESMO) - fosters alignment, but KOL forecasts often vary. This arises from:

• Overreliance on Isolated Data: A KOL may extrapolate heavily from a breakthrough trial, such as a Phase III ADC readout, while underweighting tumor heterogeneity or off-target effects.

• Domain-Specific Lenses: An immuno-oncologist might champion checkpoint inhibitors’ synergies, whereas a hematologist prioritizes CAR-T’s cellular precision - mirroring the field’s transition from broad cytotoxics to targeted modalities.

• Differential Access to Insights: Early exposure to unpublished spatial transcriptomics or ctDNA analyses can skew perspectives, akin to viewing a mosaic from varying distances.

• Interpretive Nuances: Shared datasets, like those from neoadjuvant immunotherapy trials, may diverge on biomarkers - e.g., PD-L1 expression as a robust predictor versus a flawed proxy overshadowed by tumor microenvironment dynamics.

Such variances are inherent to oncology’s complexity, where tumor evolution and patient diversity amplify uncertainty. The path forward lies not in selecting the most assertive opinion but in systematically appraising its substance.

Step 1: Evaluate Historical Accuracy

Forecasting prowess varies; a KOL’s prior calibration serves as a foundational filter. Tetlock’s Good Judgment Project underscores that superforecasters outperform through probabilistic humility, a principle applicable here: Distinguish those who foresaw ADCs’ expansion from 2021’s Enhertu approval to over 100 candidates by 2025 from skeptics who overlooked payload refinements.

Exemplar: The rollout of KRAS G12C inhibitors like sotorasib. Post-2021 approval, KOLs divided - optimists projected rapid uptake in non-small cell lung cancer (NSCLC), while others anticipated stagnation due to acquired resistance. Calibrated forecasters, integrating base rates from prior EGFR inhibitors with real-world evidence, accurately gauged sustained efficacy signals reported in 2025, informing a 15-20% market penetration in eligible cohorts.

Practical Application: Solicit documented priors, e.g., “Your 2023 view on bispecific antibodies’ solid tumor pivot?” Assess via Brier scores or hit rates. Without a record, engage judiciously. This curation elevates the panel from a spectrum of voices to a vetted consortium.

Step 2: Examine the Reasoning Framework

Robust predictions hinge on defensible logic. Pose incisive queries: Which evidence - trial endpoints, biomarker validations - anchors your outlook? Quantify confidence (e.g., 65% probability of EMA endorsement for an allogeneic CAR-T by 2026)? What threshold shifts your stance, such as null ctDNA dynamics or payer recalibrations?

Superforecasters deconstruct methodically: For CAR-T’s foray into solid tumors, segment antigen specificity, manufacturing scalability, and cytokine release syndrome mitigation. Evasion of counterevidence, like inconsistent homing in gliomas, erodes credibility.

Strategic Enhancement: Convene moderated exchanges to articulate and rebut alternatives, akin to peer review in trial design. This illuminates integrative thinkers, yielding syntheses that advance beyond silos.

Step 3: Synthesize Through Aggregation

Isolating a solitary forecast risks oversight; aggregation leverages collective acuity. Evidence shows accuracy gains of 15-20% from weighted ensembles in volatile arenas like therapy launches. Blending views on radiopharmaceutical adoption - e.g., 70% optimism for prostate cancer versus 50% caution on isotope supply - might converge at 62%, refining resource allocation.

Implementation Tip: Tabulate probabilities (e.g., “75% likelihood of ADC-immunotherapy combo guideline inclusion”) or simulate a prediction market with nominal stakes. This emulates diversified risk assessment, distilling group wisdom efficiently.

Step 4: Adopt Superforecaster Principles

Tetlock’s tenets - triage, decomposition, balanced views, belief updating, and more - democratize expertise, enabling non-oncologists to adjudicate effectively. Prioritize “Goldilocks” questions (e.g., 2026 ADC payload optimizations over decade-long resistance trajectories); Fermi-estimate impacts (10 million annual NSCLC cases × 10% KRAS eligibility × 60% response rate); anchor to precedents like TIL therapy’s 2024 melanoma nod expanding to adjuvant settings. Update iteratively with congress readouts (e.g., ASCO 2025) and log outcomes for refinement.

These habits, empirically validated, bridge KOL depth with analytical discipline, fostering resilient strategies.

The Distinct Challenges of Oncology Forecasting

Oncology’s future is obscured by genomic instability, heterogeneous responses, and regulatory scrutiny: Adaptive trial designs grapple with surrogate endpoints like progression-free survival versus overall survival versus disease-free survival; ESMO guidelines evolve amid ctDNA debates; payers scrutinize cost-effectiveness for high-price modalities like CAR-T ($400,000+ per course).

Ground deliberations in evidentiary pillars - randomized trials (e.g., MACE analogs in oncology: objective response rates), real-world databases (e.g., Flatiron Health), and ecosystem factors (biosimilar erosion on trastuzumab).

The CAR-T odyssey epitomizes 2017’s inaugural approvals for leukemia, which electrified by slashing relapses by 80% in pediatrics. However, 2024 critiques spotlighted secondary malignancies and neurotoxicity, curbing enthusiasm for solid tumors. By 2025, allogeneic iterations promise off-the-shelf access, but KOLs diverge - proponents analogize to bispecifics’ rapid scaling, forecasting 30% hematologic expansion; detractors cite homing failures in gliomas, projecting <10% solid tumor penetration absent microenvironment breakthroughs. Superforecasters, weighting epigenetic resistance (non-mutational drivers in 98% of the genome) with base rates from TIL expansions, navigated this to predict combinatorial wins, unlocking $5B+ markets by integrating with ADCs.

Parallel the ADCs surge: 2024’s 15 approvals thrilled with HER2 precision, but combination toxicities (e.g., neuropathy spikes) fueled debate - enthusiasts eyed synergies with PD-1 inhibitors for 25% response uplifts; skeptics warned of overlapping myelosuppression, echoing early taxane pitfalls. Decomposers modeling efficacy-access-toxicity triads foresaw refined payloads mitigating risks, propelling 2025’s projected 20% therapy share growth.

Transforming Conflict into Strategic Foresight

An oncology KOL assembly is a vault of unparalleled insight, awaiting orchestration. Through track record scrutiny, logic dissection, ensemble synthesis, and superforecaster rigor, discord yields precision. In a domain where foresight curtails mortality - U.S. rates down 33% since 1991 - this methodology is foundational to finding a way forward.

---

*Philip Tetlock’s Superforecasting Techniques: The Art of Accurate Prediction

Philip Tetlock, a psychologist and political scientist, revolutionized forecasting through his work on the Good Judgment Project, a tournament that tested thousands of predictors against intelligence analysts. His 2015 book Superforecasting: The Art and Science of Prediction (co-authored with Dan Gardner) reveals that “superforecasters” - those who consistently outperform experts - aren’t oracles or geniuses but practitioners of disciplined, learnable habits. Tetlock’s core techniques distill into the “Ten Commandments for Aspiring Superforecasters,” a framework emphasizing humility, analytical breakdown, and iterative learning over intuition or overconfidence. These principles, validated through rigorous experiments, apply to diverse domains like geopolitics, business strategy, and even personal decisions.

Below, I outline the ten commandments with Tetlock’s key insights and practical techniques for implementation. Each draws from patterns in superforecasters’ behaviors, such as probabilistic thinking and error-tracking.

1. Triage: Define Your Focus

• Core Idea: Not all questions are worth equal effort. Prioritize “Goldilocks” problems - challenging but solvable - avoiding the trivial (e.g., “the sun will rise tomorrow”) or the chaotic (e.g., exact stock prices in 2050).

• Technique: Rank queries by resolvability and impact. Use a simple matrix: High uncertainty + high stakes = dive deep; low on both = quick heuristics.

• Why It Works: Superforecasters allocate cognitive resources efficiently, boosting accuracy by 20-30% on targeted forecasts.

2. Break Apparent Problems into Smaller, Tractable Pieces

• Core Idea: Complex predictions crumble under holistic views; deconstruct them like a Fermi estimation (e.g., estimating global piano tuners by chaining population stats).

• Technique: Map sub-questions: For predicting a product’s market share, segment into R&D success, regulatory hurdles, and consumer adoption rates.

• Why It Works: This exposes hidden assumptions and reduces bias, turning vague hunches into precise chains of logic.

3. Strike the Right Balance Between Inside and Outside Views

• Core Idea: Blend unique case details (inside view) with historical benchmarks (outside view) to counter optimism bias, like the planning fallacy in project timelines.

• Technique: Start with base rates (e.g., “80% of startups fail”) then adjust for specifics (e.g., your team’s IP edge).

• Why It Works: Superforecasters avoid “reference class neglect,” improving calibration on timelines and probabilities.

4. Be Willing to Update Your Beliefs in Response to New Information (and Do So Quickly and Confidently When Evidence Demands It)

• Core Idea: Treat beliefs like Bayesian priors - revise them incrementally with evidence, resisting confirmation bias.

• Technique: After new data (e.g., a trial result), recalculate odds: If your 70% forecast drops to 50%, acknowledge it explicitly.

• Why It Works: This “belief updating” habit, like dental hygiene for the mind, keeps predictions agile and accurate over time.

5. Recognize That Nuance Matters: Good Forecasters Are Comfortable with Uncertainty and Complexity

• Core Idea: Embrace gray areas; synthesize opposing views into hybrid positions rather than forcing binaries.

• Technique: List counterarguments upfront and define falsifiability (e.g., “My forecast fails if X biomarker holds”).

• Why It Works: Nuance fosters resilience, as superforecasters outperform by integrating complexity without paralysis.

6. Distinguish Confidence from Certainty: Translate Vague Hunches into Precise Probabilities

• Core Idea: Avoid binary “yes/no”; express forecasts as odds (e.g., 65% chance) and calibrate them against outcomes.

• Technique: Practice with Brier scores: Track if your 80% predictions hit 80% of the time.

• Why It Works: Precision sharpens edges in competitive forecasting, revealing overconfidence gaps.

7. Balance Prudence and Decisiveness: Know When to Wait, When to Act

• Core Idea: Avoid rash calls or endless dithering; opt for “decisive humility” - act when evidence suffices.

• Technique: Set decision thresholds (e.g., act at 70% confidence) and use short-term proxies for long horizons.

• Why It Works: This equilibrium maximizes resolution (bold accuracy) alongside calibration.

8. Learn from Feedback: Keep Score of Your Predictions and Analyze Outcomes

• Core Idea: Treat forecasting as a skill to hone; journal hits/misses without rationalizing.

• Technique: Maintain a log: Post-event, ask “What surprised me? What would I change?”

• Why It Works: Feedback loops, like those in chess training, drive deliberate improvement.

9. Manage the Team: Foster Constructive Dialogue

• Core Idea: Superforecasting thrives in groups; encourage steelmanning (charitably reframing others’ views) and precise questioning.

• Technique: In meetings, probe: “What evidence would change your mind?” to build collective intelligence.

• Why It Works: Teams using these dynamics outperform individuals by aggregating diverse insights.

10. Master the Error-Balancing Bicycle: Practice Deliberately with Feedback

• Core Idea: These rules form a dynamic balance, learned through repetition - like riding a bike.

• Technique: Engage in low-stakes drills (e.g., weekly news predictions) and review quarterly.

• Why It Works: Deliberate practice embeds habits, turning novices into superforecasters over months.

Applying These Techniques Today

Tetlock’s methods extend beyond tournaments: In 2025, they’re integrated into AI-augmented tools for risk assessment and corporate strategy, with ongoing research via Good Judgment Inc. emphasizing “Bayesian Question Clustering” for long-term forecasts. To start, pick one commandment (e.g., probability journaling) and apply it to a current dilemma - your accuracy will compound. For deeper dives, Tetlock’s book remains the gold standard.

Avoiding Eminence Bias in Forecasting: Lessons from Tetlock and Beyond

Eminence bias - also known as authority bias - is the tendency to overweight opinions from high-status individuals (e.g., renowned experts or “thought leaders”) simply because of their reputation, rather than the quality of their evidence or reasoning. In contexts like pharmaceutical forecasting, where key opinion leaders (KOLs) hold sway, this can lead to flawed decisions: a famous clinician’s hunch on drug adoption might eclipse data-driven alternatives, mirroring the pitfalls Tetlock identified in Expert Political Judgment, where celebrated pundits performed no better than chance (and often worse) due to overconfidence tied to their eminence.

Tetlock’s superforecasting framework, validated through the Good Judgment Project, offers a antidote: Shift focus from pedigree to process, treating fame as a potential red flag rather than a credential. To illustrate why this matters, consider KOLs in pharma—a prime arena for eminence bias. Their status often amplifies voices that may prioritize visibility over verifiability, yet Tetlock’s principles (e.g., vetting track records and aggregating views) provide tools to neutralize it. Below, I’ll unpack how KOLs ascend, highlighting paths that can entrench bias, then detail Tetlock-inspired strategies to sidestep it.

How Do KOLs Become KOLs? The Pharma Path to Influence

Becoming a KOL isn’t a meritocracy sealed by a single breakthrough; it’s a multifaceted ascent blending expertise, networks, and industry dynamics. In the pharmaceutical industry, KOLs are typically physicians, academics, or researchers who influence peers on clinical practices, guidelines, and innovations. Pharma companies actively cultivate them as strategic partners for advisory boards, speaking gigs, and trial consultations. The process unfolds in stages:

1. Build Core Expertise and Visibility: It starts with clinical or research excellence - high-impact publications in journals like The New England Journal of Medicine, leadership in trials, or guideline contributions (e.g., NCCN panels). Digital footprints amplify this: Active presence on platforms like X (formerly Twitter) or LinkedIn, sharing insights on emerging therapies, can accelerate recognition.

2. Network and Peer Validation: Pharma scouts via surveys asking HCPs to name influential peers, or by analyzing citation networks and real-world data (RWD) from claims databases. Internal rankings often factor “friendliness” or alignment with company goals, not just objectivity. Consultants or medical affairs teams at firms like IQVIA formalize this, tiering KOLs (e.g., Tier 1 for global influencers with 50+ publications; Tier 3 for regional specialists).

3. Industry Engagement and Reinforcement: Once identified, KOLs are engaged for paid roles - speaking fees ($5,000–$20,000 per event), consulting, or advisory boards - creating a feedback loop. This boosts their eminence: More invites follow, as peers perceive them as “key” based on pharma’s endorsement. By 2025, broader criteria include patient advocates or payers, but physicians/academics dominate (80%+ of KOLs).

The catch? This system can favor charisma and availability over prescience. A KOL might rise via frequent talks for sponsors, embedding subtle biases (e.g., favoring certain drugs), yet their status alone sways forecasts - exemplifying eminence bias. As Tetlock notes, such “hedgehogs” (bold, status-driven experts) falter in accuracy compared to “foxes” (nuanced integrators).

Strategies to Avoid Eminence Bias: Tetlock’s Toolkit in Action

Tetlock’s superforecasting techniques explicitly counter biases like eminence by democratizing evaluation - anyone can apply them to KOL roundtables, shifting from “who said it” to “how well does it hold up.” Here’s how, tailored to pharma contexts:

These aren’t add-ons; they’re a “bicycle” of habits (Commandment 10) that superforecasters master through deliberate practice, like journaling outcomes quarterly. In pharma, tools like Good Judgment Inc.’s platforms now embed this for KOL engagements, yielding more resilient strategies. The result? Decisions grounded in evidence, not echoes of eminence - turning potential pitfalls into predictive power. If you’re facilitating a KOL session, start with anonymous polling to test it out.

---

There’s a lot there. In a few bullets, I’ll try to summarise:

• The Challenge of Expert Discord: In oncology advisory boards, 10 KOLs often clash on forecasts like ADC expansion or CAR-T scalability, driven by factors such as overreliance on isolated data, specialized lenses, info asymmetry, and interpretive differences - demanding a structured approach to extract value.

• Vet and Probe for Reliability: Start by auditing KOLs’ track records (e.g., KRAS inhibitor predictions) using metrics like Brier scores, then dissect their reasoning with questions on evidence, confidence levels, and falsifiability to uncover robust logic.

• Aggregate and Superforecast: Blend probabilistic views through weighted averages or prediction markets for superior accuracy (15-20% gains), while adopting Tetlock’s principles like decomposition, base rates, and iterative updates to refine group insights.

• Oncology-Specific Hurdles and Wins: Address unique complexities like genomic instability and payer scrutiny with evidentiary triads (trials, RWE, markets); case studies like CAR-T evolution show how integrated analysis unlocks billions in value, turning conflict into strategic precision.

---

So, we all agree that there’s a chance that the one KOL might be right, but how would we know?

Determining When the Lone Dissenter Is Right: A Probabilistic Path

Spotting the “one right against nine” isn’t about a eureka moment - it’s a retrospective validation or prospective hedging, rooted in Tetlock’s superforecasting ethos of evidence over intuition. In real-time, amid a KOL roundtable or scientific debate, you can’t say the outlier is correct without new data; instead, treat it as a hypothesis to test. Here’s how to navigate it systematically, blending foresight principles with historical lessons.

• Track Outcomes Relentlessly (Tetlock’s Feedback Loop): The gold standard is waiting for resolution - log predictions and revisit post-event. If the dissenter’s forecast aligns with reality (e.g., via Brier scoring), their edge emerges. In pharma, this means journaling the KOL's take on drug uptake and checking it against RWD a year later. Superforecasters shine here: They don’t claim prescience but accumulate hits through calibration, outperforming consensus by updating beliefs Bayesian-style when evidence mounts.

• Prioritize Reasoning Over Consensus (Probe for Falsifiability and Base Rates): Scrutinize the outlier’s logic: Does it decompose the problem (e.g., “What’s the base rate for similar innovations failing?”), acknowledge counters, and specify what would prove it wrong? Eminence bias tempts deference to the majority, but Tetlock warns against it - groupthink often amplifies shared blind spots. If the lone voice integrates outside views (historical analogs) better than the pack, bet provisionally on them.

• Aggregate with Weights, Not Votes (Hybrid Synthesis): Don’t pick sides; blend probabilistically, overweighting the dissenter if their track record or nuance scores higher. Tools like prediction markets reveal hidden convictions. For example, if nine models predict 90% drug failure odds but one model predicts 30% with mechanistic insight, the ensemble might land at 65%, flagging the outlier for closer watch. This avoids all-or-nothing traps, as seen in Tetlock’s tournaments where foxes (integrators) beat hedgehogs (bold majoritarians).

• Historical Precedents as Cautionary Guides: Lone vindications often surface years later through accumulating evidence. Ignaz Semmelweis, ridiculed in 1847 for insisting handwashing curbed childbirth fevers (reducing mortality from 18% to 1%), was proven right by germ theory decades after his death. Gregor Mendel, whose pea-plant genetics were dismissed as irrelevant in 1866, was rediscovered in 1900 and thus became foundational to modern biology. John Yudkin, a British nutritionist, warned in 1972 that sugar - not fat - drove heart disease, only vindicated by 2016 exposés on industry suppression. These echo oncology’s CAR-T skeptics, who, against early hype, highlighted toxicities now refined into guidelines. The lesson? Dissent thrives on testable claims; when it does, it reshapes paradigms.

In essence, you “know” through patient empiricism: Hedge on the outlier if their case is crisp, but let time and data arbitrate. As Tetlock quips, forecasting is less about being right now and more about being less wrong over time - turning solitude into strategy.