Expecting the unexpected vs not expecting the expected
The eNPV itself isn't the problem; its misuse is...
No-one expects the net present value in the eNPV.*
Many companies say this doesn’t matter, because:
‘we don’t believe it, but it is useful because the same errors go into everyone’s calculation’
‘we don’t think it is accurate, but it is directionally useful’
‘we don’t believe it, but it is a useful exercise for the team to go through’
‘we do produce an eNPV for each project, but we don’t really believe any of them…’
Many of the arguments in support of the eNPV recall the famous rebuttal to Nobel laureate Kenneth Arrow, a young statistician during the Second World War:
“The Commanding General is well aware the forecasts are no good. However, he needs them for planning purposes.”
or Dwight Eisenhower’s speech in 1957:
I tell this story to illustrate the truth of the statement I heard long ago in the Army: Plans are worthless, but planning is everything. There is a very great distinction because when you are planning for an emergency you must start with this one thing: the very definition of “emergency” is that it is unexpected, therefore it is not going to happen the way you are planning.
There are two main opportunities to upgrade the eNPV/ rNPV.
The first is simple: there is no single eNPV, by definition. An average forecast of the hundred different ways a molecule could launch, and the paths it might take to get there, is not useful. Each of those one hundred would have a different eNPV: time to market, costs, value, risks, etc. It is better for you to have all of those one hundred than to have a single compromised number that represents none of them. It is better for you to lean into that complexity than to ignore that it exists, especially when you accept that your competitors may well be running the same process, but with different assumptions.
In early stage, your forecast of risk (probability of technical success, especially) shouldbe extremely low. It makes no sense to import average attrition rates for the disease area, or any of the other surrogate numbers that are used. The economics of path dependence make risk calculation risky - choosing a linear path will lead to overestimate of probability, while ignoring paths to the side that might have been better, or the opportunity to recalculate at a point in time ahead.
(Industry-average attrition rates have a related problem: the success of therapeutic areas like oncology in phase I to market, versus areas like CNS, means oncology projects are prioritised by stealth, via tools like the eNPV.)
If, in early stage, your forecast of probability of technical success is high, you’re certainly not calculating it right. But you probably are gaming the portfolio management process. If one team unilaterally tells the whole truth, their programme will be killed because every other team inflated their confidence - the tragedy of the commons is at work. Development process means there is no incentive to reveal real probabilities. So market size and share is inflated, as is the probability of getting there - a sophistry we all know is happening.
If in reality, no-one believes the forecast for an early stage asset, or the probabilities of success (even McKinsey acknowledge that they’re all wrong, and wrong in both directions), we might ask if they are indeed useful exercises for teams to go through. Is the planning useful? Well, it would be if the goal was to produce range, rather than a single project value estimate. As soon as the team decide on a compromise TPP, and the eNPV, it will be gamed to look positive - adherence rates will be bumped up by 5%, PTS by 8%, market share by 2%, etc… If instead the team focused on good numbers for each of a wide range of paths, and were incentivised to produce accurate assumptions, the eNPVs would still be wrong (inevitably wrong) but they could be useful: the range would provide choice to portfolio managers, as well as risk mitigation strategies that recognise path dependence in early phase. The team would be having an interdependent series of conversations about opportunity, not seeking consensus on risk.
A learning company has to diagnose the parts of its system which stop the learning. The eNPV hides so many mistakes within its truthiness that it can only be dogmatism, masquerading as pragmatism. It may be hard to remove, but opportunity seeking can’t co-exist with the current process.
It is better for you to lean into that complexity than to ignore that it exists. What would that mean? Our approach to early phase planning necessarily involves the creation of alternative paths to market (different indications, probably, but certainly different patient populations and endpoints) - you can imagine that each of these would have a different eNPV. None is right, but what comes next is very important. Rather than basing decisions on the best-looking number (the worst choice), the key step is to reduce risk within those calculations. That would necessarily involve more proofs of concept, more basic science.
This leads to the usual push-back: that increases costs in early phase. It does. That is a good thing. The worst world is the one we have now, where drugs remain vulnerable to age-old expensive attrition, but we’re also launching lots of drugs that remain a long way from delivering RoI. Spending more to reduce that ratio makes a lot of sense financially. There is a lag, for sure, but getting the process right as soon as possible makes sense right now.
*I’ve used eNPV throughout as the risk-adjusted NPV is even more badly named for a similar process - it ignores so many risks, practical and opportunity risks for example, that it is under-adjusted by design.
Hard to disagree with much of the above but a few quick thoughts: I am aware of a number of companies that provide Monte Carlo type simulation to address (however imperfectly) the uncertainty inherent in drug development, and in particular the path dependency outlined above. The advantage of such approaches lies less in the range of outputs delivered (itself a function of various estimates) but in the need to think from first principles and try to understand the relevant parameters.
The second point is a bit more philosophical in nature; if we see early stage development as an information gathering exercise around patient population, measurements of efficacy and their clinical (and commercial) relevance then management needs to ask about the cost of obtaining such information. Merely trying to run a balanced portfolio based on a spread of technical success (from low to high) while appearing reasonable is unlikely to be optimal.
Finally I believe (but can’t prove) that there is a kind of inherent inertia in certain aspects of portfolio management; crudely put it’s easier to continue down a particular path (or TA) than to pivot unless there is a very compelling reason to do so (such as a command from the C suite).
Sharing an anonymous comment I received:
Thank you for sharing your insights in the article "Expecting the Unexpected vs Not Expecting the Expected." I would like to offer a few thoughts on the topics you covered.
You mentioned that "if, in the early stage, your forecast of the probability of technical success is high, you’re certainly not calculating it right." I agree, that accurately calculating this is nearly impossible, and Portfolio Management shouldn’t rely solely on these figures for decision-making. This reliance on precise numbers can be challenging, as Portfolio Management often requires quantifiable data to guide their work.
Your point about "one team unilaterally telling the whole truth and having their program killed because every other team inflated their confidence" highlights a significant issue. Programs often become inflated due to incentives that reward perceived success, which can be problematic. However, this raises the challenge of how to effectively motivate teams toward success, especially when there’s internal competition within the company or therapeutic areas.
Regarding the statement that "there is no incentive to reveal real probabilities in the development process", I believe this is due to the high uncertainty surrounding true probabilities of success, which are influenced by countless variables, including management’s willingness to invest in a clinical program (Andrew Lo and colleagues). A smaller budget only adds to these uncertainties.
When you mention that "market size and share are inflated, as is the probability of getting there", I’m not entirely convinced. Pharma has become fairly adept at forecasting, and in some cases, estimates may even be deflated, which can also lead to poor decisions.
The idea that "no one believes the forecast for an early-stage asset, or the probabilities of success" and that these exercises might be futile if they’re consistently wrong is compelling. While the forecasted path should serve as a guide rather than a definitive truth, it often ends up being a scapegoat if things go wrong.
I also agree that it would be more productive to aim for a range of estimates rather than a single project value. This approach would give Portfolio Management more options and allow for better risk mitigation strategies that account for uncertainties in early phases.
That said, I’m not entirely aligned with the notion that teams inevitably "game" the numbers to look more favorable, although I acknowledge that some stakeholders within the process might lean in this direction, creating biases. While biases can influence assumptions, the focus should be on creating a range of robust assumptions that better guide decision-making.
Risk mitigation and management should be central to the decision-making process before committing to a program, which aligns with the points you cited from Kenneth Arrow and Dwight Eisenhower’s 1957 speech (Planning involves assessing risks, even when there aren't mitigation strategies available for every potential and uncertain risk.). It’s essential to discuss the potential for termination openly and thoughtfully, as failing to do so can lead to missed opportunities or the continuation of programs that should have been reconsidered.
No one enjoys having their project critiqued, but addressing criticism directly provides an opportunity to explore solutions that could mitigate risks. Mitigation should be based on solid data that impacts safety and efficacy, rather than simply serving as a cost-saving measure through termination.
Portfolio Management serves as a valuable tool for prioritizing high-value projects with the best risk-adjusted NPV (rNPV). However, challenges arise from the system of variables, whether it’s standardized PoS from the literature that might not fit a specific development program or management’s efforts to speed up clinical programs by compromising on critical tasks (Andrew Lo and colleagues). The devil is in the details. Portfolio Management is important, but the decision-making process may need adjustments, and decision-makers might benefit from education on how to handle educated guesses. Decisions should be based on more than just values and gut feelings.