The concept of proof
Exceptional rule breaking
An exception can’t prove a rule if the concept of ‘prove’ is the one we commonly understand. The intention of the word in that sentence is ‘test’ - which makes more sense… An exception absolutely does test a rule. That’s the way science works - observation should trump prediction, and a rule is no good if there are exceptions…
Unfortunately, the concept of ‘proof’ in Proof of Concept typically suffers from the same misapplication. While it typically is a test of concept or test of mechanism, it rarely provides ‘proof’ in the ‘strong evidence’ sense. If you apply our rules of Evidential Quality and Decision Quality, most phase I or phase II studies provide extremely low evidential quality data for a pivotal study. There may well be positive signals, or some test of feasibility, but you’d demand more if you could.
But, the bigger issue is with the ‘concept’. When we coined the phrase ‘you can’t do proof of concept if you don’t have a concept to prove’, I missed the opportunity to elaborate. Companies test concepts all the time - ‘proof of concept’ studies or ‘proof of mechanism’ studies litter the landscape. Companies claim ‘success or failure’ from them, as if the ‘strong evidence’ definition applied.
Let’s unpack ‘concept’. Your ‘concept’ is that your drug is a rheumatoid arthritis drug, via an anti-inflammatory mechanism. Let’s set aside why you made that prediction, and accept it. What ‘concept’ of your molecule would you like to test? Here are some:
Your drug could be a disease modifier in RA
Your drug could reduce joint damage in RA
Your drug could reduce pain in RA
As well as ‘treating’ RA, your drug could treat other inflammatory conditions
Your balance of benefit to risk in every condition is positive
Your anti-inflammatory activity comes from reducing auto-immunity
Your anti-inflammatory activity comes from the inflammation cascade
etc…
These are not mutually exclusive, of course. But they are not mutually inclusive. Showing that your drug lowers CRP or pain in a phase 2 study may make you comfortable that there’s something there, but those data will be of low or lower or zero evidential quality for some of those concepts. You may well believe that being approved in an autoimmune inflammatory condition like RA will itself be proof of concept for your future lupus and psoriasis indications, but you’d be fooling yourself.
I remember a conversation at one company that started us down the route of re-examining the traditional path approach. In essence:
[Very smart scientist] We’re excited about our TRPV1
[Me] I bet. Where is it at?
[VSS] It’s in proof of concept
[Me] Cool. Oh (in that Columbo way…) - just help me - it may be a stupid question, which proof of concept?
[VSS] It’s in an osteoarthritis phase II
[Me] Oh, interesting… Just help me though - are there any TRPV1 receptors in the joints?
[VSS] No
[Me] So how will it work?
[VSS] It won’t
[Me] So why is it in that study?
[VSS] Because when we asked our regulatory folks how to take a pain drug forward, they said we’d need an OA proof of concept
[Me] But if/ when it fails that OA PoC, what will happen?
[VSS] It will stall in Development
[Me} Help me - instead of asking ‘how do pain drugs usually demonstrate an effect on pain?’ couldn’t you have asked ‘how might this drug show an effect in a phase II?’
[VSS] Our process doesn’t work that way
We both knew how hard it would be to change that conveyor belt… I am sure you, like we then did in the next five minutes, started thinking about concepts for that TRPV1 drug - a heat pain drug, treatment for burns patients, and so much more, none of which would derive any positive or negative signal from an OA study. That first conversation happens every day in major pharma. The follow-on conversation happens rarely, but it is by far the more interesting.
I posted this a few months ago, as a way to show how we’d unpack ‘efficacy’ for a cancer drug.
Now, consider - these are all ‘concepts’. Your drug may well ‘work’ in ovarian cancer, but which definition of ‘work’ would you like to test? Would ‘working’ in ovarian (why did you choose that tumor? Is ‘ovarian’ even one homogeneous cancer?) be a proof of likely effect in other cancers, as you’d hope?
Development is a learning process. Examining the idea of Proof of Concept is critical - ‘testing one of a hundred concepts of our drug’ is a better way to understand the current model, although it isn’t presented that way. It is presented as ‘testing the concept of our drug’. Testing is an exploratory approach - it does not lead to failure or success, but to a healthier dataset, if you have planned it that way.
The rule of linear R&D based on the prediction paradigm fails its evaluation. We’re an exception…