Medtech Monthly, Ep. 3: Breakthrough Device Analysis with Bradley Merrill Thompson
Executive Summary
In this edition of the Medtech Monthly podcast, Epstein Becker Green attorney and data scientist Bradley Merrill Thompson spoke to Medtech Insight about this analysis of the efficacy of the US Food and Drug Administration's Breakthrough Devices program.
Listen to the podcast via the player below, or scroll down to read a transcript:
Brad's been a regulatory attorney for more than 30 years. Some of you might recognize his name from his work with the Combination Products Coalition and other industry initiatives. In addition to his day job at Epstein Becker Green, he's recently turned his hand to data analysis, earning a master of applied data science from the University of Michigan's School of Information in February 2022.
And he's on the show today to talk about some research he did into the US Food and Drug Administration's Breakthrough Devices Program. The Breakthrough Devices Program is, of course, an FDA effort to get innovative devices to patients more quickly. The program was first launched as a pilot under a different name in 2011, and the final guidance came out in December 2018. The FDA’s website touts 693 devices that have received the designation, of which 54 are authorized for market.
Brad used his own Python program to analyze publicly available data from Open FDA. The analysis was published under the title "Unpacking Averages: Assessing Whether FPS Breakthrough Device Designation Is Helpful" on legal website JD Supra on August 3. And today, he's here with us to talk through what he found and what significance might be.
First, as I read through the materials that describe the program, there was really no commitment by FDA as to what they would do in exchange for a company applying for and being admitted to the program. They didn't commit to how much faster the review process would be, they didn't commit to making sure that the company would only be expected to meet the least burdensome evidentiary obligations. There just really wasn't much substance there in the way of FDA committing to do something for the company.
But the other thing that bothered me, quite honestly, is that the program sort of positioned FDA as this almost consultant who would come in and work sort of side by side with the company in order to move the development process forward. And that just runs counter to all of my experience with FDA. And I'm not being at all, I hope, I mean, in saying that, we all play the role we're responsible for playing, and FDA plays a very important role, their role is protecting the American citizens. They are the government; they are there to make sure that unsafe or ineffective products don't reach the market. And that's a very different role than sitting on the other side of the table where your job is to try to figure out how to get the best possible safe and effective devices to market in the quickest amount of time. And so while those two missions overlap, they're not identical.
And FDA can be structurally far more conservative in its decision making than someone else would be so. So to have the FDA sort of hyper-involved in the development process with that sort of acknowledged mindset that was concerning to me.
So I'll be honest with you, I had a lot of clients who came in my door and said, you know, we want to be part of this breakthrough program. And I talked him out of it. I suggested to them that, that the benefits were dubious. But all that said, I hope I kept an open mind when I did the data analysis. And I tried to do it in a transparent way so that if I have a bias, the reader can can figure out maybe where I went wrong. So I tried to show my math, as it were, to explain how I did it.
It's natural, as I said, it's part of their job. But they'll say, you know, we think the clinical trial is, is really underpowered, we'd like to see a lot more subjects in it. We'd like to see a more complex design where we've got additional controls that provide us even greater assurance of safety and effectiveness. They'll just sort of pile on or add on to the evidence that will be ultimately required in order to get through the agency.
So I've made it a habit, honestly have tried to not involve FDA unless it's absolutely necessary, because I know that I'll get that conservative advice. And I think if you talk to consultants across the industry, and I've talked to several, they will say, if you make a good-faith determination of what the clinical trial ought to look like, and it's scientifically justified, and you do all your homework and you go into present it, inevitably, you can get something through the agency review process with less data than if you went in right from the get-go and just said to FDA, “Well, what do you want? What do you want us to do?”
It's cheap and easy for them really specify a very burdensome set of evidentiary requirements. There's an art to getting something through the agency, and the agency isn't there to help you identify that quickest pathway?
And we don't have the data to know where that problem is, whether it's in the pre-submission phase or the submission review phase. But the 54 out of 693 is really appallingly low. Because another graph that FDA provides is a timing graph that tells you in what fiscal years these submissions came in. And they've been coming in since 2015. But they really been coming in in earnest since about 2019. And 2019, there were 110 submissions. If you think about it, that's three years ago. This is early September, and the end of September will be the end of fiscal year 2022.
So at the end of three years ago, there were a whole bunch of these submissions in the queue in the breakthrough program, and still only 54 at this juncture have gotten through it so. So they're dying, or they're taking incredibly long. But I think it’s more likely they're dying. Maybe it’s because the technology maybe doesn't prove to work – although typically by the time you got to FDA, you've kind of proven that the technology works reasonably well.
Maybe the business case doesn't pan out, maybe investors don't keep the money flowing. But more typically, it's that you work with FDA, and you find out that their expectations are beyond what you think is reasonable or beyond what your technology will, will allow. And so that's really discouraging. And again, I've talked to my colleagues in the industry – it's a close-knit community. And the prevailing view is a lot of products, just struggle through the program, where the same products, not through the program, could sail through the FDA [approval] process. It's definitely a scary thing.
Is there any way that the program seems to differ across the different review pathways, such as PMA, de novo and 510(k)?
But in de novo, that experience is a little bit different. With the fresh data that FDA released about a month ago for the third quarter of the fiscal year, I calculated the improvement in review time for de novo products at about 64 days over the average for all de novo products. So I can't really do it on an apples-to-apples basis, because basically, de novo is our first of a kind. So by definition, there aren't other products out there to which you can compare them. So I just compare it to the average of all first of a kind. And in that respect, it shows an improvement.
But part of me thinks that that improvement isn't so much based on a quicker review, quicker work ethic effort, but rather the fact that maybe the agency spent enough time working with a company in the pre-submission phase that they had gotten sorted up the educational curve, had learned about the product, had learned about the studies and so forth, so that naturally the review could proceed a little bit quicker.
The only reason I'm focusing on review time is because it's the only thing I have data on the bigger picture. And the thing that we don't have data on is the development time. And that's where I'm scared that that the breakthrough program actually suffers you actually takes you longer to get through the pre submission phase for a breakthrough rather than a non–breakthrough.
So cardiovascular from 163 entered the breakthrough program to eight have exited the breakthrough program and have actually been authorized by FDA. Well, that's a huge differential – 163 entering the program, a grand total of eight being authorized out of the program. And there's a couple like that. Urology didn't fare as well as it should have based on the number that were going into the program. And at the other end of the spectrum pathology devices seem to do a little bit better in getting authorization based on the on the numbers going in versus the numbers coming out.
So different areas have different success rates. I would note that the cardiovascular products were by and large PMA devices, they were higher risk PMA devices, where the neurology and orthopedics and so forth were either 510(k) or de novo. So they were going different routes, as well. But I find that interesting. That's either encouraging or discouraging depending on what what particular clinical area you're in.
It would be very important for FDA to release data on when the first encounter occurs between an applicant and the FDA. So they have data on when the breakthrough status was requested and when it was granted, and they can easily release that. And then on the non-breakthrough products, they have a very well-developed pre-submission meeting process, a Q-sub process, where folks request a meeting, they go in with their questions, they get their questions answered, and then either they'll ask for another meeting down the road, or though they'll go to a submission or not.
But it's easy to tie those data together. And if we had those data, we could tell amazing insights into the time that it takes for companies to develop the data necessary to submit to FDA. I don't want to overstate it because people consult FDA at different phases, right? Some will consult FDA real early; some will consult FDA late. I get that. But there's a meaningful average. And there's an average across the breakthrough devices. So we could handle the data responsibly and draw only appropriate conclusions from it.
But boy, if you could get FDA to release data on that, that would be huge for the industry, it'd be huge for frankly, patients and doctors and others who care about the FDA process, because it would really give us insight into how long the development process works. But that's, that's my big area of concern is not knowing how long that development process takes. And whether it's actually substantially longer for breakthrough products than it is for non-breakthrough products.
In the de novo, it's about a 64-day advantage in terms of the review cycle. That review cycle has to be viewed in the context of the development cycle. And we don't have data on that. So it kind of depends on just how difficult or problematic I think the questions are going to be that need to be answered.
If I think we can construct a reasonable case for how to put it together, I think I'd still try and avoid the breakthrough without more compelling evidence that the breakthrough program actually offers an advantage. But, you know, I would ultimately lay it out there and say, it's very uncertain.
And even within the program, and I tried to make this point a couple of times in the blog post, the experience of companies is really uneven. Some go through really quickly, some go through really slowly. The blog post really only looks at averages; it doesn't really account for all that variability. But for some companies, the breakthrough program could be absolutely horrible. So I just need to have that conversation with clients. And ultimately, it's their choice as to whether they want to do it.
I will say that some clients are persuaded that there's a marketing advantage to being in the FDA breakthrough program. And that may well justify it.
When I say marketing, really, what I'm talking about is is their ability to raise capital from private equity and others. They think that if they're in the breakthrough program, that's kind of a badge of honor, and that aventure capital folks and others will be more readily interested in investing in a company that makes a breakthrough product. So they really aren't motivated by the FDA benefit, if any, they're motivated by that marketing benefit. And that may or may not be true. I don't have data to suggest whether it is or not.
In terms of the review process, in the context of user fees, FDA has made commitments as to average target goals for reviews, they could do something similar for the breakthrough program and say, you know, our average goal instead of 90 days for a 510(k)is 60 days.
Honestly, I don't think they want to do that. I don't think they want to be that fast. I don't think they want to make a commitment. But if they were serious about the program, making some sort of commitment along those lines would be beneficial.
But the thing that I would find even more valuable again, I'm just going to sound like a nerd – and I am – making more data available to the public to evaluate the merits of the program would be hugely advantageous. And you might say, “Well, what, what data would that be?” Well, one, it would be the data I mentioned a moment ago data on first encounters, whether it's an application for a breakthrough, or whether it's a pre-submission meeting on a non-breakthrough.
But the other thing they could do very easily is at the end of the process, give the medical device company a survey for them to fill out completely anonymous, where they couldn't identify who answered the survey, because companies would always be worried; they don't want to anger anyone at FDA with a criticism. But the survey could be things like, do you think that overall, the program moderated the amount of evidence, do you think overall, that the development cycle was shorter, given the FDA interaction? Do you think the data requests were reasonable, or whatever it might be, and then at the end, you know, on an anonymized basis, annually, you know, release the experience of the companies that go through it.
To me, that's hugely helpful, because there you get right from the horse's mouth on an aggregate basis. I would think it'd be helpful to FDA in deciding whether they're doing the program the way they want to do the program. But there's no data collection effort by FDA, and so that's a disappointment. But that would be a very easy way to really assess the program from everyone's standpoint.
And then finally, if the program actually offers an advantage … I think they need to tighten up the criteria for what constitutes a breakthrough device. Right now, the criteria are really subjective. And the fear is that it's anyone who's a good friend of FDA would get admitted, while those who FDA doesn't like so much might not get admitted. And there's really no accountability around that.
The first criteria says it has to be a serious or life-threatening disease, that's relatively objective. But the other part – that it's breakthrough – is a very loosely articulated standard. If it really did amount to a substantial savings, that'd be a big competitive dynamic for a company, and they shouldn't just be handing that out willy-nilly, it really ought to be tightened, if this program is advantageous.
I think it's serving FDA’s purpose fine the way it is, and the only reason they would change it is if they had concerned that companies might not be interested in pursuing it, because it's not advantageous. And I think FDA can sort of sit back a little bit with self-confidence and say, “Well, we're the only game in town. So they either do the breakthrough, they do the regular and we frankly don't care which they do.”
And I'm also kind of excited about the possibility that you're willing to think about requesting that additional data because FDA if they shared data, and when those first encounters occurred, would just give us all sorts of insights into how the development process looks. And that's really, at the end of the day, far more important than the review time itself. We companies spent far more time in development than they do waiting for FDA review. And so getting, getting a handle on that and seeing the trends and how it varies from clinical area to clinical area that will just be immensely helpful.