The FDA's Approach to Covid-19 Is a Bloody Mess

President Trump announced a “new” treatment for Covid-19 on Sunday, just in time for the start of the Republican National Convention. From now on, per a new Emergency Use Authorization from the US Food and Drug Administration, hospitals can more easily order transfusions of Covid-19 convalescent plasma—a blood product using donations from recovered patients, and containing disease-specific antibodies. “It has been proven to reduce mortality by 35 percent,” Trump claimed. “It’s a tremendous number.”

FDA commissioner Stephen Hahn added his support. “I don’t want you to gloss over this number,” he told the press. “What that means is—and if the data continue to pan out—[out of] 100 people who are sick with Covid-19, 35 would have been saved because of the administration of that plasma.”

Trump and Hahn were right about one thing: This would be tremendous, if only it were true. It’s not.

Subscribe to WIRED and stay smart with more of your favorite Ideas writers.

In fact, as WIRED’s Adam Rogers wrote on Sunday, nothing has been proven on this topic: There haven’t yet been results from any decent-size, randomized controlled trials of convalescent plasma, so no one can say for sure whether it offers any benefits at all. But it’s somehow even worse than that: Trump’s number—the purported reduction in mortality—is even wrong on its own terms.

But let’s not jump ahead. To understand the disturbing implications of this moment for the FDA, it’s helpful to retrace some recent steps. It was Hahn, the FDA commissioner, who kicked this issue into overdrive on July 30. That’s when he called on people who have recovered from Covid-19 to donate their plasma: “You can literally save lives,” Hahn claimed, even though he couldn’t back that up with evidence, as the government launched an $8 million publicity campaign. The accompanying press release made it clear to all that a clock was ticking: “The Trump administration is imploring Americans … to dramatically increase donations of convalescent plasma by the end of August,” it said.

We learned more about what had been going on behind the scenes on August 1, at a seminar at the Mayo Clinic. Peter Marks, the head of the FDA section responsible for drug authorizations and approvals, filled in the backstory. From early on in the pandemic, the FDA had been overwhelmed by doctors making applications to use convalescent plasma on individual patients. Instead of pulling out the stops to enable a large clinical trial to get reliable data quickly, the agency asked the Mayo Clinic to set up and coordinate an Expanded Access Program and gather safety information. As the plasma became widely available through this program, and outbreaks shifted around the country, existing clinical trials ran into trouble: Few were willing to take a chance on being randomized into the control arm of a study, when the treatment was quite accessible.

By the start of August, though—and despite the dearth of trial data—Peter Marks was clearly convinced both that convalescent plasma was safe to use and that what he called “the totality of evidence” suggested it had real benefits for patients. What made him so confident? He cited four lines of research: First, studies of the use of convalescent plasma in prior outbreaks of disease, including influenza; second, related studies using animals with Covid-19; third, the small clutch of published studies involving human Covid-19 patients; and fourth, an analysis of data from the Mayo Clinic, based on 35,000 of the patients who had participated in the Expanded Access Program. These all pointed in the same direction, he said, and added up to make a compelling case for convalescent plasma.

The Mayo study was pivotal and ended up being the source for the president’s whopper-claim on Sunday that the treatment is “proven to reduce mortality by 35 percent,” and the follow-up from the White House director of communications, claiming that this benefit might, in fact, be as high as 50 percent. (“These are REAL American lives that will be saved,” she tweeted.) But of course, the Mayo study couldn’t prove that.

First things first: The Mayo study was never designed to test effectiveness. It lacked a control group, as many have pointed out. An observational study such as this one—which simply tallies up what happens without being able to minimize biases—will often exaggerate the benefits of treatment. It may even end up giving the impression that a useless or harmful treatment is actually helping. There are warning signs with this particular study, too. The patients who received transfusions of convalescent plasma in the Mayo study might not have been so representative of people hospitalized with Covid-19. For example, only 19 percent were Black. According to the Centers for Disease Control and Prevention, more than 30 percent of hospitalized Covid-19 patients in the US are Black. The CDC also reports that Black people are more than twice as likely to die of the disease than white people.

In any case, once the Mayo researchers had acquired all this observational data, they decided that maybe they could explore whether the treatment worked, after all. The donated plasma differed from batch to batch: Some of the plasma came from recovered Covid patients who had very high levels of antibodies, and some came from recovered patients with much less, or possibly even none. (Back when the study started, the researchers didn’t have a way to test for antibodies in the plasma.) So they decided to compare outcomes for people who had received transfusions of “high-,” “medium-,” and “low-” antibody plasma. Their thinking was, if convalescent plasma really works, then batches with more antibodies would be likely to provide a stronger dose and yield a greater benefit.

There were some problems with this approach, beyond the absence of a control group that could show whether some people might be better off without any plasma. Most of the plasma was gone and couldn’t be tested for antibodies. A small proportion of samples had been collected at different blood banks for quality assurance purposes and so could be used for the analysis. But it was not a random sample.

It was this analysis that showed the difference in mortality rates touted at the press conference. Among hospitalized Covid-19 patients who had received a “low antibody” dose of plasma, 13.7 percent died within a week; among those who’d gotten a “high antibody” dose, that rate was 8.9 percent. In other words, the patients who got the “best” convalescent plasma had a risk of dying that was 4.8 percentage points lower.

Now, 4.8 is also about one-third of 13.7, which is another way of describing the difference. When Trump announced a 35-percent reduction in mortality on Sunday, that’s the figure he was referring to—known as a relative risk reduction. But that’s not what it sounded like, and it’s not what Hahn would say a moment later, when he told the press that convalescent plasma could save the lives of 35 of every 100 people who get sick from Covid-19. This was obviously absurd: nothing could possibly drop the mortality rate by 35 points, because the mortality rate among all people who get sick from Covid-19 is nowhere near that high: Only around 20 percent even become severely ill. (On Monday night, Hahn admitted that he’d mixed this up.)

But even described correctly, you can’t claim that everyone who gets a transfusion of convalescent plasma could expect this result. That data applies only to the specific, somewhat non-representative sample of patients from the Mayo Clinic study and to transfusions of convalescent plasma that have a particular level of antibodies. What’s more, the number of people included in the study’s “lowest” and “highest” antibody groups was low—no more than 1,100 out of the total of 35,000. In the FDA’s supporting documentation, scientists reported that their reanalysis of the samples using a different antibody test got very different results. Change the threshold for what you call “high-,” and the numbers shift again. The point is, there are multiple levels of unreliability here.

Read all of our coronavirus coverage here.

Most of all, though, the finding isn’t solid enough to rely on, because this wasn’t a planned, randomized trial. The analysis raises an interesting hypothesis, but it can’t prove it. The Mayo study raises an equally likely, but worrying, possibility: that the vast majority of donated plasma—which doesn’t have the highest levels of antibodies and usually hasn’t been administered early in the course of illness—did not help at all, and maybe even did more harm than good to some people. Only randomized trials can disentangle this.

You don’t need to prove a treatment works, though, for the FDA to grant an Emergency Use Authorization—only that it may be effective. Still, I was shocked when I read the FDA’s documentation. It wasn’t just the weaknesses of the Mayo Clinic data. The other lines of evidence that Marks had mentioned at the seminar on August 1 were also deficient.

The authorization points to the historical studies on the use of convalescent plasma to support the “may be effective” case, for example. But these are taken almost entirely from a systematic review of research conducted up through July 2013, including no randomized trials. The conclusions of that paper are contested by a more recent systematic review from July 2020, not mentioned by the FDA, which included two randomized trials of the use of convalescent plasma for treating influenza. This one found “no convincing evidence” of benefit from those trials and expressed concern that convalescent plasma may end up doing little or nothing to help people recover from Covid-19. The new authorization also doesn’t mention a non-randomized trial of convalescent plasma during the 2016 Ebola outbreak that found no survival benefit.

Everything You Need to Know About the Coronavirus

Here’s all the WIRED coverage in one place, from how to keep your children entertained to how this outbreak is affecting the economy.

Another basis for the new authorization was the handful of studies that compared similar people who did or did not receive convalescent plasma as a treatment for Covid-19. These showed fairly positive results, but they are small, and there’s a risk of publication bias this early in the pandemic. If everyone expects a treatment to work, then a hospital where people didn’t fare so well might not be in such a rush to broadcast those results. As if on cue, another study appeared a few days ago (too late to be in the FDA documentation), finding no reduction in mortality. The “totality of evidence” in patients is weak at every point.

Could convalescent plasma be an effective treatment for Covid-19? I don’t think we have a satisfying answer. It might work for some people, but even if it does, it might not help them very much. It doesn’t seem very likely to provide important benefits for most of the Americans who will end up getting the transfusions in the coming months.

This week’s announcements do teach us something else, however. The FDA’s Emergency Use Authorizations aren’t even a short arm’s length from politics, and its leader and spokespeople are spinning data to the American people. So is the US president. Similar decisions will be coming, at some point very soon, on vaccines. If we can’t trust how those are made and communicated, the consequences will be dire.

Photographs: Ivan Valencia/Bloomberg/Getty Images; Chip Somodevilla/Getty Images; Asaad Niazi/AFP/Getty Images