The now infamous but not very helpful N95 trial (Loeb 2022)

Jul 10, 2023

When a scientific paper is immediately covered in lay media, you can almost guarantee a lot of incorrect takes. When a new paper comes out on any hotly debated topic, you are bound to see strong opinions lacking almost any nuance. The only thing that is somewhat strange about the reaction to this particular paper (Loeb 2022) is that we are talking about masks. Can you imagine anyone caring about an RCT looking at medical masks back in 2018? How we all yearn for simpler times, but living in the world we do, I guess we should try to understand what this paper actually says.

Loeb M, Bartholomew A, Hashmi M, Tarhuni W, Hassany M, Youngster I, Somayaji R, Larios O, Kim J, Missaghi B, Vayalumkal JV, Mertz D, Chagla Z, Cividino M, Ali K, Mansour S, Castellucci LA, Frenette C, Parkes L, Downing M, Muller M, Glavin V, Newton J, Hookoom R, Leis JA, Kinross J, Smith S, Borhan S, Singh P, Pullenayegum E, Conly J. Medical Masks Versus N95 Respirators for Preventing COVID-19 Among Health Care Workers : A Randomized Trial. Ann Intern Med. 2022 Nov 29. doi: 10.7326/M22-1966. Epub ahead of print. PMID: 36442064 NCT04296643

Loeb 2022 is an open-label, pragmatic, multicenter noninferiority RCT. The trial ran from May 2020 to March 2022.

Healthcare workers (HCWs) who provide direct care to patients with confirmed or suspected COVID-19. They included HCWs from 29 health care facilities: 17 acute care hospitals in Canada, 4 acute care hospitals in Pakistan, 2 long-term care facilities in Israel, and 6 acute care hospitals in Egypt. They excluded HCWs who didn't have a proper fit test, had comorbidities for severe COVID, had a prior COVID infection, or who were fully vaccinated.

HCWs were instructed to wear an N95 when providing routine care for patients with known or suspected COVID-19.

HCWs were instructed to wear a surgical mask when providing routine care for patients with COVID-19. However, they were allowed to wear an N95 at any time, based on their own point of care risk assessment.

The intervention phase lasted 10 weeks. All sites apparently had a universal masking policy in place, so all HCWs were supposed to be wearing a mask of some sort at all times while in the facility.

The primary outcome was PCR confirmed COVID-19 infection.

A total of 1,009 HCWs were included in this trial, 1,004 of whom were included in the intention to treat analysis.

The primary outcome of PCR confirmed infection occurred in 10.5% of the N95 group and 9.3% of the surgical mask group (HR 1.14, 95% CI 0.77-1.69), resulting in a conclusion of non-inferiority based on their definition.

Adverse events occurred in 10.8% of the surgical mask group and 13.6% of the N95 group. (The adverse events were almost all discomfort and skin irritation).

The different groups wore their masks for different amounts of time. The surgical mask group ‘always’ wore their mask 91% of the time, based on a self-report survey. The N95 group ‘always’ wore their mask while caring for COVID patients 81% of the time.

An unplanned secondary analysis by country was not statistically different, but appears to show relatively dramatic differences between sites. (Although I will discuss this secondary analysis below, and its implications if real, there is always an extremely high risk that these secondary analyses are noise rather than signal.)

Although this trial has been circulated as ‘proving’ that N95s are ineffective, that clearly isn't the case. It also doesn't prove that N95s are effective. Honestly, due to some fatal flaws, I am not sure the study tells us much at all. (We can see that the effect size of occasional N95 usage is going to be small, but I think that was pretty obvious before this publication.)

This big problem with this study is that they didn't really compare N95 use to surgical mask use. They compared occasional N95 use to occasional N95 use, just with somewhat different instructions. When the methods are reframed in that way, it is pretty clear that the outcome of this trial was determined before it even started.

Only 80% of the N95 group reported wearing their N95 as they were supposed to, and that is likely an overestimate as it is entirely by self-report. They don't report what is probably the most important analysis: the outcomes among those who report wearing their mask as they were supposed to (aka, the per protocol analysis). Even that would not give us a great sense of the benefits of N95s, as the surgical mask group was allowed to wear an N95 whenever they wanted, and nowhere in the manuscript do they tell us how often that occurred. We are left with a comparison of part time N95 use versus part time N95 use with no description of how part time it was in each group.

In fact, the cross-over issue is even worse than it seems. The N95 group was not asked to wear an N95 at all times. They weren't even asked to wear an N95 for all patient encounters. They were just asked to wear an N95 for all patients with suspected or confirmed COVID. However, when we were routinely testing for COVID, we would routinely find patients who were COVID positive without having any classic symptoms for COVID. We also know that COVID spreads (perhaps maximally) from asymptomatic individuals, so the patients without suspected COVID may simply not have COVID symptoms yet. Only wearing an N95 for patients with respiratory symptoms is like only opening an umbrella after your shirt is soaked through. Again, they provide us with absolutely no information on the amount of time N95s were actually worn, but you could easily wear a surgical mask for 90% of your shift and still count as being perfectly compliant with the N95 strategy in this trial.

It gets even worse than that. Because infectious disease doctors (including the authors of this paper) spent so much time early in the pandemic insisting that COVID does not spread through the airborne route, many healthcare workers were fed a false narrative that they were perfectly safe as long as they were 6 feet from their patient. In my own department, we still don't even close the doors to COVID positive patients’ rooms, and nurses will remove their N95 as soon as they step outside. The nursing station is 8 feet away from COVID patients who can't stop coughing, but infection control has insisted that everyone is perfectly safe without any personal protective equipment. This trial is even worse than that, as according to the original protocol, nurses were only instructed to wear an N95 within 3 feet of a patient with both fever and cough. Although risk of transmission is highest when in close proximity to the patient, if you are ignoring all other aspects of airborne spread you are undermining the value of airborne protection. In other words, this was a trial of 2 different masks both utilized within a droplet spread paradigm, rather than a trial of airborne precautions versus droplet precautions, which is what we really needed.

Overall, most healthcare workers in this trial seemed to have had very limited COVID exposure. Looking at the table below, most HCWs actually had 0 confirmed exposures per week, with only a very few having more than 5 confirmed weekly exposures. Not only does that mean that the theoretical benefit of masking in this population is low, it also means that these healthcare workers might have only worn their N95 for a tiny fraction of the time they were at work (as they were only supposed to wear it during these exposures.) In other words, both groups might have worn N95s for almost identical amounts of time.

The short timeframe of this trial probably under-estimates any benefit in absolute terms. They followed these individuals for about 9 weeks on average, and although not statistically significant, we see 1% fewer infections in the N95 group. If the risk of infection remains stable throughout the year, that means that over 52 weeks you might see a 5-6% decrease in infections. 1% might not convince you to wear an N95, but a 5% decrease risk of COVID (and all the annoyances that go with it) might seem like a reasonable deal. (Compare that to the absolute value of the flu shot, for example.)

Although we know healthcare workers contract COVID while doing their jobs, we also know that most contract it in the community. As dedicated as we are, we still spend most of our time outside the hospital. Of course, what you were doing with that time really depended on the timing of the pandemic, and local rules. We aren't provided with nearly enough information to make a guess at how many of these infections occurred in the hospital, versus how many were just part of community spread. Either way, I would expect any study looking at this question to require much larger numbers than they included here, because you expect the community infection rate to be the same in both groups, and you expect that to be the majority of infections. You should expect the groups to have pretty similar outcomes. These authors didn't seem to consider that in their power calculations.

Speaking of power calculations and statistical assumptions, I have discussed the many problems with non-inferiority trials before, but they are essential when trying to understand this kind of research. In particular, in order to understand non-inferiority trials, we need to understand the selection of non-inferiority margins. This is not a scientific process. The researchers can select any number they want, and this incredible subjectivity is a significant source of bias, and has completely derailed multiple trials in the past. In this trial, they decided that anything up to twice as many infections would be considered ‘non-inferior’. In other words, the surgical mask group could have 100% more infections than the N95 group and this trial would consider that as an equivalent outcome. Or using the absolute numbers they based their trial around, if 5% of the N95 group was diagnosed with COVID-19 and 10% of the surgical mask group was, surgical masks would be considered ‘noninferior’. You can make up your own mind, but that seems like an asinine assumption to base a trial around to me. If, like me, you think this non-inferiority margin is flawed, then the entire trial was fundamentally flawed from the outset. (This is a huge problem with non-inferiority trials, which are becoming popular because they are easier to run, despite often being scientifically dubious.)

This trial suggests a 15% increase in infections with surgical masks. It was not statistically significant, because the trial was designed so that it wouldn't be, but it certainly might be clinically significant, especially at a systems level.

I usually don't comment much on statistics, but there is an important problem to consider with this trial. In superiority trials, we prefer an intention to treat analysis, but the exact opposite is true when you are running a non-inferiority trial. Intention to treat analyses are more likely to make the groups seem similar, and therefore they bias towards non-inferiority. Non-inferiority trials are supposed to use a per-protocol analysis, but this trial used intention to treat, biasing the results towards the null effect. This is not best research practice, and the decision is not explained at all in the manuscript. (This also seems like a failure of peer review to me.)

There were a number of protocol changes over the course of this trial, which is certainly to be expected during a pandemic. However, from the outside, I am not sure that they all make sense. For example, they say that "follow-up was reduced from 12 to 10 weeks to minimize loss to follow-up." In what world does a 2 week change from 12 to 10 weeks make any difference? This change certainly doesn't seem like it would be worth it, considering that these protocol changes undermine the validity of the trial.

Considering that this trial ran until March 2022, the exclusion of all individuals who were vaccinated and who had prior COVID infection really makes this an incredibly select and unrepresentative population. I think that partly explains the very weird numbers in this trial. Although they included 2 sites in Israel, there were only a total of 34 individuals in the trial from Israel. That seems like an incredibly small number. It really doesn't make sense to go through all the effort to set up a trial in another country, with all the ethics and paperwork involved, and then only enroll 34 people. It isn't like this is a trial of some rare disease, where you have to wait for patients to present. Overall, they only included an average of 34 people from each hospital, which again seems like a very small number considering the normal staff size of a hospital. The logic behind this choice, and the significant risk of selection bias, are concerning to me.

Heterogeneity is usually only discussed in the context of meta-analyses, but this trial stands out among RCTs I have reviewed. It is unusual for different study sites to vary dramatically by baseline characteristics, but in this case – aside from having dramatically different healthcare systems – the different sites also had different compositions of job types, gender, job type, partial vaccination, and baseline seropositivity. In Canada, they only cared for COVID patients for an average of 60 minutes over an entire week, whereas it was more than 400 minutes in Egypt. On the other hand, ‘aerosol generating procedures’ were much more common in Canada. Despite being a single trial, it might be mixing apples and oranges. I think this heterogeneity shines through in their secondary analysis by country, where the effect size seems to differ dramatically by country.

N95s look better than surgical masks in all countries except Egypt. Clinical heterogeneity makes this hard to assess, but you could imagine many cultural reasons for such inter-country differences. Perhaps N95s were used more often in the surgical mask group in Egypt. Perhaps healthcare workers had to reuse N95s because of shortages. Perhaps the mask fitting process is different. Perhaps beards are more common, interfering with mask fit. This clinical heterogeneity makes it impossible to make definitive statements from this one study.

There is one key difference that might explain the lack of benefit in Egypt as compared to the relatively clear benefit in Canada: rates of community spread. The Egypt part of the study occurred during omicron, and healthcare workers in Egypt had an average of 45 COVID exposures per week outside the hospital. Canadian workers only had 10 exposures, and they were to a less transmissible strain of the virus. So it is quite possible that all the healthcare workers in Egypt caught COVID through community spread, whereas community spread was low enough in Canada that most of the exposures occurred in the hospital, where masks can actually help.

Conflict of interest is always important to consider whenever appraising research. There obviously isn't much direct financial conflict here, but there are some potential significant conflicts. I don't know any of the legal details, but early in the pandemic Ontario nurses fought in court for the right to wear N95s and treat COVID as an airborne threat. They lost that case, in large part because infectious disease doctors across Ontario – including many of the authors of this paper – signed legal affidavits stating COVID is not airborne and N95s should not be routinely used. The senior author of this paper is also on record in many places downplaying evidence that COVID is airborne, as well as playing up supposed harms of N95s, including CO2 retention and oxygen deprivation. Of course, we all bring our own biases to the interpretation of any science, but based on their actions over the last 3 years, many of the researchers here seem to be strongly biased against N95s. And, of course, this is an unblinded trial, which can amplify existing bias.

At this point, there isn't much doubt that COVID is airborne. We should be using whatever precautions are most effective against airborne spread. Unfortunately, this appears to be a massive gap in medical science, and most people seem more interested in fighting about it than actually generating science. Before the next pandemic, we really should have better information, not just about N95s, but also the many other options, such as elastomeric respirators and PAPRs.

I think the existing evidence clearly suggests (although not prove) that N95s provide more protection than surgical masks against respiratory infections, but the absolute difference between the two will always be small. I don't think this study changes much. It is underpowered, but I think it probably does show a small benefit from N95s. despite many sources of bias that should limit the effect size.

However, we shouldn't oversell that benefit. N95s are not parachutes. (Even parachutes might not be parachutes). Benefits will vary dramatically based on community rates of respiratory illness. There is no one size fits all answer. It is not at all clear when benefits outweigh risks. It is not clear when benefits might justify costs. For an individual, the benefit may always be so small as to be negligible. However, small benefits multiplied among many thousands of healthcare workers could be very important, especially in systems already faced with healthcare worker shortages.

Personally, as of December 2022, I am still wearing an N95 throughout my entire shift. I will not do that forever, but I might adopt it as my yearly approach to influenza season (and future respiratory pandemics). I would certainly prefer to go back to a surgical mask, as the tighter N95s sometimes give me headaches and the dermatologic impacts of N95s are not ideal, but respiratory illnesses are also miserable and I am happy to reduce my risk by a few percent each year.

This study is a mess, and I don't think you should draw many conclusions from it. The best scientific guess, based on this addition of this trial to the existing literatures, is that there is a small benefit of N95s over surgical masks at preventing the transmission of respiratory infections, but the magnitude of that benefit will depend heavily on baseline risk at any given time.

The evidence suggests N95s are better than surgical masks

COVID-19 is spread by aerosols (airborne): an evidence review

Loeb M, Bartholomew A, Hashmi M, Tarhuni W, Hassany M, Youngster I, Somayaji R, Larios O, Kim J, Missaghi B, Vayalumkal JV, Mertz D, Chagla Z, Cividino M, Ali K, Mansour S, Castellucci LA, Frenette C, Parkes L, Downing M, Muller M, Glavin V, Newton J, Hookoom R, Leis JA, Kinross J, Smith S, Borhan S, Singh P, Pullenayegum E, Conly J. Medical Masks Versus N95 Respirators for Preventing COVID-19 Among Health Care Workers : A Randomized Trial. Ann Intern Med. 2022 Nov 29. doi: 10.7326/M22-1966. Epub ahead of print. PMID: 36442064