Does this Covid-19 vaccine study suggests a 1 to 718 rate of adverse effects?

Question

As I have not much skill in reading medical research, I wanted to verify my calculation of this research result.

A big peer-reviewed research paper entitled Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting measured occurrence of adverse effects among vaccinated people, up to 21 days after each shot, vs the unvaccinated control group, and generated these results:

The sum of excess adverse effects in the vaccinated group is 139.3 per 100,000 persons.

Does this mean an adverse effects rate of ~1 to 718? Can the accuracy of this calculation be improved?

Answer 1

I am neither a medical professional nor a statistician, so caveat emptor and all that

Short answer is no.

This is because, in the article the authors state:

A total of 1,736,832 persons were eligible for inclusion in the vaccination cohort...

...The vaccination cohorts included a mean of 884,828 vaccinated persons, with a median age of 38 years

If you compare these to the numbers of persons in the adverse events cohort column recorded for each category, then you can see that a sum total of persons (22,080,670, if my maths is correct) would be well in excess of the enrolled cohort. This means that each reported event is not independent and it is likely that a person who had one adverse event also had at least one other reported event.

Thus, a simple sum of the excess events is not something that can be performed. Indeed, if this were the case, the authors would likely have done so in the paper, but they did not, leading to the conclusion that it is not a valid technique.

Now we must consider that adverse event risk ratios are calculated for vaccines and other medicines, so there must be some methodology to do so, but what that methodology is, I don't know though I suspect an odds-ratio calculation of some sort.

Note that the authors only considered some of these events to be a significant difference, even if the risk ratio or risk difference were higher than expected, so they excluded parathesia (tingling/prickling sensation) as well as vertigo (dizziness), but it looks like the excluded ones are all minor symptoms not associated with a general morbidity/mortality:

The risk was substantially higher on either the multiplicative (risk ratio) or additive (risk difference) scales in the vaccinated group than in the unvaccinated group for myocarditis (risk ratio, 3.24; 95% confidence interval [CI], 1.55 to 12.44; risk difference, 2.7 events per 100,000 persons; 95% CI, 1.0 to 4.6), lymphadenopathy (risk ratio, 2.43; 95% CI, 2.05 to 2.78; risk difference, 78.4 events per 100,000 persons; 95% CI, 64.1 to 89.3), appendicitis (risk ratio, 1.40; 95% CI, 1.02 to 2.01; risk difference, 5.0 events per 100,000 persons; 95% CI, 0.3 to 9.9), and herpes zoster infection (risk ratio, 1.43; 95% CI, 1.20 to 1.73; risk difference, 15.8 events per 100,000 persons; 95% CI, 8.2 to 24.2).

They also note the risks associated with SARS-CoV-2 infection in general and in figures 3 and 4 of the paper comparing to the risk with vaccination:

Infection substantially increased the risk of many different adverse events, including myocarditis (risk ratio, 18.28; 95% CI, 3.95 to 25.12; risk difference, 11.0 events per 100,000 persons; 95% CI, 5.6 to 15.8), acute kidney injury (risk ratio, 14.83; 95% CI, 9.24 to 28.75; risk difference, 125.4 events per 100,000 persons; 95% CI, 107.0 to 142.6), pulmonary embolism (risk ratio, 12.14; 95% CI, 6.89 to 29.20; risk difference, 61.7 events per 100,000 persons; 95% CI, 48.5 to 75.4), intracranial hemorrhage (risk ratio, 6.89; 95% CI, 1.90 to 19.16; risk difference, 7.6 events per 100,000 persons; 95% CI, 2.7 to 12.6), pericarditis (risk ratio, 5.39; 95% CI, 2.22 to 23.58; risk difference, 10.9 events per 100,000 persons; 95% CI, 4.9 to 16.9), myocardial infarction (risk ratio, 4.47; 95% CI, 2.47 to 9.95; risk difference, 25.1 events per 100,000 persons; 95% CI, 16.2 to 33.9), deep-vein thrombosis (risk ratio, 3.78; 95% CI, 2.50 to 6.59; risk difference, 43.0 events per 100,000 persons; 95% CI, 29.9 to 56.6), and arrhythmia (risk ratio, 3.83; 95% CI, 3.07 to 4.95; risk difference, 166.1 events per 100,000 persons; 95% CI, 139.6 to 193.2).

Figure 3 Figure 4

Answer 2

I think there's some missing the forest for the trees here, in the sense that adverse events (aka adverse effects) generally do not have a strict definition beyond something like:

untoward medical occurrences after exposure. These endpoints, which are not necessarily causally related [to the treatment], are called adverse events (or sometimes adverse effects) (AEs)

AEs are often enough protocol-defined. In this Israeli study:

Adverse events were defined according to diagnostic codes and short free-text phrases that accompany diagnoses in the CHS database. A complete list of the study outcomes (adverse events) and their definitions is provided in Table S2.

with the stated intent to capture "a broad set of potential short- and medium-term adverse events among vaccinated persons".

You'll note that the most common AE in table 2 (that you've included in your question) is lymphadenopathy (inflammation of the lymph nodes) which is a common but temporary (and most often benign) side effect of vaccines, but almost certainly not the most common side effect spontaneusly reported... as shown e.g. in this Canadian database, where lymphadenopathy is somewhere in the middle of the pack of the effects reported (and even below myocarditis in the Canadian database, even though that's much, much rarer in the Israeli study):

So the "broad set" was perhaps not so broad in the Israeli study, as if clearly excluded the commonly reported vaccine side effects like headache etc. While you can definitely sum up AEs according to one study's particular definition(s) and even express that total as a ratio to the population under study... it's an not incredibly useful sum/conclusion beyond that study. Whereas specific AEs are in a sense "portable" between studies, assuming they used some reasonably common definition for the individual AEs. (Myocarditis is somewhat complicated as there are [Brighton Collaboration] levels for it. Also the Canadians reported it up to 92 days after vaccination, which is a longer time window than in the Israeli study.) This is almost certainly why the Israeli paper's abstract doesn't present some sum, but the individual AEs that they found statistically significant in the vaccine's direction:

Vaccination was most strongly associated with an elevated risk of myocarditis (risk ratio, 3.24; 95% confidence interval [CI], 1.55 to 12.44; risk difference, 2.7 events per 100,000 persons; 95% CI, 1.0 to 4.6), lymphadenopathy (risk ratio, 2.43; 95% CI, 2.05 to 2.78; risk difference, 78.4 events per 100,000 persons; 95% CI, 64.1 to 89.3), appendicitis (risk ratio, 1.40; 95% CI, 1.02 to 2.01; risk difference, 5.0 events per 100,000 persons; 95% CI, 0.3 to 9.9), and herpes zoster infection (risk ratio, 1.43; 95% CI, 1.20 to 1.73; risk difference, 15.8 events per 100,000 persons; 95% CI, 8.2 to 24.2).

One has to wonder how herpes or appendicitis might be causally related to the vaccine, by the way. It could be simply behavioral like more kissing and going out to restaurants more often, although for appendicitis some connection via a higher likelihood of general inflammatory reaction can't be excluded. (For herpes zoster at least, reactivation from lymph nodes has been posited.) The Israeli paper notes that the clinical trials have noted a trend for appendicitis too, but not statistically significant, perhaps due to undepowered sample to detect a small effect. So, it's somewhat clear that they tried do "double check" some such trends they noticed/suspected, useing their larger database, while excluding AEs they probably considered trivial or "boring".