# Does this Covid-19 vaccine study suggests a 1 to 718 rate of adverse effects?

As I have not much skill in reading medical research, I wanted to verify my calculation of this research result.

A big peer-reviewed research paper entitled Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting measured occurrence of adverse effects among vaccinated people, up to 21 days after each shot, vs the unvaccinated control group, and generated these results:

The sum of excess adverse effects in the vaccinated group is 139.3 per 100,000 persons.

Does this mean an adverse effects rate of ~1 to 718? Can the accuracy of this calculation be improved?

• Also, is this the best place to ask this question? Wondered whether it better belongs to Cross Validated..
– Roy
Oct 25, 2021 at 6:23
• There may be multiple effects in the same person. Also, I'm not sure how helpful is to sum different types... or how likely it is that e.g. "herpes zoster infection" was actually caused by the vaccine. Oct 26, 2021 at 10:56
• Further, that table also lists events which are rarer in the vaccinated group than in the (ad-hoc) "controls". And things for which there is no statistically significant difference (95% CI for risk diff. contains the zero). And doesn't list common side effect like pain at the injection site. Oct 26, 2021 at 11:18
• Note that there are 25 classes of adverse events listed. With a 95% confidence interval, we'd expect between one and two of the correlations to be spurious -- that is, due to random chance causing something to fall outside the confidence interval, not due to an actual effect.
– Mark
Oct 27, 2021 at 21:05
• Downvoters: thank you for your feedback. I made an effort to write a serious, neutral question (which is my first in this site). Please let me know what should be improved.
– Roy
Oct 28, 2021 at 17:11

I am neither a medical professional nor a statistician, so caveat emptor and all that

This is because, in the article the authors state:

A total of 1,736,832 persons were eligible for inclusion in the vaccination cohort...

...The vaccination cohorts included a mean of 884,828 vaccinated persons, with a median age of 38 years

If you compare these to the numbers of persons in the adverse events cohort column recorded for each category, then you can see that a sum total of persons (22,080,670, if my maths is correct) would be well in excess of the enrolled cohort. This means that each reported event is not independent and it is likely that a person who had one adverse event also had at least one other reported event.

Thus, a simple sum of the excess events is not something that can be performed. Indeed, if this were the case, the authors would likely have done so in the paper, but they did not, leading to the conclusion that it is not a valid technique.

Now we must consider that adverse event risk ratios are calculated for vaccines and other medicines, so there must be some methodology to do so, but what that methodology is, I don't know though I suspect an odds-ratio calculation of some sort.

Note that the authors only considered some of these events to be a significant difference, even if the risk ratio or risk difference were higher than expected, so they excluded parathesia (tingling/prickling sensation) as well as vertigo (dizziness), but it looks like the excluded ones are all minor symptoms not associated with a general morbidity/mortality:

The risk was substantially higher on either the multiplicative (risk ratio) or additive (risk difference) scales in the vaccinated group than in the unvaccinated group for myocarditis (risk ratio, 3.24; 95% confidence interval [CI], 1.55 to 12.44; risk difference, 2.7 events per 100,000 persons; 95% CI, 1.0 to 4.6), lymphadenopathy (risk ratio, 2.43; 95% CI, 2.05 to 2.78; risk difference, 78.4 events per 100,000 persons; 95% CI, 64.1 to 89.3), appendicitis (risk ratio, 1.40; 95% CI, 1.02 to 2.01; risk difference, 5.0 events per 100,000 persons; 95% CI, 0.3 to 9.9), and herpes zoster infection (risk ratio, 1.43; 95% CI, 1.20 to 1.73; risk difference, 15.8 events per 100,000 persons; 95% CI, 8.2 to 24.2).

They also note the risks associated with SARS-CoV-2 infection in general and in figures 3 and 4 of the paper comparing to the risk with vaccination:

Infection substantially increased the risk of many different adverse events, including myocarditis (risk ratio, 18.28; 95% CI, 3.95 to 25.12; risk difference, 11.0 events per 100,000 persons; 95% CI, 5.6 to 15.8), acute kidney injury (risk ratio, 14.83; 95% CI, 9.24 to 28.75; risk difference, 125.4 events per 100,000 persons; 95% CI, 107.0 to 142.6), pulmonary embolism (risk ratio, 12.14; 95% CI, 6.89 to 29.20; risk difference, 61.7 events per 100,000 persons; 95% CI, 48.5 to 75.4), intracranial hemorrhage (risk ratio, 6.89; 95% CI, 1.90 to 19.16; risk difference, 7.6 events per 100,000 persons; 95% CI, 2.7 to 12.6), pericarditis (risk ratio, 5.39; 95% CI, 2.22 to 23.58; risk difference, 10.9 events per 100,000 persons; 95% CI, 4.9 to 16.9), myocardial infarction (risk ratio, 4.47; 95% CI, 2.47 to 9.95; risk difference, 25.1 events per 100,000 persons; 95% CI, 16.2 to 33.9), deep-vein thrombosis (risk ratio, 3.78; 95% CI, 2.50 to 6.59; risk difference, 43.0 events per 100,000 persons; 95% CI, 29.9 to 56.6), and arrhythmia (risk ratio, 3.83; 95% CI, 3.07 to 4.95; risk difference, 166.1 events per 100,000 persons; 95% CI, 139.6 to 193.2).

Figure 3 Figure 4

• Thanks, makes sense! If so, what would be a better way to calculate the risk for an adverse effect after vaccination?
– Roy
Oct 28, 2021 at 17:03
• BTW the comparison to Covid related adverse effects is problematic given that the chances to get infected without a vaccine are not 100%. Seems like the comparison should be Vaccine RR : Covid RR * Covid risk of infection without a vaccine. But that's unrelated to the question.
– Roy
Oct 28, 2021 at 17:08
• @roy - I don't know how to calculate the overall risk, your best bet would be to ask a statistician who specializes in clinical trials data. As for your second comment - that argument doesn't make sense at all. The data presented are for people (unvaccinated) infected with SARS-CoV-2, not the general population of infected and uninfected.
– bob1
Oct 28, 2021 at 20:13
• Thanks, I might do so.
– Roy
Oct 30, 2021 at 5:08
• Regarding covid vs vaccine AEs comparison - I meant the comparison should be - vaccinated (any covid status) vs unvaccinated (any covid status) - because this represents the actual risks you're considering before deciding on vaccination. The comparison in the research works when 100% of unvaccinated gets covid (and 0% of the vaccinated)
– Roy
Oct 30, 2021 at 5:11

I think there's some missing the forest for the trees here, in the sense that adverse events (aka adverse effects) generally do not have a strict definition beyond something like:

untoward medical occurrences after exposure. These endpoints, which are not necessarily causally related [to the treatment], are called adverse events (or sometimes adverse effects) (AEs)

AEs are often enough protocol-defined. In this Israeli study:

Adverse events were defined according to diagnostic codes and short free-text phrases that accompany diagnoses in the CHS database. A complete list of the study outcomes (adverse events) and their definitions is provided in Table S2.

with the stated intent to capture "a broad set of potential short- and medium-term adverse events among vaccinated persons".

You'll note that the most common AE in table 2 (that you've included in your question) is lymphadenopathy (inflammation of the lymph nodes) which is a common but temporary (and most often benign) side effect of vaccines, but almost certainly not the most common side effect spontaneusly reported... as shown e.g. in this Canadian database, where lymphadenopathy is somewhere in the middle of the pack of the effects reported (and even below myocarditis in the Canadian database, even though that's much, much rarer in the Israeli study):

So the "broad set" was perhaps not so broad in the Israeli study, as if clearly excluded the commonly reported vaccine side effects like headache etc. While you can definitely sum up AEs according to one study's particular definition(s) and even express that total as a ratio to the population under study... it's an not incredibly useful sum/conclusion beyond that study. Whereas specific AEs are in a sense "portable" between studies, assuming they used some reasonably common definition for the individual AEs. (Myocarditis is somewhat complicated as there are [Brighton Collaboration] levels for it. Also the Canadians reported it up to 92 days after vaccination, which is a longer time window than in the Israeli study.) This is almost certainly why the Israeli paper's abstract doesn't present some sum, but the individual AEs that they found statistically significant in the vaccine's direction:

Vaccination was most strongly associated with an elevated risk of myocarditis (risk ratio, 3.24; 95% confidence interval [CI], 1.55 to 12.44; risk difference, 2.7 events per 100,000 persons; 95% CI, 1.0 to 4.6), lymphadenopathy (risk ratio, 2.43; 95% CI, 2.05 to 2.78; risk difference, 78.4 events per 100,000 persons; 95% CI, 64.1 to 89.3), appendicitis (risk ratio, 1.40; 95% CI, 1.02 to 2.01; risk difference, 5.0 events per 100,000 persons; 95% CI, 0.3 to 9.9), and herpes zoster infection (risk ratio, 1.43; 95% CI, 1.20 to 1.73; risk difference, 15.8 events per 100,000 persons; 95% CI, 8.2 to 24.2).

One has to wonder how herpes or appendicitis might be causally related to the vaccine, by the way. It could be simply behavioral like more kissing and going out to restaurants more often, although for appendicitis some connection via a higher likelihood of general inflammatory reaction can't be excluded. (For herpes zoster at least, reactivation from lymph nodes has been posited.) The Israeli paper notes that the clinical trials have noted a trend for appendicitis too, but not statistically significant, perhaps due to undepowered sample to detect a small effect. So, it's somewhat clear that they tried do "double check" some such trends they noticed/suspected, useing their larger database, while excluding AEs they probably considered trivial or "boring".

– bob1
Oct 29, 2021 at 9:11
• Thanks, I see your point. My porpose is to find a research based risk rate for serious health issues due to vaccination. There are a lot of testimonies, and a lot of research, but I'm struggling to find a 1:X answer, based on actual medical data
– Roy
Oct 30, 2021 at 5:07
• @Roy that's a semantics question - what do you define as serious? Would the person across the street also think those are serious? The data for complications is presented in the paper you linked and the one from Fizz. For instance lymphadenopathy is normally considered non-serious, so you could exclude that, but a myocardial infarction would meet most peoples' criteria of serious. If you exclude lymphadenopathy from the data the "risk" drops significantly, if you also exclude parathesia, neutropenia and lyphopenia (other causes common) then you have almost nothing left.
– bob1
Nov 1, 2021 at 20:10
• @bob1 Thanks, for this matter I'll just use the researchers' own definitions. Some effects are mild, some more serious suspected effects are not even looked at.. So as a start I just want to understand the rates from the available data.
– Roy
Nov 2, 2021 at 5:07
• @bob1 Correcting the last comment: Thanks. I find the understanding of the rates that are available from this data is needed. Of course differentiating between different effects by severity, as well as checking about serious effects that were not measured here, is also helpful.
– Roy
Nov 2, 2021 at 5:16