Cessation of studies
We analyzed observational data from Clalit Health Services (CHS) to mimic a target study of the effects of the BNT162b2 vaccine on a wide range of potential adverse events in a population without SARS-CoV-2 infection. CHS is the largest of four integrated payor-provider-health organizations that offer compulsory health insurance in Israel. CHS insures approximately 52% of the Israeli population (> 4.7 million out of 9.0 million people), and the CHS insured population is roughly representative of the total Israeli population.17th Outpatient care is provided directly by the CHS, while inpatient care is divided between the CHS and off-grid hospitals. CHS information systems are fully digitized and feed into a central data warehouse. Data on Covid-19, including the results of all SARS-CoV-2 polymerase chain reaction tests (PCR), Covid-19 diagnoses and severity levels, and vaccinations, is collected centrally by the Israeli Ministry of Health and shared with each of the four national health organizations Every day.
This study was approved by the CHS Institutional Review Board. The study was exempt from the obligation to consent.
Eligibility criteria were an age of 16 years or older, uninterrupted membership in the health organization for a full year, no previous SARS-CoV-2 infection and no contact with the health system in the last 7 days (the latter criterion was an indicator of a health event unrelated to any subsequent vaccination and may reduce the likelihood of vaccination). Because of difficulties in distinguishing the recoding of previous events from real new events, individuals with an earlier diagnosis of that event were excluded for each adverse event.
As in our previous study on the effectiveness of the BNT162b2 vaccine10 we also excluded people from populations in which confounding could not be adequately treated – residents of long-term care facilities, people staying at home for medical reasons, health workers, and people for whom body mass index or neighborhood data were lacking (Missing Data for these variables are rare in the CHS data). A full definition of the study variables is given in Table S1 in the supplementary appendix, which is available with the full text of this article on NEJM.org.
Study design and monitoring
The target study for this study would assign eligible individuals either vaccination or no vaccination. In order to emulate this study, suitable persons who were vaccinated on that day were assigned to suitable controls who had not yet been vaccinated on each day from the start of the vaccination campaign in Israel (December 20, 2020) to the end of the study period (May 24, 2021) . Since the matching process only took into account information that was available on or before that day (and was therefore unaffected by subsequent vaccinations or SARS-CoV-2 infections), unvaccinated people who were matched on a specific day could be vaccinated at a future date, on which they could be newly vaccinated for inclusion in the study at a later date.
In an attempt to mimic randomized allocation, vaccinated subjects and unvaccinated controls were closely matched against a set of baseline variables that were subject to skill levels as potential confounders – variables that may be related to vaccination and tendency to development a wide range of adverse clinical conditions. These matching criteria included the socio-demographic variables age (divided into 2-year age groups), gender (male or female), place of residence (at city or community level), socio-economic status (divided into seven categories) and population sector (generally Jewish, Arab or ultra-Orthodox Jews). In addition, the compliance criteria included clinical factors to take into account overall clinical condition and burden of disease, including the number of pre-existing chronic conditions (which the Centers for Disease Control and Prevention considers as risk factors for severe Covid-19). [CDC] from December 20th, 2020,18th divided into four categories), the number of diagnoses documented during outpatient visits in the year preceding the index date (categorized in deciles within each age group) and the pregnancy status.
All authors drafted the study and critically reviewed the manuscript. The first three authors collected and analyzed the data. A subgroup of authors wrote the manuscript. The last author vouches for the correctness and completeness of the data as well as for the adherence to the protocol of the study. There was no commercial funding for this study and there were no confidentiality agreements.
Adverse events of interest
The set of potential adverse events for the target study was extracted from several relevant sources including the VAERS, BEST and SPEAC frameworks, information from the vaccine manufacturer and relevant scientific publications. We cast a wide net to capture a wide range of clinically meaningful short- and medium-term potential adverse events that are likely to be documented in the electronic health record. Accordingly, minor side effects such as fever, malaise and local reactions at the injection site were not included in this study. The study included a 42-day follow-up, which included a 21-day follow-up after each of the first and second doses of the vaccine. A total of 42 days was considered sufficient to identify medium-term adverse events without diluting the incidence of short-term adverse events. Adverse events that could not be plausibly diagnosed within 42 days (e.g. chronic autoimmune disease) were also not included.
Adverse events were defined using diagnostic codes and short free text phrases that accompany the diagnoses in the CHS database. For a complete list of study results (adverse events) and their definitions, see Table S2.
For each adverse event, subjects were observed from the day of matching (time zero of follow-up) to the earliest of the following events: documentation of the adverse event, 42 days, end of the study calendar, or death. We also ended the follow-up of a paired couple when the unvaccinated control received the first dose of vaccine or when one of the members of the paired couple was diagnosed with SARS-CoV-2 infection.
Risks of SARS-CoV-2 infection
To put the magnitude of the vaccine side effects in context, we also assessed the effects of SARS-CoV-2 infection on these side effects during the 42 days following diagnosis. We used the same design as the one we used to study the side effects of vaccinations, except that the analysis period began at the start of the Covid-19 pandemic in Israel (March 1, 2020) and people who have recently had contact with the health system were not excluded (as such contacts can be expected in the days leading up to diagnosis).
In this SARS-CoV-2 analysis, people with a new diagnosis of SARS-CoV-2 infection were compared with controls who were not previously infected. As with the vaccine safety analysis, people could become infected with SARS-CoV-2 after they were compared as controls the day before. 2 – infected) and could then be included in the group of SARS-CoV-2 infected persons with a newly coordinated control. The follow-up examination of each paired pair began on the date of the positive PCR test result of the infected member and ended analogously to the main vaccination analysis, this time it ended when the control member was infected or one of the persons in the matching pair was vaccinated.
The effects of vaccination and SARS-CoV-2 infection were assessed with different cohorts. Therefore, they should be treated as separate result sets and not compared directly.
Since a large proportion of the unvaccinated controls were vaccinated during the follow-up period, we decided to estimate the observation analog of the per-protocol effect if all unvaccinated subjects remained unvaccinated during the follow-up period. To do this, we censored the matched couple’s data on whether and when the control member was vaccinated. Individuals who were initially matched as unvaccinated controls and then vaccinated during the study period could be re-admitted as vaccinated individuals with a new matched control. The same procedure was followed for the SARS-CoV-2 infection analysis (ie people who were initially matched as non-infected controls and then became infected during the study period could be accepted as infected people again with a new matched control).
We used the Kaplan-Meier estimator19th Generate cumulative incidence curves and estimate the risk of any adverse event after 42 days in each group. The risks were compared with ratios and differences (per 100,000 people).
In the vaccination analysis, in order not to attribute complications due to a SARS-CoV-2 infection to the vaccination (or its lack thereof), we also censored the data of the matching couple if and when one of the members was diagnosed with SARS-CoV-2 infection . Similarly, in the SARS-CoV-2 infection analysis, we censored the matched couple’s data if and when one of the members was vaccinated. Refer to the Supplementary Methods 1 section in the Supplementary Appendix for more details.
We calculated confidence intervals using the nonparametric percentile bootstrap method with 500 iterations. As is common with safety outcomes studies, no adjustment was made for multiple comparisons. The analyzes were performed using the R software version 4.0.4.