Ischemic Stroke and COVID-19 Infection: an Analysis of Treatment Outcomes in Patients who Underwent Endovascular Thrombectomy
- Authors: Yakovlev A.I.1, Voznyuk I.A.2,3, Kharitonova T.V.4, Savello A.V.3, Prokhorova M.V.2, Kolomentsev S.V.3,5, Tsurikova N.А.6
-
Affiliations:
- Saint-Petersburg I.I. Dzhanelidze Research Institute of Emergency Medicine
- Pavlov First Saint Petersburg State Medical University Russia
- Immanuel Kant Baltic Federal University
- National Society of Neurosonology and Cerebral Hemodynamics
- Kirov Military Medical Academy
- Regional Children’s Clinical Hospital
- Issue: Vol 18, No 1 (2024)
- Pages: 55-62
- Section: Original articles
- Submitted: 16.01.2024
- Accepted: 24.01.2024
- Published: 05.04.2024
- URL: https://www.annaly-nevrologii.com/journal/pathID/article/view/1081
- DOI: https://doi.org/10.54101/ACEN.2024.1.6
- ID: 1081
Cite item
Abstract
Aim. This study aimed to compare and evaluate treatment outcomes in groups of ischemic stroke patients with or without COVID-19 infection who underwent endovascular thrombectomy (EVT).
Materials and methods. We conducted a retrospective analysis of 817 case records of IS patients aged 25 to 99 years with confirmed thrombotic occlusion of cerebral arteries and subsequent EVT who were treated in regional vascular centers in St. Petersburg from January 1, 2021
to December 31, 2021.
Results. Patients without COVID-19 had favorable outcome more often than patients with confirmed COVID-19 (35.0% vs. 7.3%, p < 0.001); mortality rate was 30% vs. 52%, respectively (p < 0.001).
Conclusions. Intercurrent COVID-19 significantly worsened prognosis and increased risk of death in ischemic stroke patients who underwent EVT.
Full Text
Introduction
Endovascular therapy has significantly expanded pathogenetic treatment options for patients with acute ischemic stroke (IS) caused by occlusion of proximal cerebral arteries and allowed decreasing hospital mortality and improving functional outcomes [1–4]. Reperfusion treatment in cerebral artery thrombosis has become more accessible with the combined use of endovascular intervention and intravenous thrombolytic therapy, expanding the “therapeutic window” for restoring blood flow from 4.5 to 6 and even up to 24 hours, increasing the rates of successful recanalization [5–7].
An established and well-equipped network of vascular centers sharing common triage principles was a key factor that allowed introducing advanced specialized medical care, including high-tech one, for stroke patients in the regions of Russia. Intensive recruitment of personnel experienced in endovascular surgery and regular training of new specialists have been performed since 2011, and this has significantly accelerated introducing endovascular thrombectomy (EVT) into routine practice of regional vascular centers. By 2014, there were 20 radiology surgeons in Saint Petersburg with up to 10 cerebral endovascular procedures in their professional journey [7–9], while today there are 77 of them. Twenty-four surgeons have performed up to 10 EVT procedures, and 10 to 100 surgeons up to 38 EVT procedures. Fourteen surgeons have performed over 100 procedures, and one of them has performed over 150 cerebral endovascular procedures for acute thrombosis and cerebral artery occlusion. The expertise gained in patient triage in St. Petersburg together with adequate staffing has led to an exponential increase in the number of endovascular procedures (i.e. from 25 in 2014 to 1123 in 2022). In 2019, 26% of the total number of EVT procedures performed in vascular centers of Russia was performed in St. Petersburg.
Considering the complex nature of the surgical interventions, regional registries to access the quality and success of the interventions were established in Moscow and St. Petersburg at the beginning of the introduction of EVT. According to the Moscow Stroke Registry, in 2019, successful recanalization (Modified Treatment in Cerebral Ischemia (mTICI) score of 2b–3) was achieved in 75% of patients; thromboaspiration, which was used in 51.6% of patients, was the most common procedure for thrombectomy; good functional recovery by the end of Day 20 (Rehabilitation Triage Scale score of 0–2) was observed in 29.2% of patients. 20-day mortality rate in the cohort of surgically treated patients was 31.8% [8].
The COVID-19 pandemic created critical challenges, significantly affecting the specialized medical care system for stroke patients and limiting triage opportunities and care availability in most regions of Russia [10]. Stroke patients were more vulnerable if having signs of COVID-19 [11–15]. A dramatic role in increased frequency of more severe disease and increased mortality was played by the mutual aggravation of the two conditions and a loss of time caused by “rehospitalization” of patients from vascular centers to so-called “green zones” in repurposed hospitals, as well as limitations associated with more common contraindications to intravenous thrombolytic therapy and EVT due to the comorbidity.
This study aimed to compare and evaluate treatment outcomes in the ischemic stroke patients with or without COVID-19 infection who underwent EVT.
Materials and methods
We conducted a retrospective analysis of 817 case records of all patients (365 men and 452 women) aged 25–99 years who were treated for IS from January 01, 2021 to December 31, 2021 in 11 vascular centers in St. Petersburg (Pokrovskaya City Hospital No 1, Hospital for War Veterans, Mariinskaya City Hospital No 16, A.M. Nikiforov Russian Emergency and Radiation Medicine Center, Aleksandrovskaya City Hospital No 17, Almazov National Medical Research Center, City Hospital No 40, City Multi-Field Hospital No 2, I.I. Dzhanelidze Research Institute of Emergency Medicine, City Hospital No 26, St. Elizaveta City Hospital No 3).
Inclusion criteria for study population:
- Ischemic stroke confirmed by neuroimaging, i.e. computed tomography (CT) or magnetic resonance imaging (MRI);
- confirmed thrombotic occlusion of proximal cerebral arteries followed by EVT;
- no history of thrombolytic therapy;
- Exclusion criteria:
- no polymerase chain reaction test for SARS-CoV-2;
- no indications for EVT;
- no control CT performed 24 hours after the surgery.
All patients underwent standard clinical and neurological examination, and the clinical diagnosis of IS was established according to ICD-10 and TOAST criteria. Neurological deficit was assessed using the National Institutes of Health Stroke Scale (NIHSS). EVT success was determined by control angiography; result was considered as success if the lumen of the cerebral artery was restored with mTICI score of 2b to 3 and as failure if mTICI score was 0 to 2a. Functional outcomes after IS were assessed using the Modified Rankin Scale (mRS) at discharge. Functional outcome was considered favorable if mRs score was 0 to 2 and unfavorable if mRs score was 3 to 5; death corresponded to mRs score of 6. We also assessed mortality on Day 90 after the onset of stroke in a cohort of discharged patients based on cases registered in the St. Petersburg State Information System. Death causes were reported based on autopsy results (for hospital deaths).
Quantitative data were presented as medians and quartiles (Me [Q1; Q3]); categorical (qualitative) data were presented as rates and percentages; 95% confidence intervals (Cis) were calculated for proportions. For all numerical data, preliminary testing for normality of distribution was performed using Shapiro–Wilk test, as well as skewness and kurtosis test by calculating p value when testing the null hypothesis about normal distribution of the variable. Student’s t-test for independent samples was used for statistical analysis of data with normal distribution. Non-parametric Wilcoxon–Mann–Whitney and Kruskal–Wallis tests were used for data without normal distribution. The association between categorical variables was evaluated using Fisher’s exact test and Pearson’s χ2 test. The relationship between quantitative characteristics was determined by calculating Pearson linear correlation coefficient. Univariate and multivariate logistic regression models were generated to assess the association of favorable functional outcome (mRs score of 0–2) at discharge and the variables of interest and their confounder variables. All р-values were based on two-sided tests. Differences were considered significant with p < 0.05. All calculations were performed using R package version 4.3.1.
Results
Demographics and clinical characteristics of the study population are presented in Table 1.
Table 1. Demographics and clinical characteristics of IS patients with or without COVID-19
Parameter | Patients with COVID-19 | Patients without COVID-19 | p |
Number of patients, n | 219 | 598 | |
Age, years | 75 [67; 84] | 73 [63; 82] | 0.023 |
Male gender, n (%) | 103 (47) | 262 (44) | 0.4 |
NIHSS score at admission | 17 [13; 20] | 15 [10; 18] | < 0.001 |
ASPECTS score by CT at admission | 10.00 [8.00; 10.00] | 10.00 [8.00; 10.00] | 0.5 |
Number and percentage of patients with 0 to 6 h from stroke to arterial puncture, n (%) | 133 (61; 95% CI 54–67) | 399 (67; 95% CI 63–70) | 0.11 |
Successful angiographic reperfusion (mTICI score of 2a to 3), n (%) | 164 (77; 95% CI 71–83) | 492 (86; 95% CI 83–89) | 0.004 |
Functional outcome at discharge, mRS score | 6.00 [4.00; 6.00] | 3.00 [2.00; 6.00] | < 0.001 |
Mortality rate on Day 90 from IS onset, n (%) | 98 (51; 95% CI 44–59) | 177 (33; 95% CI 29–37) | < 0.001 |
Note. *Quantitative data are presented as medians and quartiles (Me [Q1; Q3]). Qualitative data are presented as rates and percentages.
Based on the selection criteria, we identified a cohort of 219 patients with confirmed COVID-19 (according to PCR and chest CT results) from all 817 patients who underwent EVT; no documented confirmation of COVID-19 was obtained in 598 cases. Outcome assessment results are presented in Table 2.
Table 2. Comparative analysis of outcomes at discharge after EVT in IS patients with or without COVID-19
Outcome group | Patients with COVID-19 | Patients without COVID-19 | р | Total, n |
Discharged + died (total), n | 219 | 598 | – | 817 |
During 3 months after IS: | ||||
died, n (%) | 98 (51%; 95% CI 44–59) | 177 (33%; 95% CI 29–37) | < 0.001 | 275 |
survived, n (%) | 93 (49%; 95% CI 42–56) | 360 (67%; 95% CI 63–71) | < 0.001 | |
Number of missed values | 28 | 61 | 89 |
The data show that statistically significant differences were seen in 4 parameters:
- neurological deficit severity (NIHSS score);
- mortality rate by the end of acute IS period;
- functional outcome by mRS score by the end of treatment in the acute period;
- death number and percentage on Day 90 after disease onset.
When assessing the cohorts, we found that the neurological syndrome at admission and initiation of treatment was significantly more severe in patients with COVID-19, which supported the hypothesis on mutual aggravation of the conditions and demonstrated pathogenesis of COVID-19. NIHSS score was 17 and 15 in patients with or without COVID-19, respectively (p < 0.001).
Hospital mortality rate in the study population (n = 817) was 36% (n = 294), which was significantly higher than before the pandemic. Of those, mortality rate in patients without or with COVID-19 at the time of stroke was 30% vs. 52%, respectively (p < 0.001). A total of 89 (11%) IS cases were excluded from the analysis because their treatment did not comply with several provisions of the Procedure for providing specialized medical care for stroke patients1 due to triage problems, lack of personnel or shortage of critical care beds. In these cases, the relationship between outcomes of hospital treatment was obviously not related to the treatment option and comorbidity.
In IS patients who underwent EVT and did not have confirmed COVID-19, favorable functional outcomes (mRS score of 0–2) at discharge were more common (35.0% vs. 7.3%; p < 0.001). By Day 90 from stroke onset, outcome was known in 728/817 patients, while in 89 patients it could not be established. Mortality rate on Day 90 after stroke onset was lower in patients without diagnosed COVID-19 compared with patients without it (33% vs. 51%).
Outcomes of endovascular surgery (successful/unsuccessful) were available in 784/817 patients; no information on mTICI recanalization score was available in 33 case records. Statistical analysis confirmed significant intergroup differences (Table 1): successful recanalization was achieved in 86% of patients without signs of COVID-19 vs. 77% of patients with concomitant COVID-19 (p = 0.004).
To identify the association between favorable functional outcome with endpoints and confounders in a cohort of IS patients who required endovascular surgery, a multiple logistic regression model was generated including age, neurological deficit (NIHSS score) at admission, degree of revascularization achieved, and absence of COVID-19 (Table 3).
Table 3. Coefficients of logistic regression equation to assess the association between favorable functional outcome (mRs score of 0–2) and clinically significant variables in the study group
Parameter | Univariate analysis | Multivariate analysis | ||||
odds ratio | 95% CI | p | odds ratio | 95% CI | p | |
Age | 0.96 | 0.95–0.98 | < 0.001 | 0.97 | 0.96–0.99 | < 0.001 |
NIHSS score at admission | 0.87 | 0.84–0.89 | < 0.001 | 0.88 | 0.85–0.91 | < 0.001 |
Over 6 h from stroke onset to arterial puncture | 0.78 | 0.56–1.08 | 0.13 | 0.74 | 0.50–1.10 | 0.14 |
Successful recanalization | 3.18 | 1.98–5.37 | < 0.001 | 2.84 | 1.64–5.18 | < 0.001 |
No COVID-19 | 6.77 | 4.08–12.0 | < 0.001 | 6.82 | 3.81–13.2 | < 0.001 |
Discussion
We studied a continuous sample of IS patients who underwent EVT in vascular centers of St. Petersburg in 2021; this population was representative for the period of the widespread COVID-19 pandemic. We found that IS patients without COVID-19 who underwent EVT had a better prognosis for favorable functional outcome, higher chances of good revascularization and a lower probability of death both during hospital treatment and within 3 months after IS onset.
This may be partly caused by administrative factors such as suboptimal pre-hospital and in-hospital triage and poor availability of specialized medical care. Stroke admissions and frequency of systemic thrombolysis and/or EVT dropped during the COVID-19 pandemic in many countries [16]. Door-to-puncture time increased in EVT candidates, which was associated with the performance of chest computed tomography [17]. In our population, there was a trend towards less frequent early (up to 6 hours from the onset) EVT in patients with COVID-19 (61% vs. 67% in patients without COVID-19), which, however, did not achieve statistical significance in either univariate or multivariate analysis.
Other possible causes may include aggravation of neurological deficit due to intercurrent SARS-CoV-2 infection. Neurological symptoms in COVID-19 are not uncommon: headache, paresthesias, impaired perception of smell and taste, impaired consciousness have been reported in patients with COVID-19, and, in some cases, COVID-19 manifested with stroke [18, 19]. In our population, COVID-19 was associated with more severe neurological deficit in the hyperacute stroke phase (NIHSS score at admission). It is important to note that differences in the severity of neurological deficit were not associated with inter-group differences in the ASPECTS score, suggesting that the higher NIHSS score in patients with COVID-19 was not explained by later hospitalization.
Therefore, COVID-19 appeared to be an independent factor to aggravate IS severity, which was confirmed by multivariate analysis results. In our population, the odds of an unfavorable stroke outcome in patients with COVID-19 increased 6.82-fold (95% CI 3.81–13.2) with adjustment for age, severity of neurological deficit, EVT outcome, and time. This observation is consistent with the global trend: less favorable outcomes in patients treated for stroke have been reported in most cohort studies during the COVID-19 pandemic [18, 20]. This might be directly related to COVID-19 severity. The advantage of our study compared with observational case series that compared IS outcomes before and during the COVID-19 pandemic is that our study considered the status of SARS-CoV-2 infection and IS severity. Our results confirmed a hypothesis on the pathophysiological role of COVID-19 per se and its complications in IS patients. It is important to note that administrative measures during a pandemic should aim not only at improving patient triage but also at carefully preventing secondary infection in IS patients admitted without COVID-19.
Limitations. Our study was based on a retrospective analysis of case records; it was observational and did not consider treatment outcomes of patients who did not undergo EVT due to extreme severity of their respiratory syndrome or multiple organ involvement that complicated COVID-19. We did not consider cases with reduced EVT availability due to administrative issues because deaths in this population were expected and would likely to worsen the outcomes in the general population of stroke patients.
Conclusion
Our results demonstrated with high confidence the negative impact of COVID-19 on outcomes in patients receiving endovascular treatment for acute cerebral artery thrombosis. COVID-19 significantly worsened survival and functional outcomes in IS patients who underwent EVT. Administrative issues, which reduced the availability of timely pathogenetic therapy, significantly contributed to hospital mortality in IS patients.
1 Decree 928n of the Ministry of Health of the Russian Federation dated November 15, 2012 “On approval of the Procedure for providing medical care to stroke patients”.
About the authors
Alexander I. Yakovlev
Saint-Petersburg I.I. Dzhanelidze Research Institute of Emergency Medicine
Author for correspondence.
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0001-7648-4388
neurologist
Россия, Saint-PetersburgIgor A. Voznyuk
Pavlov First Saint Petersburg State Medical University Russia; Immanuel Kant Baltic Federal University
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0002-0340-4110
D. Sci. (Med.), Professor
Россия, Saint-Petersburg; KaliningradTatiana V. Kharitonova
National Society of Neurosonology and Cerebral Hemodynamics
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0003-4021-9421
secretary, Board of the Association
Россия, Saint-PetersburgAlexander V. Savello
Immanuel Kant Baltic Federal University
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0002-1680-6119
D. Sci. (Med.), Professor, Deputy Head, Department of neurosurgery
Россия, KaliningradMariia V. Prokhorova
Pavlov First Saint Petersburg State Medical University Russia
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0003-3412-0038
neurologist, Department of neurology for patients with acute cerebrovascular accident of the clini
Россия, Saint-PetersburgSergey V. Kolomentsev
Immanuel Kant Baltic Federal University; Kirov Military Medical Academy
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0002-3756-6214
Cand. Sci. (Med.), senior researcher, Head, Neurological department, Clinic of Nervous Diseases
Россия, Kaliningrad; Saint-PetersburgNadezhda А. Tsurikova
Regional Children’s Clinical Hospital
Email: yakovlevai92@yandex.ru
ORCID iD: 0000-0002-3946-430X
Cand. Sci. (Med.), pediatrician, Neurological department
Россия, Rostov-on-DonReferences
- Скрыпник Д.В., Анисимов К.В., Боцина А.Ю. и др. Результаты эндоваскулярного лечения пациентов с окклюзиями крупных церебральных артерий в мегаполисе. Данные Московского инсультного регистра за 2019 г. Неврология, нейропсихиатрия, психосоматика. 2020;12(5):9–17. Skrypnik D.V., Anisimov K.V., Botsina A.Yu. et al. Endovascular treatment results in patients with large cerebral artery occlusions in a metropolis. Moscow Stroke Registry data over 2019. Neurology, Neuropsychiatry, Psychosomatics. 2020;12(5):9-17. doi: 10.14412/2074-2711-2020-5-9-17
- Farooqui M., Ikram A., Suriya S. et al. Patterns of care in patients with Basilar Artery Occlusion (BAO): a population-based study. Life (Basel). 2023;13(3):829. doi: 10.3390/life13030829
- Oliveira A.J., Viana S.M., Santos A.S. Mechanical thrombectomy for acute ischemic stroke: systematic review and meta-analysis. Einstein (São Paulo). 2022;20:eRW6642. doi: 10.31744/einstein_journal/2022RW6642
- Wu L., Wu W., Tali E.T., Yuh, W.T. Oligemia, penumbra, infarction: understanding hypoperfusion with neuroimaging. Neuroimaging Clin. N. Am. 2018;28(4):599–609. doi: 10.1016/j.nic.2018.06.013
- Савелло А.В., Свистов Д.В., Сорокоумов Д.А. Внутрисосудистые методы лечения ишемического инсульта: современное состояние и перспективы. Неврология, нейропсихиатрия, психосоматика. 2015;7(4):42–49. Savello A.V., Svistov D.V., Sorokoumov D.A. Endovascular treatments for ischemic stroke: present status and prospects. Neurology, Neuropsychiatry, Psychosomatics. 2015;7(4):42–49. doi: 10.14412/2074-2711-2015-4-42-49
- Jolugbo P., Ariëns R.A.S. Thrombus composition and efficacy of thrombo- lysis and thrombectomy in acute ischemic stroke. Stroke. 2021;52(3):1131–1142. doi: 10.1161/STROKEAHA.120.032810
- Zureigat H., Alhusban M., Cobia M. Mechanical thrombectomy outcomes in COVID-19 patients with acute ischemic stroke: a narrative review. Neurologist. 2021;26(6):261–267. doi: 10.1097/NRL.0000000000000360
- Савелло А.В., Вознюк И.А., Свистов Д.В. и др. Результаты лечения ишемического инсульта с применением внутрисосудистой тромбоэмболэктомии в условиях региональных сосудистых центров в мегаполисе (Санкт-Петербург). Журнал неврологии и психиатрии им. С.С. Корсакова. Спецвыпуски. 2018;118(12-2):54–63. Savello A.V., Vozniuk I.A., Svistov D.V. et al. Outcomes of endovascular thrombectomy for acute stroke in regional vascular centers of a metropolis (St.-Petersburg). Zhurnal Nevrologii i Psikhiatrii imeni S.S. Korsakova. 2018;118(12-2):54–63. doi: 10.17116/jnevro201811812254
- Янишевский С.Н. Изменения процесса оказания помощи пациентам с инсультом в условиях эпидемии COVID-19. Артериальная гипертензия. 2020;26(3):263–269. Yanishevskii S.N. Healthcare for stroke patients in COVID-19 pandemic. Arterial Hypertension. 2020;26(3):263–269. doi: 10.18705/1607-419X-2020-26-3-263-269
- Snell J. SARS-CoV-2 infection and its association with thrombosis and ischemic stroke: a review. Am. J. Emerg. Med. 2021;40:188–192. doi: 10.1016/j.ajem.2020.09.072
- Одинак М.М., Цыган Н.В., Яковлева В.А. и др. Клинические особенности, эпидемиология и патогенез поражения нервной системы при новой коронавирусной инфекции COVID-19. Известия Российской Военно-медицинской академии. 2020;39(3):117–119. Odinak M.M., Tsygan N.V., Yakovleva V.A. et al. Clinical features, epidemiology and pathogenesis of nervous system damage in new coronaviral infection COVID-19. Russian Military Medical Academy reports. 2020;39(3):117–119.
- Avula A., Nalleballe K., Narula N. et al. COVID-19 presenting as stroke. Brain Behav. Immun. 2020;87:15–119. doi: 10.1016/j.bbi.2020.04.077
- Beristain-Covarrubias N., Perez-Toledo M., Thomas M.R. et al. Understanding infection-induced thrombosis: lessons learned from animal models. Front. Immunol. 2019;10:2569. doi: 10.3389/fimmu.2019.02569
- Morassi M., Bagatto D., Cobelli M. et al. Stroke in patients with SARS-CoV-2 infection: case series. J. Neurol. 2020;267(8):2185–2192. doi: 10.1007/s00415-020-09885-2
- Цыган Н.В., Трашков А.П., Рябцев А.В. и др. Особенности симптоматики и патогенеза повреждения центральной нервной системы при COVID-19 по данным клинических исследований (обзор). Общая реаниматология. 2021;17(3):65–77. Tsygan N.V., Trashkov A.P., Ryabtsev A.V. et al. Signs and symptoms of central nervous system involvement and their pathogenesis in COVID-19 according to the clinical data (review). General Reanimatology. 2021;17(3):65–77. doi: 10.15360/1813-9779-2021-3-65-77
- Liu R., Zhao J., Fisher M. The global impact of COVID‐19 on acute stroke care. CNS Neurosci. Ther. 2020;26:1103–1105. doi: 10.1111/cns.13442
- Fuentes B., Alonso De Leciñana M., García-Madrona S. et al. Stroke acute management and outcomes during the COVID-19 outbreak. Stroke. 2021;52:552–562. doi: 10.1161/STROKEAHA.120.031769
- Tang X., Zheng F. A review of ischemic stroke in COVID-19: currently known pathophysiological mechanisms. Neurol. Sci. 2022;43(1):67–79. doi: 10.1007/s10072-021-05679-0
- Прохорова М.В., Яковлев А.И., Вознюк И.А. и др. Воспаление и эндотелиотоксичность: патогенетические грани поражения центральной нервной системы при новой коронавирусной инфекции. Анналы клинической и экспериментальной неврологии. 2022;16(3):15–24. Prokhorova M.V., Yakovlev A.I., Voznyuk I.A. et al. Inflammation and endothelial toxicity: pathogenetic aspects of central nervous system damage due to novel coronavirus disease. Annals of Clinical and Experimental Neurology. 2022;16(3):15–24. doi: 10.54101/ACEN.2022.3.2
- Siegler J.E., Abdalkader M., Michel P., Nguyen T.N. Therapeutic trends of cerebrovascular disease during the COVID-19 pandemic and future perspectives. J. Stroke. 2022;24(2):179–188. doi: 10.5853/jos.2022.00843