Identifying stroke therapeutics from preclinical models: A protocol for a novel application of network meta-analysis

Protocol

This protocol will be registered in the international prospective register of systematic reviews (PROSPERO, CRD) following peer review. Our review protocol is reported in accordance with the Preferred Reporting Items in Systematic reviews and Meta-Analysis-Protocol guidelines (a complete checklist is available as Supplementary File 1)¹³. Post-protocol adjustments will be included in the final report.

Objectives

Primary objective. We will perform a systematic review and NMAs to address the following question: amongst in vivo models of focal ischemic stroke, what are the relative benefits of competing therapies tested in combination with the gold standard treatment alteplase¹⁴ in (i) reducing cerebral infarction size, and (ii) improving neurobehavioural outcomes?

Secondary Objective. We will also (i) assess the risk of bias of the included studies, and (ii) explore what novel considerations for statistical adjustments are necessary for NMA of preclinical studies (e.g. method of ischemic induction, timing of treatment, species, sex, and comorbidities). We will also evaluate the challenges of applying NMA to preclinical studies (e.g. consistency, heterogeneity, availability of key study covariates).

Search and study identification

An information specialist (RS) will construct a search strategy based on a previous review of comparative stroke therapies, and limit them to include studies which compared therapies to alteplase (representative search strategy is provided in Supplementary File 2)¹⁵. Search strategies will be peer reviewed by a second information specialist using the peer review of electronic search strategy method¹⁶. Searches of Ovid MEDLINE and Embase will be carried out for articles on the effects of combination therapies with alteplase (Supplementary File 2 contains the search strategy). Of note, no language or date restrictions will be used. We will also search the CAMARADES database which contains data extracted from existing preclinical systematic reviews on stroke^15,17–28. In addition to this rigorous search, we will assess bibliographies of any new studies and reviews identified. Articles in foreign languages will be translated.

Study eligibility criteria

Eligibility criteria to identify relevant studies for the current review were established in considering the Population-Intervention-Comparators-Outcomes-Study design (PICOS) framework²⁹.

Population. Preclinical in vivo models of experimentally induced focal ischemia will be sought. All species/strains of animals will be eligible. Both female and male animals will be included. Neonatal animals will be excluded; however, all other ages will be considered. Studies in which focal ischemic stroke was established by transient occlusion of the middle cerebral artery or anterior cerebral artery via any method (chemical, embolic, mechanical, thermal) will be eligible. Animal models of haemorrhagic stroke, global or hemispheric brain ischemia, models of permanent occlusion without reperfusion (e.g. photothrombosis, cauterization), or delayed reperfusion such that it is considered permanent will be excluded³⁰. Human studies and tissue culture studies will be excluded.

Intervention and comparator. Studies where the treatment in combination with alteplase (e.g. alteplase + hypothermia) is compared with alteplase alone in animals that have experimentally induced focal ischemia will be eligible. Studies that compare more than one active treatment such as alteplase + hypothermia versus alteplase +FK506 (i.e. head to head comparisons) will also be included. Studies must include alteplase as a ‘foundational’ therapeutic in experimental arms to be eligible. All delivery routes and doses will be considered. To increase potential clinical relevance (i.e. construct validity), only studies that deliver therapies within 6 hours of induction of focal ischemic stroke will be included.

Outcome measures

Study design

Controlled comparison studies testing the efficacy of therapies + alteplase versus alteplase alone will be sought.

Screening and study selection

Two reviewers (A.D. and H.S.C.), will review abstracts (Stage 1 screen) and full text reports (Stage 2 screen) from search results independently and in duplicate against the eligibility criteria below using Distiller SR® software (Evidence Partners, Ottawa, ON) to identify relevant articles. Discrepancies will be resolved through discussion with a senior team member (M.L. and D.C.). Both stages of screening will begin with a calibration exercise to ensure consistent application of eligibility criteria. A PRISMA flow diagram³⁵ will be presented to document the process of study selection.

Data extraction

Two independent reviewers (A.D. and H.S.C.) will review studies and extract data into standardized, piloted forms implemented in Microsoft Excel (Microsoft Corporation, Seattle, Washington, USA). Discrepancies will be resolved through discussion with a senior team member. We will collect data related to, but not limited to, animal characteristics (Table 1); stroke model (Table 1); intervention (Table 2a, b); and outcomes (Table 3), as well as study ID (authors, year), and study design characteristics. Measures of central tendency (e.g. mean) and dispersion (e.g. standard deviation) will be extracted as reported. Data in graphical format will be extracted using Engauge Digitizer³⁶. When measures of central tendency and dispersion or sample sizes are missing (or cannot be measured digitally), authors will be contacted; if authors do not respond, the data will be excluded.

Assessment of risk of bias and construct validity

Two independent reviewers will assess the risk of bias of each included study (quality of the design, conduct and analysis for the experiment)³⁷. We will assess the risk of bias using a modified version of the Cochrane Risk of Bias Tool for randomized trials (Table 4). Risk of bias will be summarized³⁸ with descriptive statistics and presented graphically using standard methods and radar charts. The assessment of risk of bias will play an important role in exploring potential limitations of the evidence base and establishing the feasibility of incorporating relevant adjustments in NMA models. The construct validity of included studies (i.e. degree to which experimental model and design reflect the clinical entity of stroke and its treatment) will be assessed using elements from the CAMARADES checklist alongside criteria established by expert consensus (Table 5).

Exploring the evidence and synthesizing outcome data using network meta-analysis

We will begin by exploring the pattern of treatment comparisons represented by the included set of studies using network diagrams (or using a tabular approach if necessary, should the number and pattern of comparisons be too broad to be summarized graphically). Effect estimates from all included studies will be summarized. We will summarize traits of included studies focusing on clinical (e.g. age, sex, species, stroke model, reperfusion vs. permanent model, comorbidities, severity of infarct pre-treatment, infarct location)³⁹ and methodological (e.g. risk of bias, timing of outcome assessment) features²⁷, and review these with our clinical and preclinical experts to establish the degree of homogeneity within the included studies. For NMA, given the possibility that a large proportion of the studied interventions may have been evaluated in only a single study (and many could potentially yield very large effect sizes, which may not have been substantiated by more animals in more studies), we will exclude these interventions from NMAs performed; each of these treatments removed from NMA will neither benefit from “borrowing strength” through NMA, nor end up with a summary estimate and confidence interval different from what was reported in a single study. The reported findings for the outcomes of interest from studies removed from the NMA according to these criteria will be summarized separately in descriptive tables to ensure all relevant data are summarized. This approach will also restrict the network to a more practical size and reduce the risk of computational challenges.

Where there is homogeneity of important effect modifiers, we will perform NMAs to compare interventions^9,10,40, following procedures to assess the validity of the assumptions of homogeneity, similarity, and consistency⁴¹. Based upon the extracted study characteristics, we will work with our clinical and preclinical experts to establish any additional novel aspects of preclinical studies that may be important to consider in relation to judgements regarding study homogeneity beyond those anticipated in preparing this protocol. We have anticipated different species of animals (rats, mice, gerbils, dogs, sheep, non-human primates) across studies. We also anticipate that multiple reporting formats will have been used to assess both infarct volume (e.g. mm³, % of hemisphere or total brain, etc.) and neurobehavioral changes (e.g. seconds, % of baseline).

For meta-analysis of preclinical studies, the normalized mean difference (NMD) scale is useful in serving the purpose of synthesizing the complexity of data aforementioned⁴². Prior to performing NMAs, we will perform traditional pairwise meta-analyses on the NMD scale for each comparison in the treatment networks where two or more studies are available to explore heterogeneity based on the I² statistic²⁹. To perform network meta-analyses on the NMD scale, we will use an established model from the National Institute in Health and Care Excellence’s TSD series⁴³, adapting its identity link to the log link in order to conduct the NMA on a log ratio of means (logRoM) scale. The log ratio of means of the k^th treatment and the “stroke only” control, d_k = logRoM_C,Tk, can be estimated after model fitting, and the corresponding NMD estimate is:

                                                                                             1 – exp{logRoM_C,Tk} = 1 – exp (d_k)

The NMD of the k^th treatment in comparison with alteplase (k=1) is 1 – exp (d_k – d₁).

Both fixed- and random-effect Bayesian NMAs will be performed using a common heterogeneity parameter according to established methods^10,40,43. Model fit will be assessed by comparing the model’s posterior total residual deviance with the number of unconstrained data points⁴³. Selection between models will be based on deviance information criteria (DIC), with a difference of five points suggesting an important difference⁴³. All pairwise comparisons between interventions will be expressed with both summary point estimates and corresponding 95% credible intervals. Vague prior distributions will be assigned for all measures of treatment effect, as well as for the between-study variance parameter in random effects analyses. NMAs will be performed using OpenBUGS software version 3.2.3⁴⁴ and the R Package R2OpenBUGS⁴⁵. Model convergence will be assessed using established methods including assessment of Rhat (the potential scale reduction factor) and the Gelman-Rubin convergence diagnostic to see if they are near 1⁹. Surface under the cumulative ranking (SUCRA) values, and the mean rank of each intervention (with 2.5% and 97.5% quantiles) will also be estimated for each intervention⁴⁶. Forest plots of treatment comparisons versus “stroke with no treatment” control as well as versus stroke + alteplase will be prepared for each outcome. Given the anticipated high number of interventions assessed in only a single study, a tabular approach to summarizing findings will be employed for them. We will also undertake forest plots of effects wherein interventions are ordered according to mean rank estimated from NMA.

Addressing heterogeneity and inconsistency

To check the validity of the consistency assumption (i.e., transitivity of the effect size through common comparators), a consistency model as well as an unrelated mean effects model will be fit to the data⁴⁷. We will compare their respective DIC values to check model fitting and their posterior mean deviance contribution per study to check the consistency assumption. We will also assess the magnitude of the estimated between-study SD measure from both models, as a reduction in this parameter in the inconsistency model also provides evidence of inconsistency.

The likelihood of important clinical and methodological heterogeneity between studies is anticipated by the research team to be high and may include several nuances which are unique to the pre-clinical setting. First, several vital aspects of preclinical studies from our risk of bias assessments (described earlier) may be important adjustment factors that could have an important impact on the findings from NMAs, including randomization and blinding^24,48. In this work, we will use subgroup analyses or covariate-adjusted analyses to address and explore the impact that covariates have on findings and to establish the robustness of findings from primary syntheses^49,50. We will assess the possibility to adjust for the following group level factors: animal species (e.g. mouse) and strain (e.g. C57Bl6 strain of mice), model of stroke, average animal age, percentage of female subjects, average time since stroke induction, combination therapies, cerebral blood flow, temperature, infarct location and severity, use of randomization and blinding of experimenters and outcome assessments. Alternatively, when combining data from different species, we could model animal species as an extra level in the hierarchical model for treatment effect, allowing for heterogeneity across species and assuming that treatment effects are similar across species around an overall mean effect. For the network structure, primary analyses will be performed at the treatment level. As dose may have an important effect on intervention benefits, we will also explore the range of doses associated with each intervention across studies to consider additional analyses. However, as dose response characteristics of different agents may also vary between animal species and an a priori source of information to establish appropriate dose categories is not available, any analyses pursued in relation to dose will be appropriately indicated as post-hoc. Findings from all analyses will be reported. Given the anticipated complexity of this novel application of NMA, we anticipate separate publications will be required for the primary and secondary outcomes.

Dissemination

The results of the study will be submitted for publication to a peer-reviewed journal and presented at relevant national and international conferences and scientific meetings to promote knowledge transfer.

Amendments

If amendments are required for this protocol, date of each amendment will be provided with a description for rationale for the change in this section.