Figures
Abstract
The active-controlled trial with a non-inferiority design has gained popularity in recent years. However, non-inferiority trials present some methodological challenges, especially in determining the non-inferiority margin. Regulatory guidelines provide some general statements on how a non-inferiority trial should be conducted. Moreover, in a scientific advice procedure, regulators give companies the opportunity to discuss critical trial issues prior to the start of the trial. The aim of this study was to identify potential issues that may benefit from more explicit guidance by regulators. To achieve this, we collected and analyzed questions about non-inferiority trials posed by applicants for scientific advice in Europe in 2008 and 2009, as well as the responses given by the European Medicines Agency (EMA). In our analysis we included 156 final letters of advice from 2008 and 2009, addressed to 94 different applicants (manufacturers). Our analysis yielded two major findings: (1) applicants frequently asked questions ‘whether’ and ‘how’ to conduct a non-inferiority trial, 26% and 74%, respectively, and (2) the EMA regulators seem mainly concerned about the choice of the non-inferiority margin in non-inferiority trials (36% of total regulatory answers). In 40% of the answers, the EMA recommended using a stricter margin, and in 10% of the answers regarding non-inferiority margins, the EMA questioned the justification of the proposed non-inferiority margin.
We conclude that there are still difficulties in selecting the appropriate methodology for non-inferiority trials. Straightforward and harmonized guidance regarding non-inferiority trials is required, for example on whether it is necessary to conduct such a trial and how the non-inferiority margin is determined. It is unlikely that regulatory guidelines can cover all therapeutic areas; therefore, in some cases regulatory scientific advice may be used as an opportunity for tailored advice.
Citation: Wangge G, Putzeist M, Knol MJ, Klungel OH, Gispen-De Wied CC, de Boer A, et al. (2013) Regulatory Scientific Advice on Non-Inferiority Drug Trials. PLoS ONE 8(9): e74818. https://doi.org/10.1371/journal.pone.0074818
Editor: Barbara Mintzes, University of British Columbia, Canada
Received: July 27, 2012; Accepted: August 8, 2013; Published: September 5, 2013
Copyright: © 2013 Wangge et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, has received unrestricted research funding from the Netherlands Organisation for Health Research and Development (ZonMW), the Dutch Health Care Insurance Board (CVZ), the Royal Dutch Pharmacists Association (KNMP), the private-public funded Top Institute Pharma (www.tipharma.nl; includes co-funding from universities, government, and industry), the EU Innovative Medicines Initiative (IMI), EU 7th Framework Program (FP7), the Dutch Medicines Evaluation Board, the Dutch Ministry of Health and industry (including GlaxoSmithKline, Pfizer, and others). This study was funded by the Dutch Top Institute Pharma Escher project (T6-202). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Randomized placebo-controlled trials (RCTs) are considered the gold standard to confirm the efficacy of a drug. Currently, active-controlled trials are often performed instead of or in addition to placebo-controlled trials as the basis for marketing authorization and reimbursement decisions. A previous study showed that for 48% of all new medicines approved between 1999 and 2005, at least one active-controlled trial had been conducted during the development phase [1].
An active-controlled trial may have a non-inferiority design. A non-inferiority trial intends to demonstrate that the new drug is not worse than its comparator (a drug previously shown to be more effective than a placebo) to a certain limit (the non-inferiority margin), thus indirectly showing that the new treatment is effective (i.e. more effective than a placebo). However, non-inferiority trials pose several methodological challenges, especially in determining the non-inferiority margin. Previously we found that in 22% of the non-inferiority trials, the choice of the non-inferiority margin was based merely on assumptions made by the investigators. [2].
The International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) E9 [3], the ICH E10 [4], the European Medicines Agency (EMA) guidelines [5], [6] and the US Food and Drug Administration (FDA) draft guideline on non-inferiority trials [7] are the guidelines currently available that advise on the appropriate set-up of non-inferiority trials. Most of these guidelines only have general information on how to conduct such trials. However, in their guidelines for trials in certain therapeutic areas (such as diabetes mellitus and infectious diseases), the FDA and EMA provide more explicit guidance on how to apply a non-inferiority trial methodology. [8], [9] Interestingly, in the guidelines that present a specific non-inferiority margin, there are discrepancies between the FDA and EMA. For example, in the 2008 draft FDA guidance for diabetes mellitus, a non-inferiority margin in HbA1C reduction is suggested of 0.3% or 0.4%, while the 2011 EMA guideline suggests a non-inferiority margin of 0.3%. [8] A difference of 0.1% of HbA1C may not be significant in the clinic, but this difference will result in two largely different sample size calculations in non-inferiority trials.
Apart from guidelines, regulators provide the opportunity for companies to discuss critical trial issues prior to the start of the trial, to improve the quality of pre-registration trials. Scientific advice is an important part of such discussions. In Europe, scientific advice can be sought either from the EMA or from one or more national regulatory agencies. [10] Regulatory scientific advice can be asked as often as deemed necessary by an applicant, who is not obliged to adhere to the advice received or committed to accept any result of a scientific advice procedure. In a previous study, we found that one of the top five questions posed by the applicants during a scientific advice procedure was on study design. [11] However, the questions and responses specifically related to the non-inferiority design were not assessed in detail. Therefore, it remained largely unknown whether companies often ask questions specifically related to the non-inferiority trial design, and what their questions are and what replies they receive. In our previous review, we found that most non-inferiority trials were financed by the pharmaceutical industry (73.7%). [12] In this study, we identified questions on non-inferiority trials that were posed by applicants for scientific advice in Europe in 2008 and 2009, and the responses given by the EMA. Our analysis of the questions about non-inferiority trials posed by applicants in scientific advice dialogues with the EMA could identify any complex issues in the regulation of non-inferiority trials that may benefit from more explicit regulatory guidance.
Methods
To identify which documents discussed non-inferiority trials, we used the keyword ‘inferior’ to search in the Dutch Medicines Evaluation Board (MEB) database among final letters of advice from 2008–2009 from the EMA, represented by the Committee for Medicinal Products for Human use (CHMP). This scientific advice is confidential and is therefore not publicly available; to gain access to the data, researchers must be affiliated with the EMA/MEB. Four authors (GW, MP, CGDW and HL) are affiliated with the MEB and a confidentiality agreement was signed between the parties involved (Utrecht University and MEB) prior to the start of the study. At the time of our study, scientific advice from more recent years than 2008 and 2009 was not fully available.
Each final letter of advice consisted of questions from the applicants (company questions), followed by an elaboration of the particular question, the company position, and the CHMP response to each question. We excluded documents and company questions that did not discuss non-inferiority trials on efficacy or that discussed bioequivalence trials.
The following information was collected for each scientific advice briefing document: whether it was a follow-up to a previous scientific advice application, whether the drug was classified as an orphan drug, and what the (description) therapeutic target group of the drug was. The drugs were categorized by their therapeutic target group according to their first level of Anatomical Therapeutic Chemical (ATC) classification. [13] If an ATC classification was missing, the anatomical main group was determined based on the intended indication of the product.
Each company question, accompanying company position and CHMP response was scored according to the topic of interest. The topics of interest were divided into two types: ‘general’ and ‘specific’. General topics covered discussions about the strategic/overall development process of a drug. The specific topics consisted of topics unique to non-inferiority trials and topics unrelated to non-inferiority trials. Topics unique to non-inferiority trial included questions on whether or not a non-inferiority trial should be conducted (‘whether’ questions, related to the choice of a non-inferiority study design) and topics that discussed technical issues regarding how a non-inferiority trial should be conducted (‘how’ questions, related to the type of comparator, non-inferiority margin, non-inferiority sample size calculations, per-protocol or intention-to-treat analysis, and the possible switch from a non-inferiority design to a superiority design or vice versa). Topics that were not unique to non-inferiority trials discussed aspects of clinical trials that were not specific to non-inferiority trials, for example trial inclusion-exclusion criteria and types of endpoints (See Table 1 for further details and examples).
In each company question, company position or CHMP response, multiple topics may be discussed, and all these topics were included separately in the analyses. Additional topics that were found in the CHMP response but not in the accompanying company question or position were classified as ‘extra information’. In addition the results were presented for different therapeutic target groups separately.
Author GW searched and extracted all company questions, company positions and CHMP answers in the scientific advice documents, while the classification was carried out by both GW and MP separately. In case of discrepancies (n = 5), GW and MP tried to reach agreement on the classification first, before consulting AM-T and MK to reach final consensus. Subsequently, data were analyzed by GW and MP in a descriptive way. In addition, they assessed the topics according to their therapeutic target group.
Results
Search Result and General Characteristics
In total 75 documents from 2008 (22% of the 350 final advice documents from 2008) and 91 documents from 2009 (26% of the 345 documents from 2009) contained the keyword ‘inferior’. We excluded nine documents from 2008 and one document from 2009, because they were not related to non-inferiority trials. Finally, 156 documents were included in our analysis: 66 final advice letters from 2008 and 90 final advice letters from 2009. These 156 final advice letters had been addressed to 94 different applicants (manufacturers). In total, the documents contained 278 company questions, company positions and CHMP responses related to non-inferiority trials (Figure 1).
Note: 1. Final scientific advice letters were excluded if they were not related to non-inferiority trials; 2. In the analysis, 156 final scientific advice letters were included, containing 278 different questions (including their company position and CHMP response); 3. Each company question, company position and CHMP response may contain more than one topic.
Characteristics of the documents and questions are presented in Table 2. Of the therapeutic groups, antineoplastic and immunomodulating products were discussed most often (22% of the final advice letters included), followed by alimentary tract and metabolism products (17%) and anti-infectives (16%).
Topics of Discussion
Within the 278 company questions and positions related to non-inferiority trials, a total of 587 different topics were discussed. Of these, 101 were classified as general topics, asking advice regarding the overall development strategy, which may include a non-inferiority RCT. Issues that were more specific but not unique to non-inferiority trial design were identified 132 times. The remaining 354 topics were unique to non-inferiority trials. In the CHMP responses, a total of 400 different topics were discussed. Of these, 242 topics were unique to non-inferiority trials.
Among the topics unique to non-inferiority trials, both topics of ‘whether’ and ‘how’ to conduct a non-inferiority trial frequently appeared in the company questions, company positions and CHMP responses, but ‘how’ questions were asked more frequently than ‘whether’ questions: the ‘how’ questions represented 74% of total non-inferiority topics asked and 72% of the total CHMP responses, whereas the ‘whether’ questions made up the remaining 26% of total topics asked and 28% of the total CHMP non-inferiority unique responses.
Among the ‘how’ topics, the non-inferiority margin was most frequently discussed: 98 of the 354 company questions and positions (28%) and 86 of the 242 CHMP-responses of all the topics unique to non-inferiority trials (36%). In 42 of the 86 CHMP responses that discussed non-inferiority margin (49%), the CHMP supported the applicants’ non-inferiority margin proposal, while in another 35 responses (41%), CHMP recommended a stricter margin. In the remaining 9 CHMP responses (10%), the justification of the margin was questioned by the CHMP, but no specific advice on its magnitude was given.
The topic of data analysis (sample size calculation and type I error) was the second most frequently discussed topic. This topic was discussed in 87 (25%) of the company questions and positions and in 50 (21%) of the CHMP-responses related to topics unique to non-inferiority trials. In 48 of the 50 CHMP responses that discussed a non-inferiority data analysis (96%), the CHMP supported the applicants’ data analysis plan, while in one response, the CHMP recommended that the applicant revised the alpha level and in another response, the CHMP suggested that the applicant came up with a new plan for data-interim analysis.
The topic of switching between a superiority and a non-inferiority design appeared least often in company questions and positions (6% of the total topics asked), and the topic of ‘per-protocol or intention-to-treat appeared least often in CHMP responses (3% of the total CHMP responses; see Table 3).
In addition, Table 3 shows the differences and similarities in company questions, company positions and CHMP responses between the three therapeutic target groups discussed most often. For antineoplastic and immunomodulating products and for alimentary tract and metabolism products, the topics of data analysis mostly appeared in company questions and company positions (29% and 31%, respectively), while for anti-infective drugs the non-inferiority margin was discussed most often (40% of total topics asked within the therapeutic area). Most company questions and positions relating to other drugs dealt with the non-inferiority study design (28% of the total topics asked).
Among the CHMP responses, the non-inferiority margin was the topic that appeared most often in all three therapeutic target groups (38% of the total CHMP responses in antineoplastic and immunomodulating drugs, 28% in alimentary and metabolism drugs and 57% in anti-infective drugs).The non-inferiority margin was also discussed most often in the CHMP responses for other drugs (35% of the total CHMP responses).
‘Extra information’, i.e. unsolicited answers given by the CHMP, pertained to ‘how’ to perform a non-inferiority study more often than to ‘whether’ to perform a non-inferiority trial (data not shown). In alimentary tract and metabolism products, extra information was mostly given about ‘whether’ to perform a non-inferiority trial.
Discussion
Our content analysis of 2008 and 2009 scientific advice on non-inferiority trials provided by the EMA shows that applicants frequently ask questions about ‘whether’ and ‘how’ to conduct such trials. In addition, the EMA’s main concern seems to be the non-inferiority margin applied in non-inferiority trials.
Interestingly, more than 25% of the questions were ’whether’ questions, and thus it seemed that applicants frequently have doubts about the need for a non-inferiority trial. This result illustrates that more explicit guidance is necessary on fundamental issues in non-inferiority trials, such as in what situations a non-inferiority trial could or should be applied. However, we realize that one general guideline may not be feasible for all therapeutic areas, for example in cases where the efficacy of the current standard therapy against placebo has not yet been fully established (e.g. anti-depressants). [14].
Our second finding shows that non-inferiority margins and data analysis were the specific topics discussed most frequently. This finding applied to all therapeutic areas. Moreover, in 40% of the CHMP responses on non-inferiority margins, a stricter margin was recommended. This concern was previously acknowledged by European regulators. [15], [16] Although the data analysis was often discussed by applicants, it seems that the CHMP did not regard this area as problematic. The CHMP mostly supported the applicants’ data-analysis proposal or had no additional remarks. A similar finding was also found for the topic of per-protocol and intention-to-treat analysis. This may have been the result of the straightforward nature of the data analysis and the clear guidelines. [3], [5] However, the large proportion of ‘how’ questions confirms that the methodology of non-inferiority trials is not straightforward, in particular the determination of the non-inferiority margin. [2] These facts underline the need for additional guidance for the applicants on technical issues, such as given previously by the EMA guidance [6] and draft FDA guidelines on non-inferiority trials. [7].
Our subgroup analysis showed that the non-inferiority trial design for alimentary tract and metabolism products (including diabetes products) is of specific concern to the CHMP since the CHMP often recommends a non-inferiority design for these products, even if the applicant does not ask for guidance on this subject. Apparently, the use of non-inferiority trials to confirm drug efficacy is still complex in this therapeutic area. The CHMP released their revised guidance on anti-diabetic drugs in 2011. [8] Besides superiority trials, this guidance recommends the use of non-inferiority trials in diabetes patients. This may help to clarify in which cases non-inferiority trials should be performed.
The 2011 guidance on anti-diabetic drugs described above included the recommendation of a non-inferiority margin of 0.3% HbA1C. A similar specific requirement was previously proposed by the EMA for anti-infective drugs, recommending a specific value of the non-inferiority margin (10%). [17] Although the numbers are small, we found that the specific requirements still resulted in questions on the non-inferiority margin in anti-infective drugs. Recently, in a version updated in 2011, the value of 10% was replaced by a general statement in the guideline on how to determine a non-inferiority margin. [18] This approach is in line with the draft FDA guideline 2010 [7], which recommends determining a non-inferiority margin based on historical data instead of using a single fixed value as margin. It remains to be seen whether this new approach will lead to a reduction or an increase in scientific advice questions related to non-inferiority trials. In the meantime, it is essential that regulators are aware of the difficulties faced by applicants, and scientific dialogue between both parties can support the regulators in improving guidance on non-inferiority trials. This study did not assess the clinical impact of different scientific recommendations for non-inferiority studies by regulatory authorities on public health, but this is an interesting topic that should be explored in a follow-up study. In a previous study, we conducted a survey among clinical experts and demonstrated that assessing an appropriate non-inferiority margin with an acceptable clinical impact is not a straightforward task, even for clinical experts. One of the reasons may lie in the difficulty to understand the non-inferiority concept and in the need for a better structured method. A survey and a Delphi method may work, but further research is required. [19].
Non-inferiority trials are considered controversial [20] because they provide the opportunity for patients to receive medicine that provides no extra benefit but could be less safe. Indeed, in a previous review we showed that the ethical concerns regarding non-inferiority trials are mainly related to exposing patients to drugs without the intention of showing that these drugs have any additional benefits. [21] However, even if showing superiority of an outcome is considered to be an integrated part of all trials, a non-inferiority aim for efficacy outcomes could be important and necessary. Patients may respond differently to medicinal products, and therefore it may be beneficial for patients to have more therapeutic choices on the market that are equally effective. In addition, marketing approval decisions will not be based solely on the result of one trial or on a single non-inferiority trial. The needs for transparency about the conduct of non-inferiority studies, including insight into the questions posed by sponsors conducting non-inferiority trials and into the interpretation of the results are essential for good non-inferiority trials practice.
The limitations of this study also need to be addressed. Firstly, our data came from final scientific advice documents, which did not give detailed information on the reasoning behind the advice given. To discuss best practices and understand choices in non-inferiority trials, analyses of scientific advice questions for specific therapeutic drug groups are needed, supplemented by a more in-depth qualitative study. Secondly, the advice documents contained only a few (statistical) details on the non-inferiority trials. The type of study was not always clear, for example whether it was an explorative study or a pivotal confirmative phase III study. In the final scientific advice documents, the process of determining the non-inferiority margin was mostly described in general terms. Thus, it was difficult to analyze the statistical reasoning behind the proposed non-inferiority margin. However, the questions in the scientific advice documents show the potential problems in non-inferiority trials faced by the applicants.
We conclude that difficulties still exist in selecting the appropriate methodology of non-inferiority trials. Straightforward and harmonized guidance on non-inferiority trials is needed, such as in which cases these trials need to be conducted and how the non-inferiority margin is to be determined. It is unlikely that regulatory guidelines (either as one general guideline or special sections on non-inferiority trials in disease-specific guidelines) can cover all therapeutic areas; therefore, in some cases regulatory scientific advice may be used as an opportunity for tailored advice.
Author Contributions
Conceived and designed the experiments: GW MP AKMT MJK CCGDW. Performed the experiments: GW MP. Analyzed the data: GW MP. Wrote the paper: GW MP MJK OHK CCGDW AdB AWH HGL AKMT.
References
- 1. van Luijn JC, van Loenen AC, Gribnau FW, Leufkens HG (2008) Choice of comparator in active control trials of new drugs. Ann Pharmacother 42(11): 1605–12.
- 2. Wangge G, Klungel OH, Roes KCB, de Boer A, Hoes AW, et al. (2010) Interpretation and inference in noninferiority randomized controlled trials in drug research. Clin Pharmacol Ther 88(3): 420–3.
- 3.
ICH Expert Working Group (1998) Statistical principles for clinical trials E9.
- 4.
ICH Expert Working Group (2001) Choice of control group and related issues in clinical trials. E-10.
- 5.
Committee for Medicinal Products for Human Use (CHMP) (2005) Guideline on the choice of the non-inferiority margin. EMEA/CPMP/EWP/2158/99.
- 6.
Committee for Proprietary Medicinal Products (CPMP) (2000) Points to consider on switching between superiority and non-inferiority. CPMP/EWP/482/99.
- 7.
Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER) (2010) DRAFT GUIDANCE: Guidance for industry non-inferiority clinical trials.
- 8.
Committee for Medicinal Products for Human Use (CHMP) (2011) Guideline on clinical investigation of medicinal products in the treatment of diabetes mellitus. CPMP/EWP/1080/00 Rev 1*.
- 9.
Center for Drug Evaluation and Research (CDER) (2010) Guidance for industry, antibacterial drug products: use of non-inferiority trials to support approval.
- 10.
European Medicines Agency (2010) European medicines agency guidance for companies requesting scientific advice and protocol assistance. EMEA-H-4260-01-Rev 6.
- 11. Putzeist M, Mantel-Teeuwisse AK, Wied CCG, Hoes AW, Leufkens HG (2011) Regulatory scientific advice in drug development: Does company size make a difference? Eur J Clin Pharmacol 67(2): 157–64.
- 12. Wangge G, Klungel OH, Roes KCB, de Boer A, Hoes AW, et al. (2010) Room for improvement in conducting and reporting non-inferiority randomized controlled trials on drugs: A systematic review. PLoS One 27 5(10): e13550.
- 13.
WHO Collaborating Centre for Drug Statistics Methodology (2012) ATC/DDD index. Available: http://www.whocc.no/atc_ddd_index/via the Internet. Accessed 2012 Jan 31.
- 14. Temple R, Ellenberg SS (2000) Placebo-controlled trials and active-control trials in the evaluation of new treatments. part 1: Ethical and scientific issues. Ann Intern Med 19 133(6): 455–63.
- 15. Brown D, Day S, Hemmings R, Wright D (2008) Assessing the impact of ICH E9. Pharm Stat 7(2): 77–87.
- 16. Lewis JA (2002) The European regulatory experience. Stat Med 15 21(19): 2931–8.
- 17.
Committee for Proprietary Medicinal Products (CPMP) (1995) Guideline on the evaluation of medicinal products indicated for treatment of bacterial infections. CPMP/EWP/558/95.
- 18.
Committee for Medicinal Products for Human Use (CHMP) (2011) Guideline on the evaluation of medicinal products indicated for treatment of bacterial infections. CPMP/EWP/558/95 rev 2.
- 19. Wangge G, de Boer A, Klungel OH, Hoes AW, Knol MJ (2013) Expert-opinion on non-inferiority margin: A case study of oral anti-coagulant agents for prophylaxis of venous thromboembolic events after orthopedic surgery. Thromb Res 131(4): 368–71.
- 20. Garattini S, Bertele V (2007) Non-inferiority trials are unethical because they disregard patients’ interests. Lancet 370: 1875–1877.
- 21. Wangge G, Klungel OH, Roes KC, de Boer A, Hoes AW, et al. (2013) Should non-inferiority drug trials be banned altogether? Drug Discov Today 18(11–12): 601–4.