Figures
Abstract
Setting
Research capacity is weakest in low and middle-income countries (LMICs) where operational research is highly relevant and needed. Structured Operational Research and Training Initiative (SORT IT) courses have been developed to train participants to conduct and publish operational research and influence policy and practice. Twenty courses were completed in Asia, Africa, Europe and the South Pacific between 2009 and 2014.
Objectives
In the 20 completed SORT IT courses, to assess where the research was conducted, who was trained, who became facilitators in subsequent courses and course outcomes.
Results
There were 236 participants (41% female) including 64 nationalities who conducted research in 59 countries, mostly from Asia and Africa (mean course duration = 9.7 months). Most participants (68%) were from government health programs and non-governmental agencies. A total of 213(90%) participants completed all milestones successfully with 41(19%) becoming subsequent course facilitators, 88% of whom were from LMICs. Of 228 manuscripts submitted to scientific journals, 197(86%) were either published or in press; in 86%, the principal investigator (first author) was a LMIC national. Papers were published in 23 scientific journals (impact factor 0.5–4.4) and covered 21 disease categories (median publication time = 5.7 months). Published papers (186) had 94,794 cumulative article views/downloads. Article views/downloads for immediate open access articles were double those from closed access journals.
Citation: Zachariah R, Rust S, Berger SD, Guillerm N, Bissell K, Delaunois P, et al. (2016) Building Global Capacity for Conducting Operational Research Using the SORT IT Model: Where and Who? PLoS ONE 11(8): e0160837. https://doi.org/10.1371/journal.pone.0160837
Editor: Pietro Cipresso, IRCCS Istituto Auxologico Italiano, ITALY
Received: March 2, 2016; Accepted: July 26, 2016; Published: August 9, 2016
Copyright: © 2016 Zachariah et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The 2013 World Health Report of the World Health Organization (WHO) was entitled “Research for Universal Health Coverage”.[1] One of its key messages was that universal health coverage, with access to high-quality services, cannot be achieved without evidence from research.[1] Importantly, it emphasized the need to strengthen research capacity in public health programs, close to the supply and demand of health services.[1, 2]
Research capacity, is lowest in Low- and Middle-Income Countries (LMICs) where research evidence is highly relevant and most needed.[3, 4] Meeting this challenge requires developing and implementing effective models for capacity building. [2, 5, 6] One such model is the Structured Operational Research and Training Initiative (SORT IT)–a global partnership led by the Special Program for Research and Training in Tropical Diseases at WHO (WHO/TDR). SORT IT supports countries to conduct operational research around their own priorities, to build improved and sustainable research capacity, and to make evidence-informed changes in the delivery of health services. [7]
The SORT IT training courses impart the practical skills of conducting and publishing research in order to influence policy and practice.[8] There is an emphasis on publishing in peer-reviewed scientific journals as an indicator of successful completion of planned research [2] and a scientific quality control standard. Publication enhances the credibility of research findings,[9] and provides robust documentation of evidence for policy review. Measuring the real impact of research is a long-term effort, however, the published paper is an important milestone in the journey to achieving impact on the ground. [10]
Between 2009 and 2014, 20 SORT IT courses were completed world-wide. Previous studies on these courses have reported high publication yields and self-reported influence on policy and practice.[11–14] However, to date, there has been no detailed assessment of “where” this research has been done “who” has been trained and who amongst those trained became facilitators. This information would be useful to assess patterns in the capacity being built by SORT IT.
Furthermore, assessing whether research evidence is available in a timely and “openly accessible” manner is important for operational research. [10, 15, 16] Internet downloads and article views are useful parameters with which to judge research access and this assessment has yet to be done.
We aimed to describe SORT IT participant profiles, course outcomes and characteristics of published papers. Specific objectives were to report on the:
- geographic location of the research projects, the socio-demographic profiles of participants, their nationalities and institutional affiliations.
- number and profile of participants who successfully completed the course, published their research and became facilitators on subsequent courses.
- characteristics of publications, including time to publication, immediate open access publication costs, and article views and downloads.
Methods
Design and participants
This was a retrospective cohort study involving all participants (and outcomes) of 20 completed SORT IT courses conducted in Europe (France, Luxembourg and Estonia), Asia (India and Nepal), Africa (Kenya and Ethiopia) and the South Pacific (Fiji). In the courses conducted in France and Luxembourg, most participants were nationals from Africa or Asia. Other courses chose participants from their respective region or country. All reported courses were conducted between 2009 and 2014.
Participant profiles
All applicants submitted a detailed application containing their socio-demographic information, nationality, project location, institutional affiliations, field of work and ideas for a research project. A maximum of 12 candidates, who fulfilled strict selection criteria, were enrolled per course[8]. These criteria were designed to maximize individual and institutional commitment to course completion. Most candidates, who were chosen, were new to research.[17]
SORT IT and criteria for success
The details of SORT IT courses, including participant selection criteria, milestones, mentorship and the metrics of assessment have been described previously.[8] In brief, the course was comprised of three modules conducted over a period of 10 to 12 months: Module 1 (5–6 days) focused on development of a study protocol, leading to Module 2 (5–6 days) on efficient data capture and analysis. Module 3 (5–7 days) focused on writing a manuscript for scientific publication.[10] Specific milestones needed to be achieved if participants were to proceed from one module to the next. A participant was judged to have successfully completed the course if s/he completed all milestones, including submission of a completed manuscript to a peer-reviewed journal within four weeks of completing Module 3. Milestone achievement was closely monitored by course coordinators.
Successful course participants, who showed high motivation, active participation and quick learning skills as judged by senior facilitators at each course, were invited to participate in future courses as facilitators. A dedicated SORT IT facilitator inventory was used to compile data on the numbers of former participants who became facilitators on subsequent SORT IT courses.
Publication outputs and characteristics
Publication outputs were assessed and reported by course coordinators on a quarterly basis and they included original articles and viewpoint articles. Viewpoint articles stemming directly from iterative discussions within the courses and that challenged “business as usual” were included. For manuscripts, the date of initial and subsequent submission to any journal, name of the journal and eventual date of publication were recorded. This allowed calculation of “time to publication”.
Information on whether a given publication was immediate open access, closed or delayed open access was sought from the journal website on the censor date. Immediate open access publishing costs were based on journal rates.
The number of publication downloads were sourced from the journal websites or Ingenta publishing (if an Ingenta journal). Citation counts were retrieved from Google Scholar. Cumulative publication outputs were assessed for the 20 courses, which were completed between March 1st 2010 and December 30th 2014 and censored on 1st July 2015.
Data collection and analysis
Information on socio-demographic characteristics of participants, course outcomes, publication outputs and new facilitators were obtained from participant application forms and/or course files. EpiData software was used for data entry (version 3.1 EpiData Association, Odense, Denmark) and STATA (Version 11, CDC, Atlanta) for analysis (S1 Variables, S1 Dataset). Maps were generated using the StatPlanet software, (http://www.statsilk.com/license) which is available free-of-charge for non-commercial use.
Ethics
The study met the Médecins Sans Frontières Ethics Review Board (Geneva, Switzerland) approved criteria for studies of routinely collected data. The Ethics Advisory Group of the International Union Against Tuberculosis and Lung Disease (Paris, France) also exempted the study from ethics review. This study was a review of records and involved anonymized and de-identified data prior to analysis. Hence, the need for individual informed consent was waived by the named ethics committees.
Results
Geographical location and research projects
There were a total of 20 completed SORT IT courses lasting between seven and 12 months (mean = 9.7 months) between protocol development and manuscript completion. Course locations were Europe -Paris (4), Luxembourg (3), Tallinn (1); Africa-Nairobi (1) and Addis Ababa; (2), Asia- Kathmandu (3), Chennai (2), Hyderabad (1); and South Pacific- Fiji (3). A total of 236 participants were enrolled with research projects from 59 countries.
Fig 1 shows the location of SORT IT operational research projects. Of note, research projects were conducted in 59 countries, the great majority located in LMICs. [18]
Socio-demographic characteristics and institutional affiliations
Of 236 participants (41% female, age 27–61 years), there were 64 nationalities including: 19 countries from Africa, 16 from Asia, nine from Eastern Europe, eight from the North and South Pacific, seven from Western Europe, three from Latin America, and one each from USA and Australia respectively. Individuals with nationalities from Western Europe, USA and Australia were all from institutional headquarters and directly involved with research projects in LMICs.
Table 1 shows the sociodemographic profile, work sites and institutional affiliations of participants. Most (68%) were medical care providers (doctors, nurses or clinical officers) working within public health programs of Ministries of Health (MOH) or with non-governmental organizations (NGOs). Occupations outside the health sector included education, agriculture, and social welfare. Participants working with vulnerable populations comprised 5% of the cohort.
Course completion, publication outputs and subsequent facilitation by participants
Course completion and publication outputs are shown in Fig 2. Of 236 participants, 213 (90%) completed all milestones (“successful course completion”). There were 23 (10%) participants who did not complete the course and were considered “unsuccessful”. The main reasons for this unfavorable outcome included: failure to collect or analyze data (6); changed jobs or left for further studies abroad (6); did not complete a paper submission (5); and did not obtain ethics clearance (3) (Fig 2).
A total of 228 manuscripts were submitted to peer-reviewed scientific journals (Fig 2). Of these, 197 (86%) were either published (186) or in press (11) on the censor date.
Of the 213 participants who achieved successful course completion, 41(19%) became new facilitators on subsequent courses (mean number of modules facilitated = 5.7, range 1–29). The average number of facilitators generated per course was two. Table 2 shows the characteristics of these new facilitators. Most (88%) were from LMICs and 37% were female. The most frequent institutional affiliations were Ministry of Health (37%) and International NGOs (34%). Participant facilitators comprised 25 nationalities including 16(39%) from Asia, 10 (24%) from Africa, five (12%) from East Europe, four (10%) from West Europe and two (5%) each from the Pacific and USA and one each from Latin America and Australia.
Of new facilitators, 14 (34%) were from the Union and MSF and they undertook 127 (57%) of 224 subsequent facilitations. Thirteen (nine from LMICs) of the Union-MSF facilitators were operational research fellows, who are obliged (as part of their terms of reference) to be involved in research teaching.
Characteristics of published papers
A total of 186 papers were published in 23 peer-reviewed scientific journals after a median time of 5.7 months from first submission (Inter-quartile range = 4.3–7.7, range 1–46 months).
The six most common journals were Public Health Action (PHA, 58%), PLoS ONE (16%), Transactions of the Royal Society of Tropical Medicine and Hygiene (5%), International Journal of Tuberculosis and Lung Disease (4%) and Biomed Central Journals (4%). For the 22 of the 23 indexed journals (PHA was not yet but scheduled to be indexed at the time of analysis), journal impact factors ranged from 0.5 to 4.4.
Table 3 shows the characteristics of published papers. In 160 (86%) of 186 publications, the principal investigator (and first author) was from a LMIC. Publications covered 21 disease categories, the most common being tuberculosis (TB) and HIV/AIDS.
Internet article views, downloads and citations
Published papers (n-186) had a cumulative total of 74,802 article views (mean = 402, range 0–5512), a cumulative total of 19,992 downloads (mean = 107, range 0–1092), and 504 citations.
Of 155 (82%) open access publications, there were 87,810 (563 per published paper) cumulative article views and downloads compared to 6,402 for 24 closed access publications (267 per published paper).
Open access costs
The total cost of 155 immediate open access publications was 226,115 Euros (average cost per paper = 1,459 Euros, range = 150–4500) (Table 3). The least expensive journal was the Pan African Medical Journal while the most expensive was Lancet Global Health.
Fig 3 shows the trend in open access publications for the period 2009 to 2013 (2014 was excluded as there were only four publications at censor date). A progressive and linear trend was noted, reaching ≥80% in later years. (Chi-square for trend = 13.94, P<0.001)
Discussion
This is the first study to assess the “where” and “who” of the SORT IT global capacity building initiative in operational research. Most participants were from a wide range of LMICs, and almost one-in-five become facilitators in subsequent courses. There were high publication outputs, in a timely manner, with the majority in immediate open access journals.
These findings are of importance for two reasons. First, a few years ago, Ian Chalmers and Paul Glasziou, estimated that close to 85% of all research investment was being wasted.[19, 20] Their message was that medical research is wasteful if it is not published and if it does not contribute to improvements in health care delivery. As the SORT IT program demands research completion and publication, it reduces research waste. Furthermore, two other studies have already shown that between 55% to 74% of SORT IT publications had an influence on policy and/or practice. [12, 21] Second, SORT IT focuses on participants from (or directly involved with) LMICs, where the greatest research gaps exist and where it is vital to improve research capacity. [18]
The study strengths are that information on all participants, their milestone achievements and publication outputs were carefully archived, allowing rigorous tracking of participants and publication outputs. A limitation is that the follow-up time was different for the various courses and we may have underestimated the publication rate and the number of article views and downloads. Of note, the Journal, Public Health Action, was not accessible through PubMed at the time of censoring, and being the choice for over half of all published papers, its absence may have underestimated the reported article views and downloads. In addition, we only recorded facilitation on SORT IT courses and may have not counted contributions to other courses, thereby underestimated this parameter.
This study reports a number of important findings. First, the great majority of participants were from public health programs of MoH or NGOs, which represents a step towards implementers embracing research as part of their work. Furthermore, roughly one-in-five participants became facilitators on subsequent SORT IT courses, thereby demonstrating improved research and training capacity. This finding is further endorsed by two other studies, [11, 22] showing that research output continued well beyond SORT IT course completion with about 50% of participants completing new research studies and 40% getting published.
Second, although the majority of participants were from the traditional health sector, there were a few participants that worked with vulnerable and often excluded groups such as commercial sex workers, and drug users. As well, there was one participant each from non-health sectors (education, agriculture and social science) which is encouraging. In addition, four in ten successful participants were women and this is important in terms of gender equity. Streamlining the SORT IT “way of learning” to mesh with the new Sustainable Development Goals (SDGs) may help expand the use of the SORT IT model to other domains. [23] The fact that publications covered a wide spectrum of disease categories suggests that the approach could be applied to a wider range of health systems.
Third, the average time from first submission to publication was less than six months, which is timely for making relevant evidence available to public health programs. The relatively short time period is due to two principal factors–choice of journals which have quicker turn-around times and are supportive of operational research and sustained support by experienced mentors during the peer-review process and re-submission. The fact that the majority of publications were in categories of TB and HIV/AIDS is also related to data availability–both disease control programs have structured quarterly reporting and monitoring systems which facilitate operational research. As individuals and countries evolve in their basic capacity to conduct and publish operational research, the need to diversify the portfolio of disease categories and take on more sophisticated research designs is evident.
Fourth, although it is encouraging that about one in five participants became facilitators, there is no yardstick with which to compare this outcome. A proportion of almost 20% becoming facilitators is not unreasonable given the intense nature of the course and the need for facilitators to commit for several months (or years) until publication, SORT IT is unlike other training courses where facilitators come for a week or two, teach and then leave “hands-free” with no post-course mentorship commitment. SORT IT facilitators need to have a combination of several qualities including technical competence, an interest for teaching and high motivation for hard and largely pro-bono work. Importantly, most course participants came from busy and overworked public health programs and they already find difficulty getting dedicated time away from their usual workload to complete their SORT IT course requirements. Thus, they may have even less opportunity to become facilitators. Another factor hindering availability may be socio-cultural factors preventing some female facilitators from travelling to distant course locations. Therefore, “interest and competence” in facilitation may not necessarily translate into “ability” and “willingness” to do so. We know of several participants who were competent to facilitate, but were unable or unwilling to commit to this role.
A considerable proportion of all mentoring was done by facilitators from two NGOs (The Union and MSF) that were pioneers in its development and have continued their institutional commitment to SORT IT. These two institutions have introduced innovative operational research fellowships which included facilitation as part of individual capacity development.
With roughly two participants being identified as potential facilitators on each SORT IT course, it will take about three to four years for a country to reach the current minimum SORT IT requirement of 6–8 facilitators for 10–12 participants. Possible ways for generating new SORT IT facilitators may include: formulating objective criteria for identifying potential facilitators on all courses; pairing senior and junior facilitators to enhance on-the-job learning; building facilitator pools around countries or regional courses; negotiating with institutions to allow dedicated time for facilitation; providing nominal honorariums; and extending the operational research fellowship program to public health programs. We also need to identify and bring in experienced operational researchers with good understanding of public health programs from outside the SORT IT network to facilitate at country level.
Finally, in order to foster dissemination of operational research studies, and to meet WHO requirements, we chose “open access” journals for publication wherever possible. The benefit of this approach is that it fosters access and dissemination of published evidence in LMICs, without cost to readers. A significant down side is the high cost for open access which highlights the need for better planning and dedicated funding.[16]
In conclusion, in a wide range of LMICs, where operational research is much needed, the SORT IT model contributed to building research capacity and produced relevant evidence. There was a very high publication rate. Most participants were from LMICs, 40% were women, and their studies were focused on programmatic issues; almost 20% went on to become facilitators in other courses. The SORT IT model deserves to be taken up and expanded further to address the urgent health needs at program level and contribute towards achieving universal health coverage.
Supporting Information
S1 Variables. Documentation sheet.
Variables and codes.
https://doi.org/10.1371/journal.pone.0160837.s002
(DOCX)
Acknowledgments
We thank the facilitators and organizers of all courses for their motivation and dedication to teaching and mentorship. We are also very grateful to WHO/TDR, WHO Regional and Country Offices, Bloomberg Philanthropies, the Department for International Development (DFID) UK, USAID, MSF, The Union and Ministries of Health for supporting the various courses. This paper is an output of the Structured Operational Research and Training Initiative (SORT IT), a global partnership led by the Special Program for Research and Training in Tropical Diseases at the World Health Organization (WHO/TDR), Geneva, Switzerland.
Author Contributions
- Conceived and designed the experiments: RZ SR SDB NG KB PD AJR AMVK PLO JCR ADH AR.
- Performed the experiments: RZ SR AJR AMVK PLO JCR ADH AR.
- Analyzed the data: RZ SR SDB NG KB PD AJR AMVK PLO JCR ADH AR.
- Wrote the paper: RZ SR SDB NG KB PD AJR AMVK PLO JCR ADH AR.
- Revisions: RZ ADH AR.
References
- 1.
WHO. The world health report 2013: research for universal health coverage. World Health Organization, Geneva. Available:http://apps.who.int/iris/bitstream/10665/85761/2/9789240690837_eng.pdf; (Accessed 13th March 2014).
- 2. Rujumba J, Byamugisha R. Publishing operational research from 'real life' programme data: a better form of accountability. Trop Med Int Health. 2011. Epub 2011/09/29. pmid:21951461.
- 3. McKee M, Stuckler D, Basu S. Where there is no health research: what can be done to fill the global gaps in health research? PLoS Med. 2012;9(4):e1001209. Epub 2012/05/01. PMEDICINE-D-12-00103 [pii]. pmid:22545025; PubMed Central PMCID: PMC3335864.
- 4. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol. 2009;114(6):1341–5. Epub 2009/11/26. 00006250-200912000-00028 [pii]. pmid:19935040.
- 5. Walley J, Khan MA, Shah SK, Witter S, Wei X. How to get research into practice: first get practice into research. Bull World Health Organ. 2007;85(6):424. Epub 2007/07/20. doi: S0042-96862007000600004 [pii]. pmid:17639235; PubMed Central PMCID: PMC2636352.
- 6. Pappaioanou M, Malison M, Wilkins K, Otto B, Goodman RA, Churchill RE, et al. Strengthening capacity in developing countries for evidence-based public health: the data for decision-making project. Soc Sci Med. 2003;57(10):1925–37. Epub 2003/09/23. doi: S0277953603000583 [pii]. pmid:14499516.
- 7. Zachariah R, Harries AD, Ishikawa N, Rieder HL, Bissell K, Laserson K, et al. Operational research in low-income countries: what, why, and how? Lancet Infect Dis. 2009;9(11):711–7. Epub 2009/10/24. doi: S1473-3099(09)70229-4 [pii] pmid:19850229.
- 8. Ramsay A, Harries AD, Zachariah R, Bissell K, Hinderaker SG, Edginton M, et al. The Structured Operational Research and Training Initiative for public health programmes. Public Health Action. 2014;4(2):79–84. pmid:26399203
- 9. Zachariah R, Tayler-Smith K, Ngamvithayapong-Yana J, Ota M, Murakami K, Ohkado A, et al. The published research paper: is it an important indicator of successful operational research at programme level? Trop Med Int Health. 2010;15(11):1274–7. Epub 2010/10/27. pmid:20976874.
- 10. Zachariah R, Ford N, Maher D, Bissell K, Van den Bergh R, van den Boogaard W, et al. Is operational research delivering the goods? The journey to success in low-income countries. Lancet Infect Dis. 2012. doi: S1473-3099(11)70309-7 [pii] pmid:22326018.
- 11. Guillerm N, Tayler Smith K, Berger SD, Bissell K, Kumar AMV, Ramsay A, et al. What happens after participants complete a Union-MSF structured operational research training course? Public Health Action. 2014;4(2):89–95. pmid:26399205
- 12. Zachariah R, Guillerm N, Berger S, Kumar AM, Satyanarayana S, Bissell K, et al. Research to policy and practice change: is capacity building in operational research delivering the goods? Trop Med Int Health. 2014. Epub 2014/06/10. pmid:24909292.
- 13. Atia AM, Mumina A, Tayler-Smith K, Boulle P, Alcoba G, Elhag MS, et al. Sodium Stibogluconate and Paromomycin for treating visceral leishmaniasis under routine conditions in eastern Sudan: when will we get a better treatment? Trop Med Int Health. 2015. Epub 2015/10/02. pmid:26427033.
- 14. Bissel K, Harries AD, Reid AJ, Edginton M, Hinderaker SG, Satyanarayana S, et al. Operational research training: the course and beyond Public Health Action. 2012;2(3):92–7. pmid:26392960
- 15. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20. Epub 2011/12/20. pmid:22179294.
- 16. Zachariah R, Kumar AMV, Reid AJ, Van den Bergh R, Isaakidis P, Draguez B, et al. Open access for operational research publications from low and middle-income countries: who pays? Public Health Action. 2014;4(3):141–4.
- 17.
Melin G, Allinson R, Grudin M, Mostert B, Ploeg M, Sadeski F, et al. Evaluation of the Developing Operational Research Capacity in the Health Sector Project. Final Report. Technopolis Group, Stockholm, Sweden. Available: http://www.technopolis-group.com/?report=evaluation-developing-operational-research-capacity-health-sector-project (Accessed 16th June 2016)2015.
- 18.
COHRED. Council on health research for development.Geneva, Switzerland. Available: http://www.cohred.org/our-mission/ (Accessed 20th January 2016) 2013.
- 19. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9. Epub 2009/06/16. pmid:19525005.
- 20. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76. Epub 2014/01/15. pmid:24411647.
- 21. Kumar AMV, Shewade HD, Tripathy JP, Guillerm N, Tayler-Smith K, Dar Berger S, et al. Does research through Structured Operational Research and Training (SORT IT) courses impact policy and practice? Public Health Action. 2016;1(1–6).
- 22. Guillerm N, Tayler-Smith K, Dar Berger S, Bissell K, Kumar AM, Ramsay A, et al. Research output after participants complete a Structured Operational Research and Training (SORT IT) course. Public Health Action. 2015;5(4):266–8. Epub 2016/01/15. pmid:26767182; PubMed Central PMCID: PMC4682620.
- 23.
UN. Sustainable Develpment Goals. Transforming Our World—the 2030 Agenda for Sustainable Development. Available: https://sustainabledevelopment.un.org/?menu=1300 (Accessed 20th january 2016)2015.