Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2014

Open Access 01-12-2014 | Research article

A critical analysis of test-retest reliability in instrument validation studies of cancer patients under palliative care: a systematic review

Authors: Carlos Eduardo Paiva, Eliane Marçon Barroso, Estela Cristina Carneseca, Cristiano de Pádua Souza, Felipe Thomé dos Santos, Rossana Verónica Mendoza López, Sakamoto Bianca Ribeiro Paiva

Published in: BMC Medical Research Methodology | Issue 1/2014

Login to get access

Abstract

Background

Patient-reported outcome validation needs to achieve validity and reliability standards. Among reliability analysis parameters, test-retest reliability is an important psychometric property. Retested patients must be in a clinically stable condition. This is particularly problematic in palliative care (PC) settings because advanced cancer patients are prone to a faster rate of clinical deterioration. The aim of this study was to evaluate the methods by which multi-symptom and health-related qualities of life (HRQoL) based on patient-reported outcomes (PROs) have been validated in oncological PC settings with regards to test-retest reliability.

Methods

A systematic search of PubMed (1966 to June 2013), EMBASE (1980 to June 2013), PsychInfo (1806 to June 2013), CINAHL (1980 to June 2013), and SCIELO (1998 to June 2013), and specific PRO databases was performed. Studies were included if they described a set of validation studies. Studies were included if they described a set of validation studies for an instrument developed to measure multi-symptom or multidimensional HRQoL in advanced cancer patients under PC. The COSMIN checklist was used to rate the methodological quality of the study designs.

Results

We identified 89 validation studies from 746 potentially relevant articles. From those 89 articles, 31 measured test-retest reliability and were included in this review. Upon critical analysis of the overall quality of the criteria used to determine the test-retest reliability, 6 (19.4%), 17 (54.8%), and 8 (25.8%) of these articles were rated as good, fair, or poor, respectively, and no article was classified as excellent. Multi-symptom instruments were retested over a shortened interval when compared to the HRQoL instruments (median values 24 hours and 168 hours, respectively; p = 0.001). Validation studies that included objective confirmation of clinical stability in their design yielded better results for the test-retest analysis with regard to both pain and global HRQoL scores (p < 0.05). The quality of the statistical analysis and its description were of great concern.

Conclusion

Test-retest reliability has been infrequently and poorly evaluated. The confirmation of clinical stability was an important factor in our analysis, and we suggest that special attention be focused on clinical stability when designing a PRO validation study that includes advanced cancer patients under PC.
Appendix
Available only for authorised users
Literature
1.
Ferrell B, Paice J, Koczywas M: New standards and implications for improving the quality of supportive oncology practice. J Clin Oncol. 2008, 26 (23): 3824-3831. 10.1200/JCO.2007.15.7552.CrossRefPubMed
2.
Frost MH, Reeve BB, Liepa AM, Stauffer JW, Hays RD, Mayo FDAP-ROCMG: What is sufficient evidence for the reliability and validity of patient-reported outcome measures?. Value Health. 2007, 10 (Suppl 2): S94-S105.CrossRefPubMed
3.
Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E, Stein RE: Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002, 11 (3): 193-205. 10.1023/A:1015291021312.CrossRefPubMed
4.
Richardson LA, Jones GW: A review of the reliability and validity of the Edmonton Symptom Assessment System. Curr Oncol. 2009, 16 (1): 55-PubMedPubMedCentral
5.
6.
7.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC: The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010, 19 (4): 539-549. 10.1007/s11136-010-9606-8.CrossRefPubMedPubMedCentral
8.
Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC: Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012, 21 (4): 651-657. 10.1007/s11136-011-9960-1.CrossRefPubMed
9.
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, Bouter LM, de Vet HC: Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007, 60 (1): 34-42. 10.1016/j.jclinepi.2006.03.012.CrossRefPubMed
10.
Axelsson B, Sjoden PO: Assessment of quality of life in palliative care–psychometric properties of a short questionnaire. Acta Oncol. 1999, 38 (2): 229-237. 10.1080/028418699431663.CrossRefPubMed
11.
Cohen SR, Mount BM: Living with cancer: “good” days and “bad” days–what produces them? Can the McGill quality of life questionnaire distinguish between them?. Cancer. 2000, 89 (8): 1854-1865. 10.1002/1097-0142(20001015)89:8<1854::AID-CNCR28>3.0.CO;2-C.CrossRefPubMed
12.
Llobera J, Esteva M, Benito E, Terrasa J, Rifa J, Pons O, Maya A: Quality of life for oncology patients during the terminal period. Validation of the HRCA-QL index. Support Care Cancer. 2003, 11 (5): 294-303.PubMed
13.
Mystakidou K, Tsilika E, Kouloulias V, Parpa E, Katsouda E, Kouvaris J, Vlahos L: The “Palliative Care Quality of Life Instrument (PQLI)” in terminal cancer patients. Health Qual Life Outcomes. 2004, 2: 8-10.1186/1477-7525-2-8.CrossRefPubMedPubMedCentral
14.
Steinhauser KE, Clipp EC, Bosworth HB, McNeilly M, Christakis NA, Voils CI, Tulsky JA: Measuring quality of life at the end of life: validation of the QUAL-E. Palliat Support Care. 2004, 2 (1): 3-14.CrossRefPubMed
15.
Lopes Ferreira P, Pinto Barros A: Measuring quality of life in palliative care. Acta Med Port. 2008, 21 (2): 111-124.PubMed
16.
Suarez-del-Real Y, Allende-Perez S, Alferez-Mancera A, Rodriguez RB, Jimenez-Toxtle S, Mohar A, Onate-Ocana LF: Validation of the Mexican-Spanish version of the EORTC QLQ-C15-PAL questionnaire for the evaluation of health-related quality of life in patients on palliative care. Psychooncology. 2011, 20 (8): 889-896. 10.1002/pon.1801.CrossRefPubMed
17.
Leppert W, Majkowicz M: Validation of the Polish version of the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire - Core 15 - Palliative Care in patients with advanced cancer. Palliat Med. 2013, 27 (5): 470-477. 10.1177/0269216312458823.CrossRefPubMed
18.
Kim KU, Yoon SJ, Lee JL, Ahn HS, Park HJ, Lee SI, Jo MW: Validation of the Korean version of the McMaster Quality of Life Scale in terminal cancer patients. J Palliat Care. 2006, 22 (1): 40-45.PubMed
19.
Kim SH, Choi YS, Lee J, Oh SC, Yeom CH, Lee MA, Kim DG, Moon Do H, Kim DY, Koh SJ: Reliability and validity of the Hospice Quality of Life Scale for Korean cancer patients. J Pain Symptom Manage. 2009, 37 (2): 156-167. 10.1016/j.jpainsymman.2008.02.008.CrossRefPubMed
20.
Serra-Prat M, Nabal M, Santacruz V, Picaza JM, Trelis J, Grupo Catalan de Estudio de la Efectividad de los Cuidados P: Validation of the Spanish version of the palliative care outcome scale. Med Clin. 2004, 123 (11): 406-412.CrossRef
21.
Eisenchlas JH, Harding R, Daud ML, Perez M, De Simone GG, Higginson IJ: Use of the palliative outcome scale in Argentina: a cross-cultural adaptation and validation study. J Pain Symptom Manage. 2008, 35 (2): 188-202. 10.1016/j.jpainsymman.2007.02.045.CrossRefPubMed
22.
Guo H, Fine PG, Mendoza TR, Cleeland CS: A preliminary study of the utility of the brief hospice inventory. J Pain Symptom Manage. 2001, 22 (2): 637-648. 10.1016/S0885-3924(01)00296-2.CrossRefPubMed
23.
Lo RS, Woo J, Zhoc KC, Li CY, Yeo W, Johnson P, Mak Y, Lee J: Cross-cultural validation of the McGill Quality of Life questionnaire in Hong Kong Chinese. Palliat Med. 2001, 15 (5): 387-397. 10.1191/026921601680419438.CrossRefPubMed
24.
Shahidi J, Khodabakhshi R, Gohari MR, Yahyazadeh H, Shahidi N: McGill Quality of Life Questionnaire: reliability and validity of the Persian version in Iranian patients with advanced cancer. J Palliat Med. 2008, 11 (4): 621-626. 10.1089/jpm.2007.0173.CrossRefPubMed
25.
Harding R, Selman L, Agupio G, Dinat N, Downing J, Gwyther L, Mashao T, Mmoledi K, Moll T, Sebuyira LM, et al: Validation of a core outcome measure for palliative care in Africa: the APCA African Palliative Outcome Scale. Health Qual Life Outcomes. 2010, 8: 10-10.1186/1477-7525-8-10.CrossRefPubMedPubMedCentral
26.
Hearn J, Higginson IJ: Development and validation of a core outcome measure for palliative care: the palliative care outcome scale. Palliative Care Core Audit Project Advisory Group. Qual Health Care. 1999, 8 (4): 219-227. 10.1136/qshc.8.4.219.CrossRefPubMedPubMedCentral
27.
Wilson KG, Graham ID, Viola RA, Chater S, de Faye BJ, Weaver LA, Lachance JA: Structured interview assessment of symptoms and concerns in palliative care. Can J Psychiatry. 2004, 49 (6): 350-358.PubMed
28.
Agra Y, Badia X: Spanish version of the Rotterdam Symptom Check List: cross-cultural adaptation and preliminary validity in a sample of terminal cancer patients. Psychooncology. 1998, 7 (3): 229-239. 10.1002/(SICI)1099-1611(199805/06)7:3<229::AID-PON302>3.0.CO;2-R.CrossRefPubMed
29.
Chang VT, Hwang SS, Feuerman M: Validation of the Edmonton Symptom Assessment Scale. Cancer. 2000, 88 (9): 2164-2171. 10.1002/(SICI)1097-0142(20000501)88:9<2164::AID-CNCR24>3.0.CO;2-5.CrossRefPubMed
30.
Stiel S, Matthes ME, Bertram L, Ostgathe C, Elsner F, Radbruch L: Validation of the new version of the minimal documentation system (MIDOS) for patients in palliative care : the German version of the edmonton symptom assessment scale (ESAS). Schmerz. 2010, 24 (6): 596-604. 10.1007/s00482-010-0972-5.CrossRefPubMed
31.
Moro C, Brunelli C, Miccinesi G, Fallai M, Morino P, Piazza M, Labianca R, Ripamonti C: Edmonton symptom assessment scale: Italian validation in two palliative care settings. Support Care Cancer. 2006, 14 (1): 30-37. 10.1007/s00520-005-0834-3.CrossRefPubMed
32.
Mystakidou K, Cleeland C, Tsilika E, Katsouda E, Primikiri A, Parpa E, Vlahos L, Mendoza T: Greek M.D. Anderson symptom inventory: validation and utility in cancer patients. Oncology. 2004, 67 (3–4): 203-210.CrossRefPubMed
33.
Pautex S, Berger A, Chatelain C, Herrmann F, Zulian GB: Symptom assessment in elderly cancer patients receiving palliative care. Crit Rev Oncol Hematol. 2003, 47 (3): 281-286. 10.1016/S1040-8428(03)00043-X.CrossRefPubMed
34.
Stiel S, Pollok A, Elsner F, Lindena G, Ostgathe C, Nauck F, Radbruch L: Validation of the symptom and problem checklist of the German Hospice and Palliative Care Evaluation (HOPE). J Pain Symptom Manage. 2012, 43 (3): 593-605. 10.1016/j.jpainsymman.2011.04.021.CrossRefPubMed
35.
Carvajal A, Centeno C, Watson R, Bruera E: A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer patients. Eur J Cancer. 2011, 47 (12): 1863-1872. 10.1016/j.ejca.2011.03.027.CrossRefPubMed
36.
Radbruch L, Sabatowski R, Loick G, Jonen-Thielemann I, Elsner F, Hormann E: MIDOS–validation of a minimal documentation system for palliative medicine. Schmerz. 2000, 14 (4): 231-239. 10.1007/s004820070028.CrossRefPubMed
37.
Kwon JH, Nam SH, Koh S, Hong YS, Lee KH, Shin SW, Hui D, Park KW, Yoon SY, Won JY, et al: Validation of edmonton symptom assessment system in korean patients with cancer. J Pain Symptom Manage. 2013, 46 (6): 947-956. 10.1016/j.jpainsymman.2013.01.012.CrossRefPubMed
38.
Aoun SM, Monterosso L, Kristjanson LJ, McConigley R: Measuring symptom distress in palliative care: psychometric properties of the Symptom Assessment Scale (SAS). J Palliat Med. 2011, 14 (3): 315-321. 10.1089/jpm.2010.0412.CrossRefPubMed
39.
Pereira F, Santos C: Estudo de adaptação cultural e validação da Functional Assessment of Cancer Therapy-General em cuidados paliativos. Rev Enferm Referênci. 2011, 5: 10-
40.
Sterkenburg CA, King B, Woodward CA: A reliability and validity study of the McMaster Quality of Life Scale (MQLS) for a palliative population. J Palliat Care. 1996, 12 (1): 18-25.PubMed
41.
Fanelli D: Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PLoS One. 2010, 5 (4): e10271-10.1371/journal.pone.0010271.CrossRefPubMedPubMedCentral
42.
Guo Y, Logan HL, Glueck DH, Muller KE: Selecting a sample size for studies with repeated measures. BMC Med Res Methodol. 2013, 13: 100-10.1186/1471-2288-13-100.CrossRefPubMedPubMedCentral
43.
Hopkins WG: Measures of reliability in sports medicine and science. Sports Med. 2000, 30 (1): 1-15. 10.2165/00007256-200030010-00001.CrossRefPubMed
44.
45.
Fayers P, Machin D: Quality of Life. Assessment, Analysis and Interpretation. 2007, Chichester: John Wiley, 2
46.
47.
Streiner DL, Norman GR: Health Measurement Scales. A Practical Guide to Their Development and use. 2008, New York: Oxford University Press, 4CrossRef
48.
Jim HS, Small B, Faul LA, Franzen J, Apte S, Jacobsen PB: Fatigue, depression, sleep, and activity during chemotherapy: daily and intraday variation and relationships among symptom changes. Ann Behav Med. 2011, 42 (3): 321-333. 10.1007/s12160-011-9294-9.CrossRefPubMedPubMedCentral
49.
Dimsdale JE, Ancoli-Israel S, Ayalon L, Elsmore TF, Gruen W: Taking fatigue seriously, II: variability in fatigue levels in cancer patients. Psychosomatics. 2007, 48 (3): 247-252. 10.1176/appi.psy.48.3.247.CrossRefPubMed
50.
Vrijman C, Homan ML, Limpens J, van der Veen W, Wolkerstorfer A, Terwee CB, Spuls PI: Measurement properties of outcome measures for Vitiligo: a systematic review. Arch Dermatol. 2012, 148 (11): 1302-1309. 10.1001/archdermatol.2012.3065.CrossRefPubMed
51.
Abma FI, van der Klink JJ, Terwee CB, Amick BC, Bultmann U: Evaluation of the measurement properties of self-reported health-related work-functioning instruments among workers with common mental disorders. Scand J Work Environ Health. 2012, 38 (1): 5-18. 10.5271/sjweh.3190.CrossRefPubMed
52.
Reimers AK, Mess F, Bucksch J, Jekauc D, Woll A: Systematic review on measurement properties of questionnaires assessing the neighbourhood environment in the context of youth physical activity behaviour. BMC Public Health. 2013, 13: 461-10.1186/1471-2458-13-461.CrossRefPubMedPubMedCentral
53.
Werner MU, Petersen MA, Bischoff JM: Test-retest studies in quantitative sensory testing: a critical review. Acta Anaestheriol Scand. 2013, 57 (8): 957-963. 10.1111/aas.12150.CrossRef
54.
Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979, 86 (2): 420-428.CrossRefPubMed
Metadata
Title
A critical analysis of test-retest reliability in instrument validation studies of cancer patients under palliative care: a systematic review
Authors
Carlos Eduardo Paiva
Eliane Marçon Barroso
Estela Cristina Carneseca
Cristiano de Pádua Souza
Felipe Thomé dos Santos
Rossana Verónica Mendoza López
Sakamoto Bianca Ribeiro Paiva
Publication date
01-12-2014
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2014
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-14-8

Other articles of this Issue 1/2014

BMC Medical Research Methodology 1/2014 Go to the issue

Research article

Improving the error rates of the Begg and Mazumdar test for publication bias in fixed effects meta-analysis

Research article

Network meta-analysis of (individual patient) time to event data alongside (aggregate) count data

Research article

A randomized controlled trial of the effectiveness of a pre-recruitment primer letter to increase participation in a study of colorectal screening and surveillance

Research article

Challenges and strategies for cohort retention and data collection in an indigenous population: Australian Aboriginal Birth Cohort

Research article

Testing the treatment effect on competing causes of death in oncology clinical trials

Technical advance

Developing a weighting strategy to include mobile phone numbers into an ongoing population health survey using an overlapping dual-frame design with limited benchmark information