Top

Published in:

Open Access 01-12-2017 | Research article

Testing of the assisting software for radiologists analysing head CT images: lessons learned

Authors: Petr Martynov, Nikolai Mitropolskii, Katri Kukkola, Monika Gretsch, Vesa-Matti Koivisto, Ilkka Lindgren, Jani Saunavaara, Jarmo Reponen, Anssi Mäkynen

Published in: BMC Medical Imaging | Issue 1/2017

Abstract

Background

Assessing a plan for user testing and evaluation of the assisting software developed for radiologists.

Methods

Test plan was assessed in experimental testing, where users performed reporting on head computed tomography studies with the aid of the software developed. The user testing included usability tests, questionnaires, and interviews. In addition, search relevance was assessed on the basis of user opinions.

Results

The testing demonstrated weaknesses in the initial plan and enabled improvements. Results showed that the software has acceptable usability level but some minor fixes are needed before larger-scale pilot testing. The research also proved that it is possible even for radiologists with under a year’s experience to perform reporting of non-obvious cases when assisted by the software developed. Due to the small number of test users, it was impossible to assess effects on diagnosis quality.

Conclusions

The results of the tests performed showed that the test plan designed is useful, and answers to the key research questions should be forthcoming after testing with more radiologists. The preliminary testing revealed opportunities to improve test plan and flow, thereby illustrating that arranging preliminary test sessions prior to any complex scenarios is beneficial.

Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B. Content-based image retrieval in radiology: current status and future directions. J Digit Imaging. 2011; doi:10.1007/s10278-010-9290-9.

Aisen AM, Broderick LS, Winer-Muram H, Brodley CE, Kak AC, Pavlopoulou C, Dy J, Shyu CR, Marchiori A. Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology. 2003; doi:10.1148/radiol.2281020126.

Moltz JH, D’Anastasi M, Kiessling A, Pinto dos Santos D, Schülke C, Peitgen HO. Workflow-centred evaluation of an automatic lesion tracking software for chemotherapy monitoring by CT. Eur Radiol. 2012; doi:10.1007/s00330-012-2545-8.

de Oliveira JE, Machado AM, Chavez GC, Lopes AP, Deserno TM, Araújo Ade A. MammoSys: a content-based image retrieval system using breast density patterns. Comput Methods Prog Biomed. 2010; doi:10.1016/j.cmpb.2010.01.005.

Cho HC, Hadjiiski L, Sahiner B, Chan HP, Helvie M, Paramagul C, Nees AV. Similarity evaluation in a content-based image retrieval (CBIR) CADx system for characterization of breast masses on ultrasound images. Med Phys. 2011; doi:10.1118/1.3560877.

Hsu W, Antani S, Long LR, Neve L, Thoma GRSPIRS, Web-based A. Image retrieval system for large biomedical databases. Int J Med Inform. 2009;78(Suppl. 1):S13–24. doi:10.1016/j.ijmedinf.2008.09.006. CrossRefPubMed

Avni U, Greenspan H, Konen E, Sharon M, Goldberger J. X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. IEEE Trans Med Imaging. 2011; doi:10.1109/TMI.2010.2095026.

Geldermann I, Grouls C, Kuhl C, Deserno TM, Spreckelsen C. Black box integration of computer-aided diagnosis into PACS deserves a second chance: results of a usability study concerning bone age assessment. J Digit Imaging. 2013; doi:10.1007/s10278-013-9590-y.

Kumar A, Kim J, Bi L, Fulham M, Feng D. Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images. Int J Comput Assist Radiol Surg. 2013; doi:10.1007/s11548-013-0896-5.

10.

Mourão A, Martins F, Magalhães J. Multimodal medical information retrieval with unsupervised rank fusion. Comput Med Imaging Graph. 2014; doi:10.1016/j.compmedimag.2014.05.006.

11.

Markonis D, Holzer M, Baroz F, De Castaneda RLR, Boyer C, Langs G, Müller H. User-oriented evaluation of a medical image retrieval system for radiologists. Int J Med Inform. 2015; doi:10.1016/j.ijmedinf.2015.04.003.

12.

Ballerini L, Li X, Fisher RB, Rees JA. Query-by-example content-based image retrieval system of non-melanoma skin lesions. Medical content-based retrieval for clinical decision support 2009. Lect Notes Comput Sci. 2010;5853:31–8.CrossRef

13.

Shyr C, Kushniruk A, Wasserman WW. Usability study of clinical exome analysis software: top lessons learned and recommendations. J Biomed Inform. 2014; doi:10.1016/j.jbi.2014.05.004.

14.

Martynov P, Mitropolskii N, Kukkola K, Mutanen L, Reponen J, Mäkynen A. CARDS: The decision support tool for radiologists examining head CT images. ECR 2016 / B-0233. 2016; doi:10.1594/ecr2016/B-0233.

15.

Sivic J, Zisserman A. Video Google: a text retrieval approach to object matching in videos. International Conference on Computer Vision. 2003;2:1470–7.

16.

Lux M, Marques O. Visual information retrieval using java and LIRE. Synthesis Lectures on Information Concepts, Retrieval, and Services. 2013;5(1) doi:10.2200/S00468ED1V01Y201301ICR025.

17.

Brooke JSUS. A ‘quick and dirty’ usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, editors. Usability evaluation in industry. London: Taylor & Francis; 1996. p. 189–94.

18.

Lund AM. Measuring usability with the USE questionnaire. STC SIG newsletter, usability. Interface. 2001;8(2):3–6.

19.

Brooke JSUS. A retrospective. J Usability Stud. 2013;8(2):29–40.

Title: Testing of the assisting software for radiologists analysing head CT images: lessons learned
Authors: Petr Martynov
Nikolai Mitropolskii
Katri Kukkola
Monika Gretsch
Vesa-Matti Koivisto
Ilkka Lindgren
Jani Saunavaara
Jarmo Reponen
Anssi Mäkynen
Publication date: 01-12-2017
Publisher: BioMed Central
Published in: BMC Medical Imaging / Issue 1/2017
Electronic ISSN: 1471-2342
DOI: https://doi.org/10.1186/s12880-017-0229-1

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Testing of the assisting software for radiologists analysing head CT images: lessons learned

Abstract

Background

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2017

Cardiovascular magnetic resonance in the evaluation of heart valve disease

Distal radius plate of CFR-PEEK has minimal effect compared to titanium plates on bone parameters in high-resolution peripheral quantitative computed tomography: a pilot study

T2 relaxation time for intervertebral disc degeneration in patients with upper back pain: initial results on the clinical use of 3.0 Tesla MRI

Diffusion tensor MR imaging characteristics of cerebral white matter development in fetal pigs

Time-resolved tracking of the atrioventricular plane displacement in Cardiovascular Magnetic Resonance (CMR) images

Evaluating segmental liver function using T1 mapping on Gd-EOB-DTPA-enhanced MRI with a 3.0 Tesla