Top

Journal of Imaging Informatics in Medicine

Published in:

01-12-2017

Enabling Real-Time Volume Rendering of Functional Magnetic Resonance Imaging on an iOS Device

Authors: Joseph Holub, Eliot Winer

Published in: Journal of Imaging Informatics in Medicine | Issue 6/2017

Abstract

Powerful non-invasive imaging technologies like computed tomography (CT), ultrasound, and magnetic resonance imaging (MRI) are used daily by medical professionals to diagnose and treat patients. While 2D slice viewers have long been the standard, many tools allowing 3D representations of digital medical data are now available. The newest imaging advancement, functional MRI (fMRI) technology, has changed medical imaging from viewing static to dynamic physiology (4D) over time, particularly to study brain activity. Add this to the rapid adoption of mobile devices for everyday work and the need to visualize fMRI data on tablets or smartphones arises. However, there are few mobile tools available to visualize 3D MRI data, let alone 4D fMRI data. Building volume rendering tools on mobile devices to visualize 3D and 4D medical data is challenging given the limited computational power of the devices. This paper describes research that explored the feasibility of performing real-time 3D and 4D volume raycasting on a tablet device. The prototype application was tested on a 9.7” iPad Pro using two different fMRI datasets of brain activity. The results show that mobile raycasting is able to achieve between 20 and 40 frames per second for traditional 3D datasets, depending on the sampling interval, and up to 9 frames per second for 4D data. While the prototype application did not always achieve true real-time interaction, these results clearly demonstrated that visualizing 3D and 4D digital medical data is feasible with a properly constructed software framework.

Di Salle F, Formisano E, Linden DE, Goebel R, Bonavita S, Pepino A et al.: Exploring brain function with magnetic resonance imaging. Eur J Radiol 30:84–94, 1999 http://www.ncbi.nlm.nih.gov/pubmed/10401589 CrossRefPubMed

Rosen BR, Savoy RL: fMRI at 20: Has it changed the world? Neuroimage 62:1316–1324, 2012. doi: 10.1016/j.neuroimage.2012.03.004 CrossRefPubMed

Bandettini P: Functional MRI today. Int J Psychophysiol 63:138–145, 2007. doi: 10.1016/j.ijpsycho.2006.03.016 CrossRefPubMed

Neuroimaging Informatics Technology Initiative (NIfTI), (2011). http://nifti.nimh.nih.gov

Westover L: Footprint evaluation for volume rendering. ACM SIGGRAPH Comput Graph 24:367–376, 1990. doi: 10.1145/97880.97919 CrossRef

Botsch M, Kobbelt L: High-quality point-based rendering on modern GPUs, 11th Pacific Conf. on Computer Graph. Appl. 2003. Proceedings. (2003). doi:10.1109/PCCGA.2003.1238275.

Neophytou N, Mueller K: GPU accelerated image aligned splatting. Fourth Int. Work. Vol. Graph. 2005:197–242, 2005. doi: 10.1109/VG.2005.194115

Moller T, Machiraju R, Mueller K, Yagel R: Classification and local error estimation of interpolation and derivative filters for volume rendering, in: Proc. 1996 Symp. Vol. Vis., San Fransisco, CA, 1996: pp. 71–78. doi:10.1109/SVV.1996.558045.

Pfister H, Hardenbergh J, Knittel J, Lauer H, Seiler L: The VolumePro Real-Time Ray-casting System, in: Siggraph 1999, Comput. Graph. Proceedings, 1999: pp. 251–260. doi:10.1145/311535.311563.

10.

Wu Y, Bhatia V, Lauer H, Seiler L: Shear-image order ray casting volume rendering, in: Proc. 2003 Symp. Interact. 3D Graph. - SI3D ‘03, 2003: pp. 152–162. doi:10.1145/641508.641510.

11.

Rezk-Salama C, Engel K, Bauer M, Greiner G, Ertl T: Interactive volume on standard PC graphics hardware using multi-textures and multi-stage rasterization, in: Proc. ACM SIGGRAPH/EUROGRAPHICS Work. Graph. Hardw., 2000: pp. 109–118. doi:10.1145/346876.348238.

12.

Engel K, Kraus M, Ertl T: High-quality pre-integrated volume rendering using hardware-accelerated pixel shading, in: Proc. ACM SIGGRAPH/EUROGRAPHICS Work. Graph. Hardw., 2001: pp. 9–16. doi:10.1145/383507.383515.

13.

Pfister H: Hardware-Accelerated Volume Rendering, in: Vis. Handb., Elsevier, 2005: pp. 229–258. doi:10.1016/B978-012387582-2/50013-7.

14.

Weiskopf D: GPU-Based Interactive Visualization Techniques, Springer Berlin Heidelberg, 2006. doi:10.1007/978-3-540-33263-3.

15.

Fox PT, Raichle ME: Focal physiological uncoupling of cerebral blood flow and oxidative metabolism during somatosensory stimulation in human subjects. Proc Natl Acad Sci U S A 83:1140–1144, 1986. doi: 10.1073/pnas.83.4.1140 CrossRefPubMedPubMedCentral

16.

Kaufman A, Mueller K: Overview of volume rendering, in: Vis. Handb., 1st ed., Elsevier Inc., Amsterdam, 2005: pp. 127–174.

17.

Lichtenbelt B, Crane R, Naqvi S: Introduction to volume rendering. Upper Saddle River, NJ: Prentice-Hall, Inc., 1998

18.

Levoy M: Display of surfaces from volume data. IEEE Comput Graph Appl 8:29–37, 1988. doi: 10.1109/38.511 CrossRef

19.

Drebin RA, Carpenter L, Hanrahan P: Volume Rendering, Proc. SIGGRAPH ‘88. Comp Graph 22:65–74, 1988. doi: 10.1145/54852.378484 CrossRef

20.

He T, Hong L, Kaufman A, Pfister H: Generation of transfer functions with stochastic search techniques, in: Proc. 7th Conf. Vis. ‘96, San Fransisco, CA, 1996: pp. 227–234. doi:10.1109/VISUAL.1996.568113.

21.

INMOS: Graphics Databook, 2nd edition. Melksham, Wiltshire: Redwood Press Limited, 1990

22.

Kruger J, Westermann R: Acceleration Techniques for GPU-based Volume Rendering, in: IEEE_VIS, Seattle, Washington, USA, 2003: pp. 38–44. doi:10.1109/VIS.2003.10001.

23.

Whitted T: An improved illumination model for shaded display. ACM SIGGRAPH Comput Graph 13:14, 1979. doi: 10.1145/965103.807419 CrossRef

24.

Levoy M: Efficient ray tracing of volume data. ACM Trans Graph 9:245–261, 1990. doi: 10.1145/78964.78965 CrossRef

25.

Cai W, Sakas G: Data intermixing and multi-volume rendering. Comput Graph Forum 18:359–368, 1999. doi: 10.1111/1467-8659.00356 CrossRef

26.

Manssour IH, Furuie SS, Nedel LP, Freitas CMDS: A framework to visualize and interact with multimodal medical images. Vol Graph:385–398, 2001. doi: 10.1007/978-3-7091-6756-4_27

27.

Beyer J, Hadwiger M, Wolfsberger S, Bühler K: High-quality multimodal volume rendering for preoperative planning of neurosurgical interventions. IEEE Trans Vis Comput Graph 13:1696–1703, 2007. doi: 10.1109/TVCG.2007.70560 CrossRefPubMed

28.

Ferre M, Puig A, Tost D: A framework for fusion methods and rendering techniques of multimodal volume data. Comput Animat Virtual Worlds 15:63–77, 2004. doi: 10.1002/cav.1 CrossRef

29.

Ferré M, Puig A, Tost D: Rendering Techniques for Multimodal Data, in: Proc. SIACG 2002 1st Ibero-American Symp. Comput. Graph., 2002: pp. 305–313

30.

Wilson B, Lum EB, Ma K: Interactive Multi-volume Visualization, in: Comput. Sci. — ICCS 2002, Springer Berlin Heidelberg, 2002: pp. 102–110. doi:10.1007/3-540-46080-2_11.

31.

Noon C, Foo E, Winer E: Interactive GPU volume raycasting in a clustered graphics environment. Med Imaging 2012 Image-Guided Proced Robot Interv Model 8316:1–9, 2012. doi: 10.1117/12.911646

32.

Van Gelder A, Kim K: Direct volume rendering with shading via three-dimensional textures, in: Proc. 1996 Symp. Vol. Vis., Acm, San Fransisco, CA, 1996: p. 23–30,. doi:10.1109/SVV.1996.558039.

33.

DeGusta M: Are Smart Phones Spreading Faster than Any Technology in Human History?, MIT Technol. Rev. (2012)

34.

Noon CJ: A Volume Rendering Engine for Desktops, Laptops, Mobile Devices and Immersive Virtual Reality Systems using GPU-Based Volume Raycasting by, Iowa State University, 2012

35.

Metal - Apple Developer: https://developer.apple.com/metal/, accessed May (2017)

36.

Vulkan - Industry Forged: https://www.khronos.org/vulkan/, accessed May (2017)

37.

Khronos Group, OpenGL ES: https://www.khronos.org/opengles/, accessed May (2017)

38.

Noon C, Holub J, Winer E: Real-time volume rendering of digital medical images on an iOS device, in: C.G.M. Snoek, L.S. Kennedy, R. Creutzburg, D. Akopian, D. Wüller, K.J. Matherson, et al. (Eds.), IS&T/SPIE Electron. Imaging. Int. Soc. Opt. Photonics, 2013. doi:10.1117/12.2005335

39.

Forsberg L: NeuroPub, (2013)

40.

Miller RB, Response time in man-computer conversational transactions, Proc. December 9–11, 1968, Fall Jt. Comput. Conf. Part I. (1968) 267–277. doi:10.1145/1476589.1476628.

41.

Holub J, Winer E, Visualizing fMRI data using volume rendering in Virtual Reality Joseph, in: Interservice/Industry Training, Simulation, Educ. Conf., Orlando, FL, 2015: pp. 1–11.

Title: Enabling Real-Time Volume Rendering of Functional Magnetic Resonance Imaging on an iOS Device
Authors: Joseph Holub
Eliot Winer
Publication date: 01-12-2017
Publisher: Springer International Publishing
Published in: Journal of Imaging Informatics in Medicine / Issue 6/2017
Print ISSN: 2948-2925
Electronic ISSN: 2948-2933
DOI: https://doi.org/10.1007/s10278-017-9986-1

At a glance: The STEP trials

Springer Medicine

Enabling Real-Time Volume Rendering of Functional Magnetic Resonance Imaging on an iOS Device

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 6/2017

Radiologist Digital Workspace Use and Preference: a Survey-Based Study

Computer-Assisted Diagnosis System for Breast Cancer in Computed Tomography Laser Mammography (CTLM)

Redefining the Practice of Peer Review Through Intelligent Automation Part 2: Data-Driven Peer Review Selection and Assignment

Can an Offsite Expert Remotely Evaluate the Visual Estimation of Ejection Fraction via a Social Network Video Call?

Accuracy Validation of an Automated Method for Prostate Segmentation in Magnetic Resonance Imaging

Computer-Aided Diagnosis of Lung Nodules in Computed Tomography by Using Phylogenetic Diversity, Genetic Algorithm, and SVM