Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Journal of NeuroEngineering and Rehabilitation
Published in: Journal of NeuroEngineering and Rehabilitation 1/2019

Open Access 01-12-2019 | Research

Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home

Authors: Jirapat Likitlersuang, Elizabeth R. Sumitro, Tianshi Cao, Ryan J. Visée, Sukhvinder Kalsi-Ryan, José Zariffa

Published in: Journal of NeuroEngineering and Rehabilitation | Issue 1/2019

Login to get access

Abstract

Background

Current upper extremity outcome measures for persons with cervical spinal cord injury (cSCI) lack the ability to directly collect quantitative information in home and community environments. A wearable first-person (egocentric) camera system is presented that aims to monitor functional hand use outside of clinical settings.

Methods

The system is based on computer vision algorithms that detect the hand, segment the hand outline, distinguish the user’s left or right hand, and detect functional interactions of the hand with objects during activities of daily living. The algorithm was evaluated using egocentric video recordings from 9 participants with cSCI, obtained in a home simulation laboratory. The system produces a binary hand-object interaction decision for each video frame, based on features reflecting motion cues of the hand, hand shape and colour characteristics of the scene.

Results

The output from the algorithm was compared with a manual labelling of the video, yielding F1-scores of 0.74 ± 0.15 for the left hand and 0.73 ± 0.15 for the right hand. From the resulting frame-by-frame binary data, functional hand use measures were extracted: the amount of total interaction as a percentage of testing time, the average duration of interactions in seconds, and the number of interactions per hour. Moderate and significant correlations were found when comparing these output measures to the results of the manual labelling, with ρ = 0.40, 0.54 and 0.55 respectively.

Conclusions

These results demonstrate the potential of a wearable egocentric camera for capturing quantitative measures of hand use at home.
Literature
1.
Anderson KD. Targeting recovery: priorities of the spinal cord injured population. J Neurotrauma. 2004;21:1371–83.CrossRef
2.
Kalsi-Ryan S, Curt A, Verrier MC, Fehlings MG. Development of the graded redefined assessment of strength, sensibility and Prehension (GRASSP): reviewing measurement specific to the upper limb in tetraplegia. J Neurosurg Spine. 2012;17:65–76.CrossRef
3.
Linacre JM, Heinemann AW, Wright BD, Granger CV, Hamilton BB. The structure and stability of the Functional Independence Measure. Arch Phys Med Rehabil. 1994;75:127–32.PubMed
4.
Heinemann AW, Linacre JM, Wright BD, Hamilton BB, Granger C. Relationships between impairment and physical disability as measured by the functional independence measure. Arch Phys Med Rehabil. 1993;74:566–73.CrossRef
5.
Catz A, Itzkovich M, Agranov E, Ring H, Tamir A. SCIM–spinal cord independence measure: a new disability scale for patients with spinal cord lesions. Spinal Cord. 1997;35:850.CrossRef
6.
Itzkovich M, Gelernter I, Biering-Sorensen F, Weeks C, Laramee MT, et al. The spinal cord Independence measure (SCIM) version III: reliability and validity in a multi-center international study. Disabil Rehabil. 2007;29:1926–33.CrossRef
7.
Noorkõiv M, Rodgers H, Price CI. Accelerometer measurement of upper extremity movement after stroke: a systematic re- view of clinical studies. J Neuroeng Rehabil. 2014;11:144.CrossRef
8.
Lemmens RJM, Timmermans AAA, Janssen-Potten YJM, Janssen-Potten SANTDP, et al. Accelerometry measuring the outcome of robot-supported upper limb training in chronic stroke: a randomized controlled trial. PLoS One. 2014;9:e96414.CrossRef
9.
10.
Brogioli M, Popp WL, Albisser U, Brust AK, Frotzler A, et al. Novel sensor technology to assess independence and limb-use laterality in cervical spinal cord injury. J Neurotrauma. 2016;33:1950–7.CrossRef
11.
Marino RJ. Domains of outcomes in spinal cord injury for clinical trials to improve neuro- logical function. J Rehabil Res Dev. 2007;44:113.CrossRef
12.
Waddell KJ, Strube MJ, Bailey RR, Klaesner JW, Birkenmeier RL, et al. Does task-specific training improve upper limb performance in daily life post- stroke? Neurorehabil Neural Repair. 2017;31:290–300.CrossRef
13.
Li C, Kitani KM. Pixel-level hand detection in ego-centric videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2013. p. 3570–7.
14.
Li C, Kitani KM. Model recommendation with virtual probes for egocentric hand detection. In: Proceedings of the IEEE International Conference on Computer Vision; 2013. p. 2624–31.
15.
Serra G, Camurri M, Baraldi L, Benedetti M, Cucchiara R. Hand segmentation for gesture recognition in ego-vision. In: Proceedings of the 3rd ACM inter- national workshop on interactive multimedia on mobile & portable devices:31-36ACM; 2013.
16.
Betancourt A, Morerio P, Marcenaro L, Barakova E, Rauterberg M, Regazzoni C. Towards a unified framework for hand-based methods in first person vision, in Multimedia & Expo Workshops (ICMEW), 2015 IEEE International Conference on:1-6IEEE2015.
17.
Fathi A, Rehg JM. Modeling actions through state changes. In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition; 2013. p. 2579–86.
18.
Likitlersuang J, Zariffa J. Arm angle detection in egocentric video of upper extremity tasks, in World congress on medical physics and biomedical engineering, June 7–12, 2015. Toronto, Canada: Springer; 2015. p. 1124–7.
19.
Bambach S, Lee S, Crandall DJ, Yu C. Lending a hand: detecting hands and recognizing activities in complex egocentric interactions. In: Proceedings of the IEEE Inter- national Conference on Computer Vision; 2015. p. 1949–57.
20.
Zariffa J, Popovic MR. Hand contour detection in wearable camera video using an adaptive histogram region of interest. J Neuroeng Rehabil. 2013;10:114.CrossRef
21.
Khan AU, Borji A. Analysis of Hand Segmentation in the Wild, arXiv preprint arXiv:1803.03317; 2018.
22.
Fathi A, Farhadi A, Rehg JM. Understanding egocentric activities, in Computer vision (ICCV), 2011 IEEE international conference on:407-414IEEE; 2011.
23.
Fathi A, Ren X, Rehg JM. Learning to recognize objects in egocentric activities. In: Computer vision and pattern recognition (CVPR), 2011 IEEE conference on:3281- 3288IEEE; 2011.
24.
Ishihara T, Kitani KM, Ma WC, Takagi H, Asakawa C. Recognizing hand-object interactions in wearable camera videos. In: Image processing (ICIP), 2015 IEEE international conference on:1349- 1353IEEE; 2015.
25.
Matsuo K, Yamada K, Ueno S, Naito S. An attention-based activity recognition for egocentric video. In: Proceedings of the IEEE conference on computer vision and pat- tern recognition workshops; 2014. p. 551–6.
26.
Pirsiavash H, Ramanan D. Detecting activities of daily living in first-person camera views. In: Computer vision and pattern recognition (CVPR), 2012 IEEE conference on:2847- 2854IEEE; 2012.
27.
Ren X, Philipose M. Egocentric recognition of handled objects: Benchmark and analysis. In: Computer vision and pattern recognition workshops, 2009. CVPR work- shops 2009. IEEE computer society conference on:1-8IEEE; 2009.
28.
Likitlersuang J, Zariffa J. Interaction detection in egocentric video: towards a novel outcome measure for upper extremity function. IEEE J Biomed Health Inform. 2016;22(2):561–9.CrossRef
29.
Association American Occupational Therapy. Occupational therapy practice framework: domain and process. Am J Occup Ther. 2008;62:625–83.CrossRef
30.
Ren S, He K, Girshick R, Sun J. Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems; 2015. p. 91–9.
31.
Viola P, Jones MJ. Robust real- time face detection. Int J Comput Vis. 2004;57:137–54.CrossRef
32.
Jones MJ, Rehg JM. Statistical color models with application to skin detection. Int J Comput Vis. 2002;46:81–96.CrossRef
33.
Dollár P, Zitnick CL. Structured forests for fast edge detection. In: Proceedings of the IEEE International Conference on Computer Vision; 2013. p. 1841–8.
34.
Betancourt A, Lopez MM, Regazzoni CS, Rauterberg M. A sequential classifier for hand detection in the framework of egocentric vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops; 2014. p. 586–91.
35.
Cartas A, Dimiccoli M, Radeva P. Detecting hands in egocentric videos: towards action recognition. Computer Aided Systems Theory -- EUROCAST 2017. Cham: Springer International Publishing. 2018. p. 330–8.CrossRef
36.
Farnebäck G. Two-frame motion estimation based on polynomial expansion. Image Analysis. Berlin: Springer. 2003. p. 363–70.CrossRef
37.
Likitlersuang J, Sumitro ER, Pi T, Kalsi-Ryan S, Zariffa J. Abstracts and Workshops 7th National Spinal Cord Injury Conference November 9–11, 2017 Fallsview Casino Resort Niagara Falls, Ontario, Canada. J Spinal Cord Med. 2017;40:813–69.CrossRef
38.
Likitlersuang J, Sumitro ER, Theventhiran P, Kalsi-Ryan S, Zariffa J. Views of individuals with spinal cord injury on the use of wearable cameras to monitor upper limb function in the home and community. J Spinal Cord Med. 2017;40:706–14.CrossRef
39.
Redmon J, Farhadi A. YOLO9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 7263–71.
Metadata
Title
Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home
Authors
Jirapat Likitlersuang
Elizabeth R. Sumitro
Tianshi Cao
Ryan J. Visée
Sukhvinder Kalsi-Ryan
José Zariffa
Publication date
01-12-2019
Publisher
BioMed Central
Published in
Journal of NeuroEngineering and Rehabilitation / Issue 1/2019
Electronic ISSN: 1743-0003
DOI
https://doi.org/10.1186/s12984-019-0557-1

Other articles of this Issue 1/2019

Journal of NeuroEngineering and Rehabilitation 1/2019 Go to the issue

Research

Adapting myoelectric control in real-time using a virtual environment

Research

Technology-aided assessments of sensorimotor function: current use, barriers and future directions in the view of different stakeholders

Research

Long term at-home treatment with transcranial direct current stimulation (tDCS) improves symptoms of cerebellar ataxia: a case report

Research

Grip control and motor coordination with implanted and surface electrodes while grasping with an osseointegrated prosthetic hand

Research

Effects of repeated waist-pull perturbations on gait stability in subjects with cerebellar ataxia

Research

Tarsal fusion for pes equinovarus deformity improves gait capacity in chronic stroke patients