Skip to main content
Top
Published in: International Journal of Computer Assisted Radiology and Surgery 1/2018

Open Access 01-01-2018 | Original Article

Simulation of facial expressions using person-specific sEMG signals controlling a biomechanical face model

Authors: Merijn Eskes, Alfons J. M. Balm, Maarten J. A. van Alphen, Ludi E. Smeele, Ian Stavness, Ferdinand van der Heijden

Published in: International Journal of Computer Assisted Radiology and Surgery | Issue 1/2018

Login to get access

Abstract

Purpose

Functional inoperability in advanced oral cancer is difficult to assess preoperatively. To assess functions of lips and tongue, biomechanical models are required. Apart from adjusting generic models to individual anatomy, muscle activation patterns (MAPs) driving patient-specific functional movements are necessary to predict remaining functional outcome. We aim to evaluate how volunteer-specific MAPs derived from surface electromyographic (sEMG) signals control a biomechanical face model.

Methods

Muscle activity of seven facial muscles in six volunteers was measured bilaterally with sEMG. A triple camera set-up recorded 3D lip movement. The generic face model in ArtiSynth was adapted to our needs. We controlled the model using the volunteer-specific MAPs. Three activation strategies were tested: activating all muscles \((\hbox {act}_\mathrm{all})\), selecting the three muscles showing highest muscle activity bilaterally \((\hbox {act}_3)\)—this was calculated by taking the mean of left and right muscles and then selecting the three with highest variance—and activating the muscles considered most relevant per instruction \((\hbox {act}_\mathrm{rel})\), bilaterally. The model’s lip movement was compared to the actual lip movement performed by the volunteers, using 3D correlation coefficients \((\rho )\).

Results

The correlation coefficient between simulations and measurements with \(\hbox {act}_\mathrm{rel}\) resulted in a median \(\rho \) of 0.77. \(\hbox {act}_3\) had a median \(\rho \) of 0.78, whereas with \(\hbox {act}_\mathrm{all}\) the median \(\rho \) decreased to 0.45.

Conclusion

We demonstrated that MAPs derived from noninvasive sEMG measurements can control movement of the lips in a generic finite element face model with a median \(\rho \) of 0.78. Ultimately, this is important to show the patient-specific residual movement using the patient’s own MAPs. When the required treatment tools and personalisation techniques for geometry and anatomy become available, this may enable surgeons to test the functional results of wedge excisions for lip cancer in a virtual environment and to weigh surgery versus organ-sparing radiotherapy or photodynamic therapy.
Appendix
Available only for authorised users
Literature
1.
4.
go back to reference Gladilin E, Zachow S, Deuflhard P, Hege H (2001) Towards a realistic simulation of individual facial mimics. In: Thomas E, Bernd G, Heinrich N, Hans-Peter S (eds) Proc. Vis. Model. Vis. Conf. 2001 (VMW ’01). Aka GmbH, Stuttgart, Germany, pp 129–134 Gladilin E, Zachow S, Deuflhard P, Hege H (2001) Towards a realistic simulation of individual facial mimics. In: Thomas E, Bernd G, Heinrich N, Hans-Peter S (eds) Proc. Vis. Model. Vis. Conf. 2001 (VMW ’01). Aka GmbH, Stuttgart, Germany, pp 129–134
5.
go back to reference Koch RM, Roth SHM, Gross MH, Zimmermann AP, Sailer HF (2002) A framework for facial surgery simulation. In: Proceedings of 18th ]Spring Conference on Computer Graphics—SCCG ’02. ACM Press, New York, USA, p 33 Koch RM, Roth SHM, Gross MH, Zimmermann AP, Sailer HF (2002) A framework for facial surgery simulation. In: Proceedings of 18th ]Spring Conference on Computer Graphics—SCCG ’02. ACM Press, New York, USA, p 33
7.
go back to reference Chabanas M, Payan Y (2001) Finite element model of the face soft tissue for computer-assisted maxillofacial surgery. In: Proceedings of fifth international symposium on computer methods in biomechanics and biomedical engineering (CMBBE 2001). Rome, Italy, pp 1–6 Chabanas M, Payan Y (2001) Finite element model of the face soft tissue for computer-assisted maxillofacial surgery. In: Proceedings of fifth international symposium on computer methods in biomechanics and biomedical engineering (CMBBE 2001). Rome, Italy, pp 1–6
9.
go back to reference Buchaillard S, Brix M, Perrier P, Payan Y (2007) Simulations of the consequences of tongue surgery on tongue mobility: implications for speech production in post-surgery conditions. Int J Med Robot Comput Assist Surg 3:252–261. doi:10.1002/rcs.142 CrossRef Buchaillard S, Brix M, Perrier P, Payan Y (2007) Simulations of the consequences of tongue surgery on tongue mobility: implications for speech production in post-surgery conditions. Int J Med Robot Comput Assist Surg 3:252–261. doi:10.​1002/​rcs.​142 CrossRef
10.
14.
go back to reference Nazari MA, Perrier P, Chabanas M, Payan Y (2010) Simulation of dynamic orofacial movements using a constitutive law varying with muscle activation. Comput Methods Biomech Biomed Eng 13:469–482. doi:10.1080/10255840903505147 CrossRef Nazari MA, Perrier P, Chabanas M, Payan Y (2010) Simulation of dynamic orofacial movements using a constitutive law varying with muscle activation. Comput Methods Biomech Biomed Eng 13:469–482. doi:10.​1080/​1025584090350514​7 CrossRef
15.
go back to reference Groleau J, Chabanas M, Marécaux C, Payrard N, Segaud B, Rochette M, Perrier P, Payan Y (2007) A biomechanical model of the face including muscles for the prediction of deformations during speech production. In: Manfredi C (ed) 5th Int. Work. Model. Anal. Vocal Emiss. Biomed. Appl. MAVEBA ’2007. Firenze University Press, Firenze, Italy, pp 173–176 Groleau J, Chabanas M, Marécaux C, Payrard N, Segaud B, Rochette M, Perrier P, Payan Y (2007) A biomechanical model of the face including muscles for the prediction of deformations during speech production. In: Manfredi C (ed) 5th Int. Work. Model. Anal. Vocal Emiss. Biomed. Appl. MAVEBA ’2007. Firenze University Press, Firenze, Italy, pp 173–176
16.
go back to reference Nazari MA, Payan Y, Perrier P, Chabanas M, Lobos C (2008) A continuous biomechanical model of the face: a study of muscle coordination for speech lip gestures. In: Sock R, Fuchs S, Laprie Y (eds) 8th International Seminar on Speech Production. INRIA, Strasbourg, France, pp 321–324 Nazari MA, Payan Y, Perrier P, Chabanas M, Lobos C (2008) A continuous biomechanical model of the face: a study of muscle coordination for speech lip gestures. In: Sock R, Fuchs S, Laprie Y (eds) 8th International Seminar on Speech Production. INRIA, Strasbourg, France, pp 321–324
19.
go back to reference Wu T, Martens H, Hunter P, Mithraratne K (2013) Estimating muscle activation patterns using a surrogate model of facial biomechanics. In: 2013 35th annual international conference of the IEEE engineering in medicine and biology society, IEEE, Osaka, Japan, pp 7172–7175 Wu T, Martens H, Hunter P, Mithraratne K (2013) Estimating muscle activation patterns using a surrogate model of facial biomechanics. In: 2013 35th annual international conference of the IEEE engineering in medicine and biology society, IEEE, Osaka, Japan, pp 7172–7175
22.
go back to reference Eskes M, van Alphen MJA, Smeele LE, Brandsma D, Balm AJM, van der Heijden F (2017) Predicting 3D lip movement using facial sEMG: a first step towards estimating functional and aesthetic outcome of oral cancer surgery. Med Biol Eng Comput 55:573–583. doi:10.1007/s11517-016-1511-z CrossRefPubMed Eskes M, van Alphen MJA, Smeele LE, Brandsma D, Balm AJM, van der Heijden F (2017) Predicting 3D lip movement using facial sEMG: a first step towards estimating functional and aesthetic outcome of oral cancer surgery. Med Biol Eng Comput 55:573–583. doi:10.​1007/​s11517-016-1511-z CrossRefPubMed
23.
go back to reference van Dijk S, van Alphen MJA, Jacobi I, Smeele LE, van der Heijden F, Balm AJM (2016) A new accurate 3D measurement tool to assess the range of motion of the tongue in oral cancer patients: a standardized model. Dysphagia 31:97–103. doi:10.1007/s00455-015-9665-7 CrossRefPubMed van Dijk S, van Alphen MJA, Jacobi I, Smeele LE, van der Heijden F, Balm AJM (2016) A new accurate 3D measurement tool to assess the range of motion of the tongue in oral cancer patients: a standardized model. Dysphagia 31:97–103. doi:10.​1007/​s00455-015-9665-7 CrossRefPubMed
24.
go back to reference Lapatki BG, Stegeman DF, Zwarts MJ (2005) Selective contractions of individual facial muscle subcomponents monitored and trained with high-density surface EMG. In: Beurskens CHG, van Gelder RS, Heymans PG, Manni JJ, Nicolai JA (eds) Facial Palsies Complementary Approaches, Lemma, the Netherlands, pp 89–108 Lapatki BG, Stegeman DF, Zwarts MJ (2005) Selective contractions of individual facial muscle subcomponents monitored and trained with high-density surface EMG. In: Beurskens CHG, van Gelder RS, Heymans PG, Manni JJ, Nicolai JA (eds) Facial Palsies Complementary Approaches, Lemma, the Netherlands, pp 89–108
26.
go back to reference Lloyd JE, Stavness I, Fels S (2012) ArtiSynth: a fast interactive biomechanical modeling toolkit combining multibody and finite element simulation. In: Payan Y (ed) Soft tissue biomechanical modeling for computer assisted surgery, 1st edn. Springer, Berlin, pp 355–394CrossRef Lloyd JE, Stavness I, Fels S (2012) ArtiSynth: a fast interactive biomechanical modeling toolkit combining multibody and finite element simulation. In: Payan Y (ed) Soft tissue biomechanical modeling for computer assisted surgery, 1st edn. Springer, Berlin, pp 355–394CrossRef
30.
go back to reference Stavness I, Lloyd JE, Payan Y, Fels S (2011) Coupled hard-soft tissue simulation with contact and constraints applied to jaw-tongue-hyoid dynamics. Int J Numer Method Biomeds Eng 27:367–390. doi:10.1002/cnm.1423 CrossRef Stavness I, Lloyd JE, Payan Y, Fels S (2011) Coupled hard-soft tissue simulation with contact and constraints applied to jaw-tongue-hyoid dynamics. Int J Numer Method Biomeds Eng 27:367–390. doi:10.​1002/​cnm.​1423 CrossRef
35.
go back to reference Lapatki BG (2010) The facial musculature: characterisation at a motor unit level. Radboud University Nijmegen, Nijmegen Lapatki BG (2010) The facial musculature: characterisation at a motor unit level. Radboud University Nijmegen, Nijmegen
37.
go back to reference Koch RM, Gross MH, Carls FR, von Büren DF, Fankhauser G, Parish YIH (1996) Simulating facial surgery using finite element models. In: Proceedings of 23rd annual conference on computer graphics and Interact Technolog—SIGGRAPH’96. ACM Press, New York, USA, pp 421–428 Koch RM, Gross MH, Carls FR, von Büren DF, Fankhauser G, Parish YIH (1996) Simulating facial surgery using finite element models. In: Proceedings of 23rd annual conference on computer graphics and Interact Technolog—SIGGRAPH’96. ACM Press, New York, USA, pp 421–428
39.
42.
go back to reference Kraaijenga SAC, van der Molen L, Stuiver MM, Teertstra HJ, Hilgers FJM, van den Brekel MWM (2015) Effects of strengthening exercises on swallowing musculature and function in senior healthy subjects: a prospective effectiveness and feasibility study. Dysphagia 30:392–403. doi:10.1007/s00455-015-9611-8 CrossRefPubMed Kraaijenga SAC, van der Molen L, Stuiver MM, Teertstra HJ, Hilgers FJM, van den Brekel MWM (2015) Effects of strengthening exercises on swallowing musculature and function in senior healthy subjects: a prospective effectiveness and feasibility study. Dysphagia 30:392–403. doi:10.​1007/​s00455-015-9611-8 CrossRefPubMed
43.
go back to reference van der Molen L, van Rossum MA, Rasch CRN, Smeele LE, Hilgers FJM (2014) Two-year results of a prospective preventive swallowing rehabilitation trial in patients treated with chemoradiation for advanced head and neck cancer. Eur Arch Oto Rhino Laryngol 271:1257–1270. doi:10.1007/s00405-013-2640-8 CrossRef van der Molen L, van Rossum MA, Rasch CRN, Smeele LE, Hilgers FJM (2014) Two-year results of a prospective preventive swallowing rehabilitation trial in patients treated with chemoradiation for advanced head and neck cancer. Eur Arch Oto Rhino Laryngol 271:1257–1270. doi:10.​1007/​s00405-013-2640-8 CrossRef
44.
go back to reference Kim D-H, Lu N, Ma R, Kim Y-S, Kim R-H, Wang S, Wu J, Won SM, Tao H, Islam A, Yu KJ, Kim T, Chowdhury R, Ying M, Xu L, Li M, Chung H-J, Keum H, McCormick M, Liu P, Zhang Y-W, Omenetto FG, Huang Y, Coleman T, Rogers Ja (2011) Epidermal electronics. Science 333:838–843. doi:10.1126/science.1206157 CrossRefPubMed Kim D-H, Lu N, Ma R, Kim Y-S, Kim R-H, Wang S, Wu J, Won SM, Tao H, Islam A, Yu KJ, Kim T, Chowdhury R, Ying M, Xu L, Li M, Chung H-J, Keum H, McCormick M, Liu P, Zhang Y-W, Omenetto FG, Huang Y, Coleman T, Rogers Ja (2011) Epidermal electronics. Science 333:838–843. doi:10.​1126/​science.​1206157 CrossRefPubMed
Metadata
Title
Simulation of facial expressions using person-specific sEMG signals controlling a biomechanical face model
Authors
Merijn Eskes
Alfons J. M. Balm
Maarten J. A. van Alphen
Ludi E. Smeele
Ian Stavness
Ferdinand van der Heijden
Publication date
01-01-2018
Publisher
Springer International Publishing
Published in
International Journal of Computer Assisted Radiology and Surgery / Issue 1/2018
Print ISSN: 1861-6410
Electronic ISSN: 1861-6429
DOI
https://doi.org/10.1007/s11548-017-1659-5

Other articles of this Issue 1/2018

International Journal of Computer Assisted Radiology and Surgery 1/2018 Go to the issue