Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

In 2017, Shin et al collected two-class MI EEG-fNIRS data from 29 subjects [22]. The MI tasks were left hand and right hand grabbing. For each MI task, 30 samples were collected. This dataset had been open accessed. Later, there have been many left-hand vs. right-hand multimodal MI studies [23, 25, 41]. However, these data are not open accessed. In addition, these studies contain only two classes. This small class set restricts applications of multimodal MI-BCI. There are some multiclass EEG-fNIRS motor execution decoding studies [26, 39, 40]. However, there are gaps between movement execution and MI brain signals, which are difficult to use to produce MI decoding methods. Increasing the number of classes will enlarge the command set of MI-BCI, thus broadening the application prospects of MI-BCI. A multimodal MI dataset with more classes will be helpful for future multimodal MI decoding studies. To the best of our knowledge, no five class EEG-fNIRS MI decoding studies have been published. In this study, a five-class EEG-fNIRS dataset including left hand, right hand, both hand, feet, and rest MI tasks was built. It will be helpful for developing multiclass multimodal MI decoding methods.

Decoding MI from EEG has been studied for decades. In 1999, CSP was employed to decode movement intention from EEG signals [14]. Later, many improved versions of CSP were proposed [42–44]. In 2008, the FBCSP, one of the most classic CSP-based methods, was proposed by Kai Keng et al [42]. The FBCSP alleviated the subject-specific frequency band selection problem and achieved 90.3% decoding accuracy in MI classification tasks of the right hand and right foot. In 2012, FBCSP achieved the best decoding accuracy and won BCI competition IV on dataset 2a [15, 36]. More recently, Miao et al proposed common time-frequency-spatial patterns to extract features from multiple time windows and achieved 85% decoding accuracy on a three-class MI decoding task. Some Riemannian geometry-based methods have also been proposed in MI decoding [45, 46]. In 2012, Barachant et al proposed the Riemannian minimum distance to mean method and achieved a decoding accuracy of 63.2% in a four-class MI decoding task.

In recent years, DL has made great progress in vision processing and natural language processing. Many DL-based methods have been proposed for decoding MI tasks [47] and have become a trend for single modality MI decoding [48]. In 2017, Schirrmeister et al proposed a CNN-based shallow network (shallow CNN) and a CNN-based network (deep CNN) [16]. They achieved decoding accuracies of 73.7% and 70.9%, respectively, on a four-class MI decoding task. In 2018, another CNN-based neural network, EEGNet, was proposed by Lawhern et al to decode MI from EEG data [17]. Compared with shallow CNN, EEGNet contains fewer parameters and can be applied in many EEG decoding tasks, including steady-state visual evoked potentials and event-related potentials. In 2018, Sakhavi et al proposed the C2CM for MI decoding. They first applied the FBCSP spatial filter to EEG data. The envelope of the resulting data was taken as a feature. A small CNN-based network was proposed to classify manually designed features [18]. They achieved 74.46% decoding accuracy on a four-class MI decoding task. In 2021, Li et al proposed a TS-SEFF network that utilizes both temporal and spectral EEG features [49]. They achieved 74.71% decoding accuracy on a four-class MI decoding. Liu et al proposed the multiscale multitask CNN (MSMT-CNN) network to extract space-time-frequency EEG representations and proposed a multitask learning framework to enhance the feature representation [50]. In the same year, Mane et al proposed FBCNet. A filter bank is adopted to filter EEG signals into multiple subbands. A multiview CNN was designed to decode multiband EEG and achieved 76.2% decoding accuracy on the four-class MI task [51]. In 2022, Ma et al proposed a TD-Attn network that uses a time-distributed attention network to classify CSP features. They achieved 46.15% decoding accuracy on five-class same lamb MI decoding task [52]. Also in 2022, Pan et al proposed an MAtt network that takes the covariance matrix of the EEG signal as input. A manifold attention module is proposed to capture the spatial-temporal information in the covariance matrix time series. They achieved 74.71% decoding accuracy on four-class MI decoding [53].

As neuroimaging techniques have improved in the last few decades, many commercial brain fNIRS devices have appeared. The number of fNIRS channels has increased, and the devices have become more portable [54]. Therefore, fNIRS data have become easier to access for MI studies. Compared with EEG, fNIRS signals have lower temporal resolution and higher spatial resolution. Therefore, the feature extraction and classification methods of EEG and fNIRS are different. In 2017, Shin et al extracted the signal mean and signal slope over time from oxygenated hemoglobin (HbO) signals as features and adopted an linear discriminant analysis (LDA) classifier in two-class MI decoding, which achieved a decoding accuracy of 66.5% [22]. In the same year, Qureshi et al proposed using signal mean, signal peak, signal slope, signal standard deviation signal kurtosis and signal skewness as features and SVM as the classifier to predict class labels. They achieved 82.3% decoding accuracy on MI vs. rest decoding [20]. In 2020, Hosni et al used the same feature set as Qureshi did and conducted channel selection based on generalized linear mode analysis. They achieved an average decoding accuracy of 85.4% from eight amyotrophic lateral sclerosis patients [21]. In the same year, Chhabra et al proposed a deep CNN to decode fNIRS signals and achieved 72.35% decoding accuracy in the four-class MI task [55].

In 2017, Shin et al proposed a stacking-based method that used single modality prediction results as the input for the meta classifier to decode MI tasks and achieved a decoding accuracy of 74.2% in the two-class MI task [22]. In 2018, Chiarelli et al proposed using manual features and a deep neural network to decode MI from EEG-fNIRS data. ERD and ERS were used as the EEG features, while average values of deoxygenated hemoglobin (HbR) and HbO concentration changes were used as the fNIRS features. They achieved a decoding accuracy of 83.28% on two-class (left-hand/right-hand) MI decoding [23]. In 2020, Hasan et al proposed a computationally efficient method for EEG-fNIRS signal decoding. They used mean, skewness, kurtosis, and peak values as features for both EEG and fNIRS and then conducted channel selection based on the Pearson product-moment correlation coefficient. They achieved an accuracy of 75.28% on classification of the four motor execution tasks versus rest [26]. In 2020, Sun et al proposed a DL-based EEG-fNIRS decoding neural network that uses two CNN base networks to extract EEG and fNIRS features separately and employs polynomial fusion to combine the multimodal features. They achieved 77.53% decoding accuracy in two classes of MI decoding (left hand and right hand) [27]. In 2021, Wang et al proposed a feature concatenation-based and DF-based method to decode EEG-fNIRS data. They achieved 84.45% decoding accuracy on well-trained subjects on two-class MI decoding (left hand and right hand) [25]. These EEG-fNIRS decoding methods use feature fusion or DF rather than consider the intramodality relationship between EEG and fNIRS signals.

Fifteen right-handed subjects participated in the experiment, including nine males and six females, aged 25.6 ± 2.1 (mean ± standard deviation). All subjects had no history of neurological or psychiatric disease. They were informed about the experimental details and signed consent forms before the experiment started. This study was approved by the ethical committee of the Institute of automation, Chinese Academy of Sciences.

EEG and fNIRS signals were simultaneously recorded during the experiment. EEG data were collected with electrodes, and each electrode turned into one channel of EEG signal. fNIRS data were collected with probes, including the light source and detector. Each pair of sources and detectors composed one channel of the fNIRS signal. The EEG and fNIRS shared one standard 128-channel cap for signal collection. The specific channel positions of EEG and fNIRS are shown in figure 1. The EEG signal was recorded by a LiveAMP amplifier (Brain Products Company), which has 62 channels. The EEG channel setting was the same as a standard 64-channel electrode cap except that the CB1 and CB2 channels were not collected, as these two channels are too far from the sensorimotor area. Channel FCz was taken as the reference channel. Channel AFz was taken as the ground. The sampling rate was 1000 Hz. The impedance of all EEG channels was kept below 10 kΩ during the experiment. The EEG data were bandpass filtered between 0.5 Hz and 100 Hz during recording. A notch filter of 50 Hz was also applied to remove the power-line interference. fNIRS data were recorded by an NIRsports amplifier (NIRx Medical Technologies company). There were 16 sources and 16 detectors used, and each pair of adjacent sources and detectors formed one fNIRS channel. In total, 52 fNIRS channels were recorded. The sampling rate was 10.2 Hz. To synchronize the EEG signal, fNIRS signal and the MI task, triggers were sent to a synchronize box by the MI task control program.

Zoom In Zoom Out Reset image size

Each subject participated in two days of the experiment. The tasks in the two days were exactly the same, while on the first day, no signal was collected. The aim of the first day of the experiment was to ensure that the subject was familiar with the experimental paradigm and practice to perform MI tasks well. On the second day of the experiment, multimodal MI signals were collected.

In the experiment, a subject sat in front of a 19 inch (1280 × 1024 pixels) screen and acted following the instructions presented on the screen. Each subject was asked to perform ten runs of the MI task. Each run consisted of 25 trials. Figure 2 shows the timing of the trials. Each trial contained three phases. The first phase was the preparing phase, when a red circle was presented on the screen for 2 s. In this phase, the subject was asked to be mentally prepared for the following task. The second phase was the task phase and lasted for 10 s. During this phase, one of the following texts in Chinese was shown on the screen: 'left hand', 'right hand', 'both hand', 'both feet', 'rest'. The subject was asked to perform the corresponding MI task according to the text instructions. In the third phase, the subject was asked to rest. The rest phase lasted for a random length from 9 s to 11 s. Random duration can help avoid the influence of the expectation of the subject on the neural signal. The timing of our experiment is designed following the studies of Shin et al [22].

Zoom In Zoom Out Reset image size

There were a total of five kinds of tasks, four MI tasks and one rest task. These tasks are widely used in single modality MI studies [56]. In the MI tasks of hands, the subject imagined repeatedly grasping a palm-sized ball with their best effort at an approximately 1 Hz rate. Subjects imagined left, right or both hand grasping as the screen instructed. In the MI task of feet, the subject imagined riding a bicycle with the best effort. In the rest task, the subject relaxed and rested. Before each run started, the subject was asked to sit in a comfortable pose and be relaxed. During the whole session, the subject was asked to not move any part of their body, no swallow and no facial actions, keep their visual attention around the center of the screen. There were short breaks between sessions, and the subject could relax and move their body during the break time.

The raw EEG signal was first downsampled to 200 Hz. Then, 44 channels around motor cortices ('Fz', 'F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 'FC1', 'FC2', 'FC3', 'FC4', 'FC5', 'FC6', 'FT7', 'FT8', 'Cz', 'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'T7', 'T8', 'CPz', 'CP1', 'CP2', 'CP3', 'CP4', 'CP5', 'CP6', 'TP7', 'TP8', 'Pz', 'P1', 'P2', 'P3', 'P4', 'P5', 'P6', 'P7', 'P8') were picked for further use. These channels are close to sensorimotor brain areas that are activated during MI tasks [57, 58]. Other channels were too far from the sensorimotor brain area and were discarded to avoid the disturbance of unrelated information. A fifth-order IIR bandpass filter is applied to extract the 4–40 Hz signal. For each trial, a 0–10 s EEG signal with respect to the start of the second phase was extracted as one EEG data sample for further analysis.

The Beer‒Lambert law describes how light attenuates when traveling through different materials [59] The raw fNIRS signal is first resampled to 10 Hz and then converted to concentration changes in HbR and HbO by the modified Beer–Lambert law [60]. Then, a fourth-order IIR bandpass filter is applied to extract 0.01–0.10 Hz signal. Baseline correction is performed by subtracting the signal from −3 s to 0 s with respect to the start of the second phase. Then, HbO and HbR signals from 0 s to 15 s were extracted as one fNIRS data sample for further analysis.

The multimodal dataset is represented as $\mathcal{D}\,{\text{ = }}\,\left\{ {(x_i^E,x_i^N,{y_i})} \right\}_{i = 1}^n$, where $n$ is the total number of samples. $x_i^E$ is the ith preprocessed multichannel EEG data $x_i^E \in {\mathbb{R}^{1 \times 44 \times 2000}}$. $x_i^N \in {\mathbb{R}^{2 \times 52 \times 150}}$ is the ith multichannel fNIRS data, which consists of HbR data ${x^{{\text{HbR}}}} \in {\mathbb{R}^{1 \times 52 \times 150}}$ and HbO data ${x^{{\text{HbO}}}} \in {\mathbb{R}^{1 \times 52 \times 150}}$. ${y_i}$ is the label of the ith multimodal data sample.

In the following chapters, superscript $s$ is used to represent the spatial dimensionality of one array, and $t$ is used to represent the temporal dimensionality. Superscript $E$ and $N$ are used to indicate EEG modality and fNIRS modality. Letters i, j, and k will be repeatedly used as sample indexes.

The overall decoding network is illustrated in figure 3. It mainly consists of an EEG feature extractor, a fNIRS feature extractor, a fusion module, and a classifier. Intramodality alignment losses and intermodality alignment loss were also proposed. The intramodality alignment losses pull the features from the same class closer and push features from different classes further. The intermodality constrains pull the EEG and fNIRS features from the same sample closer and push the EEG and fNIRS features from different samples further.

Zoom In Zoom Out Reset image size

The EEG feature extractor mainly consists of two convolution layers, a power activation layer, and a pooling layer. The first convolution layer carries out temporal convolution along the temporal axis of the input EEG data to extract temporal information. The shape of the temporal convolution kernel is 1 $\times$ 25, and the number of kernels is 40. Then, the second convolution layer is applied along the spatial axis of EEG data to extract spatial information from multichannel EEG data. The kernel shape is the shape of the spatial kernel, which is $44\ \times$ 1, and the number of kernels is 40. The kernel length is the same as the channel number of EEG data to capture the global spatial pattern. The Pow operation is selected as activation for EEG, which can reflect the power features. Finally, a slicing and pooling layer is used to conduct data augmentation and reduce the dimensionality of EEG features. Concretely, a sliding window is first applied to the EEG feature with a window length of 476 temporal points and a window step of ten temporal points. Then, in each window, a mean pooling layer of temporal length 75 and temporal step 15 is applied to reduce the temporal dimensionality. With a sliding window, we could augment EEG data by generating more data samples. Adding a sliding window before the convolution layers will greatly increase the repeated temporal computation. A sliding window was applied after temporal convolution, which means that for each input EEG data sample, only one temporal convolution operation along the full temporal length was conducted. The output features should be the same as if a sliding window was performed before the convolution operations. By this means, considerable computational effort can be saved.

The fNIRS feature extractor mainly contains temporal pooling, temporal differential, spatial convolution and slicing layers. Temporal pooling and differential layers are applied in parallel to the input fNIRS data. The length and step of the pooling layer are 30 and 1 temporal points, respectively. Mean pooling was used to capture the coarse temporal trend of fNIRS data. The differential layer calculates the mean difference of each sliding window with a window length of 30 and a window step of 1 temporal point. The differential layer could capture the fine local temporal features of fNIRS data. The outputs of the pooling layer and differential layer are concatenated along the convolution channel dimension. Similar to the EEG feature extractor, a spatial convolution layer is adopted to extract the spatial pattern of multichannel fNIRS data. Data augmentation is also applied to fNIRS features at the end of the fNIRS feature extractor. The slicing layer picks every 30-th temporal array to compose a final fNIRS feature, which results in 30 augmentations of the fNIRS feature with a temporal length of 4 for each input fNIRS data sample. For example, the 1,31,61,91th temporal points compose one augmentation. The 2,32,62,92th temporal points compose another augmentation. Each augmentation contains full temporal information in the 15 s fNIRS data.

An attention-based fusion module was designed to merge the EEG features and fNIRS features. EEG features were first transferred to query and value with two linear mapping layers. Concretely, each temporal point of one EEG feature $E \in {\mathbb{R}^{{s^E} \times {t^E}}}$ is a vector ${E_i} \in {\mathbb{R}^{{s^E} \times 1}}$, and ${s^E}$ and ${t^E}$ are the spatial and temporal dimensionalities of the EEG feature. A linear map is applied to each vector ${E_i}$, resulting in a new vector ${Q_i} \in {\mathbb{R}^{d \times 1}}$, where $d$ is the dimensionality of the quarry array and equals ${s^E}$ in our network. Then, the query array $Q \in {\mathbb{R}^{d \times {t^E}}} = \{ {Q_i}\} _{i = 1}^{{t^E}}$ was obtained. In the same way, a value $V \in {\mathbb{R}^{d \times {t^E}}}$ array for EEG and a key array $K \in {\mathbb{R}^{d \times {t^N}}}$ for fNIRS were obtained. The attention map is calculated as $A \in {\mathbb{R}^{{t^E} \times {t^N}}} = {Q^{\textrm T}} \times K$, where T represents the matrix transport operation. The attention map is averaged along dimension two, which results in an attention vector $a \in {\mathbb{R}^{{t^E}}}$. The attention vector represents the attention weights of each temporal point in the EEG feature. The temporal weighted EEG feature is calculated as ${E^w} = \{ {E_i} \times {a_i}\} _{i = 1}^{{t^E}}$. Similar to most attention modules, a residual block was also adopted to stabilize the model performance. Attention to fNIRS features was also applied; that is, fNIRS features were used to generate queries and values while using EEG features to generate keys. The resulting weighted fNIRS feature is concatenated with the weighted EEG feature along the temporal dimension.

Finally, a one-layer fully connected neural network along with a log softmax activation layer is adopted as a classifier. As feature augmentation was performed in the prior network, the output of augmentation from one single original sample was averaged to obtain the final output. Dropout and batch norm are employed to enhance model performance by reducing the effect of overfitting and making the training process more stable [61, 62]. The network structure and specific parameters are summarized in table 1.

The heterogeneity of EEG and fNIRS data makes the multimodal fusing process difficult. EEG and fNIRS features are distributed in their own specific space. Straight fusing EEG and fNIRS features, for example, concatenating EEG and fNIR features, leads to poor multimodal decoding accuracy. Therefore, two intramodality alignment losses and one intermodality alignment loss were proposed for extracting better feature representations.

4.3.1. Intramodality center loss

A center loss was added to the fNIRS features. The center loss pulls the features from the same class closer:

$$\begin{equation}{l_{{\text{center}}}} = \sum\limits_{k = 1}^{{n_c}} \sum\limits_{i = 1}^{{n_m}} {\left( {N_i^{\,k} - {{\overline N}^{\,k}}} \right)^2}\end{equation} \tag{ 1 }$$

$$\begin{equation}{\overline N ^{\,k}} = \frac{1}{{{n_m}}}\sum\limits_{i = 1}^{{n_m}} {N_i^{\,k}} \end{equation} \tag{ 2 }$$

where ${n_c}$ is the number of classes and ${n_m}$ is the number of samples belonging to one class (same for each class). $N_i^{\,k}$ is the fNIRS feature of sample $i$ from class $k$. ${\overline N ^{\,k}}$ is the average of fNIRS features from class $k$.

4.3.2. Intramodality contrastive loss

$$\begin{equation}{l_{\operatorname{intra} }} = \sum\limits_{i = 1}^n {(l_i^E + l_i^N)} \end{equation} \tag{ 3 }$$

$$\begin{align}l_i^E &= - \frac{1}{{{n_m} - 1}}\sum\limits_{j = 1}^n {{\textbf{I}}_{i \ne j}} \cdot {{\textbf{I}}_{{y_i} = {y_j}}}{} \nonumber\\ &\quad\cdot \log \frac{{{{\text{e}}^{{\text{sim}}({E_i},{E_j})/\tau }}}}{{\sum\limits_{k = 1}^n {{\textbf{I}}_{i \ne k}} \cdot {{\text{e}}^{{\text{sim}}({E_i},{E_k})/\tau }}}}\end{align} \tag{ 4 }$$

$$\begin{align}l_i^N &= - \frac{1}{{{n_m} - 1}}\sum\limits_{j = 1}^n {{\textbf{I}}_{i \ne j}}\nonumber\\ &\quad \cdot {{\textbf{I}}_{{y_i} = {y_j}}} \cdot \log \frac{{{{\text{e}}^{{\text{sim}}({N_i},{N_j})/\tau }}}}{{\sum\limits_{k = 1}^n {{\textbf{I}}_{i \ne k}} \cdot {{\text{e}}^{{\text{sim}}({N_i},{N_k})/\tau }}}}\end{align} \tag{ 5 }$$

$$\begin{equation}{\text{sim}}(u,v) = {u^{\text{T}}}v/\left\| u \right\|\left\| v \right\|\end{equation} \tag{ 6 }$$

where n is the number of samples from all classes. $\tau$ is a temperature hyper parameter and is set to 0.1. ${\textbf{I}}$ represents the indicator function, ${{\textbf{I}}_{i \ne j}} = 1$ if $i \ne j$; otherwise, ${{\textbf{I}}_{i \ne j}} = 0$. Info noise-contrastive estimation (infoNCE) loss makes similar sample pairs stay close to each other while dissimilar ones are far apart [63]. The proposed intramodality contrastive loss takes the form of infoNCE loss. The key to adopting infoNCE loss into our network is to define positive and negative sample pairs. In our study, in one modality, pairs of features from the same class are positive pairs, and pairs of features from different classes are negative pairs.

The difference between center loss and intramodality contrastive loss lies in the fact that center loss measures the similarity of two features by squared error, while intramodality contrastive loss measures the cosine distance. In addition, center loss only pulls features from the same class together, while intramodality contrastive loss also pushes the features from different classes further.

4.3.3. Intermodality contrastive loss

$$\begin{equation}{l_{\operatorname{inter} }} = \frac{1}{n}\sum\limits_{i = 1}^n {(l_i^E + l_i^N} )\end{equation} \tag{ 7 }$$

$$\begin{equation}l_i^E = - \log \frac{{{{\text{e}}^{{\text{sim}}({E_i},{N_i})/\tau }}}}{{\sum\limits_{k = 1}^n {{\textbf{I}}_{{\text{k}} \ne i}}{{\text{e}}^{{\text{sim}}({E_i},{N_k})/\tau }} + \sum\limits_{k = 1}^n {{\textbf{I}}_{{\text{k}} \ne i}}{{\text{e}}^{{\text{sim}}({E_i},{E_k})/\tau }}}}\end{equation} \tag{ 8 }$$

$$\begin{align}l_i^N = - \log \frac{{{{\text{e}}^{({\text{sim}}({N_i},{E_i})/\tau )}}}}{{\sum\limits_{k = 1}^n {{\textbf{I}}_{{\text{k}} \ne i}}{{\text{e}}^{({\text{sim}}({N_i},{N_k})/\tau )}} + \sum\limits_{k = 1}^n {{\textbf{I}}_{{\text{k}} \ne i}}{{\text{e}}^{({\text{sim}}({N_i},{{\text{E}}_k})/\tau )}}}}.\end{align} \tag{ 9 }$$

Here, EEG and fNIRS feature one single data sample and are defined as a positive pair. EEG and fNIRS features from different data samples are negative pairs.

4.3.4. Overall optimization objective

The classification employs cross-entropy loss:

$$\begin{equation}{l_{{\text{ce}}}} = - {{\textrm E}_{{x^{\,E}},{x^{\,N}},y\sim \mathcal{D}}}{\text{CE}}\left( {{\text{Net}}\left( {{x^{\,E}},{x^{\,N}}} \right),y} \right)\end{equation} \tag{ 10 }$$

where $\operatorname{CE} \left( \cdot \right)$ represents the cross-entropy loss function and $\operatorname{Net} \left( x \right)$ represents our whole network.

The overall optimization objective is:

$$\begin{equation}{l_{{\text{all}}}} = {l_{{\text{ce}}}} + {w_1}{l_{{\text{center}}}} + {w_2}{l_{{\text{intra}}}} + {w_3}{l_{{\text{inter}}}}\end{equation} \tag{ 11 }$$

where ${w_1}$, ${w_2}$, and ${w_3}$ are hyperparameters of the loss weights.

Stratified 5-fold cross-validation was conducted on our dataset and a public dataset [22], which is referred to as the MI2 dataset in the rest of the paper. The MI2 dataset contains EEG and fNIRS data of left-hand and right-hand MI from 29 subjects in a comparison experiment to evaluate the decoding accuracy of the proposed method. An ablation study was also conducted where each part of the proposed method was removed in turn to verify their effectiveness.

The comparison methods include EEG single modality methods, fNIRS single modality methods and multimodal methods.

• (a)  
  EEG methods: EEG single modality methods include FBCSP, EEGNet, shallow CNN, deep CNN, and C2CM.

  • 1.  
    FBCSP [42] followed by LDA is a classic MI decoding method. It is implemented using the MNE toolbox [64]. https://mne.tools/stable/index.html.
  • 2.  
    EEGNet [17] is a compact DL-based MI decoding model. It is implemented by adopting the source code from https://github.com/vlawhern/arl-eegmodels.
  • 3.  
    Shallow CNN and Deep CNN [16] extract EEG features with temporal and spatial convolution layers. A fully connected layer is then used to output the predicted result. The codes were adopted from https://github.com/TNTLFreiburg/braindecode.
  • 4.  
    C2CM [18] first uses FBCSP to filter EEG data. Then, it takes the envelope of filtered signals as a feature and uses small convolutional layers to learn temporal and spatial information. As we do not have access to the source code, this method was reimplemented following the original paper.
  • 5.  
    FBCNet [51] employs a filter bank to create multiview EEG features and uses a multiview CNN with temporal variance layers to classify EEG signals. The codes were adopted from https://github.com/ravikiran-mane/FBCNet.
  • 6.  
    TD-Attn [52] extracts CSP features as input and uses a time-distributed attention network to classify EEG signals. We adopted the codes provided by the original author.
  • 7.  
    Matt [53] is a Riemannian geometric-based DL method that takes the covariance matrix time series of EEG signals as input and applies an attention mechanism to capture the spatial-temporal information. The codes were adopted from https://github.com/CECNL/MAtt.
• (b)  
  fNIRS methods: The fNIRS single modality method extracts the means and slope of HbR and HbO signals as features and uses shrinkage LDA (SLDA) as a classifier. This fNIRS decoding method is referred to as SLDA and was reimplemented following the paper of Shin et al [22].
• (c)  
  Multimodal method: multimodal methods, including DF and polynomial fusion.

  • 1.  
    The DF method takes vote of the output of the FBCSP and SLDA. The method was reimplemented following Wang et al [25].
  • 2.  
    PFNet [27] first extracts EEG and fNIRS features with a CNN. Then, it fuses the multimodal features with the polynomial fusion method. Finally, a fully connected layer is adopted to output the prediction result. The code was adopted from https://github.com/sunzhe839/tensorfusion_EEG_NIRS. As the original code did not use a validation set, we added a validation process to the code. In short, the model that gains the highest validation accuracy in training epochs was selected as the final network. The validation process is the same as that in our method.

Figure 4 shows the average time-frequency maps of four MI tasks and one resting task across all subjects for EEG electrode positions C3, Cz, and C4. Blue indicates ERD. For simplicity, we use LH, RH, BH, F, R to represent left hand, right hand, both hand, feet, and resting MI tasks in the following context. For four MI tasks (LH, RH, BH, and F), the time-frequency maps show obvious ERDs in the alpha and beta frequency bands. For LH, RH, and BH, ERD is strong at C3 and C4 in the alpha and beta bands, which starts at approximately 0.5 s after MI tasks begin. Compared with the hand MI tasks, there was a weak ERD pattern at C3, C4, and Cz in the feet MI task. In the resting state, no obvious ERD is observed in the alpha and beta frequency bands.

Zoom In Zoom Out Reset image size

According to the time-frequency maps, 11.5–13.5 Hz is selected to obtain the average topological maps of EEG signals across all subjects (figure 5). Obvious ERD is observed on motor cortices in topological maps of LH, RH, BH, and F MI tasks. For the resting state, the activation in the motor area is relatively low. For LF and RH, contralateral dominance was observed, and electrodes from contralateral sides of the MI task hand showed much stronger ERD than ipsilateral side electrodes. For BH, both left-side and right-side electrodes show strong ERD. For feet MI, ERD is observed on central and both sides electrodes of the motor area.

Zoom In Zoom Out Reset image size

Figure 6 shows the average HbO waveform for three fNIRS channels (CCP3-CCP5, CCP2-CCP1, CCP6-CCP4) across all subjects. The channels were selected from the left motor area, middle motor area and right motor area. The positions of the three fNIRS channels are close to the positions of EEG electrodes C3, Cz, and C4. For the HbO waveforms of LH, RH, BH, and F, the HbO values increased as the MI task began and reached the highest point at approximately 8 s. For the resting state, the increase in the HbO value was relatively low. One-way repeated-measures ANOVA showed that MI class had a significant main effect on the waveform values in 2.2–13.5 s in the CCP3-CCP5 channel, in 7.5–13.8 s in the CCP2-CCP1 channel and in 7.0–12.5 s in the CCP6-CCP4 channel (p < 0.05). In the CCP3-CCP5 channel in the left hemisphere, this time range is much wider than that in the CCP2-CCP1 channel and CCP6-CCP4 channel. Figure 7 shows the HbO waveform of LH vs. BH and RH vs. BH. LH and BH waveforms are significantly different on CCP3-CCP5channel in the left hemisphere from 4.6 to 7.7 s. LH and BH waveforms tend to be different on CCP2-CCP1 channel in the mid hemisphere from 13.7 to 16.6 s. RH and BH waveforms are significantly different on CCP6-CCP4 channel in the right hemisphere from 6.4 to 11.3 s. RH and BH waveforms are also significantly different on CCP2-CCP1 channel from 15.3 s to 20 s.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

According to the HbO waveforms, the time range of 8–10 s was selected to obtain the average topological map of HbO across all subjects (figure 8). In figure 8, LH, RH, BH, and F MI show obvious activation in the motor brain area, while the rest state shows no activation. Contralateral dominance was observed for the left- and right-hand HbO signals. For both hand MI, both left and right side channels show obvious activation. For feet MI, left side, right side and post middle channels in motor brain area shows activation.

Zoom In Zoom Out Reset image size

Overall, the EEG signals correspond to the fNIR signals. Compared to the rest state, the hands and feet MI induced obvious activation of the motor brain area for both EEG and fNIRS signals. Contralateral dominance is observed on both EEG and fNIRS signals. For feet MI, central channels show relatively higher activation on EEG and fNIRS signals.

The decoding accuracy of our method and all compared methods on our dataset is presented in table 2. Each column refers to each subject in our experiment, and each row refers to one decoding method. One-way repeated-measures ANOVA showed a significant main effect of the decoding method on classification accuracy (F = 17.09, p < 0.01). Post hoc analysis shows that our method is significantly better than all compared methods (post hoc test, all p < 0.01).

Among EEG single modality methods, shallow CNN achieves the highest decoding accuracy of 58.7%, and its performance is significantly higher than other EEG single modality methods (all p < 0.05). SLDA, a fNIRS single modality method, achieves a decoding accuracy of 41.4%, which is significantly lower than most of the EEG single modality methods (FBCSP, EEGNet, Shallow CNN, C2CM, FBCNet, Matt, all p < 0.05). This finding suggests that the EEG signal contains more useful information for MI decoding than the fNIRS signal. Our method achieves significantly higher decoding accuracy than the best single modality method Shallow CNN (p < 0.01). As our method uses information from two modalities based on our framework, we can obtain higher decoding performance.

Our multimodal method achieves a decoding accuracy of 64.4%, which is 6.4% higher than that of the multimodal compared method, DF (p < 0.01). These results show that our method can better utilize information from EEG and fNIRS data and enhance the decoding accuracy of MI-BCI.

The decoding accuracy of our method and all compared methods on the MI2 dataset is presented in table 3. On this public dataset, the decoding accuracy of fNIRS data from some subjects is below chance level; thus, this kind of fNIRS data could not contribute useful information in multimodal learning. Hence, the decoding accuracies of all subjects, subjects whose fNIRS decoding accuracy was below 50% (chance level) and subjects whose fNIRS decoding accuracy was above 50% were separately calculated. One-way repeated-measures ANOVA showed a significant main effect of decoding method on classification accuracy for all the cases (all p < 0.01). Post hoc analysis shows that our method is significantly better than all compared methods except MAtt under the case fNIRS > 50% (all p < 0.01). For case all subjects, our method achieves significantly higher decoding accuracy than all compared methods except shallow CNN and MAtt. For case fNIRS < 50%, the decoding accuracy of our method is not significantly different from that of any of the compared methods. These results show that our method can enhance MI decoding accuracy when the fNIRS modality is informative.

An ablation study was conducted to evaluate the effectiveness of each part in our method. Table 4 shows the decoding accuracy of different models. Each column refers to one subject. Rows 'fNIRS' and 'EEG' represent our single modality network, without any proposed loss and fusion module. 'w/o inter', 'w/o intra', and 'w/o center' refer to our network trained without inter- and intramodality contrastive loss and center loss, respectively. In the 'w/o attn' case, EEG features and fNIRS features are fused by concatenation instead of our attention-based multimodal fusion module (all losses used). One-way repeated-measures ANOVA showed a significant main effect of model on decoding accuracy (F = 22.20, p < 0.01). The post hoc test shows that our method achieves significantly higher decoding accuracy than each compared model (p < 0.05). This indicates that each of our proposed loss and fusion models can enhance decoding accuracy. Our full multimodal method achieves 4.6% higher decoding accuracy than EEG single modality. This shows that by incorporating the fNIRS modality, we could further enhance the decoding accuracy of MI-BCI. It is noteworthy that without center loss, the decoding accuracy dropped by 8.6%. This indicates that center loss plays an important role in our method.

To explore the effect of center loss on multimodal fusion, t-distributed stochastic neighbor embedding was applied to visualize the data distribution from the network. One network with center loss and one without center loss were trained on one folder of data from subject 1, which are named Net1 and Net2, respectively. Both networks are trained, and then these two networks are used to extract the features of all data from subject 1. Figures 9(a) and (b) show the distribution of raw EEG and fNIRS signals. Figures 9(c) and (d) show the distribution of EEG and fNIRS features extracted by Net2 without center loss. Figures 9(e) and (f) show the distribution of EEG and fNIRS features extracted by Net1 with center loss. Figures 9(g) and (h) show the distribution of fused multimodal features through Net2 and Net1, respectively.

Zoom In Zoom Out Reset image size

In figures 9(a) and (b), the maximum overlap was observed among different classes of raw EEG and raw fNIRS signals. After processing by our feature extractor, the samples of each class are clustered into multiple clusters for both EEG and fNIRS features (figures 9(c)–(f)). Thereinto, clusters of EEG features are easier to distinguish (figures 9(d) and (f)), while fNIRS features of different classes are entangled together (figures 9(c) and (e)). After processed by our multimodal fusion module of Net₂ without center loss, the fused feature (figure 9(g)) shows better separability than the fNIRS feature (figure 9(c)) but worse separability than the EEG feature (figure 9(d)). After processing by our multimodal fusion module of Net1 with center loss, the fused feature (figure 9(g)) shows better separability than fNIRS (figure 9(e)) and EEG (figure 9(f)) single modality features. Notably, the EEG feature of BH is separated into two clusters in figure 9(f), while the fused feature of BH forms a single cluster between the left hand and right hand in figure 9(h). These results show that with center loss, our network could extract better multimodal feature representations.

To explore how our attention-based multimodal fusion module works, the average output attention weights for the fNIRS feature and the average temporal waveform of the fNIRS signal from subject 1 were visualized (figure 10). In figure 10(a), attention weights gradually grow over time, which shows a similar pattern to the fNIRS waveform in figure 10(b). As suggested in figure 6, time points from 7.5 s to 12 s are significantly different between classes. Our network gives higher weights to these time points. This shows that our fusion module learns to allocate weights according to the nature of the input feature and makes our model focus on time points that are more valuable for classification.

Zoom In Zoom Out Reset image size

To explore how the multimodal fusion enhances the decoding performance of each MI class, the confusion matrixes of the single modality decoding results, multimodal decoding result and the subtraction of the EEG confusion matrix from the multimodal confusion matrix were further plotted (figure 11). For EEG single modality MI decoding (figure 11(a)), LH and RH show similar decoding accuracies (57.3% and 57.5%), which are higher than the decoding accuracy of BH (42.3%). The feet and rest classes achieve 76.1% and 79.3% accuracies, respectively, which are higher than the accuracies of LH, RH and BH. BH is prone to be misclassed with LH and RH (BH → LH: 24.8%; BH → RH: 23.3%; LH → BH: 24.9%; RH → BH: 24.8%). fNIRS single-modality decoding shows similar results to EEG decoding (figure 11(b)). F and Rest achieve higher decoding accuracy than hand tasks. LH and RH are prone to be misclassed with BH. In figure 11(c), the multimodal classification accuracy of each class is higher than that of the single modal classification. As shown in figure 11(d), with multimodal fusion, the classification accuracies of LH, RH, BH, F, and R increased 5.3%, 5.3%, 3.7%, 2.3% and 4.0%, respectively, compared with EEG single modality decoding. Compared with other classes, the LH and RH classes have greater improvement in accuracy and greater reduction in misclassification with BH. Thus, for hand MI, multimodal fusion increases the decoding accuracy of LH and RH by reducing their misclassification with BH. These results demonstrate that multimodal fusion improves the decoding performance of each class.

Zoom In Zoom Out Reset image size

Our modality-specific feature extractors showed good performance. The experimental results show that the single modality decoding accuracies are comparable to the best single modality decoding methods (our EEG: 59.8% ± 3.7%, shallow CNN 58.7% ± 3.6%, p = 0.092; our fNIRS 43.2% ± 3.4%, SLDA 41.4% ± 3.6%, p < 0.05). This good performance depends on our reasonable model design based on the characteristics of EEG and fNIRS data during MI tasks. The EEG feature extractor uses temporal convolution to extract patterns in the time series and applies pow activation to extract energy information. Indeed, MI tasks induce event-related synchronization and desynchronization of EEG data. The fNIRS modality network calculates the temporal mean and slope of fNIRS data with low temporal resolution in multiple time windows and then employs spatial convolutions to extract spatial information from all fNIRS channels.

It is worth noting that center loss was found to play a very important role in multimodal MI decoding in our ablation study and visualization results (table 4 and figure 9). Also, multimodal decoding accuracy without center loss is worse than EEG single modality decoding accuracy (table 4). The distribution of the fused feature (figure 9(g)) is worse than that of the EEG single modality feature (figure 9(f)). This implies that fusing EEG and fNIRS data without proper guidance may damage the fusion feature representation and reduce the decoding accuracy. The fused multimodal feature with center loss shows a much better distribution than the fused multimodal feature without center loss (figure 9(h) vs. Figure 9(g)). This indicates that center loss can guide multimodal fusion rather than enhance single modality feature representation.

Our intramodality contrastive loss and intermodality contrastive loss significantly improved the decoding accuracy (table 4). It may be due to that our intramodality contrastive loss can enhance the single modality feature representation and that our intermodality contrastive loss can serve as a pathway for the two modalities to guide each other during network training. EEG data and fNRIS data are easily disturbed by noise. When one modality of a sample is disturbed by strong noise, the extracted feature may be far from the class center. This would confuse the classifier during training. In this case, aligning features of noisy modalities to features of the other modality would indirectly pull the features of the noisy modality closer to their class center through intermodality contrastive loss, which may alleviate the problem of noise disturbance and enhance the robustness of our method. Hence, the proposed contrastive losses improve the decoding performance in MI tasks.

Finally, the attention-based EEG-fNIRS fusion module dynamically focuses on different time points, alleviates the impact of nonstationarity and noise and enhances multimodal decoding accuracy.

The mean multimodal decoding accuracy across subjects is higher than that of widely researched EEG single modal decoding accuracy. However, for different subjects, the improvement in decoding accuracy is different. For a few subjects, multimodal decoding accuracy is even decreased. To explore what might affect the accuracy improvement, we analyzed the relationship between multimodal accuracy improvement (subtraction of EEG decoding accuracy from multimodal decoding accuracy) and the single modality accuracies as well as the difference between EEG and fNIRS accuracy on our dataset.

Multimodal accuracy showed no relation with EEG accuracy (r = 0.219, p = 0.214), a positive relationship with fNIRS accuracy (r = 0.612, p < 0.01, figure 12(a)) and a negative relationship with the difference between EEG and fNIRS accuracy (r = 0.727, p < 0.01, figure 12(b)).

Zoom In Zoom Out Reset image size

Higher fNIRS decoding accuracy implies that our decoding model is able to extract useful information from fNIRS data. This may be helpful to enhance the decoding accuracy when combined with EEG data. A larger difference between EEG decoding accuracy and fNIRS decoding accuracy implies that the separability of fNIRS and EEG features extracted by our network are very different. In this case, it may be difficult for our intermodality contrastive loss to align the fNIRS and EEG feature distributions and to affect the ability of our network to extract good multimodal feature representations. Thus, when the difference between EEG and fNIRS decoding accuracy is very large, our multimodal alignment and fusion modules may be hard to work well.

In this study, considering unrelated information may damage the decoding performance. Forty-four EEG channels were manually selected from the motor sensory area. The decoding accuracy of the EEG signal with all channels and the EEG signal with selected channels were compared. Two representative EEG decoding methods, the FBCSP and shallow CNN, are selected to perform this experiment. The results are presented in table 5. Each column refers to one subject. Two-way repeated-measures ANOVA showed a significant main effect of channel selection on decoding accuracy (F = 5.45, p < 0.05) and a significant interaction between the decoding method and channel selection (F = 11.52, p < 0.01). For shallow CNN, the decoding accuracy of selected channels is significantly higher than the accuracy of all channels (p < 0.01). For the FBCSP, the decoding accuracy of the selected channels was not significantly higher than the accuracy of all channels (p = 0.428). These results show that our manually selected EEG channels are feasible.

In our experiments, offline five-fold cross-validation was adopted to evaluate the effectiveness of our method. Here, a simulated online experiment was performed to evaluate the performance of our method by using the first eight sessions of data as the training set and the remaining two sessions of data as the test set. Table 6 presents the results of offline cross validation and simulated online validation. Each column refers to one subject. Our method achieves 64.3% decoding accuracy in the simulated online experiment and 64.4% decoding accuracy in the offline experiment. The paired t test shows that the decoding accuracies of the offline and simulated online experiments are not significantly different. In addition, the decoding accuracy of our method in the simulated online experiment is significantly higher than the accuracies of all compared methods in the offline experiments (all p < 0.05). These results suggest that our proposed method may work well in online MI-BCI application scenes. In future work, online experiments should be conducted to solidly evaluate the decoding methods.

Although our proposed method based on EEG and fNIRS data enhances the multimodal MI decoding accuracy. There are still several limitations. First, each time one uses MI-BCI, a large amount of training data is needed to train the decoding model. This process is time consuming and impacts the practicality of MI-BCI. Transferring knowledge from existing data to facilitate the model training of new subjects can alleviate this problem. There have been some studies on single modality transfer learning of MI decoding. However, multimodal transfer learning of MI decoding needs further research.

Second, there is a practical problem in MI-BCI that the online decoding accuracy declines over a long period of time. However, many single-modality incremental learning methods have been proposed. The online update method of the multimodal MI-BCI decoding model needs further study.