Experimental design and signal selection for construction of a robot control system based on EEG signals
© Yoshioka et al.; licensee Springer. 2014
Received: 8 October 2014
Accepted: 6 November 2014
Published: 26 November 2014
Aiming to develop an noninvasive BMI control system with EEG (electroencephalogram) signals to control external devices such as prostheses and robots for rehabilitation and/or power support, four different tasks corresponding to different brain excitation degrees are designed. Their EEG spectra are analyzed with short-time fast Fourier transform (STFFT), and their features of mu and beta rhythms corresponding to the different tasks are extracted. The extracted features are used to control a one-joint robot arm and their corresponding results are compared. The results show that the EEG signal when a subject is holding a weight is comparatively more stable than the EEG signals in other tasks such as motor imagery. This implies an effective way for power assist by EEG signals.
Brain-machine interfaces (BMIs) are a technology that allow interaction between humans and artificial devices. They rely on continuous, real-time interaction between living neuronal tissue and artificial effectors. The past decade has seen a fast growing interest to develop various effective BMIs that can be invasive or noninvasive. Invasive BMIs, which derive the subject’s intent from neuronal action potentials or local field potentials recorded within the brain, are being studied mainly in nonhuman primates -. Recently, prosthetic devices controlled by a human subject with invasive BMI is reported . Though these invasive BMIs have made great success, they face substantial technical difficulties and entail significant clinical risks: they require that recording electrodes be implanted in the cortex and function well for long periods, and they risk infection and other damages to the brain. The efforts to develop them, despite these disadvantages, are based on the widespread belief that only invasive BMIs will be able to provide users with real-time multidimensional control of a robotic arm or a neuroprosthesis.
On the other hand, noninvasive BMIs for human users derive the user’s intent from scalp-recorded electroencephalogram (EEG) activity. Because these systems do not require invasive surgical implants, they can be used for wide range of applications. Up so far, they have been mainly used for cursor control and communication by means of selection of letters or items on a computer screen -. Recently, an electrical powered wheelchair is controlled by noninvasive BMI . However, despite having the great advantage of not exposing the subject to the risks of brain surgery, EEG-based techniques provide communication channels of limited capacity. In this study, we not only use EEG signals as communication tool but also use them as control signals of external artificial devices. Therefore, one of our main purposes here is to find and extract stable EEG signals by task design and use the signals to control an external robot for power assist.
In awake people, primary sensory or motor cortical areas often display 8 to 12 [Hz] EEG activity when they are not engaged in processing sensory input or producing motor output. This idling activity is called μ rhythm when focused over the somatosensory or motor cortex. These μ rhythms are usually associated with 13 to 30 [Hz] β rhythms. While some β rhythms are harmonics of μ rhythms, some are separable from them by topography and/or timing and thus are independent EEG features. Note that they do not require actual movement, and they occur also with motor imagery . A voluntary movement results in a circumscribed desynchronization (decrease) in the μ and lower β bands. This decrease is called event-related desynchronization (ERD) and begins in the contralateral rolandic region about 2 s prior to the onset of a movement and become bilaterally symmetrical immediately before execution of movement. After a voluntary movement and with relaxation, the power in the brain rhythms increases. This phenomenon called event-related synchronization (ERS) is dominant over the contralateral sensorimotor area and reaches a maximum around 600 ms after movement offset .
Because BMIs do not depend on nerves or muscles, BMIs are conventionally and mainly used to provide communication and control to people with severe neuromuscular disorders such as amyotrophic lateral sclerosis (ALS), brainstem stroke, cerebral palsy, and spinal cord injury. Besides, to achieve the above purposes, this study also aims to construct a BMI that can be used to control robotic devices to help healthy people (e.g. caregivers) with power assistance.
In this study, aiming to develop a BMI control system with EEG signals to control external devices such as prostheses and robots for rehabilitation and/or power support, four different tasks corresponding to different brain excitation degrees are designed. Their EEG spectra are analyzed with short-time fast Fourier transform (STFFT), and their features of μ and β rhythms corresponding to the different tasks are extracted. The extracted features are used to control a one-joint robot arm and their corresponding results are compared. The results show that the EEG signal when a subject is holding a weight is comparatively much stabler than the EEG in other cases such as motor imagery. This implies the possibility of power assistance using EEG signals.
Overview of the research
Several factors suggest that μ and/or β rhythms from the sensorimotor cortex could be good signal features for EEG-based communication and control. In this study, these two sensorimotor μ and β rhythms, generated in motor imagery or motion, are used to control a robotic arm.
Design four different experimental tasks: idling task, gazing task, motion imagery task, and motion task. In these four tasks, the brain excitation degree gradually increase task by task. Hence, their corresponding μ and/or β rhythms should have different changes.
Acquire the EEG signals in each task into computer for analyzing.
Divide the obtained EEG signals at a time interval of 256 ms with 206-ms overlap to get a series of the short-time windows.
Perform STFFT to the EEG signals in each divided short-time window repetitively to get their power spectra. In this way, the power spectra of EEG signals are obtained while their time information is also kept. The time information of EEG signals is very important for control.
Take the obtained power spectrum magnitudes of μ and β rhythms as the features of the motion imagery or motion.
Convert the power spectrum magnitudes of μ and β rhythms into the target angles of the robot arm and control the robot arm.
With the above steps, a one-joint robot arm is controlled with EEG signals and the control results are obtained. These results are compared to determine which of the tasks is the most suitable for robotic control and power assist.
Measurement and signal processing of EEG
Acquisition of EEG signals
Extraction of μ and β rhythms
Sensorimotor rhythms (SMRs) include an arch-shaped μ rhythm, usually with a frequency of 10 Hz (range 8 to 12 Hz), often mixed with a β rhythm (around 20 Hz, range 13 to 30 Hz) and a γ component (around 40 Hz) recorded over somatosensory cortices, most preferably over C3 and C4. The SMR is related to the motor cortex with contributions of somatosensory areas such as that the β component arises from the motor and μ component from the sensory cortex. SMR is blocked by movements, movement imagery, and movement preparation; thus, it is seen as ‘idling’ rhythm of the cortical sensory region. In this study, the μ and β rhythms in movement imagery and/or movement are used as the EEG features to control a robot arm. To extract such μ and β rhythms and further to control a robot arm with the extracted EEG features, STFFT is employed so that both of the spectra of the μ and β rhythms and the time information are obtained.
where , X k is the obtained spectrum of x n , and k is the number of frequencies.
Task design for motion imagery and motion
In order to extract the features of the μ and β rhythms in motion imagery and/or motion, four different tasks are designed, in which the subject is asked to image to flex his nondominant arm (for example, left arm) but without real movement (only motion imagery) or to perform a real actually motion of flexing his elbow joint for holding up a dumbbell. The four tasks are so designed to generate the different EEG signals from relaxed idling task to excited motion task. When the subject is performing each task of motion imagery or actual motion, he is sitting in a chair wearing the head cap for EEG measurement. The chair height is adjusted to fit the subject’s height and let him feel comfortable.
The detail of designed each task is explained as follows.
Task 1: idling task
Task 2: gazing task
The subject is opening his eyes and continuously gazing at a marker at the endpoint of the robot arm as shown in Figure 4 task 2.
Task 3: motion imagery task
In this task, the subject is asked to continuously image to flex his left arm but not really perform an actual motion while he is gazing at the marker at the endpoint of the robot arm as shown in Figure 4 task 3.
This task is designed to get the subject’s EEG in motion imagery. It is expected that his μ rhythm would be further decreased in this task. Same as the purpose of task 2, the second purpose of this task is to see what happens to the subject’s β rhythm in motion imagery.
Task 4: motion task
In this task, the subject is actually holding a 4 kg dumbbell in 90° angle of his left elbow joint while he is still gazing at the marker at the endpoint of the robot arm as shown in Figure 4 task 4. Of course, to conduct the holding task of a dumbbell, the subject will first have an intent, or saying, image, to flex his left arm. This task is in fact a combination of motion imagery and real motion. In this paper, we simply define it as motion task.
Spectrum characteristics of each task and some remarks
As described above, tasks 1 to 4 are designed according to the degree of brain excitation with/without the existence of motion. It is expected that the spectrum of μ rhythm will decline and the spectrum of β rhythm will change, probably will increase, task by task.
Since the spectrum of β rhythm is much less than that of μ rhythm, the spectrum of EEG is normalized with the ratio X r =X(t)/X(max,t), where X(t) and X(max,t) are the spectrum and the maximum spectrum in the frequency band 1 to 30 Hz during the time interval t, respectively.
Meanwhile, we can also find that the β rhythm basically increases from tasks 1 to 4. This shows the fact that the spectrum in β band increases according to the excitation degree of the brain. In this study, these two features of μ and β rhythms are used to control the robot.
The subjects are two young men. Before experiments, they gave the consent for the study, which was reviewed and approved by the Ethics Committee of our university. In order to get the EEG signals in motion imagery and/or motion, each subject who is wearing a head cap with embedded electrodes is asked to perform the defined four tasks, and his μ rhythm and β rhythm extracted from the EEG are used to control a robot arm.
where X(m a x,t) is the maximum spectrum in the frequency band 1 to 30 Hz during the time duration t, and K μ and K β are the weight coefficients for μ and β rhythms, respectively. Since the spectrum magnitudes of μ rhythm and β rhythm are different and the spectrum magnitude of β rhythm is much less than that of μ rhythm, the two different weight coefficients respectively for μ rhythm and β rhythm are necessary. Further, in the measured EEG signals, there are a lot of noises such as artifacts. These contaminated EEG signals in μ and β bands may lead to a sharp change of the target angle θ d . This would result in an abrupt movement of the robot arm. To prevent this, a 5-Hz low-pass filter is employed to smooth the output of the target angle θ d . Moreover, a threshold is set to the filtered target angle θ d to avoid frequently trivial motions of the robot arm. Finally, this target angle θ d is used as the command for position control of the robot arm.
Results and discussion
Remarks on experimental results
In task 2 and task 3 of subject A (Figure 12), from start to about 12 s, the angle of the robot arm in task 3 is larger than that in task 2. However, after that, this difference has disappeared. Further, after 21 s, the situation is reversed, that is, the angle of the robot arm in task 3 decreases to be less than the angle in task 2. For subject B (Figure 13), from start to about 9 s, the angles of the controlled robot arm in task 2 are almost the same as the angles in task 3. But from 9 to 23 s, the angle of the robot arm in task 3 is larger than that in task 2. However, after 17 s, the angle in task 3 decreases and then has a slight rise.
The reason for such changes could be interpreted as follows. Task 3 is mainly related to the motion imagery and such kind of action of the subjects to perform motion imagery is greatly dependent on his concentration. It could be postulated that after 12 s (subject A) or 17 s (subject B), the concentration of the subjects starts to decline and after 21 s, the subject A’s concentration further recedes. Though we could not observe the similar phenomenon for subject B, it could be considered as an individual difference.
The experimental results tell us two facts. One is that the subject training for motion imagery is necessary to keep his/her concentration. On the other hand, long-time motion imagery may not be a good way for robot control or external device control since the subject could not keep his concentration over dozen seconds.
For both subjects A and B, different from the results of tasks 2 and 3, during the entire experiment duration, the angles of the robot arm in the task 1 are almost always the minimum in all of four tasks. Contrarily, the angles corresponding to task 4 are basically always the maximum in all of four tasks. In other words, compared with the results in tasks 2 and 3, the angles of the robot arm in tasks 1 and 4 are more stable. This phenomenon is meaningful.
The result of the task 1 is natural because the brain excitation of the subject is the lowest in all of the four tasks. Though it is comparatively stable, this task has no practical application value since the subjects in this task have not done and thought anything.
However, in task 4, the subjects are holding a 4 kg dumbbell in 90 ° angle of his left elbow joint while gazing at the marker at the endpoint of the robot arm. Since their brain excitation is the highest, the angles of the robot arm are basically always the maximum in all of four tasks. This result tells us such a fact that the EEG signals of the subjects are comparatively stable when the subjects are holding or touching something. Furthermore, this fact implies that EEG signal can be used for robotic power assist to help a subject holding or moving a heavy load.
In this study, aimed to construct a robot control system using EEG signals, four different tasks with different human’s brain excitation degree are designed, and their spectra of μ and β rhythms of EEG signals in the four tasks are extracted with STFFT. These spectra are used to control a one-joint robot arm. The experimental results show that the classic motion imagery cannot be simply used to real robot control since the subject’s concentration cannot be kept over dozen seconds. Contrarily, since the EEG signals in the task of holding a load are comparatively stable, we conclude that this holding task has a great application potential to be used to construct a robotic power assist system to help person to hold or move a heavy load. This is one of our future work.
- Chapin JK, Moxon KA, Markowitz RS, Nicolelic MAL: Real-time control of a robot arm using simulataneously recorded neurons in the motor cortex. Nat Neurosci 1999, 2: 664–670. 10.1038/10223View ArticleGoogle Scholar
- Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs AJ, Srinivasan MA, Nicolelic MAL: Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 2000, 408: 361–365. 10.1038/35042582View ArticleGoogle Scholar
- Nicolelic MAL: Action from thoughts. Nature 2001, 409: 403–407. 10.1038/35053191View ArticleGoogle Scholar
- Taylor DM, Tillery SIH, Schwartz AB: Direct cortical control of 3d neuroprosthetic. Science 2002, 296: 1829–1832. 10.1126/science.1070291View ArticleGoogle Scholar
- Carmena JM, Lebeddev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelic MAL: Learning to control a brain-machine interface for reaching and grasping by primates. PLOS Biol 2003, 1: 193–208. 10.1371/journal.pbio.0000042View ArticleGoogle Scholar
- Lebeddev MA, Nicolelic MAL: Brain-machine interfaces: past, present and future. Trends Neurosci 2006, 29: 536–546. 10.1016/j.tins.2006.07.004View ArticleGoogle Scholar
- Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB: Cortical control of a prosthetic arm for self-feeding. Nature 2008, 453: 1098–1101. 10.1038/nature06996View ArticleGoogle Scholar
- Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP: Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 2006, 442: 164–171. 10.1038/nature04970View ArticleGoogle Scholar
- Wolpaw JR, Birbraumer N, McFarland DJ, Pfurtscheller G, Vaughan TM: Brain-computer interfaces for communication and control. Clin Neurophysiol 2002, 113: 767–791. 10.1016/S1388-2457(02)00057-3View ArticleGoogle Scholar
- Wolpaw JR, McFarland DJ: Control of a two-dimensional movements signal by a noninvasive brain-computer interface in humans. Proc Natl Acad Sci U S A 2004, 101: 17849–17854. 10.1073/pnas.0403504101View ArticleGoogle Scholar
- Dornhege G, Millan J, Hinterberger T, McFarland JD, Muller KR: Towards Brain-Computer Interfacing. The MIT Press, Cambridge; 2007.Google Scholar
- Iturrate I, Antelis JM, Kubler A, Minguez J: A noninvasive brain-actuated wheelchair based on a P300 neurophysiological protocol and automated navigation. IEEE Trans. Robot 2009, 25: 614–627. 10.1109/TRO.2009.2020347View ArticleGoogle Scholar
- Bashashati A, Fatourechi M, Ward RK, Brich GE: A survey of signal processing algorithms in brain-computer interfaces based on electrical brain signals. J Neural Eng 2007, 4: 32–57. 10.1088/1741-2560/4/2/R03View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.