Experimental design and signal selection for construction of a robot control system based on EEG signals

Aiming to develop an noninvasive BMI control system with EEG (electroencephalogram) signals to control external devices such as prostheses and robots for rehabilitation and/or power support, four different tasks corresponding to different brain excitation degrees are designed. Their EEG spectra are analyzed with short-time fast Fourier transform (STFFT), and their features of mu and beta rhythms corresponding to the different tasks are extracted. The extracted features are used to control a one-joint robot arm and their corresponding results are compared. The results show that the EEG signal when a subject is holding a weight is comparatively more stable than the EEG signals in other tasks such as motor imagery. This implies an effective way for power assist by EEG signals.

Brain-machine interfaces (BMIs) are a technology that allow interaction between humans and artificial devices. They rely on continuous, real-time interaction between living neuronal tissue and artificial effectors. The past decade has seen a fast growing interest to develop various effective BMIs that can be invasive or noninvasive. Invasive BMIs, which derive the subject’s intent from neuronal action potentials or local field potentials recorded within the brain, are being studied mainly in nonhuman primates [1]-[7]. Recently, prosthetic devices controlled by a human subject with invasive BMI is reported [8]. Though these invasive BMIs have made great success, they face substantial technical difficulties and entail significant clinical risks: they require that recording electrodes be implanted in the cortex and function well for long periods, and they risk infection and other damages to the brain. The efforts to develop them, despite these disadvantages, are based on the widespread belief that only invasive BMIs will be able to provide users with real-time multidimensional control of a robotic arm or a neuroprosthesis.

On the other hand, noninvasive BMIs for human users derive the user’s intent from scalp-recorded electroencephalogram (EEG) activity. Because these systems do not require invasive surgical implants, they can be used for wide range of applications. Up so far, they have been mainly used for cursor control and communication by means of selection of letters or items on a computer screen [9]-[11]. Recently, an electrical powered wheelchair is controlled by noninvasive BMI [12]. However, despite having the great advantage of not exposing the subject to the risks of brain surgery, EEG-based techniques provide communication channels of limited capacity. In this study, we not only use EEG signals as communication tool but also use them as control signals of external artificial devices. Therefore, one of our main purposes here is to find and extract stable EEG signals by task design and use the signals to control an external robot for power assist.

In awake people, primary sensory or motor cortical areas often display 8 to 12 [Hz] EEG activity when they are not engaged in processing sensory input or producing motor output. This idling activity is called μ rhythm when focused over the somatosensory or motor cortex. These μ rhythms are usually associated with 13 to 30 [Hz] β rhythms. While some β rhythms are harmonics of μ rhythms, some are separable from them by topography and/or timing and thus are independent EEG features. Note that they do not require actual movement, and they occur also with motor imagery [9]. A voluntary movement results in a circumscribed desynchronization (decrease) in the μ and lower β bands. This decrease is called event-related desynchronization (ERD) and begins in the contralateral rolandic region about 2 s prior to the onset of a movement and become bilaterally symmetrical immediately before execution of movement. After a voluntary movement and with relaxation, the power in the brain rhythms increases. This phenomenon called event-related synchronization (ERS) is dominant over the contralateral sensorimotor area and reaches a maximum around 600 ms after movement offset [13].

Because BMIs do not depend on nerves or muscles, BMIs are conventionally and mainly used to provide communication and control to people with severe neuromuscular disorders such as amyotrophic lateral sclerosis (ALS), brainstem stroke, cerebral palsy, and spinal cord injury. Besides, to achieve the above purposes, this study also aims to construct a BMI that can be used to control robotic devices to help healthy people (e.g. caregivers) with power assistance.

In this study, aiming to develop a BMI control system with EEG signals to control external devices such as prostheses and robots for rehabilitation and/or power support, four different tasks corresponding to different brain excitation degrees are designed. Their EEG spectra are analyzed with short-time fast Fourier transform (STFFT), and their features of μ and β rhythms corresponding to the different tasks are extracted. The extracted features are used to control a one-joint robot arm and their corresponding results are compared. The results show that the EEG signal when a subject is holding a weight is comparatively much stabler than the EEG in other cases such as motor imagery. This implies the possibility of power assistance using EEG signals.

Several factors suggest that μ and/or β rhythms from the sensorimotor cortex could be good signal features for EEG-based communication and control. In this study, these two sensorimotor μ and β rhythms, generated in motor imagery or motion, are used to control a robotic arm.

In order to steadily and effectively get EEG signals, in this study, four different tasks corresponding to the different brain excitation degrees are designed. Their features of μ and β rhythms corresponding to the different tasks are extracted with their EEG spectra by STFFT, and a one-joint robot arm is controlled by the extracted features. Then, the control results using the EEG signals from these four tasks are compared to determine which task is more appropriate for robot control. The procedures of the research are as follows.

1. 1.

   Design four different experimental tasks: idling task, gazing task, motion imagery task, and motion task. In these four tasks, the brain excitation degree gradually increase task by task. Hence, their corresponding μ and/or β rhythms should have different changes.

2. 2.

   Acquire the EEG signals in each task into computer for analyzing.

3. 3.

   Divide the obtained EEG signals at a time interval of 256 ms with 206-ms overlap to get a series of the short-time windows.

4. 4.

   Perform STFFT to the EEG signals in each divided short-time window repetitively to get their power spectra. In this way, the power spectra of EEG signals are obtained while their time information is also kept. The time information of EEG signals is very important for control.

5. 5.

   Take the obtained power spectrum magnitudes of μ and β rhythms as the features of the motion imagery or motion.

6. 6.

   Convert the power spectrum magnitudes of μ and β rhythms into the target angles of the robot arm and control the robot arm.

With the above steps, a one-joint robot arm is controlled with EEG signals and the control results are obtained. These results are compared to determine which of the tasks is the most suitable for robotic control and power assist.

Measurement and signal processing of EEG

Acquisition of EEG signals

In this study, the EEG signals of one of two subjects (two healthy 20’s young men) are measured while the subject is sitting in a chair as shown in Figure 1. The electrodes embedded in a head cap are positioned in the International 10-20 system of electrode placement. The cap is so worn that the position C _z is right at the top of the subject’s scalp, and the cap is firmly held on the head with a belt passed through the subject’s chest. The embedded electrodes do not directly contact to the scalp of the subject. Instead, electrolytic gel is poured into the electrode holes with a special syringe to suppress the contact impedance between the electrodes and the scalp. The EEG signals over the contralateral sensorimotor area (C₄ to A₂) are measured. The signals are assumed to be corresponding to the movement of the subject’s left arm. Position C _z (the top of the scalp) is used as ground, and the right wrist is used as body earth. The sampling frequency is 1 kHz. The EEG signals are amplified 10,000 times by an amplifier (g.BSamp, g.tec Co., Austria). Then, the signals are acquired through 12-bit A/D converters in a multi-functional interface board and further processed online on a PC running real-time operation system. In our case, shield room is not used in consideration of our BMI system which would be used in daily life. The extracted EEG signals are used to drive a one-joint robot arm like the subject flexing his elbow joint. The robotic arm is also controlled with real-time operation system.

Measurement of EEG signals.

Full size image

Extraction of μ and β rhythms

Sensorimotor rhythms (SMRs) include an arch-shaped μ rhythm, usually with a frequency of 10 Hz (range 8 to 12 Hz), often mixed with a β rhythm (around 20 Hz, range 13 to 30 Hz) and a γ component (around 40 Hz) recorded over somatosensory cortices, most preferably over C₃ and C₄. The SMR is related to the motor cortex with contributions of somatosensory areas such as that the β component arises from the motor and μ component from the sensory cortex. SMR is blocked by movements, movement imagery, and movement preparation; thus, it is seen as ‘idling’ rhythm of the cortical sensory region. In this study, the μ and β rhythms in movement imagery and/or movement are used as the EEG features to control a robot arm. To extract such μ and β rhythms and further to control a robot arm with the extracted EEG features, STFFT is employed so that both of the spectra of the μ and β rhythms and the time information are obtained.

Here, we briefly describe the concept of STFFT. A complex Fourier series of function x(t) is defined as

By inner product over the interval [−T/2,T/2], the Fourier coefficients C _k are defined by the formula

Further, the discrete Fourier transform about the discrete signal x _i with sampling interval T _s is expressed as

To the amplified and sampled sequence of N EEG signals x _n (n=0,1,⋯,N−1), its discrete FFT (DFFT) is given as

where W=e^{-j\frac{2\pi }{N}}, X _k is the obtained spectrum of x _n , and k is the number of frequencies.

As aforementioned, STFFT is used in this study. The data x _n to be transformed is divided into N blocks (saying, N windows, corresponding to time shift t) with an overlap to reduce artifacts at the boundary. Each block is discrete Fourier transformed, then the complex result is added to a matrix, which records magnitude and phase for each point in time and frequency. This process is repeated so that both the spectra and time information of the EEG signals are obtained. For example, let t=0,10,20,⋯, [ms], then

Shortening the window width will increase the time resolution, but it will increase the computation cost. The concept of the STFFT is shown in Figure 2 and one example of obtained spectra of μ rhythm by the STFFT is shown in Figure 3.

Concept of discrete short-time FFT for EEG signals.

Full size image

Spectra of μ rhythm obtained by short-time FFT.

Full size image

Task design for motion imagery and motion

In order to extract the features of the μ and β rhythms in motion imagery and/or motion, four different tasks are designed, in which the subject is asked to image to flex his nondominant arm (for example, left arm) but without real movement (only motion imagery) or to perform a real actually motion of flexing his elbow joint for holding up a dumbbell. The four tasks are so designed to generate the different EEG signals from relaxed idling task to excited motion task. When the subject is performing each task of motion imagery or actual motion, he is sitting in a chair wearing the head cap for EEG measurement. The chair height is adjusted to fit the subject’s height and let him feel comfortable.

The detail of designed each task is explained as follows.

Task 1: idling task

In this task, the subject just closes his eyes in relaxation state, and he is not doing and not thinking anything as shown in Figure 4 task 1.

Four tasks designed to obtain the μ and β rhythms in motion imagery and/or motion.

Full size image

This task is designed to get the subject’s EEG in idling (relaxed) state. It is expected to get the highest μ rhythm in all of four tasks. The results of STFFT for this task are shown in Figure 5, in which we find that μ rhythm is greatly appeared.

Results of short-time FFT for task 1. μ rhythm is greatly generated but almost no β rhythm.

Full size image

Task 2: gazing task

The subject is opening his eyes and continuously gazing at a marker at the endpoint of the robot arm as shown in Figure 4 task 2.

In this task, the subject just concentrates himself to gaze at the marker and is not doing and not thinking anything else. This task is designed to get the subject’s EEG when he opens his eyes and gazes at an object. The brain excitation degree of the subject in this task is higher than that in task 1. It is expected that his μ rhythms would be decreased or saying, suppressed, and the β rhythms would have changes accompanying the changes of μ rhythms in this task. Figure 6 shows the results of STFFT for this task 2. We find that μ rhythms are suppressed and there are a lot of noises as indicated by ‘A’ in other frequency bands that is probably caused by eye-blinking artifacts.

Results of short-time FFT for task 2. μ rhythm is suppressed and there are many mixed noises as indicated by ‘A’ probably caused by eye-blinking artifacts.

Full size image

Task 3: motion imagery task

In this task, the subject is asked to continuously image to flex his left arm but not really perform an actual motion while he is gazing at the marker at the endpoint of the robot arm as shown in Figure 4 task 3.

This task is designed to get the subject’s EEG in motion imagery. It is expected that his μ rhythm would be further decreased in this task. Same as the purpose of task 2, the second purpose of this task is to see what happens to the subject’s β rhythm in motion imagery.

Figure 7 shows the results of task 3 with STFFT. We find that μ rhythms are suppressed but β rhythms are generated. This motion imagery task confirmed such a fact that motion imagery can really result in the changes of μ and β rhythms.

Results of short-time FFT for task 3. μ rhythm is suppressed; meanwhile, β rhythm is generated in motor imagery.

Full size image

Task 4: motion task

In this task, the subject is actually holding a 4 kg dumbbell in 90° angle of his left elbow joint while he is still gazing at the marker at the endpoint of the robot arm as shown in Figure 4 task 4. Of course, to conduct the holding task of a dumbbell, the subject will first have an intent, or saying, image, to flex his left arm. This task is in fact a combination of motion imagery and real motion. In this paper, we simply define it as motion task.

This task is designed to get the subject’s EEG in motion task. It is expected that his μ rhythm would be suppressed to minimum and his β rhythm would be further increased. The results of this task with STFFT are shown in Figure 8. We find that μ rhythm is really suppressed and β rhythm is greatly generated as we expected.

Results of short-time FFT for task 4. μ rhythm is suppressed and β rhythm is greatly produced in motion and motion imagery.

Full size image

Spectrum characteristics of each task and some remarks

As described above, tasks 1 to 4 are designed according to the degree of brain excitation with/without the existence of motion. It is expected that the spectrum of μ rhythm will decline and the spectrum of β rhythm will change, probably will increase, task by task.

Since the spectrum of β rhythm is much less than that of μ rhythm, the spectrum of EEG is normalized with the ratio X _r =X_(t)/X_(max,t), where X_(t) and X_(max,t) are the spectrum and the maximum spectrum in the frequency band 1 to 30 Hz during the time interval t, respectively.

The EEG between C₄ and A₂ in each task is measured in 24 trials. The measurement time is 30 s, and the window width of STFFT (i.e. the analytical period) is 100 ms. The averages of the spectrum ratios (here, we sometimes still call it as power spectrum) in 24 trials for 30 s are shown in Figure 9. From this figure, we can find that μ rhythm greatly occurs in task 1 and it is really suppressed in other three tasks as we expected. On the other hand, the suppression extent of μ rhythms in other three tasks is not so gradually intensified as the degree of brain excitation is increased task by task. The differences among the spectra of the other three tasks indeed exist, but they are not obvious. This is different from our expectation, in which we expected that the suppression of μ rhythm would be gradually intensified as the increases of the brain excitation degree.

Averages of normalized power spectrum of each task in 24 measurement times for 30 s.

Full size image

Meanwhile, we can also find that the β rhythm basically increases from tasks 1 to 4. This shows the fact that the spectrum in β band increases according to the excitation degree of the brain. In this study, these two features of μ and β rhythms are used to control the robot.

Experiments

The subjects are two young men. Before experiments, they gave the consent for the study, which was reviewed and approved by the Ethics Committee of our university. In order to get the EEG signals in motion imagery and/or motion, each subject who is wearing a head cap with embedded electrodes is asked to perform the defined four tasks, and his μ rhythm and β rhythm extracted from the EEG are used to control a robot arm.

Control approach

The flowchart of controlling a robot arm using EEG signals is shown in Figure 10. The EEG signals from the electrodes are amplified by an amplifier and input to a personal computer for A/D conversion via an interface board. Then, STFFT is implemented to the converted data series to obtain their spectra X_(k,t) (k means frequency band and t indicates time). Further, μ rhythm (X_(μ,t)) and β rhythm X_(β,t) are extracted from the computed X_(k,t). The target joint angle θ _d of the robot arm is calculated according to the obtained spectra of μ rhythm(X_(μ,t)) and β rhythm X_(β,t) with the following formula,

Control flowchart.

Full size image

where X_{(m a x,t)} is the maximum spectrum in the frequency band 1 to 30 Hz during the time duration t, and K _μ and K _β are the weight coefficients for μ and β rhythms, respectively. Since the spectrum magnitudes of μ rhythm and β rhythm are different and the spectrum magnitude of β rhythm is much less than that of μ rhythm, the two different weight coefficients respectively for μ rhythm and β rhythm are necessary. Further, in the measured EEG signals, there are a lot of noises such as artifacts. These contaminated EEG signals in μ and β bands may lead to a sharp change of the target angle θ _d . This would result in an abrupt movement of the robot arm. To prevent this, a 5-Hz low-pass filter is employed to smooth the output of the target angle θ _d . Moreover, a threshold is set to the filtered target angle θ _d to avoid frequently trivial motions of the robot arm. Finally, this target angle θ _d is used as the command for position control of the robot arm.

Experimental procedures

The experimental procedures are basically same as the steps for four tasks described in Section ‘Task design for motion imagery and motion’, in which the subjects are asked to perform each task for 38 s. The EEG is measured between C ₄to A ₂ and the reference point is C _z . The ground of the measurement system is right wrist. This ground is separated from the ground of the motor driving system to prevent the subject from the possible electrical shock. The angle of the robot arm is controlled to investigate the performance of each task. The experiment system is shown in Figure 11.

Experimental system.

Full size image

Experimental results

The experimental results of subjects A and B are shown in Figures 12 and 13, respectively. In the two figures, the horizontal axes indicate time and the vertical axes represent the controlled angle of the robot arm. The experimental time from the start is 38 s for all of four tasks.

Experimental results of subject A.

Full size image

Experimental results of subject B.

Full size image

Remarks on experimental results

Figures 12 and 13 show the experimental results of controlling the robot arm using extracted EEG features from the designed four tasks. We find:

In task 2 and task 3 of subject A (Figure 12), from start to about 12 s, the angle of the robot arm in task 3 is larger than that in task 2. However, after that, this difference has disappeared. Further, after 21 s, the situation is reversed, that is, the angle of the robot arm in task 3 decreases to be less than the angle in task 2. For subject B (Figure 13), from start to about 9 s, the angles of the controlled robot arm in task 2 are almost the same as the angles in task 3. But from 9 to 23 s, the angle of the robot arm in task 3 is larger than that in task 2. However, after 17 s, the angle in task 3 decreases and then has a slight rise.

The reason for such changes could be interpreted as follows. Task 3 is mainly related to the motion imagery and such kind of action of the subjects to perform motion imagery is greatly dependent on his concentration. It could be postulated that after 12 s (subject A) or 17 s (subject B), the concentration of the subjects starts to decline and after 21 s, the subject A’s concentration further recedes. Though we could not observe the similar phenomenon for subject B, it could be considered as an individual difference.

The experimental results tell us two facts. One is that the subject training for motion imagery is necessary to keep his/her concentration. On the other hand, long-time motion imagery may not be a good way for robot control or external device control since the subject could not keep his concentration over dozen seconds.

For both subjects A and B, different from the results of tasks 2 and 3, during the entire experiment duration, the angles of the robot arm in the task 1 are almost always the minimum in all of four tasks. Contrarily, the angles corresponding to task 4 are basically always the maximum in all of four tasks. In other words, compared with the results in tasks 2 and 3, the angles of the robot arm in tasks 1 and 4 are more stable. This phenomenon is meaningful.

The result of the task 1 is natural because the brain excitation of the subject is the lowest in all of the four tasks. Though it is comparatively stable, this task has no practical application value since the subjects in this task have not done and thought anything.

However, in task 4, the subjects are holding a 4 kg dumbbell in 90 ° angle of his left elbow joint while gazing at the marker at the endpoint of the robot arm. Since their brain excitation is the highest, the angles of the robot arm are basically always the maximum in all of four tasks. This result tells us such a fact that the EEG signals of the subjects are comparatively stable when the subjects are holding or touching something. Furthermore, this fact implies that EEG signal can be used for robotic power assist to help a subject holding or moving a heavy load.

In this study, aimed to construct a robot control system using EEG signals, four different tasks with different human’s brain excitation degree are designed, and their spectra of μ and β rhythms of EEG signals in the four tasks are extracted with STFFT. These spectra are used to control a one-joint robot arm. The experimental results show that the classic motion imagery cannot be simply used to real robot control since the subject’s concentration cannot be kept over dozen seconds. Contrarily, since the EEG signals in the task of holding a load are comparatively stable, we conclude that this holding task has a great application potential to be used to construct a robotic power assist system to help person to hold or move a heavy load. This is one of our future work.

Affiliations

1. Department of Systems Life Engineering, Maebashi Institute of Technology, Kamisadori 460-1, Maebashi, 371-0816, Gunma, Japan
   • Masataka Yoshioka
   • , Chi Zhu
   • , Kazuyuki Imamura
   •  & Feng Wang
2. Department of Bioengineering, Faculty of Engineering, National University of Singapore, 9, Engineering Drive 1, Singapore City, 117576, Singapore
   • Haoyong Yu
3. Department of Automation, College of Information Technical Science, Nankai University, No. 94 Weijin Road, Tianjin, 300071, People’s Republic of China
   • Feng Duan
4. Department of Bioengineering, School of Engineering, Santa Clara University, 500, El Camino Real, Santa Clara, 95053, CA, USA
   • Yuling Yan

Corresponding author

Correspondence to Chi Zhu.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

MY and CZ proposed the basic idea of this study and designed the experiments. Moreover, MY implemented the experiments, analyzed and interpreted the experiment data. He also wrote the draft of this manuscript. CZ revised and complete the manuscript. KI and FW advised the experiment design and the measurement of EEG signals. HY, FD and YY advised the method of signal processing. All authors read and approved the final manuscript.

Reprints and Permissions