Learning by imitation with the STIFF-FLOP surgical robot: a biomimetic approach inspired by octopus movements
© Malekzadeh et al.; licensee Springer. 2014
Received: 30 June 2014
Accepted: 25 September 2014
Published: 16 October 2014
Transferring skills from a biological organism to a hyper-redundant system is a challenging task, especially when the two agents have very different structure/embodiment and evolve in different environments. In this article, we propose to address this problem by designing motion primitives in the form of probabilistic dynamical systems. We take inspiration from invertebrate systems in nature to seek for versatile representations of motion/behavior primitives in continuum robots. We take the perspective that the incredibly varied skills achieved by the octopus can guide roboticists toward the design of robust motor skill encoding schemes and present our ongoing work that aims at combining statistical machine learning, dynamical systems, and stochastic optimization to study the problem of transferring movement patterns from an octopus arm to a flexible surgical robot (STIFF-FLOP) composed of two modules with constant curvatures. The approach is tested in simulation by imitation and self-refinement of an octopus reaching motion.
Time allowed biological organisms to refine and find smart ways to behave and survive in their environment. Hence, imitation of the models, systems, and elements of nature for the purpose of solving complex human problems (biomimetics) has given rise to new technologies in robotics, both at the level of hardware and software.
In minimally invasive surgery (MIS), widely used in the abdominal operations in the last 30 years , tools go through narrow openings and manipulate soft organs to perform surgical tasks. However, due to the limited flexibility and maneuverability of the available tools in MIS, up to four or five trocar accesses are required during operation. To cope with this issue, novel robotic surgical instruments have been developed to give higher flexibility and dexterity to the surgeons . Such systems are actuated by motors moving rods , gears , or a combination of cables and rods . Yet, stiffness control of such rigid mechanical structures remains a largely unexplored area. The actuation techniques presented in - are not able to have different level of stiffness in different parts of the robot. In , the robot consists of two snake-like robots, one covering the other. They use cable-driven mechanism to stiff the inner robot while the outer longer robot can be compliant at the tip. By using this technique, it is not possible to have a robot compliant at the proximal part and stiff at the distal part. A motor-driven robot has been designed in  with two arms and the motors embedded on the arms. This robot provides only stiff configuration at the tip suitable to perform surgical tasks in single-port laparoscopy (SPL).
The STIFF-FLOP project aims at developing biologically inspired manipulators and soft robotic arms to manipulate objects while controlling the stiffness of selected body part . They are inspired by biological systems such as an octopus arm or elephant trunk. There are several examples of this kind of robots with high degrees of freedom (DOF) such as the OCTARM continuum robot  or the Clemson University elephant trunk . In spite of being inherently compliant when interacting with objects, their rigid nature imposes limitations in their usability in the medical field. Hence, some other manipulators with no rigid structures were developed, e.g., the Air-Octor , which combines pneumatic tendon actuation or the arm developed within the OCTOPUS project .
None of the mentioned systems answer to the MIS requirements since flexibility and softness come generally with limited force; while high forces are difficult to be achieved without any rigid supports . A different approach has been followed in the STIFF-FLOP project  based on variable and controllable stiffness techniques using composite granular jamming, membrane coupling, and pneumatic actuation .
There is a wide range of bioimitation mechanisms, ranging from the blind copy of actions (mimicry) to the understanding of the intent underlying an observed action (goal-level imitation), associated with self-training capability required to match the extracted intent. Several approaches in machine learning exist to cover this span of imitative behaviors.
The core idea of acquiring skills by imitation has been labeled over the years with various names such as teaching by showing, robot coaching, programming by demonstration, or learning from demonstration (LfD) . A survey of LfD techniques addressing robotics control problems has been presented in . In , Billard et al. present an overview of different machine learning approaches in robot programming by demonstration.
In programming by demonstration, providing good demonstration plays a key role to speed up the learning process. This is not always easy, specially when the contexts for the demonstration and the reproduction are not sufficiently similar. This may happen, for example, when the structure of the demonstrator agent and the robot are very different. The generalization capability is also often limited to a certain range of situations, which shows the importance of designing representations of skills and tasks that can be trained by various learning strategies. In particular, one key challenge is to allow the imitation system to continuously adapt the learned model when new demonstrations are available, by also exploiting in parallel exploration, auto-calibration, and adaptation strategies. In some situations, the design of efficient self-refinement algorithms is crucial for the robot to learn how to perform a task in new situations . A robust and flexible representation of skills and movements is one of the keys to enable robots to jointly exploit several learning methodologies. In this article, we study this problem in the case of biological or robotic systems characterized by a hyper-redundant kinematic chain.
In ,, we introduced a high-level skill transfer approach by learning context-dependent reward functions in the same vein as in inverse reinforcement learning (IRL)  problems. We first used a statistical approach to extract the underlying intents of a demonstrated action (by kinesthetic teaching with a 7-DOF Barrett WAM), in the form of context-dependent objective functions, and then transferred these high-level goals to the STIFF-FLOP robot by using a stochastic reward-weighted self-refinement strategy so that the robot could find a policy matching the extracted objectives.
We use a representation of the octopus arm movement which enables us to transfer motion skills to a STIFF-FLOP robot with two modules. The representation, based on Euler angles and offset values following the standard Denavit-Hartenberg convention in robotics, is invariant to rotation and translation in Cartesian coordinate system. The use of rotation-translation invariant representations is indeed highly preferable if we want to exploit biological motion data in skill transfer applications. Otherwise, the extracted features can hardly be generalized to other situations.
Our approach then exploits the interaction between statistical models and continuous dynamical systems. In particular, the use of statistics allows the system to discover relevant invariants and correlations among the control variables in the form of joint probability distributions, information that is later exploited in the reproduction phase to reproduce appropriate movements when confronted with perturbations.
In this article, we present a first set of experiments to validate that the proposed approach can effectively enable the STIFF-FLOP robot to mimic a motion recorded from an octopus arm.
2Octopus reaching motion
Octopuses are considered to be among the most developed and intelligent animals in the invertebrate kingdom, with impressive skills that can be attributed to the high maneuverability of their arms and the capacity of the peripheral nervous system to process sensory information and control arm movements . The octopus can use its arms for various tasks such as locomotion, food gathering, hunting, and sophisticated object manipulation.
Motion primitives can be regarded as a minimal set of movements, which can be combined in different ways to give rise to rich movement repertoires and facilitating the acquisition of skills . In , Zelman et al. demonstrated that the kinematics of octopus arm movements can be described by a reduced set of motion primitives, which required the analysis of different types of arm movements. They showed that a weighted combination of 2D Gaussian functions, in the form of a Gaussian mixture model (GMM), could be used to span arm behavior in 3D space.
The encoding approach that we propose takes inspiration from this work but provides two important novelties. It first extends the approach to a dynamical system to handle perturbations in the environment. It then exploits the GMM in a different way. In , each Gaussian is used as a parametric function to fit the local shape of a surface (as bell-shaped components), while it is used in our application to fit the joint distribution of the data. Combined with a regression approach, this allows us to exploit the statistical properties of the model if multiple demonstrations are available. Our work also differs in the sense that the analysis and encoding are not the final targets. Namely, we do not aim at classifying octopus movements or studying movements from a physiological perspective. Instead, we study the problem of designing technical approaches that could be used to transfer movement skills from biological systems to the STIFF-FLOP robot. This goal is reflected by the proposed encoding approach that has closer links to robotics systems, with a reversible representation with three Euler angles and a scalar offset, depending on two input variables (time and arm-index). The representation facilitates retargeting to the robot by representing the two agents as kinematic chains with Denavit-Hartenberg parameters.
In , a motion capture system for 3D tracking of octopus arm movements was developed to create a database containing several types of arm movements from octopuses of different ages and sizes. The problem is very challenging. Indeed, the soft property of the octopus arm and its flexible ability to change shape does not allow the use of conventional shape recognition algorithms. In contrast to human motion analysis, it is also not possible to attach markers on the octopus arms as it immediately rejects them .
In this paper, we exploited one of the most representative reaching movement from this database, gratefully provided by Prof. Binyamin Hochner, Shlomi Hanassy, and Alexander Botvinnik from the Department of Neurobiology, Hebrew University, Jerusalem, Israel, and Prof. Tamar Flash from the Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel.
Recording and storing such data require discrete sampling in both time and space. The raw data consists of discrete Cartesian position data with (on average) 100 points along the octopus arm and (on average) 100 time steps. To reduce the computational time, we re-sampled the data in both arm-index and time with 50 instances (surface of 2,500 points).
3.1 Spatiotemporal representation of octopus movements
In order to generate a mathematical model of the octopus movements, the arm was approximated as a robot with a high number of links. The continuous arm is approximated by a kinematic chain with a high number of revolute joints, describing the local curvature and torsion, alternated with prismatic joints describing the local elongation. The forward kinematics of such system can thus be described by a set of three Euler angles and a scalar offset. The ZYX decomposition of the Euler angles resulted in the most appropriate selection for the motion range in our dataset. This representation is reversible; i.e., by having the angles and offset, the points in Cartesian space can be retrieved and vice-versa.
The original raw data (noisy Cartesian positions) are first resampled and smoothed through a two-dimensional polynomial fitting (surface fitting) with a 7-degree polynomial. The degree for the polynomial is set experimentally by testing different orders. The arm movement is translated to keep the base of the octopus at the origin. The data is then globally rescaled to match the size of the robot.
3.2 Dynamical system with evolution over time and arm-index
Dynamical movement primitives (DMP) is a popular tool in robotics to learn and reproduce movements while being robust to perturbations . It consists of a spring-damper system modulated by a non-linear force profile encoded as a series of predefined basis functions associated with local force behaviors (or alternatively, by the path of a virtual spring-damper system producing a non-linear force profile). We introduced in  a probabilistic formulation of DMP by jointly learning the basis functions and the local force behaviors. The trajectory of the attractor point is then encoded by means of a statistical model (as explained later), whence the trajectory can be later extracted by regression.
The novelty in the current work is the use of surface attractors instead of trajectory attractors (generalization of the generic spring-damper system to a spatiotemporal dynamical system). We use both arm-index s and time t (rather than only time) as input variables, which enables the approach to encode the movement of invertebrate arm movements with a compact model (namely, by encoding the movement of all points along the arm).
where and are respectively the stiffness of the spring and the damping ratio of the damper. and are the velocity and acceleration calculated by differentiating x for each s with respect to t. In other words, the observations x of the motion are converted into the trajectory of virtual attractors y for each arm-index. Similarly as x=[θ z ,θ y ,θ x ,Δ L] in Figure 2, y can thus be represented as a set of attractor surfaces.
3.3 A probabilistic model of continuous movements
where D is the dimension of variables and K is the number of Gaussian components.
where N is the number of datapoints. In the E-step, h i (ξ j ) is the posterior probability , computed by the Bayes theorem. This is the probability that the Gaussian component k is responsible for ξ j . EM guarantees the improvement of the likelihood at each iteration, converging to a local optimum . The EM algorithm is initialized with k-means clusteringb to start the iterative procedure from a good initial estimate. Here, K is determined empirically.
where the superscripts I and O represent input and output variables.
In contrast to other regression methods, GMR does not model the regression function directly, but models the joint probability of the data. It then derives the regression function from the joint probability model. Density estimation can thus be learned in an off-line phase (with the GMM encoding approach presented above) and extracted in real time at reproduction, since GMR has a low computation time independent from the number of datapoints used to train the model.
We will use the term DS-GMR to refer to this combination of dynamical system and Gaussian mixture regression.
3.4 Self-refinement with reward-weighted regression
The model parameters above can then be optimized through a self-refinement algorithm with an objective or reward function defined according to the different objectives of the task. The self-refinement process consists of searching for new solutions in the policy parameter space, associate a reward to each trial, and then sample from the exploration space by a weighted combination of the most successful trials obtained so far.
Dayan and Hinton originally suggested that a self-refinement problem can be tackled by EM to avoid gradient computation . They introduced the idea of treating immediate rewards as probabilities of a fictitious event, in which case probabilistic inference techniques can be used for optimization. They maximize the reward by solving a sequence of probability matching problems, where the task parameters are chosen at each step to match a fictitious distribution determined by the average rewards experienced on the previous steps. Although there can be large changes in the task parameters from one step to the next, there is a guarantee that the average reward is monotonically increasing. From this simple idea, various reward-weighted policy learning approaches emerged -. Indeed, several research fields converged to similar algorithmic solutions, with approaches such as the cross-entropy method (CEM)  or the covariance matrix adaptation evolution strategy (CMA-ES) .
One option for implementing such self-refinement is to use an EM-based stochastic optimization algorithm to refine the GMM parameters encoding the movement, similarly as in . The procedure corresponds to an EM algorithm in which the reward signal is treated as a pseudo-likelihood, which can easily be extended to multi-optima policy search .
Another line of research is to explore methodologies that could move toward more structured techniques of exploration. In , it was proposed to speed up the search process by redefining it as an iterative reward-weighted regression problem. This is particularly relevant for the subclass of problems in which we have access to the goal or to the highest value that a reward can take (e.g., reaching the center of a target, be as close as possible to a reference trajectory, etc.).
In this case, the reward (or objective) is expressed in the form of a vector, and the aim of the optimization is treated as an iterative reward-weighted regression problem in the augmented space formed by the policy parameters and the achieved goals. At each iteration, a Gaussian distribution showing the joint probability is fit to the augmented data containing the best policies and the corresponding goals obtained so far. The input of the regression problem is represented by the desired goal ζ I ∗. The output is a candidate optimal policy that is tested on the system and that will be associated with the corresponding outcome.
where Σ 0 is a predefined minimal exploration noise. With the joint probability in Eq. (5), the conditional probability can then be computed to obtain a subsequent policy (see Eq. (3)), by stochastic sampling from the retrieved Gaussian distribution in the policy parameter space.
Thus, at each iteration of the algorithm, a new policy ζ O is evaluated and associated with the achieved goal ζ I and the corresponding scalar reward r (calculated through a reward function of exponential form). This iterative process continues until convergence or a maximum number of iterations is attained.
Similarly to EM-based stochastic optimization, the process can be extended to a GMM/GMR representation for multi-policy search .
In the proposed experiment, the process starts from a set of randomly generated policies, where the initial number has been determined empirically. The objective function is also explicitly defined.
4STIFF-FLOP robot kinematics
The first prototype of the STIFF-FLOP robot, currently under development, will be composed of two cylindrical modules. Each section will consist of a soft cylinder with three chambers disposed concentrically around the axis, where air is inflated to bend the link in the desired orientation. A central chamber filled with hard grain-shaped particles is used to stiffen the link at a desired orientation by air suction.
In the local frame of each module, the rest position (no chamber is inflated) corresponds to the module aligned along the vertical axis e 3, with a rest length L 0, see Figure 6 -left.
Here, γ corresponds to all possible points from the base of the module to the tip (γ=0 corresponds to the base).
We will use the Cartesian position Q i of the tip in the rest frame of the base as internal variables. They will replace the role of joint angle commonly used as internal variables in kinematic models of standard manipulators.
The use of the constant curvature coordinates allows us to evaluate the possible limits that the hardware possesses. The only limitation on the possible configurations are placed in the fact that the inflation mechanism only allows a limited range of elongation, depending on the bending of the module and its orientation in space. The length of the robot can be obtained as a function of the constant curvature coordinates as L i =ρ i β i .
Once the robot is bent in a given direction, a range of possible elongations can be obtained by fixing the curvature radius ρ i and varying the arclength β i , which is achieved by inflating the chambers. The geometry of the system suggests that this elongation also depends on the bending direction (i.e., which chamber is inflated to get that curvature). As a result, we will have joint limitations such as β min(α i ,ρ i )<β i <β max(α i ,ρ i ), which will need to be obtained experimentally (no workspace analysis and joint limit measurements have been performed on the prototype so far). As a starting hypothesis, we will consider limitations corresponding to the nominal elongation when all the chambers are inflated (80%), thus limiting the possible lengths of the robot to L 0<L i <L 0+0.8L 0.
with R 00=I. For two links, this results into y=Q 1+R 01 Q 2 and R 03=R 01 R 12.
5Results and discussion
The model is first tested to emulate two typical octopus reaching motion primitives by manually setting the model parameters. It is then tested if these parameters could be autonomously learned from real octopus motions and transferred to the STIFF-FLOP robot.
5.1 Emulation of octopus motion primitives
The model is first employed to emulate typical movements used by octopus to reach for food. These movements involve bend propagation and elongation of the proximal part of the arm. Note here that the reaching motion differs from the fetching of the grasped food to the mouth, where the octopus arm is configured into a quasi-articulated structure with three segments and three rotary joints ,,. In contrast, reaching movements are usually generated by a combination of two motor primitives: the propagation of a bend along the arm and an elongation of the arm ,. These two motor primitives may be combined with different weights to create a broader spectrum of reaching movements.
The advantage of the DS-GMR representation to encode such kind of patterns is that it provides not only information about individual bending/elongation patterns but also information about their local correlations throughout the movement, in the form of a full covariance matrix.
5.2 Imitation of octopus motion by the STIFF-FLOP robot
In this section, we test if the proposed encoding technique can be used to autonomously transfer skills from real octopus movements to the STIFF-FLOP robot.
The DS-GMR model is exploited to transfer the movement to a two-link STIFF-FLOP robot. The constant curvature constraint of each link results in a segmentation of the continuous surfaces into a given number of subsurfaces, corresponding to the number of modules (here, two).
In order to map the movement to the STIFF-FLOP robot, we consider the Frenet frames at the base of the robot (R 1), in the middle point between the two modules (R n/2), and at the tip (R n/2), where n is the number of segments used for the discretization of the continuous kinematic chain.
where, for example, is the scalar in the second row and third column of matrix , and Δ L is the offset.
This approach for the transfer of skills from a continuous arm to an arm with piecewise constant curvatures can provide a good initial estimate but does not guarantee the best fit. Indeed, even though the movement is quite similar, the different structure of the STIFF-FLOP robot can cause some dissimilarity in the learned movement.
The self-refinement approach is thus used to refine this initial estimate, by using as policy parameters the centers of the GMM, as well as the first eigencomponents of ordered eigendecompositions of the covariances (directions of the main axes of the ellipsoids).
where δ is a bandwidth coefficient set experimentally. With three 3D points at different time instances, the goal has nine dimensions.
where L 0, L, and β are the module minimum length, length, and curvature, respectively (see Figure 6).
We presented an approach based on statistical dynamical systems to encode movements in biological or robotic systems with a continuous kinematic structure. We showed that the approach could be employed to learn skills from demonstration, and that it could be combined with a self-refinement strategy based on iterative reward-weighted regression.
The aim is to extract relevant motion primitives from biological systems and map those to the STIFF-FLOP robot, while creating a set of natural motion patterns as building blocks. These building blocks could later be combined and reorganized differently to build new types of movements and skills.
The case study with octopus reaching movements, of course, does not target as a final goal to replicate reaching movements in surgery. But such typical motion repertoire allows us to test the efficiency of the proposed encoding approach and to exploit existing databases of biological movements. Even if the same exact movements will not be used in the surgical application, it provides us with a very important starting point to the design of robust models capable of encoding various skills in flexible continuum robots. In other words, the presented work is primarily aimed at exploiting typical behaviors observed in invertebrate systems to test an encoding strategy and guide us toward the design of new modeling techniques that could be used to encode a broader range of skills and patterns, including new skills that will be relevant to surgical applications.
In future work, we aim at applying this learning approach to the real STIFF-FLOP platform, by considering diverse motion/feedback skill primitives that could be used to assist doctors in surgical operations. We plan to exploit the context-dependent learning interface that we proposed in  as a high-level imitation approach to extract and exploit the most relevant cost functions that could explain the observation of octopus movements. In particular, we plan to study how the octopus exploits the degrees of freedom of its flexible arm to perform various tasks, in the form of objective functions that could vary depending on the ongoing situation. The variability information could then be exploited to regulate the stiffness and damping of the robot, with appropriate synergies among curvature, torsion, and elongation variables.
a Note that in this implementation, the evolution is only semi bi-dimensional in the sense that the evolution over s with t=0 is first computed, followed by evolutions over t for s∈[0,1], providing a continuous surface.
b k-means is a method of clustering that aims to partition observations into K clusters in which each observation belongs to the cluster with the nearest mean. The center of each cluster after applying k-means, has the minimum sum of distances from all other points in that cluster.
We gratefully acknowledge Prof. Binyamin Hochner, Shlomi Hanassy, and Alexander Botvinnik from the Department of Neurobiology, Hebrew University, Jerusalem, Israel, and Prof. Tamar Flash from the Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel, who kindly provided us with the 3D octopus motion data processed from vision sensing and used in this article. We would also like to thank them for their truly inspiring work and for their recommendation concerning the interpretation of the collected octopus behaviors. This work was supported by the STIFF-FLOP European project (FP7-ICT-287728).
- Reynolds W: The first laparoscopic cholecystectomy. JSLS 2001, 5(1):89–94.Google Scholar
- Vitiello V, Lee S, Cundy TP, Yang G: Emerging robotic platforms for minimally invasive surgery. IEEE Rev Biomed Eng 2013, 6: 111–126. 10.1109/RBME.2012.2236311View ArticleGoogle Scholar
- Degani A, Choset H, Wolf A, Zenati MA (2006) Highly articulated robotic probe for minimally invasive surgery In: Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), 4167–4172, Orlando, FL, USA.Google Scholar
- Piccigallo M, Scarfogliero U, Quaglia C, Petroni G, Valdastri P, Menciassi A, Dario P: Design of a novel bimanual robotic system for single-port laparoscopy. IEEE/ASME Trans Mechatronics 2010, 15(6):871–878.Google Scholar
- Bajo A, Goldman RE, Wang L, Fowler D, Simaan N (2012) Integration and preliminary evaluation of an insertable robotic effectors platform for single port access surgery In: Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), 3381–3387, Saint Paul, MN, USA.Google Scholar
- Smith KK: Trunks, tongues, and tentacles: moving with skeletons of muscle. Am Sci 1989, 77: 29–35.Google Scholar
- McMahan W, Chitrakaran V, Csencsits M, Dawson D, Walker ID, Jones BA, Pritts M, Dienno D, Grissom M, Rahn CD (2006) Field trials and testing of the OctArm continuum manipulator In: Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), 2336–2341, Orlando, FL, USA.Google Scholar
- Walker ID (2000) Some issues in creating “invertebrate” robots In: Proc. intl symp. on adaptive motion of animals and machines, Montreal, Canada.Google Scholar
- McMahan W, Jones BA, Walker ID (2005) Design and implementation of a multi-section continuum robot: air-octor In: Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), 2578–2585, Edmonton, Canada.Google Scholar
- Martinez RV, Branch JL, Fish CR, Jin L, Shepherd RF, Nunes R, Suo Z, Whitesides GM: Robotic tentacles with three-dimensional mobility based on flexible elastomers. Adv Mater 2013, 25(2):205–212. 10.1002/adma.201203002View ArticleGoogle Scholar
- Cianchetti M, Ranzani T, Gerboni G, De Falco I, Laschi C, Menciassi A (2013) STIFF-FLOP surgical manipulator: Mechanical design and experimental characterization of the single module In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3567–3581, Tokyo, Japan.Google Scholar
- STIFF-FLOP European project official website, (2014) [http://www.stiff-flop.eu/] .
- Jiang A, Xynogalas G, Dasgupta P, Althoefer K, Nanayakkara T (2012) Design of a variable stiffness flexible manipulator with composite granular jamming and membrane coupling In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2922–2927, Vilamoura, Portugal.Google Scholar
- Kuniyoshi Y, Inaba M, Inoue H (1989) Teaching by showing: Generating robot programs by visual observation of human performance In: Proc. intl symposium of industrial robots, 119–126, Tokyo, Japan.Google Scholar
- Riley M, Ude A, Atkeson C, Cheng G (2006) Coaching: An approach to efficiently and intuitively create humanoid robot behaviors In: Proc. IEEE-RAS intl conf. on humanoid robots (humanoids), 567–574, Genova, Italy.Google Scholar
- Billard A, Calinon S, Dillmann R, Schaal S: Robot programming by demonstration. In Handbook of robotics. Edited by: Siciliano B, Khatib O. Springer, Secaucus; 2008:1371–1394. 10.1007/978-3-540-30301-5_60View ArticleGoogle Scholar
- Argall BD, Chernova S, Veloso M, Browning B: A survey of robot learning from demonstration. Robot Auton Syst 2009, 57(5):469–483. 10.1016/j.robot.2008.10.024View ArticleGoogle Scholar
- Calinon S, Bruno D, Malekzadeh MS, Nanayakkara T, Caldwell DG: Human-robot skills transfer interfaces for a flexible surgical robot. Comput Methods Programs Biomed 2014, 116(2):81–96. Special issue on new methods of human-robot interaction in medical practice Special issue on new methods of human-robot interaction in medical practice 10.1016/j.cmpb.2013.12.015View ArticleGoogle Scholar
- Malekzadeh MS, Bruno D, Calinon S, Nanayakkara T, Caldwell DG (2013) Skills transfer across dissimilar robots by learning context-dependent rewards In: Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), 1746–1751, Tokyo, Japan.Google Scholar
- Ng AY, Russell SJ (2000) Algorithms for inverse reinforcement learning In: Intl conf. on machine learning, 663–670, Stanford, CA, USA.Google Scholar
- Zelman I, Titon M, Yekutieli Y, Hanassy S, Hochner B, Flash T: Kinematic decomposition and classification of octopus arm movements. Front Comput Neurosci 2013, 7: 60. 10.3389/fncom.2013.00060View ArticleGoogle Scholar
- Yekutieli Y, Sagiv-Zohar R, Aharonov R, Engel Y, Hochner B, Flash T: Dynamic model of the octopus arm. I. Biomechanics of the octopus reaching movement. J Neurophysiol 2005, 94(2):1443–1458. 10.1152/jn.00684.2004View ArticleGoogle Scholar
- Gutfreund Y, Flash T, Fiorito G, Hochner B: Patterns of arm muscle activation involved in octopus reaching movements. J Neurosci 1998, 18(15):5976–5987.Google Scholar
- Gutfreund Y, Flash T, Yarom Y, Fiorito G, Segev In, Hochner B: Organization of octopus arm movements: a model system for studying the control of flexible arms. J Neurosci 1996, 16(22):7297–7307.Google Scholar
- Flash T, Hochner B: Motor primitives in vertebrates and invertebrates. Curr Opin Neurobiol 2005, 15(6):660–666. 10.1016/j.conb.2005.10.011View ArticleGoogle Scholar
- Zelman I, Galun M, Akselrod-Ballin A, Yekutieli Y, Hochner B, Flash T: Nearly automatic motion capture system for tracking octopus arm movements in 3d space. J Neurosci Methods 2009, 182(1):97–109. 10.1016/j.jneumeth.2009.05.022View ArticleGoogle Scholar
- Ijspeert A, Nakanishi J, Pastor P, Hoffmann H, Schaal S: Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Comput 2013, 25(2):328–373. 10.1162/NECO_a_00393MathSciNetView ArticleMATHGoogle Scholar
- Calinon S, Li Z, Alizadeh T, Tsagarakis NG, Caldwell DG (2012) Statistical dynamical systems for skills acquisition in humanoids In: Proc. IEEE intl conf. on humanoid robots (humanoids), 323–329, Osaka, Japan.Google Scholar
- Dempster AP, Laird NM, Rubin DB: Maximum likelihood from incomplete data via the EM algorithm. J Roy Stat Soc 1977, 39(1):1–38.MathSciNetMATHGoogle Scholar
- Wu C: On the convergence properties of the EM algorithm. Ann Stat 1983, 11: 95–103. 10.1214/aos/1176346060MathSciNetView ArticleMATHGoogle Scholar
- MacQueen JB (1967) Some methods for classification and analysis of multivariate observations In: Proc. of the 5th Berkeley symp. on mathematical statistics and probability, 281–297, Berkeley, CA, USA.Google Scholar
- Ghahramani Z, Jordan MI: Supervised learning from incomplete data via an EM approach. In Advances in Neural Information Processing Systems. Edited by: Cowan JD, Tesauro G, Alspector J. Morgan Kaufmann Publishers, Inc., Burlington; 1994:120–127.Google Scholar
- Dayan P, Hinton GE: Using expectation-maximization for reinforcement learning. Neural Comput 1997, 9(2):271–278. 10.1162/neco.19184.108.40.2061View ArticleMATHGoogle Scholar
- Peters J, Schaal S (2007) Using reward-weighted regression for reinforcement learning of task space control In: Proc. IEEE intl symp. on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), 262–267, Honolulu, HI, USA.Google Scholar
- Rueckstiess T, Sehnke F, Schaul T, Wierstra D, Sun Y, Schmidhuber J: Exploring parameter space in reinforcement learning. Paladyn J Behav Robot 2010, 1(1):14–24. 10.2478/s13230-010-0002-4Google Scholar
- Theodorou E, Buchli J, Schaal S: A generalized path integral control approach to reinforcement learning. J Mach Learn Res 2010, 11: 3137–3181.MathSciNetMATHGoogle Scholar
- Kober J, Peters J: Imitation and reinforcement learning: Practical algorithms for motor primitives in robotics. IEEE Robot Autom Mag 2010, 17(2):55–62. 10.1109/MRA.2010.936952View ArticleGoogle Scholar
- Stulp F, Sigaud O (2012) Path integral policy improvement with covariance matrix adaptation In: Proc. Intl Conf. on Machine Learning (ICML), 1–8, Edinburgh, Scotland.Google Scholar
- Kroese DP, Rubinstein RY: The cross-entropy method: A unified approach to combinatorial optimization. In Monte-Carlo simulation and machine learning. Springer, New York; 2004.Google Scholar
- Hansen N: The CMA evolution strategy: A comparing review. In Towards a new evolutionary computation. Studies in Fuzziness and Soft Computing. Edited by: Lozano J, Larranaga P, Inza I, Bengoetxea E. Springer, Berlin-Heidelberg; 2006:75–102. 10.1007/3-540-32494-1_4View ArticleGoogle Scholar
- Calinon S, Kormushev P, Caldwell DG: Compliant skills acquisition and multi-optima policy search with EM-based reinforcement learning. Robot and Autonom Syst, Elsevier 2013, 61(4):369–379. 10.1016/j.robot.2012.09.012View ArticleGoogle Scholar
- Kormushev P, Calinon S, Saegusa R, Metta G: Learning the skill of archery by a Humanoid robot iCub. In Proc. IEEE intl conf. on humanoid robots (humanoids), Nashville. TN, USA; 2010:417–423.Google Scholar
- Sumbre G, Fiorito G, Flash T, Hochner B: Neurobiology: Motor control of flexible octopus arms. Nature, Nature Publishing Group, New York City, NY, USA 2005, 433(7026):595–596.Google Scholar
- Hanassy S (2008) Reaching movements of the octopus involve both bend propagation and arm elongation. MSc thesis, Medical Neurobiology Department, Jerusalem, Israel: The Hebrew University.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.