Talk:Evolutionary Robotics
Very helpful overall summary of the field with good coverage. I have a few suggestions to expand it a bit:
Novelty is presented mainly as a way to avoid local optima, but researchers have recently been using it to do more than that. In particular, an interesting use of novelty is to search for a collection of discoveries than are each interesting or useful in their own right. My group has been calling this application of novelty "Quality Diversity" (QD) algorithms, which are usually some kind of combination of novelty with a quality measure get a diverse set of high-quality options (or a "repertoire"). (Examples are the NSLC algorithm and MAP-Elites algorithm.) Some of the work in this area is reviewed in the Background of our 2015 GECCO paper on QD: http://dl.acm.org/citation.cfm?id=2754664 (though more recent overviews are coming out in 2016).
> Authors’ reply: In the section titled “Guiding the search in evolutionary robotics”, we have added a new subsection called “Combining fitness and novelty” in which the combination of fitness and novelty is discussed both from an optimisation perspective, and in terms of Quality Diversity algorithms/Illumination algorithms.
> Reviewer's reply: My only new comment after the authors' edits is to suggest pointing readers interested in QD to our comprehensive review of the subject just published in Frontiers: http://journal.frontiersin.org/article/10.3389/frobt.2016.00040/full?&utm_source=Email_to_authors_&utm_medium=Email&utm_content=T1_11.5e1_author&utm_campaign=Email_publication&field=&journalName=Frontiers_in_Robotics_and_AI&id=202845
> Authors’ reply: We now point readers to Pugh et al. (2016) for a comprehensive review of Quality Diversity algorithms.
One notable example of QD is the recent Nature paper http://www.nature.com/nature/journal/v521/n7553/full/nature14422.html from Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. In it, they collect a repertoire of gaits for a quadruped with different leg usages. Interestingly, a particular gait is later chosen from among the collected options to suit the conditions of a robot that may be damaged, so it is relevant also to the "Damage Recovery" part of the ER article.
> Authors’ reply: In the “Damage Recovery” subsection, we now explicit mention that the Intelligent Trial and Error algorithm used in Cully et al.’s Nature paper employs the MAP-Elites algorithm to construct the behaviour-performance map.
Finally, multiagent HyperNEAT is also a good example of a multirobot system that has been used to evolve behaviors for multiple robots in the real world. For example: http://link.springer.com/article/10.1007%2Fs12065-012-0086-3 It's also an example of indirect encoding applied to multirobot systems. Jacob Schrum, Joel Lehman, and Sebastian Risi have also recently gone beyond multiagent HyperNEAT with a "multi-brain HyperNEAT": http://arxiv.org/abs/1604.07806
> Authors’ reply: In the subsection “Control for multirobot systems”, we have added a description of multiagent HyperNEAT and its advantages in the synthesis of control for groups of robots, along with a brief description of multibrain HyperNEAT.
Of course, it is easy enough to go on forever with references possibly of interest in ER, but I think those above are helpful in sketching a part of the more recent horizons the field is beginning to probe.
> Reviewer's Recommendation: Aside from the one additional suggested reference I link above, I am satisfied with the authors' revision. I approve publication assuming the one suggested reference is added. Congratulations to the authors on a very useful and important summary of the ER field.
In your definition of evolutionary robotics (first 4 lines of the manuscript) you stress the utilization of algorithms inspired by natural evolution to create the control and the body plans of robots. It is important to specify that ER enable to generate robots that adapt to their environment through a process analogous to natural evolution. Also, although the co-evolution of body and brain is certainly one of the most important topics in ER, only a part of the research in the field address this aspect. So I thing the definition should be reformulated by clarifying that evolutionary robotics enable the evolution robots in general terms, not necessarily the evolution of robots in which the morphology is co-evolved with the brain.
> Authors’ reply: We have revised the introduction section to address your comments and to aid clarity.
I think it is also important to clarify that Evolutionary Robotics differs from Artificial Life for the usage of physical robots. This has important implications: a stronger emphasis on the importance of embodiment and situatedness, the presence of constraints the prevent the realization of physically irrealistic simulations, and the possibility to synthesize physical agents (robots) able to operate in a physical environment.
> Authors’ reply: In the introduction section, we have also added a discussion on the differences between Evolutionary Robotics and Artificial Life regarding the usage of physical robots. We now explicitly state that ER puts a strong emphasis on embodiment and situatedness in a physical environment, and on the close interaction between brain, body, and environment.
You correctly point out that evolutionary robotics is used to carry on research in evolutionary biology, cognitive science. I think the cited work by Harvey at al. (2005) nicely cover the cognitive science domain. For biology I suggest to cite Floreano and Keller (2010), Evolution of Adaptive Behaviour in Robots by Means of Darwinian Selection. PLOS Biology that provide a general discussion of this issue.
> Authors’ reply: We now cite Floreano and Keller (2010) in the introduction section, and on the section “The role of evolutionary robotics in other fields of research” (previously titled “Evolved robots give back to biology”)
The section on Fitness-based evolution and the relation fitness-based evolution and novelty-based evolution should be rewritten. The usage of a task-dependent fitness function does not implies the usage of a multi-components fitness or the usage of a fitness that do not evaluated the agents on the basis of their behaviour. On the contrary, as discussed for example in Nolfi and Floreano (2000), see page 65-68, the best fitness functions are behavioural, internal, and implicit. Behavioural means that the robots should be rewarded on the basis of the outcome of their behaviour and not on the basis of how their behaviour should be realized. Implicit means that it should be based on a single component (on the minimum number of components) rather than on many components that specify how the desired behaviour should be realized. Internal means that it is calculated on the basis of information that is available to the robot. An example of fitness that fit this description is that used in predator-prey evolutionary studies in which the predator robots are rewarded simply on the basis of whether they manage to capture prey and prey are rewarded on the basis of whether they manage to escape or not from predators, see Nolfi S. (2012). Co-evolving predator and prey robots. Adaptive Behavior, 20 (1):10-15.
> Authors’ reply: We have revised the sections on “fitness-based evolution” and on “novelty-based evolution” to improve clarity and for completeness. We have included a description (with examples) of the three-dimensional fitness space framework. In the novelty-based evolution section, we have also clarified the relation between fitness-based evolution and novelty-based evolution.
Concerning the pioneering contributions in the field, the first experiments carried out with evolving robots were performed at Sussex University, U.K. (Cliff at al., 1993), at the Swiss Federal Institute of Technology in Lausanne (Floreano and Mondada, 1994), at the University of Southern California, USA (Lewis et al., 1992), and at Italian National Research Council (Nolfi et al., 1994). In the case of the first three works listed above evolution was carried out on hardware. In the last work, evolution was carried out in simulation and the evolved robots were post-evaluated on hardware. The first experiment involving the co-evolution of brain and body in robots was realized by Lipson and Pollack (2000). Also in this case the robots were evolved in simulation and then post-evaluated on hardware. The work of Sims (1994), which is the basis of the Lipson and Pollack work, was realized in simulation only and was also based on a partially irrealistic simulations, consequently it cannot be considered an evolutionary robotics work. If you extend the field of evolutionary robotics to studies carried out in simulation only, you should include many other studies carried out within Artificial Life. Of course, several researchers including Brooks (1992), that you cite, hypothesized the possibility to realize evolutionary robotics experiments. But in case you want to discuss also precursors, you might want to cite Valentino Braitenberg and Alan Turing as well, at least.
-Cliff, D. et al. (1993) Explorations in evolutionary robotics Adaptive Behavior 2, 73-110 -Floreano D. and Mondada F. (1994). Automatic creation of an autonomous agent: genetic evolution of a neural-network driven robot. In D. Cliff, P. Husbands, J. Meyer and S.W. Wilson (Eds.), From Animals to Animats 3: Proceedings of Third Conference on Simulation of Adaptive Behavior. Cambridge, MA: MIT PressIBradford Books. -Lewis M.A., Gaff A.H., Solidum A. (1992). Genetic programming approach to the construction of a neural network for control of a walking robot. Proceedings of the IEEE International Conference on Robotics and Automation. Nice. France. -Nolfi S., Floreano D., Miglino O. & Mondada F. (1994). How to evolve autonomous robots: different approaches in evolutionary robotics,In R.Brooks and P.Maes (Eds.), Proceedings of the International Conference Artificial Life IV. Cambridge Mass: MIT Press, 190-197 -Lipson, H. and Pollack, JB (2000) Automatic design and manufacture of robotic lifeforms Nature (6799) 406, 974-978
> Authors’ reply: We have revised the section titled “Early pioneering contributions” according to the comments above. We now cite Turing, who introduced the concept of evolutionary search and discussed the potential of machine intelligence, and Braitenberg, who evoked the possibility of evolving robots in one of his thought experiment. Afterwards, we describe the early pioneering contributions in evolutionary robotics. Instead of the suggested article by Cliff et al (1993), we cite Harvey et al. (1994) because it was the first study of the authors in which a real robot was employed. We then point to Brooks (1992) as an example of the cross fertilisation between of ER and other robotics domains. Finally, we revised the reference to the work of Sims. His work is now briefly mentioned in the “body plan evolution” section as the source of inspiration of Lipson and Pollack’s study. We explicitly acknowledge that the work of Sims was in the artificial life domain.
I also suggest that move the section pioneering contribution outside the section research directions.
> Authors’ reply: We have placed the section on pioneering contributions, now called “Early pioneering contributions”, before the section “Research directions” to give an idea of progress and current horizons in the field.
I thing that the discussion of the usage of evolutionary robotics results to progress our comprehension of biological system (the section evolved robots give back to biology) should be moved out from the research direction section. You can discuss this aspect in the introduction when you discuss the relation with biology and cognitive science.
> Authors’ reply: We briefly discuss the topic in the introduction, but we have kept the section in the same place (now called “The role of evolutionary robotics in other fields of research”) because the topic is arguably a research direction, and the section allows us to expand on the subject and to provide examples without expanding the already long introduction section.
Concerning the bootstrap problem and behavioural decomposition, a possible alternative solution consist in the exploitation of the multi-level and multi-scale nature of behaviour, in the possibility to generate higher-level behaviour by recombining lower level behaviours in a modular way without necessarily having a system that is modular at the control level, see: Nolfi S. (2009). Behavior and cognition as a complex adaptive system: Insights from robotic experiments. In C Hooker (Ed.), Handbook of the Philosophy of Science. Volume 10: Philosophy of Complex Systems. General editors: Dov M. Gabbay, Paul Thagard and John Woods. Elsevier and Tuci E., Ferrauto T., Zeschel A., Massera G., Nolfi S. (2011). An Experiment on behaviour generalisation and the emergence of linguistic compositionality in evolving robots, IEEE Transactions on Autonomous Mental Development, (3) 2: 176-189.
> Authors’ reply:.We have revised the text to include a description of design for emergence techniques in which behaviour is considered a multi-layer system with different levels of organisation unfolding over different time scales (Nolfi, 2005, 2011; Yamashita and Tani, 2008).
Concerning the utilization of common/shared simulation platforms and benchmarks, some projects constitute a initial step in this direction, see for example the MorphEngine, http://www.cs.uvm.edu/~jbongard/MorphEngine/, or FARSA, https://sourceforge.net/p/farsa/wiki/Home/, Massera G., Ferrauto T., Gigliotta O., and Nolfi S. (2013). FARSA: An open software tool for embodied cognitive science. In P. Lio', O. Miglino, G. Nicosia, S. Nolfi and M. Pavone (Eds.), Proceeding of the 12th European Conference on Artificial Life. Cambridge, MA: MIT Press.
>Authors’ reply: We realise that different general purpose simulation platforms have been introduced in evolutionary robotics throughout the years. There are thus numerous initial steps. However, no evolutionary robotics platform has yet gained sufficient traction to be considered prevalent. This contrasts with, for example, mainstream robotics platforms such as ROS and MOOS, which have been widely adopted by the research community, and accelerated innovation in a number of different ways.
You might want to include between recommended readings this recent review of the field: Nolfi S., Bongard J., Husband P. & Floreano D. (2016). Evolutionary Robotics, in B. Siciliano and O. Khatib (eds.), Handbook of Robotics, II Edition. Berlin: Springer Verlag.
>Authors’ reply: We have included the book chapter in the recommended reading section, along with two other publications: an article on the open issues in the field, and a review by Bongard (2013) that discusses how ER can contribute to other fields of research.
The English should be improved.
>Authors’ reply: Aside from the substantial revisions discussed above, we have made a large number of low-level improvements to the text.
