Oral-History:Hiroshi Ishiguro
About Hiroshi Ishiguro
Hiroshi Ishiguro studied undergraduate Computer Science at the University of Yamanashi and later received a D.Eng. in Systems Engineering from Osaka University in 1991. Following graduation he began working as a research assistant in the Department of Electrical Engineering and Computer Science at Yamanashi. In 1992 he moved to the Department of Systems Engineering at Osaka University. In 1994 he became an Associate Professor in the Department of Information Science at Kyoto University where he began research in distributed vision using omnidirectional cameras. From 1998 to 1999 he served as a visiting scholar at the Department of Electrical and Computer Engineering at the University of California, San Diego (UCSD), and in 1999 as a visiting researcher at ATR Media Information Science Laboratories developing interactive humanoid robots, such as Robovie. In 2000 he transferred to the Department of Computer and Communication Sciences as an associate professor at Wakayama University, later becoming a full professor in 2001 before moving back to Osaka. He is currently a professor of the Department of Systems Innovation in the Graduate School of Engineering Science at Osaka University (2009-), and group leader of the Hiroshi Ishiguro Laboratory at ATR Intelligent Robotics and Communication laboratories (2011-) and ATR fellow. His current research interests include interactive robots, android robots and perceptual information infrastructure. His contributions to the field of robotics include, publishing more than 300 papers in major journals and conferences, such as Robotics Research and IEEE PAMI, and the development of several humanoids and androids (e.g. Robovie, Repliee, Geminoid, Telenoid, Elfoid, etc.)
In this interview, Ishiguro discusses his career and his accomplishments in the field of robotics. Outlining his various research projects (SLAM, Geminoid, Robovie, etc.) he then comments on the influences of theater and art on his work in humanoid and interactive robotics. He recounts the developments and challenges which arose during his research, such as Android Science and the difficulty of creating a human-like appearance, and reflects on the future challenges and applications of robotics.
About the Interview
HIROSHI ISHIGURO: An Interview Conducted by Peter Asaro and Selma Sabanovic, IEEE History Center, 17 May 2012.
Interview #721 for Indiana University and IEEE History Center, The Institute of Electrical and Electronics Engineers Inc.
Copyright Statement
This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to Indiana University and to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.
Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, IEEE History Center, 445 Hoes Lane, Piscataway, NJ 08854 USA or ieee-history@ieee.org. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user. Inquiries concerning the original video recording should be sent to Professor Selma Sabanovic, selmas@indiana.edu.
It is recommended that this oral history be cited as follows:
Hiroshi Ishiguro, an oral history conducted in 2012 by Peter Asaro and Selma Sabanovic, Indiana University, Bloomington Indiana, for Indiana University and the IEEE.
Interview
INTERVIEWEE: Hiroshi Ishiguro
INTERVIEWER: Peter Asaro and Selma Sabanovic
DATE: 17 May 2012
PLACE: St. Paul, MN
Early Life and Education
Hiroshi Ishiguro:
Is it okay?
Peter Asaro:
Yeah.
Hiroshi Ishiguro:
So, my name is Hiroshi Ishiguro. I am Japanese. And I grow up in Shiga Prefecture in Japan. I studied the computer science when I was an undergrad student. And I studied the control theory and artificial intelligence when I was a student in graduate school in Osaka University. But I was not so serious student when I was undergrad student. I tried to be a professional oil painter, but eventually I give up to be a professional oil painter. And then I studied computer science and artificial intelligence very hard. And now, I became a roboticist.
Peter Asaro:
And who was your advisor for your first graduate program?
Hiroshi Ishiguro:
There was in graduate school Professor Saburo Tsuji. And he’s a pioneer, one of the pioneer of computer visions in the world I believe. And he was a very famous professor in that area. When I was in his laboratories, I saw that computer vision is over. We need to extend to other areas, then the idea of the vision guided vehicle. And then I move into the pure robotics, but intelligent robotics, not traditional robotics.
Early Robotics Projects
Selma Sabanovic:
So, what was the first robotic platform that you worked with?
Hiroshi Ishiguro:
It was a very simple robot. It’s a combination of a camera and mobile platform. But after that, I have developed the robot that has four cameras, each camera independently moves like the clocks.
Selma Sabanovic:
And what were some of the early projects that you worked on with the mobile robots and vision?
Hiroshi Ishiguro:
The purpose is – today, the people say it’s SLAM, simultaneous mapping and navigation. So, around 1990, we studied about some sort of a SLAM. But we used this for navigating the robots. And sometime I have the developed the omni-directional cameras. And by using omni-directional cameras, I have developed the – another simple robot for the navigation.
Peter Asaro:
Was that your Ph.D. thesis project?
Hiroshi Ishiguro:
Yes. My Ph.D. was for the active vision and the omni-directional visions, panoramic visions. The active vision is in how we can actively control the cameras focusing on the environment. And panoramic or omni-direction vision is how we can use the three hundred sixty degree views for localizing the robot or navigating the robot in the environment. So, I focus on that kind of two different types of the vision systems.
Peter Asaro:
Did you integrate those into the same platform?
Hiroshi Ishiguro:
Honestly speaking, no, right? But before my project, nobody seriously think about how to move the camera. In my Ph.D. thesis, I focus on the two types of movement. One is the tracking the person or tracking something and by actively control the cameras. Another is just panning the cameras to get the three hundred sixty degree views. So, that is omni-directional or panoramic vision.
Selma Sabanovic:
Was there a specific application that you were thinking of at the time?
Hiroshi Ishiguro:
Right, actually I think I could develop a pretty interesting technology. By using omni-directional cameras, the robot does not – robot could develop the environmental map and localize itself without any internal sensors like inertial sensors. I think I like that work very much. But when I submitted that paper to the top class conference in computer visions, the reviewer said that this is kind of a daydream. <laughs> And he rejected it. But eventually, I have submitted that paper to the ICRA, I don’t remember it is ICRA or IROSI think that I got a pretty good evaluation. And after that, I started to write many, many papers on that.
Peter Asaro:
How did you decide to be in Professor Tsuji’s lab?
Hiroshi Ishiguro:
Well, because he’s the most famous professor in computer vision area in Japan.
Selma Sabanovic:
And how did you get interested in computer vision?
Hiroshi Ishiguro:
As I said, I tried to be an oil painter. Oil painter world closely related to the computer graphics and computer visions. I was interested in how the human recognize the visions.
Selma Sabanovic:
So, even early on, your work with robots was inspired by thinking about human abilities?
Hiroshi Ishiguro:
Yes, always. As an oil painter, I was thinking what is human, how we can represent ourselves on the canvas. So, I believe I am doing the same things.
Peter Asaro:
Did you continue your work in vision?
Hiroshi Ishiguro:
Yeah. And after when I was an undergrad student – sorry, the graduate school. But recently, I was not so seriously working in the computer visions areas. But I’m interested in more human/robot interaction.
Selma Sabanovic:
And so, after Osaka, you went to Kyoto University?
Hiroshi Ishiguro:
Yes. After getting a Ph.D. in Osaka University, I became the assistant professor in Osaka University – sorry, the Yamanashi University and in Osaka University – I’m back to Osaka University and spend two years. And then I became associate professor in Kyoto University. And then I was in UCSD one year, back to Kyoto University again, and move to the Wakayama University and become a full professor in that university, and back to the Osaka University.
Peter Asaro:
So, did your research projects change as you changed locations?
Hiroshi Ishiguro:
Yes, right. I have some – I have a policy. Three years maximum for one project. The first year, we prepare something. The second year, we run the experiments. And for those other years, we write papers. Then, we should start something new. In order to start something new, the best strategy is to change the environment. If we move to another university, different university, the people expect me to do something new. I can have a good pressure from the environment. Then I can do something. So, always I’m thinking – that is not a typical Japanese, like American. But I like that kind of the lifestyle.
Selma Sabanovic:
And so, what was – can you tell us a little bit about the environment in the different places. If you start with Yamanashi University, who was there? What kinds of things were being done?
Hiroshi Ishiguro:
Yeah, actually when I give up to be a professional oil painter, I was one – well, I couldn’t decide which laboratory is good or not. Eventually, I have met Professor Mori. And he was the developing a vision guided robot. And I liked that project. And but his laboratory was not so popular. And therefore, I could join. And I seriously started computer visions. And after that, I moved to Osaka University.
Professorship
Selma Sabanovic:
So, in your first – right after your Ph.D., what kind of work did you do then?
Hiroshi Ishiguro:
After Ph.D.?
Selma Sabanovic:
Mm-hmm.
Hiroshi Ishiguro:
Well, I continued in my work a little bit, a couple of years, maybe three years, and well when I was the assistant professor in Yamanashi University and Osaka University. But when I moved to Kyoto University as an associate professor. In Japan, the associate professor is not independent. We belong to the professor’s laboratories. And the director of that laboratory was Professor Ishida. And he’s a very famous professor, a researcher in artificial intelligence. He asked me not to write journal paper or conference paper anymore. Before joining to the Kyoto University, I have written so many papers. Every month, I have published a journal paper. And he said it’s good. But that isn’t enough. Now, what I have to do is to change – to develop something new. That can change this world, right? So, that is a very important mission of all researchers. And I started to think what I can do with a robot, with computer vision. And I started two projects. One was the project with an interactive robot. The other is the sensor network, multiple camera system. After that, I’m always doing two things. One is for ubiquitous computing or sensor network. And the other is interactive robots.
Interactive Robotics
Peter Asaro:
How did you first approach interactive robotics?
Hiroshi Ishiguro:
First approach was when I started the project in Kyoto University, we didn’t have a good platform. So, that I have developed the little big robot. I forget the name of that robot. It was a commercial product made U.S. And we have imported. And then I have designed the pan tilt camera controls and put it on the robot. And I have developed some sort of not so simple matter, the mobile platform with the pan tilt camera. And then we started work with that robot. A roll of the robot was not just the move around the environment but to interact, to talk something with the people. And we focused on the interaction with people.
Selma Sabanovic:
And so, what kinds of questions, in terms of interaction, were you initially interested in?
Hiroshi Ishiguro:
Yeah, how robots – the robot interact with people. Other time, the camera was – well, the quality of the camera was okay, but the computer was not so powerful. Therefore, we have – obviously, we had limitations with the vision process. And for example, today digital camera has the function of face tracking, but we didn’t have that kind of function at that time. But still, robot needs to get some information from the cameras and detect some actions. And so, the question was how we can continue the interactions between robots and humans. And the roll of the robot was to guide a person to the destination. But during the guidance, the robot needs to interact with the person. So, we tried to develop that kind of – and the guide robot.
Selma Sabanovic:
And what were some of the things that you found the guide robot needed to be able to do?
Hiroshi Ishiguro:
Well, first of all, the robot need to find the person. But the computer vision, the cam – well, the computer vision technique was not so rich. Right, therefore, we needed to develop kind of a situated intelligence. We need to put some knowledge about the environment. And we needed to limit the – how can I say it, the possibility for analyzing the visions. So, I call it "situated vision." And we can redesign that kind of – the vision process for the robot.
Selma Sabanovic:
And what kinds of aspects of the environment was the robot paying –
Hiroshi Ishiguro:
Well, it was just corridors in the university. I wanted to have some robot that moves in outdoor. But it was not so easy. And we could develop the robot that guide people in indoor, in the building.
Selma Sabanovic:
And was this project funded by the university, or outside?
Hiroshi Ishiguro:
Basically, the university does not have much money in Japan. Usually, we need to submit a proposal to the government, like NSF. We call it JST in Japan, JST, and JSPS. I don’t remember exactly, but probably I was getting some money from the JSPS and JST.
Robovie Project
Selma Sabanovic:
Were you related to the Robovie project?
Hiroshi Ishiguro:
Yes. Robovie project, I have started. When I developed the interactive robot in Kyoto University, then I wanted to develop much better robot. Then I went to the ATR, and I started work in ATR. And actually, my group was – I started my group with just another two people. And we are three. And I have developed the Robovie by myself. And after that, the ATR robotics group keep growing. And now, I don’t remember, but probably we have more than forty or fifty people there.
Selma Sabanovic:
And who were some of the people that you worked with at the time?
Hiroshi Ishiguro:
Professor Imai from Kyoto University and Professor Ono from Hokkaido University, now they became professors, and they have their own laboratories there.
Selma Sabanovic:
And so, what were some of the things when you were developing Robovie that you did to kind of improve on existing platforms?
Hiroshi Ishiguro:
The Robovie – when I developed the Robovie what I wanted – I wanted to have some robust in the platform, interactive platform with two arms, and of course the two eyes, and next two arms, and tactile sensors, and cameras, ultrasonic sensors. I have just gathered very reliable sensors and actuators. And I have designed and developed that, the Robovie. And I think it was good. We didn't have any problem with the Robovie. And the robot could work four hours with the batteries. Therefore we could have many field tests with the Robovie. And after building a Robovie, what I did was the field test. And we had a field test in elementary school. I think – well at least my memory that was the first try in the world. And we checked the student reactions. Right? Robovie was speaking in English. And Robovie was a kind of foreign student. And by speaking – and the Robovie interact with the Japanese students in English. I guess Robovie could give motivation to study the English for Japanese students.
Peter Asaro:
So, interacting with – robots interacting with humans seems to have been an interest of yours from very early on. But perhaps at those early times it wasn't as common. Why did you have the idea that robots would need to interact with people?
Hiroshi Ishiguro:
Okay. Right. The way we see the robotics usually we develop a robot that works in the machine shops in the world. Industrial robot was our main focus as application of robot. But that is over I think. At that time, we needed to find some new application areas. And then I was seriously thinking about that. And as Professor Ishida says, how we can change this world with my technology. And then usually – and the new applications, the U.S. people and some European people, they were thinking to develop military application with the robot. But in Japan, of course we don't have a military. And it's illegal to work on the military applications. But on the other hand, as you know, the Japan has pretty good technology for assembling small stuff and many companies are making the good product electric equipments. So, that I thought, in the near future, in electric equipments, they're going to be kind of a robot. And if we improve the interface more and more, then they going to be a kind of an interactive robot. So, that was my idea. Therefore, I started there with more interactive robots project.
Peter Asaro:
And when did you first start thinking about humanoid robots in particular? And was that a response to what you learned about earlier?
Hiroshi Ishiguro:
Yeah, well the Robovie – Robovie's also a kind of a human like robot. Now, I had a basic idea that the human brain has the function to recognize humans, not for other – well, for example, computer. The monitors and just the text, they are not so good. And if someone cannot read my Japanese, well probably you cannot read the Japanese. So, that even if we show someone Japanese text on the screen, it is not a good interface. Well, even if we cannot speak in Japanese, by taking some gestures, we can communicate. That means the human needs to have a human like interface. The problem, I have a designed a kind of a human-like robot. So, that was the Robovie. But when I have developed the Robovie and then Mitsubishi Heavy Industry, they made a copy of a Robovie. And they developed the Wakamaru as kind of a brother robot of the Robovie. But I didn't like the design. I asked to give a more humanlike design to the Wakamaru. But that famous industrial designer, they gave us a kind of insect-like appearance. But it was okay as an industrial design. But, for me, I wanted – well, my policy is that the robot needs to be more human. Probably we have some strong reason for this kind of a human like appearance. And then I have decided to start the android project for to understand the importance of a human appearance. And of course the purpose was not just make a copy of humans, but to understand the important factors for the human/robot interactions and the importance of human-like appearance. We are getting much improvements from the very human-like appearance. So, we have – as you know, the way I developed the Geminoid, it's my copy. We did many studies on that, and we could find that kind of human nature.
The Android Project
Peter Asaro:
And where did you find funding for the android project?
Hiroshi Ishiguro:
Well, the first funding was coming from JSPS. It was a very small amount of that. But in 2004, we had the large expo in Aichi. And one of their focus was robotics. And then the government gave us a good amount of money. Therefore I could develop the first android, female android, I’m sorry, the second android. The first android was my daughter's. And after that, well I guess I could establish kind of a research area for the android. And then I could get another, the big amount of money for building the Geminoid, my Geminoid and another Geminoid.
Peter Asaro:
And what motivated you to make replicas of specific people, yourself and your daughter?
Hiroshi Ishiguro:
Well, I started to have a kind of virtual appearance, virtual of humans. It was the average face. I don’t know the exact English name of that society, the Japanese society for the face. They are studying about the human face and human facial expressions. And they have the computer graphics for taking the average of many humans face. And one of the interesting things is if we take the average of human face, that face is going to be very cute, handsome, smart. And then I have decided to use that face, the virtual face. But actually, android was not so human like. It was like a doll. We don’t know how we can represent human likeness in detail. Therefore I give up to use that kind of partial face. And I have hired professional person who can make a copy of the human. Actually, she was working Hollywood. And she was pretty good for making a perfect copy of human. And that is the reason why we are using the human model. But important thing is not to finish the android. So, we need to put the real person here and android here. And that professional person doing some perfect makeup for the android. But I will say again. We don’t know what is important, the factors for making the human likeness. For example, if we have a single picture, we can make the android of that picture. But that is not enough. Picture does not contain enough information for representing the human likeness. Actually, we need the real person there. Then we can have a perfect copy. So therefore, we are always using the real human model for the android.
Peter Asaro:
And how did you go about thing to study facial expression and movement and replicate that in the systems?
Hiroshi Ishiguro:
Still facial expressions for the android is limited. But that is a kind of a very important know-how for the companies. We worked with Kokoro. It’s a company who are building the android. They have a many know hows, how to use a silicon materials for representing the animal skin and human skins. And always we are improving how to represent the more better human expressions. And so, that is – I would say that is just know how. And we are improving the know how every time.
Selma Sabanovic:
And you’ve done also a lot of research on human presence, so how you can show that with robots. And a number of it is with tele-operated robots. Do you think that’s necessary to have a human kind of put their –?
Hiroshi Ishiguro:
No, the most important thing is for representing a human presence is subconscious movement. When we are sitting, we never stop. So, my eyes are always moving. And shoulder is also moving. Head is moving, subconsciously, slightly. And if the computer stop like this, it’s quite strange. That means a dead person. So, subconscious movement is quite important. And another important is the actions. When I touch, or someone touches to my hand like that, usually I take some reaction. That kind of reaction is quite important. And then to have that, to implement that kind of reaction, I have the developed the very soft sensitive skin sensors. In early stage of android project, I have spent more than one million US dollars for developing that. But it was good. We could develop very sensitive soft skin sensors. And then android became more human like. And that is a basic human likeness. In addition to that, an android – it’s better if android can talk, right? Android don’t need to talk, for example, recently, we are exhibiting the android for the show windows. Probably you know our activities in Hong Kong and Takashimaya. It was in Shinjuku. My idea was a mannequin is not ideal for the show windows. What we want to do with the show window is to show the somewhat futuristic theme in a kind of the elegant life. We want to show that. But to show that, of course we want to use a human. But we cannot use it. Human cannot stay in the window. So, therefore, I think an android is much better, better than of course the human and the mannequin. And we have implemented that kind of subconscious movement. And we have also implemented the simple interactive behaviors, not talking. And the android – if someone look at the android, android can smile back like, that’s kind of – well a number of interactions. That subconscious movement and number of interactions is quite fundamental. And it’s important to be a human. And then, in addition to that, if android can speak, it’s much better.
Selma Sabanovic:
And did you realize the importance of the subconscious movement through showing the android to people? Or was it from reading certain theoretical works?
Hiroshi Ishiguro:
I don’t remember exactly. But I had that kind of intuition. When I saw the static android, I thought this is strange. This is kind of a dead person. We should make it move.
Selma Sabanovic:
Were you thinking about Masahiro Mori’s uncanny valley at that point?
Hiroshi Ishiguro:
Well, I first do the uncanny valley issues when I developed my first android with my daughter’s body. At that time, I didn’t have enough money for buying the number of motors and actuators. Therefore, I have just installed eight motors on the head. But I didn’t install any motors. But anyway, my purpose was to have a human-like appearance. It was okay. Appearance as okay. But movement, when the head is moved, the body was shaking like this, right? It was like a zombie. And that is uncanny valley. My daughter’s copy was in the bottom of uncanny valley. And then I found – well, eventually, I found the – I guess then someone gave me information about the Professor Mori’s uncanny valley. And then we started to study about that.
Peter Asaro:
What was the most surprising thing that you learned from your early humanoids?
Hiroshi Ishiguro:
When I developed my daughter’s copy, of course, the movement was jerky. But therefore, we just put it on, how can I say – okay, I will explain like this. So, suppose this is my daughter’s copy. I try to give a hug, right? And then, I get close to my face to the android face. At that moment, I got a smell of my daughter. It’s a virtual thing. Probably, you remember that – did you try that kind of thing? There was my android, female android.
Selma Sabanovic:
Mm-hmm.
Hiroshi Ishiguro:
The human-like appearance, they give us some memories about the smell, just giving a hug, or just closely watching the face of the daughter’s android, I could remember the smell of my daughter. But it was not remembering. It was quite realistic the feeling.
Selma Sabanovic:
So, you actually experiencing basically her being there.
Hiroshi Ishiguro:
Right, right.
Technological Developments for Robotics
Peter Asaro:
Can you talk a bit more about the skin sensors that you developed and what the challenges were and how you solved them.
Hiroshi Ishiguro:
The challenge was we didn’t have any in the soft skin sensors before our project. So, as you know, this skin is very flexible, stretchable. In addition to that, this is quite sensitive. So, the question was how we can develop this kind of very soft and stretchable and sensitive and dense sensors. And the idea was – my idea was to work with the material science, the professors. And he had good technology for printing. But his printing technology is stretchable. And we used silicone rubber plate. And we printed the wiring. And we put the small piezo film sensors, which is a kind of stretchable electric circuit. And other was our skin sensor. But honestly speaking, we are not using that sensor today because the mass production is different issue. So, there is no market for that, that sensor yet. The problem is we couldn’t reduce the cost. It costs a lot.
Selma Sabanovic:
So, the androids that Kokoro produces are not – they don’t necessarily have kind of sensors in the skin?
Hiroshi Ishiguro:
They have but we are using just the commercial products of piezo film. Our sensor is good. But it takes a lot of cost for producing that.
Selma Sabanovic:
When did you make the sensors?
Hiroshi Ishiguro:
I think I started that project in very early stage of android project. Probably, I started android project 2001, 2000 or 2001, I guess. And I spend six or seven years before having the final product.
Selma Sabanovic:
Were there any other technological things that you needed to develop while you were building your android?
Hiroshi Ishiguro:
Of course, we needed to develop the control, the programs for the new manufacturers. And this is how the model is quite easy to control. But pneumatic computer is nonlinear, and it’s also compliant. It’s quite difficult to control that. And additions, for android, female android has forty-three actuators, pneumatic actuators. If we move just one actuator, the other actuator also moving. So, we – that means that we have to control all actuators simultaneously. Therefore, we have developed various computer programs for controlling that. We tried the ideas that I got, neural networks, or traditional control theories. And it was a student project.
Interactions and Collaborations
Selma Sabanovic:
Is there other – at Osaka University, there are some other also robots that use pneumatic –
Hiroshi Ishiguro:
Geminoid, we –
Selma Sabanovic:
Crawling, and then there’s CB2, are they all connected with yours? Or does everybody develop their own?
Hiroshi Ishiguro:
Yes. Actually, I’m developing the robot for – the baby robots, right? Six years ago, Professor Hosoda, he started the big project. And I and Professor Kuniyoshi and Professor Hosoda, and another, you know, the neuroscientist there. We are the member of that project and each of us there, we had a group. And my group is – the role of my group was to develop the platform for baby robots, and, you know, they study about the social aspect of the robot. I said, its interactive aspect, interactive function robots. And another time in the project, you know, I have the rob – the small baby robot and big one – and baby robot is CB2. The big one, you know, I used pneumatic actuators, but that is a kind of full spec android. And I have installed around 60 pneumatic actuators. It was so complicated, and then one that was a big challenge is how to control the 60 pneumatic actuators right? Then honestly speaking I think – well, it's almost impossible. But, we also have so many muscles, and how we can learn to stand up and to walk, right? So, usually, the caregiver need to help them, right? They're probably not – we took the same strategies, and how the robot can learn based on the help of caregivers, right?
And, actually caregivers that lift up like this, right? By holding the two arms? And then, robot could understand which actuator is the important to stand up. And robot could learn something. That kind of standing up behaviors and the walking behaviors. With the help of caregivers.
Selma Sabanovic:
Also, in the university, there are quite a few robotics groups. How much do you interact with other, or are your labs very separate? What is the environment like?
Hiroshi Ishiguro:
Well, at Osaka University we have two faculties, you know, one is engineering, while the other is engineering science. And both the faculty has the mechanical engineering and control theory groups. And artificial intelligence groups, right? That is a unique aspect of Osaka University. Usually, university has a science faculty, a science faculty and a faculty of engineering, but we have, you know, I'm in the engineering science, right? So, you know, I think the – we have good activities. I don't remember the exactly how many robots we have for the robotics s but, maybe we have around 10 or you know. And then each group has a little bit different advantages in a different areas, and they – well, we can always try to collaborate and try to get the bigger fund and grant from the government by working together.
Selma Sabanovic:
And when you went to use UCSD, what did you do there and who did you work with?
Hiroshi Ishiguro:
I work with the Mohan Trivedi and he was, you know, in Tennessee and he just came from the Tennessee, and joined to the UCSD and the – I also work with Ramesh Jain. Ramesh Jain. He was a big part of – famous people, in computer vision. And then Mohan Trivedi – Professor Mohan Trivedi was a famous professor in the robotics, but, you know – and both are my host.
And I also worked with people in cognitive science and the neuroscience. And UCSD has a pretty good groups for the neuroscience, like Sejnowski, Professor Sejnowski, and... Cognitive Science Group is also the very, very traditional. Alright? And they have the Don Norman, they had it right, there are many friends there. So, you know, basically, the, my role was to do something with robot and with multiple camera systems. Actually Mohan Trivedi prepared the laboratory space and student for me. My stay was just one year, but still that he could – he kindly did prepare, a pretty good environment, research environment for me. And they're probably not – I could have done many things at UCSD and same time, you know, by – we are talking with the different area – peoples in different areas in neuroscience and cognitive science I could start that interdisciplinary, the works there.
And then right now, I'm hiring the neuroscientist and cognitive scientist in my group, because that is quite important for studying the android and the Geminoids, right?
Android Science
Selma Sabanovic:
And you've developed kind of a concept called Android Science that is this interdisciplinary space. Can you tell us a little bit about that and how you decided to do that?
Hiroshi Ishiguro:
Yeah. Actually, you know, there are two types – two ways how we can understand human cell, neuroscience and cognitive science, their approach is a kind of bottom-up approach, they kind to identify the brain functions on some sort of, well cognitive functions, right? And try to build up the big pictures about the humans. We need the text book of cognitive science and neuroscience, but still we have many unknowns, right? Robotics people cannot build a robot based on that kind of text book. But in order to have the – the robotics and robot, we need to have some other ideas. And the system-level hypothesis about the, and mechanism of humans, right? Of course we don't have a perfect knowledge about humans, but we have some, you know, the knowledge about the system or robots, right? Therefore, if we can try to – you know, we can try to build a human-like robot, right? And with our hypothesis and with their hypothesis, of course, and then if a robot became the human-like that means well, we could verify the hypothesis, right? And then we can improve the robot more, and neuroscience and cognitive science people they can use our robot as their platform, right? By using the androids, they can have some, the experiments. And they can brush up their ideas and then they can give us another hypothesis. So that kind of, and collaboration is quite important. Then, therefore, I say it is not just engineering, right? It's kind of a science, right? And android is a terminology for the engineering, right, but science is a science, right, so therefore I call it android science.
Selma Sabanovic:
And who are some of the cognitive science or neuroscience people that you work with?
Hiroshi Ishiguro:
The – many of them, well, for Japanese – Japanese, Professor Hiraki from Tokyo University, and the Professor Itakura from Kyoto University. And these people are very good cognitive scientists. And neuroscientists – well, for example Doctor Kawato from ATR, right? They are recently we are running a cooperative project in ATR and Osaka University, and we also have some other, you know, neuroscientists in Osaka University.
For example, Javier Movellan, you know, from UCSD and he is also running the very similar project to our Professor Asada’s project. Actually, we – I helped to develop the Diego-San his robot's name. It's a kind of brother robot of CB2, right? And I helped to develop that robot. And well, that mean that we are sharing the kind of common platform. And he was in Carnegie Mellon, the department of cognitive science Carnegie Mellon. And he moved to the UCSD, right? And he's not pure cognitive scientist and neuroscientist, but he is pretty good for neuroscience and cognitive science. And from 2000, when I was in UCSD in 1999 and 2000, and after that we were always working together, and we always exchanging information.
Peter Asaro:
And you gave a good description of how the cognitive functions kind of map from cognitive science into robotics, but how do you see the neural models mapping? Do you think you can directly model these kinds of neural structures or do you just get inspiration?
Hiroshi Ishiguro:
Well, for example, we can learn something. For example, right, how can I say that, there is a model, the human brain model to control the bodies. It's based on the central pattern generators and thereby using a neural network that we can compose the central pattern generators and you know, we can use that model for controlling the android behaviors. So that is just my example, but, there are many works about the brain functions. And they have some model. But still, it's difficult to implement that kind of human brain model into the robots. It's quite – especially for the human-like robot, right? And very different. There is a challenge in UCSD. It's not – it's a neural institute, and there is a professor, I sorry, I forget the name of that. It's quite famous, their project. They're neuroscientists and on the other hand they are also building robot. And what they are building is, you know, just a mobile robot with cameras, and maybe there are two cameras and sensors. It's quite simple robot. For that kind of a simple robot, we can directly install the brain model. But honestly speaking, the robot cannot do anything like a human, right? It's quite basic, the behaviors, right, and therefore I mean, well, to have a more human-like robot, still... we can get some ideas. We can get some hint from neuroscience, as I explained, right? But, you know, we cannot freely install their, the model into the robot, too early I think.
Robots and Theater
Peter Asaro:
In what ways have acting and performance helped you understand human-robot interactions?
Hiroshi Ishiguro:
Acting performance? Well –
Peter Asaro:
The actoid and...
Hiroshi Ishiguro:
Actuators? Sorry –
Peter Asaro:
Acting? Even performance? Theatre?
Hiroshi Ishiguro:
Oh, theatre. <laughs> Theatre is also the android science for me, you know. Psychology and the cognitive science, they are just focusing on the particular behaviors, a kind of common behaviors in all situations. But human is situated, right? You know, the psychology, they are the observation about human is – is quite general, but human is not so general. We are very situated, right? Actually, the director has much knowledge about human, and director knows well – for example, when we're interacting, so, how much the time interval we need? You know, when you say something, if I quickly response to it, right, and let us consider two cases. Quick response or I take some time, right? Just three seconds or five seconds and if I take time so you think I'm thinking something, right? By changing the time interval, we can change the meaning, right? So the theatre director knows that kind of things very, a lot, right? But cognitive scientist and psychologist, their interest is more, how can I say, more general. Not so situated. They cannot explain well the human behaviors according to the situations, but the theatre director can explain that, right? What we want to have is, theatre director knowledge, right? Because we want to use the robot in real situation and we want to know how, you know, well, what kind of human-like behavior that we can install to the robot, right? I'm running the many things from the theatre directors and actually, we are submitting many patents on that. It's a kind of mental level knowledge, right? And the cognitive science and the psychologists, they are focusing on the very small low-level stuff, but – and the theatre director is looking at more global, more higher-level behavior of humans. And then we can learn many things. That is motivations. That is why I'm working for the android theatres. You know, that is not just for fun. <laughs>
Peter Asaro:
And what are some of the directors you've collaborated with?
Hiroshi Ishiguro:
Oriza Hirata. Oriza Hirata he's very famous. Most famous director for the small theatres, the theatres up to 200 people, 300 people, and he – he's also a professor in Osaka University, professor with communications. And he knows, and because you know he has deep knowledge about the, you know, human communications and cognitive science, right, so. And he's also the very famous in Coliers in France. The most important things is to get the good reputation in France. Then, you know, an artist can be a real artist.
Peter Asaro:
And what about actors?
Hiroshi Ishiguro:
Actors, yeah, actor is, well, they are normal, you know, Oriza Hirata, he has the own theatre companies, named the Seinendan. Seinendan means just young people group. And they're, doing well, but the problem is it's quite difficult to survive as the actor or actress, right? If they appeared in TV, right, you know, they can survive, but for small theatres, right, it's quite different. Usually they are taking some part-time jobs or, you know, well, not for everybody. Actually, that theatre will be supported by government, right? Not so much, like, France. France is spending a lot of money, almost same as research grant, you know. We know that, you know the France is supporting artists quite well, pay very well, you know. If you want to survive as artist you should go to the France, right? Well in Japan it's, you know, Japan is support the maybe the one of 10 of France. But still, you know, Seinendan, they are getting probably enough support from the government. You know, because of our professor, Oriza Hirata’s name.
Selma Sabanovic:
I have to just run up and get the next person, but please continue, sorry.
Peter Asaro:
And I'll continue on this line. So, I sat in on the workshop on the performing robots, but can you tell us a bit more about how you are using robots in theatrical performance?
Hiroshi Ishiguro:
Yeah well, what we want to do is, we want to change this historical, the traditional, the theatres. You know, so far, we thought theatres should be played by human, but that is not true, right? Sometimes the robot android can represent the human likeness more than human, so that was our challenge. And there is a little different activity about robot theatre and the old – it looks like the puppet robot, our field is different, right. Our field is the real theatre, I mean, like more traditional theatres, right? We want to have professional activities with robot and android and not just, a kind of exhibition. And actually we could – we got a good success in Europe and France, especially in France. And we got a pretty good reputation, therefore we are getting many, many offers from Europe and Asia, and we are always traveling. So.
Peter Asaro:
Great. And –
Hiroshi Ishiguro:
I forgot your question. Probably I could answer some – no, no...
Peter Asaro:
What are the biggest technical challenges to do with this kind of performance?
Hiroshi Ishiguro:
Problem is not – not to stop the robot. Robot need to keep working and we cannot – well, well once the robot stop, the robot got problem, right? We need to stop the theatre there. But, you know, it's too bad. So we need to have very reliable robots, and we need to have – very reliable computer program for controlling that. We need to have different the know-hows, right, how to develop the very robust robot and robot programs. So, and my staffs work on that maybe two years and always, you know, well, now, well, it's okay, you know, we can trust our robot, but in the beginning of the theatre activities, you know, everybody was worried about <laughs> the, you know, and when the robot stop, we were carefully watching that. <laughs> But it was okay now. The next step is to make it fully autonomous. I mean, you know, in order to have fully autonomous robot we need to have a sensor network. And as I said, I'm developing two types of robots systems. One is a sensor network one, the other is a robot. And we could establish the good technologies for the sensor network. And now we are preparing for the sensor network for the android theatres and robot theatres and then well, we don't need to have any operators to monitor the system and the whole system. The computer monitors the whole activities, and perfectly control the robot.
Peter Asaro:
How are you building the sensor networks?
Hiroshi Ishiguro:
Our idea is to use the laser scanners, very tiny laser scanners. And laser scanner is much better than cameras. But we going to combine laser scanners and cameras, as, you know, the Kinect. Now, many robotics people are using Kinect but, we don't need to use the Kinect. The Kinect has limited range for the scanning, but one dimensional, laser scanner has much longer range for the scanning. And well, we have spent more than three or four years for developing that kind of robot systems that we can use in the daily situation.
Peter Asaro:
So otherwise they are mostly tele-operated?
Hiroshi Ishiguro:
Not tele-operation, you know, but somebody to watch them, monitor them. And someone need to push a button to take the timing. Not every time, right, maybe every 10 minutes or 20 minutes or something. But still it's troublesome, right? We want to make it perfectly autonomous.
Peter Asaro:
Do you give your robot actors speaking lines?
Hiroshi Ishiguro:
Speaking line?
Peter Asaro:
Do they speak or do they just...
Hiroshi Ishiguro:
You mean the robot or –
Peter Asaro:
Robot.
Hiroshi Ishiguro:
Robot is, yeah, speaking, actually. It's important to speak something, right? It's a theatre, right?
Peter Asaro:
Yeah. And do you have trouble regulating that speech or is it just pre-recorded?
Hiroshi Ishiguro:
Pre-recorded. Yeah, but for the robot we are using the synthesizer. Voice synthesizers. And for the android, we cannot use voice synthesizers; the quality is not so enough, right, because Android need to have human voice.
Peter Asaro:
And are they better at drama or comedy?
Hiroshi Ishiguro:
Now, we are going to have another drama, right? And now, android theatre is just 20 minutes. And – well, this April we extended it to the 40 minutes. But this September we going to have 90 minutes. Android theatres and android, we going to use the – we cannot – I can not tell you the details about this. You know, we – android is moving around on the stage. And, yes.
Peter Asaro:
And when you decided to make the Geminoid that looked like yourself, did you consider this a kind of self-portrait?
Hiroshi Ishiguro:
Well, it looks like twin brother, not like a portrait or photograph, right? Because, you know, it's moving, right? And it's talking. You know, if someone control the Geminoid it became really different person, right? But still, it has my appearance. That is the twin brother. Okay. And the more important things is I don't know my face, voice, behaviors. You know that? You know your own face? You don't know that. Every morning you are watching the mirror but the mirror is about you, right? You know, flipped? And photograph is your face. Everybody is watching this face. And human face is not symmetrical. Therefore, the photograph and the mirror, these are completely different, right? So, the mirror is not your face, it's about your face. But still, you know, by watching a mirror, you – you are confirming the hair style and the face; that it looks okay, but it is not okay. It is not your face, right? Same things for the voice, right? If we record our voice, it sounds like a different person's voice, right? So human does not know the appearance and voice and behavior also. It's quite difficult to have a perfect knowledge about ourself. Therefore we just believe. But when we see the very similar person there, I can guess that it's, you know, my copy there, but I don't have any strong feeling that that is myself.
Students
Peter Asaro:
And who have been some of your most successful students who have continued working in robotics?
Hiroshi Ishiguro:
My students? Obviously, well, Doctor Kanda. He was, you know – now he is a group leader in my project in ATR. He's quite successful. And he's a young leader in human-robot interactions, conference and society. And I going to retire and he going to take my position, I think. But I am interested in more – recently, I'm interested in more artistic work or, you know, different science, so that I think he is going to take my role and I can go to the different areas. There are many student I have, you know. Many good students. There are many people, student, they contributed to the android and the RoBoVie.
Challenges
Peter Asaro:
And what do you see as the biggest challenges to human-robot interaction going forward?
Hiroshi Ishiguro:
Biggest challenge is, well, to make it commercialized first, but, unfortunately, Japanese company were not so good, but you know, in U.S. for example, InTouch Health, VGo's and the Willow Garage, they are building pretty good platforms for tele-operated robots. But that is for the interaction, right? Alright, so, what was the questions?
Peter Asaro:
So the biggest challenges for HRI going forward.
Hiroshi Ishiguro:
Okay alright, the biggest challenge is to develop a new market first. Then the company going to spend more money, and we can improve the robot more and more. So, the meaning of improvement is to make it autonomous, you know. Well, and if we develop the market for the interactive robot by using the tele-operated robot, right, then we, well the company can spend more money for make it autonomous. And the tele-operation gives us a lot of data and based on that data we can make it autonomous, right. So I call this approach is sustainable robotics, right? How we can continue? How we can continue developing robots, and in order to do that we need a market, right? We have spent so much money, so I think it's time to develop the market with the technology that we have developed so far. And otherwise, probably the people cannot trust us, right, and the government, right? And, but I believe, we can develop a real market and then, and us, we are improving the computers every year. We can have more rapid improvement of robots. Then robots going to be autonomous soon, right? Without any practical use, it's quite difficult to spend much money for developing better robot.
Advice to Young People
Peter Asaro:
What's your advice to young people who are interested in a career in robotics?
Hiroshi Ishiguro:
Well, my advice is to have fundamental question, right? Don't watch the simple applications, or don't focus on the particular areas, particular research areas. And if the student they have very fundamental question like human likeness or human presence, or human consciousness and then, you know, they can take some interdisciplinary approach and they can develop a new research areas. So that is the research science, and that is a true – well, a scientific approach, so my message is, have a fundamental question.
Peter Asaro:
Is there anything we missed or something you would like to add?
Hiroshi Ishiguro:
No. <laughs> Well, another message is, it's quite important to enjoy. Don't watch the professor. Don't hear the – stupid senior researcher's opinion. Just have own opinions, and let's enjoy. And then, you know, you can develop a much better robot.
Reflections
Peter Asaro:
And what do you consider to be your greatest accomplishment in robotics so far?
Hiroshi Ishiguro:
Well, let me see, so many good works, right? Robot is a kind of an integration of the important studies, the important result, but in the sense of integrations, the most successful robot is the Honda Asimo. It's almost at commercial level, right? But now, the U.S. Government started the Humanoid Challenge. I want to see what's happen there, right? Probably you may have much better integrations for building humanoids. I'm not sure. But, that is going to be a good trigger; that motivates many people, right? But anyways, but so far, I think Asimo is the one, the good result of integration. And I hope to use that kind of platform for my study, also. Actually I'm working with Honda, but we need to have some common platform everybody can use, right? So, in the near futures I hope, you know, I want to see that kind of humanoid, in, you know, our daily life. And then we can change our society again. You know, computer changes our society pretty much, but next time is robot. Robot may change our style – lifestyle more. That is a possibility.
Peter Asaro:
How do you see the relationship between academia and industry within Japan?
Hiroshi Ishiguro:
We have to change certain, you know, technology development is getting faster and faster, right? So we need to erase the boundary between industry and academia. For example, the Media Lab in MIT, well, they are doing very well. They are getting new questions, new issues from the industries, and from the, you know, our daily situations, right, and they are developing some new devices. And so we need to quickly catch the – the research issues from the industry or in real life, and then well, university or academia will provide some answer, provide some solutions to that, so that the important things is to erase the boundary. But still, you know, I am feeling some gaps – some barriers between industry and academia. So, of course we need to keep some traditional fundamental research areas, and for artificial intelligence and the media technologies, we need to take a little bit different style. With collaboration for the industry, academia cooperation. But I'm not sure yet, right, what kind of – much better the collaboration we can have; what is more – the better situations, therefore working together. But, you know, that is also another – the innovations we need. Someone, they innovate. The new universities or new college for doing that. But I'm not sure. We have to change university more and move, always by changing the topics in universities, we can develop our futures, right? So that is the role of universities. And working with universities – and National University in Japan, and I am also running the Robotic program at COE, Centre Of Excellence Program; and the Centre Of Excellence Program is quite important for changing the – well, graduate school in university and for having better futures.
Peter Asaro:
What do you think are the most promising applications for robotics in human society?
Hiroshi Ishiguro:
There are so many. So many applications we have, you know. Well, for example – well, my interest is interaction of communication, okay? Well, how can I say it, how about this idea: the convenience stores, right? And the staff in convenience store, I think the humanoid and android we can use there. And the ticketing machines and the problem of the current railway station is we can – it's difficult to find the staff, the human staff, right? When we going to ask something, right, so, we need access to the ticketing machine or the display there, but that they are not so good for the elderly people. Well, people just want to talk to somebody, right? But if we use the android or humanoid there, right, the people can easily, they talk to, ask questions. So, if we can improve our current system more, so. And another example is – this is a secret. I cannot tell detail, but, you know, okay. Let me – how about this example: we can have more realistic museums, movie stars, by using android. Well, I guess that's kind of over, but, you know. How about this idea: if I get old, if suppose I need to stay in a hospital, right? And in such case, I definitely use my android for teaching my class, okay? So, a tele-operated robot, a tele-operated android, they can change our lifestyle, especially for the elderly people, right? Of course, the robot, it support elders, but the robot can work instead of, you know, elders. Not completely, right? The elderly can tele-operate the robot, right? So, elderly people have much knowledge and experience, right? They going to be good teachers, but the problem is that they cannot visit the school, right? In such case, you know, they want to use a tele-operated android. So, I think that is a very possible applications. And now I'm building another – my copy, and that's going to be a more portable. And so, that – what I'm going to do after this is I'm going to just send the android to the foreign countries for having this kind of interviews, right? I can stay in Japan, or, you know, I put them, the android, in my office and I never go to the university anymore, I just travel around. <laughs> So.
Peter Asaro:
Well, I'm glad we got the real you on video.
Hiroshi Ishiguro:
I'm very serious.
Peter Asaro:
Yeah, we'll come back and interview your android later.
Hiroshi Ishiguro:
I just send android next time.
Peter Asaro:
Thank you. Is there anything else you'd like to add?
Hiroshi Ishiguro:
No. I think that's enough.
Peter Asaro:
That's good.
