Oral-History:Ken Goldberg
About Ken Goldberg
Ken Goldberg grew up in Bethlehem, Pennsylvania. He earned a B.S. in electrical engineering from the University of Pennsylvania in 1984 and a Ph.D. in computer science from Carnegie Mellon University in 1990. After graduating, Goldberg taught computer science at the University of Southern California until 1995 when he moved to the University of California, Berkeley. Goldberg is currently a distinguished professor of New Media at Berkeley, Editor-in-Chief of the IEEE Transactions on Automation Science and Engineering, and an IEEE Fellow. Much of Goldberg's work has been at the intersection of robotics, art, and social interactions.
In this interview, Goldberg discusses the development of his work in robotics and automation, his contributions to IEEE, and the future of his research.
About the Interview
KEN GOLDBERG: An Interview Conducted by Selma Šabanović and Peter Asaro, IEEE History Center, 17 November 2010
Interview # 697 for Indiana University and the IEEE History Center, The Institute of Electrical and Electronics Engineers, Inc.
Copyright Statement
This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to Indiana University and to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.
Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, IEEE History Center, 445 Hoes Lane, Piscataway, NJ 08854 USA or ieee-history@ieee.org. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user. Inquiries concerning the original video recording should be sent to Professor Selma Šabanović, selmas@indiana.edu.
It is recommended that this oral history be cited as follows:
Ken Goldberg, an oral history conducted in 2010 by Selma Šabanović and Peter Asaro, Indiana University, Bloomington Indiana, for Indiana University and the IEEE.
Interview
INTERVIEWEE: Ken Goldberg
INTERVIEWER: Selma Šabanović and Peter Asaro
DATE: 17 November 2010
PLACE: Berkeley, CA
Early Life and Education
Asaro:
So, we’re just going to be asking your career and experiences in robotics.
Goldberg:
Great. I’m happy to talk about it.
Asaro:
So, can you start by telling us where you were born, where you grew up?
Goldberg:
I was born in Nigeria and only lived there for six months. My family was there – my parents were there as teachers in a little village called Ikenne. It’s always amazed me that they decided to have a baby in this remote village that didn’t even have running water when they first got there. But, my father was an engineer, a metallurgist, and my mother was a teacher, a psychologist. And so, we lived there for about six months and then my mom said, “We got to get out of here.” They had netting over my bed because there were spiders and scorpions and all kinds of stuff like that; so, to keep them off of the baby. And then, we moved back to Columbus, Ohio, and my father did his Ph.D. there at Ohio State. And then, we moved to Bethlehem, Pennsylvania, which was a steel town. He was a steel—he did research in steel. So, that’s where I grew up. So, it’s a little town in Pennsylvania.
Asaro:
Where did you do your undergraduate studies?
Goldberg:
So then, I did undergrad at Penn, which was actually where both my parents had attended as well. I started out actually doing metallurgy like my father, but I realized I didn’t like it. And so, I ditched that I wandered over to the EE department, ended up majoring in electrical engineering, but one part of that story – well, I took a year off. I did a year abroad my junior year in Scotland, and that’s when I discovered – actually, the way it worked there, they had a fair of different types of courses you could take. I was wandering around there and there was a guy sitting at this table with AI. And so, I took a full year course in AI. It had aspects of robotics and all these different parts in it, and that’s when I sort of discovered robotics. But when I came back, I’ll never forget, I was walking in the electrical engineering building at Penn and I saw a sign for the ESP lab. I thought, “They’re doing research in ESP here? I just have to check this out.” So, I went and it turned out to be a robotics lab. ESP stood for Experimental Sensory Processing, and it was Ruzena Bajcsy’s lab. So, that she was the one who started me off on doing research in robotics, and I did it as an undergrad. I just hung out with the grad students for the last two years I was there.
Graduate Research in Robotic Manipulation at Carnegie Mellon
Šabanović:
So then, after undergraduate where did you go?
Goldberg:
Then, I went to Carnegie Mellon and I worked with two faculty there; Marc Raibert who was doing the running machines, and he was also doing tactile sensing. I was very interested in how do you give robots the sense of touch. So, there were all these interesting problems. It’s still not a solved problem actually, having a reliable sensor. Human sense of touch is incredible, and we can detect a very microscopic little ridge if you run your finger across it. We don’t have anything that can do that as a robotic device. But, when I was at Carnegie Mellon, I met Matt Mason and he was doing work on the theory of manipulation and much more the mathematical modeling and geometry, and I really got excited by that. That’s what I ended up doing my dissertation on.
Asaro:
So, what was your dissertation project?
Goldberg:
Dissertation was on the mechanics of manipulation. So, what we looked at was the problem of orienting parts, grasping and orienting parts and in particular, what we looked at was a very simple gripper, the parallel jaw gripper, and that’s the simplest gripper. Actually, that’s one. It just opens and closes and it’s just binary, on/off, but you can pick things up with that. In fact, we argue that you can do remarkable things because if you look at someone eating Chinese food with chopsticks, that’s a parallel jaw gripper basically. We started looking at how you could use it to orient parts on a factory assembly line. We noticed – the first thing was that parts could get caught in certain unstable configurations. This was due to frictional properties of the gripper. It turned out that if you could remove the friction, you could avoid these unstable configurations. The problem was then when you tried to lift it off the table, the part would slip away. So, we ended up coming up with this – I had this mechanism that I designed that was just purely mechanical that would basically have low friction with linear bearings, but would have high friction inside so that as you lifted the part, it would be stable. I got a patent for that. We filed the patent while I was at Carnegie Mellon; then it was issued a couple years later, but my dissertation was on the mathematics of orienting parts in a plane. And so, we looked at the idea of how these grippers could go through a sequence of operations and have the part end up in a unique final configuration even though you’d never sense the orientation of the part. Here’s the analogy. Let’s say I want you to put a ball, there’s a ball somewhere in a room and you need to put that ball in the center of the room. Now, you’re not allowed to – I’m not going to tell you where the ball is. It’s a very interesting problem, right, because you would first think that it’s impossible; how can I get the ball to the center of the room, but the answer is you could imagine where you sweep from the outside toward the center and if you do that carefully, at the end, the ball will be at the center of the room even though you’ve never sensed it. And that was sort of the idea we used with the geometry of grasping, and we were able to show that you could generate these sequences of grasps. As long as you knew the shape of the part, you could go through these operations without ever sensing what was going on, and the part would end up in a unique final orientation. It was like a sort of mathematical puzzle, and then we showed that you could compute that for any shape, any polygonal shape and that there was always a solution for every polygon. So, that was the dissertation topic.
Asaro:
Great, and what was the next project you worked on after that?
Goldberg:
Well, let’s see. I did a little work also on stochastic manipulation when I was at CMU, which was looking at probabilistic models. So again, you didn’t know exactly what was going on. I would say a fundamental issue, just to step back, in robotics is uncertainty, that the world is just very uncertain. We are very, very graceful and adept at dealing with that. We do it every day all day long and we just have techniques to naturally handle it. But, robots get very stuck because as soon as something isn’t where they expect it or there’s an error, they basically get jammed. So, there are different ways of modeling this and at that time, I was really interested in using ideas from decision theory, which is a branch of statistics. So, I took courses in the statistics department and we started to apply them to doing things like stochastically orienting the part. So in other words, you wouldn’t do it perfectly, but with probability you could succeed.
Asaro:
And that was also Carnegie Mellon.
Robotic Art Projects at Carnegie Mellon
Goldberg:
That was also Carnegie Mellon. Should I tell you about the art projects we were doing?
Asaro:
Yeah.
Goldberg:
Okay. So, the other thing we were doing is I was really interested in making art and in some ways, it was a struggle for me because I wanted to do art and my mother, who’s very practical, said to me, “Look, you can do all the art you want, but first get your engineering degree.” And so, I’m actually really always appreciative, thankful for her of doing that because I now know how brutal the art world is, but the nice thing was that because I came from engineering and done engineering background, I had a very different approach to making art than most artists. But, I was able to keep doing art and at Carnegie Mellon, I had a number of friends in the art department and we at one point rigged up a robot to paint. So, we put a brush on it. And of course, obviously you can have a plotter that would perfectly reproduce, but I wasn’t interested in that. I wanted to see how the brush, the imprecision, the probabilistic aspects could come into play by having a loose brush that had a spring on it and it would dip into paint and it would slobber the paint all over the place. But, the resulting paintings were not so obviously from a robot. The problem was they were terrible paintings. It was really interesting to me. By the time I graduated, I had a show in the student art gallery and they gave me the whole gallery, and I had tons of these paintings. They were all terrible, but what everybody was doing; everybody was in the back of the room watching the video we had made of the robot doing the painting. That got me excited because I started to realize that it was the robot, the performance of it that was much more interesting or compelling than the paintings. So, I started moving more in that direction. So, since then, I’ve done a number of installations involving robots and now, actually I’m interested in involving people who work with technology and oftentimes their physical bodies. So, I’m working on a piece now at the Contemporary Jewish Museum that uses a computer, a robotic camera if you will, but it will basically track the motion of visitors in the gallery very precisely and then direct sound right to where they are. So, that was all going on. I would say I had my day job and then at night I would come in with the paint brushes and we’d rig these things up, and the professors were very tolerant, although one night the robot went a little bit haywire and spilled paint all over this guy’s – actually, it wasn’t my lab. It was the lab next door, and he came in and he was a little bit miffed that I had been painting in this lab. But, that was the great thing about Carnegie Mellon. They were very open to all kinds of hijinks and unorthodox experiments with robots, and I think it’s part of the reason it’s such a great place to do robotics.
Šabanović:
Did the robot have any logic to the way it was sloshing paint around, or was it just random?
Goldberg:
No, I should say. So, what I would do is I would draw these pictures. They pretty figurative and I would not claim to be a great figurative painter, but I would do them with a mouse. So, that was the code. Then the robot would execute those, but it was always very proximate. And I really wanted to get this kind of gestural, kind of abstract expressionist painting. Actually, this is one from a later project, but all the oranges were generated by the robot or painted by the robot. Now, you can see a certain similarity to them even though no two are exactly alike, but that was a joint project I did with an artist in LA when I moved to USC.
Robotics, Art, and the World Wide Web at USC
Šabanović:
So, has your research in, or not your research, but your work in art fed into your work in robotics?
Goldberg:
Oh, absolutely. Here’s the best way. When I started at USC, I was – so, I graduated from Carnegie Mellon and I was very lucky to get a position at USC with George Bekey, who I hope you interview because he’s fantastic, and actually, they have a number of great researchers there. It’s a real I would say center of robotics. There I was still doing this work on manipulating parts and geometry and all those things, but I had a lab and I did this art project where we built this big crazy robot that was going to paint on a big scale, like a 20 foot scale. That’s what this came from. In the course of that, I had a number of students who were kind of a little bit unorthodox, some undergrads and characters, and one day they came to me and they said, “You got to check this out.” This was 1993 and they walked me down to the lab and they showed me the World Wide Web and I was like, “Wow, this is wild.” Then they said, “Listen, we want to do something. We want to do something with this.” We started talking about having a robot on the web. We said, “Let’s make an art project.” It didn’t seem to me like this was going to be a research project, but I could definitely see art. And so, we started working and it was a nighttime thing where they were coming in like all hours of the night. We had a sense that someone might beat us to it. So, we worked like crazy from January and we launched it in August 1994. It got all kinds of attention. It was in the London Times and Newsweek and it was in ABC News. A week later, another team launched on in Australia. It was like they were that close to – we were close to being scooped. The reason it was interesting was because it was – I always saw it as an art project. In fact, we worked – there was a whole story around. We had a sort of fictitious story that it was all from a radioactive zone in the desert and people had to come in and help us excavate this radioactive site. Of course, you couldn’t tell the scale, but you were operating the robot and you were blowing air in the sand and so you were doing this thing. And so, it really attracted interest partly because if I had just said, “Oh, assemble the water pump,” that might not have been very compelling, but I was really tuned into the idea of making an art project that I knew would – how to get people engaged with something like that, have a narrative around it. And so, after that, it was so much fun to see people coming in and operating our robot and literally, those are online 24 hours a day and you could hear in my office upstairs all day long, people driving it and moving it around. Tens of thousands of people were doing it. So then, we got some funding and we did a next version called the Telegarden. That was the garden that was operating, controlled over the web, and that was much bigger because we had a better robot, color camera, but there we had plants. And so, people could move around, look at the plants. They could water them and then they could plant seeds. So, coming back to the question of research, in the context of that, first of all, I started getting exciting by the idea that there was some real engineering involved and how do you design a system that could be operated by non-experts from anywhere in the world? It had to be robust. In fact, that garden was online for nine years pretty constantly. And so, we had to think a lot about how to make it bulletproof and do a lot of error detection. But, the other part was that the whole field of networked robots started – people started talking about that. So, we formed an IEEE society under robotics automation, the technical committee on networked robots in about ’95 or ’96 or so. Pretty soon, we had 200 members. We published a book; we published two books actually with MIT Press about them, and there were all these technical issues about the time delays and how to do them and now, it’s actually interesting and that it would hard to imagine a robot that’s not on the network. So, it was a really interesting arc for me and that I probably would have never gotten interested and worked on that area if I hadn’t been doing art.
Founding the Technical Committee on Networked Robots
Šabanović:
So, who else was with you in the founding of the technical committee on networked robots?
Goldberg:
There was a person, Roland Siegwart, who you hopefully will also interview. Can I get the book?
Asaro:
Yeah.
Šabanović:
I can get you the book. Keep talking.
Goldberg:
Is it beyond the webcams? Is it sitting over there? It should be. <Crew talk>
Goldberg:
So, you have done a little bit of background research.
Asaro:
Yeah.
Goldberg:
Am I telling you things you already know?
Šabanović:
Oh, but this is kind of important, yeah.
Asaro:
What’s on the record.
Goldberg:
Oh, okay. Okay. All right. Well, just stop me if I’m going off. But, Roland was a co-author on that, a co-editor and he was working on network robots in Europe and so, we started doing this and then we brought in Wolfram Burgard. And then over time, it’s evolved and there’s a number of new people. Dezhen Song, my student, is now one of the co-directors. Nori Hagata from Japan and Korea was involved. I have the whole list online, but it was really interesting to see that – we gave a talk on the robot controlled by the World Wide Web at a IEEE conference where we talked about all the technical aspects and how it worked. I was amazed. I’ll never forget. We walked in and the room was just packed. I mean it was just like so exciting because it was brand new. The web was new to everybody and everybody was just trying to figure out what you could do with it.
Asaro:
What was that first one call with the radiation?
Goldberg:
The radiation; called the Mercury Projects and that’s online. There’s archives on it, lots of information.
Asaro:
What was the Australian project? I’m curious.
Goldberg:
The Australian was stacking blocks. It was kind of – a guy named Trevelyan, who’s actually a really well-known researcher, and one of his students - I want to say Taylor. I can look it up for you. In fact, it’s probably in that book. I think they wrote a paper, one of the chapters, yeah.
Work with Networked Telerobots
Šabanović:
Did you have a feeling for how people used your systems at all?
Goldberg:
Those telerobots? Networked telerobots?
Šabanović:
Mm-hmm.
Goldberg:
Sure, because we have tons of documentation. That was the other nice thing about it is when I did art projects that were in a museum environment, we would come in at the end and you’d see like a guest book where someone might have written a few things, which really didn’t have any real contact with watching people use it. But on the web, you could have all these logs. At the end of every day, I could see who had visited, where they heard about it, how much time they spent. So, we have all kinds of information about what they were doing. With the garden, there was all kinds of stories. I mean there were aphid infestations and actually, the biggest one, we made this huge mistake, which is that when we got the garden all set up and we brought in plants to sort of inspire people because it would start out just bare. So, we brought in these plants and then like about two weeks later they all died and we were like some mysterious fungus or something. So, we brought in all the new plants again and they died. And like after the third time, I called in a friend of mine. I was like, “What’s going on here?” He’s like, “Tell me about the system.” He’s like, “What’s your light cycle here?” I was like it’s 24 hours because people are using it all over the world. He’s like, “You idiot. Plants need darkness as much as they need light,” and we were burning them up. It just never had occurred to me. So, you needed a dark cycle and a light cycle. So then, we put headlights on the robots so it could be dark at night and they could still see what was going on. So, there was that and then there was an early version of like a chat system. So, there was a Village Square and people would get together and talk and it was actually one relationship that began. People got married that had met in the garden, and the thing I was most surprised by was an e-mail I got one day from a student who said, “How do I know there’s really a garden here?” It was one of those questions that completely because we were done there running the plumbing and running all the cables and all this. I have to say, this was a fantastic group of students that were involved. George Bekey was also on that team. Steven Gentner and Jeff Wiegley were the two most active, who really put amazing amounts of time into it. So, there was no question for us that there was a garden, but this kid was asking a very interesting question and I kept thinking because there were fake systems out there around that time. People were making lots of hoaxes and there was lots of disinformation as there is today. So, how do you know? I thought, actually, you could fake it, right? We could have just taken a camera and taken pictures and pretended the whole thing and just indexed into pre-stored images and it could be faked. So, right around that time I moved up to Berkeley and I was offered a position here. When I came up I learned that one of my heroes, Hubert Dreyfus, the philosopher, is here on campus. His brother actually is in my department and I went and I said, “How can I find Hubert Dreyfus?” He said, “Let’s have lunch with him.” And so, I got to meet my absolute hero and not only that, but he was super interested in robots and technology, and we had this great conversation. I told him about this story. He said, “Well, this is a philosophical question.” He said, “This is epistemology.” He said, “It’s about knowledge.” So, we started this four-year project together where he introduced me to a number of philosophers and we invited them to write essays and we ended up calling it telepistemology, which is what can you know at a distance, and how the Internet was actually reengaging that question, the same way that the telescope engaged that for Descartes. So, we’re actually now very good friends. We meet about once a month and we’ve taught a course, two courses on philosophy and technology; most recently, a course on Heidegger. I sent you that article, but that was around telerobots and a bigger question about how technology changes and shapes our behavior.
Telepistemology: Robotics, Philosophy, and Social Networks
Šabanović:
So, has your questioning into telepistemology gone any further after that?
Goldberg:
Yeah, because then that became the central question that sort of guided my art projects for the next ten years. And so, we had a laboratory where people burned hundred dollar bills. We did a Ouija board. That was in the Whitney Biennial, in the year 2000 and a number of projects that tried to look at that from different perspectives, always trying to question what was knowledge, what is knowable, how deception and agency interact, and also the question of the physical body, which is something that’s always interesting to me because robotics is at this interface between the computer, the abstract digital world of pure representation and then the physical world that we live in. And so, the robots are always trying to bridge that gap in various ways. And so, I’m really interested in how embodiment plays a role.
Šabanović:
So, in your project with Dreyfus, what kinds of things did you look at? Did you build any systems for that?
Goldberg:
I’m trying to think. I mean we talked a lot about the projects we worked on. I would say probably the best outcome was the courses that we taught that had to do with – we looked at questions about efficiency and flexibility, and how the technologies of the 20th Century are all about efficiency and the technologies of the 21st Century are about flexibility. So, these are things that are very, very adaptable. Think about nanotechnology or stem cells, new kinds of polymers. They don’t just do one thing; they do many things. The computer is another example of this and of course, robots too.
Šabanović:
I mean in terms of telepistemology, how do you see the current since currently there’s somewhat of a rising craze in telepresence from what I can tell. Do you think that that research has been in any way building upon some of the stuff that you did, or have they really developed further, or is it just finally come into a sellable form without really challenging kind of how we know or how we can embody ourselves in the mechanism?
Goldberg:
Yeah, I think those questions are still there. I mean one of the things – we try to do certain things, but the technology was really not ready at that time, and we do not develop our own video codecs and things like that. We were using lots of off-the-shelf tools and then recombining them. And so, at that time, having high-speed video was just not practical. There was no wireless when we were doing our experiments around 2000. So, like Wi-Fi wasn’t commonplace. Now, a lot of the tools are much better. As you know, the G4 with the iPhone gives you really very, very respectable teleconferencing, video teleconferencing. So, it would be interesting to try and do some of these experiments again and see, but I still think the idea of being able to navigate, to essentially be present remotely is very compelling and very interesting.
Šabanović:
What do think some of the ways to achieve that would be? So, I mean, for example, in your Telegarden, one of the things you were talking about is that you were very embodied and you were obviously constantly interacting with the system and the system was reacting to you. And so, how reactive was the system to the users when they were – I mean could you see yourself moving stuff around?
Goldberg:
Right. Yeah, you could. One of the things though you’re getting at is we were very interested in how the people, how people were responding. It was, in a way, the social network was almost more interesting than what was going on with the robot. They were interacting with each other through the robot. So, the robot system had to be designed in a certain that was compelling for them and kind of, I would say, realistically reflected what they were doing so that they’d want to spend time there, but their interactions with each other was really interesting. So, we developed a robot system; this was with Eric Paulos and John Canny and a few others that we called the Tele-Actor. Here, the idea was we wanted to put robots into much more complicated environments. In other words, we were thinking of this evolution. We started with digging in the sand, which was like hunting and gathering, and then we moved into agriculture with the Telegarden and then we wanted to go to a social environment. We wanted to bring these telerobot systems into like a cocktail party or an art opening and all kinds of things like that. So, we looked around and the robots, at that time at least, were just not reliable enough. They were just too clunky. So, we said, “Well, listen. We’re not so concerned about that. Why don’t we just finesse that question and we’ll replace – in this case, robots have been replacing humans. We’ll replace the robot with a human.” And so, we’ll have a person who puts on the helmet and goggles and basically then all the information is transmitted back to the Net. And so, people anywhere in the world can observe what’s going on. And then, the key idea was collaborative tele-operation. So, you had many people basically providing input, basically where they wanted to go and other things by indicating with markers on the screen, and all this information would come down to the Tele-Actor who was at the art opening or the cocktail party or at one point, we wanted to be invited to a dinner at the White House. And then, she or he would see a display and then make a judgment call about what the group wants me to do. The idea was that the Tele-Actor is not a slave. They aren’t just following orders. It’s a whole other story, but it was about the Tele-Actor like an actor, taking direction, but then obviously having to improvise in these very complex situations. So, we had an NSF grant. We did that for three years and it was a very fun project. We took it out of all kinds of environments and she was on stage at the Opera House in San Francisco at one point. And so, I think that kind of thing could – I still think that could be interesting. I mean how often do you want to – there’s some event. You have a conflict. You have to be two places at once. But if you could imagine that this could be worked out and that you could just dial in and you could essentially go to that party or that other event and really not just watch a video after the fact, but actually be present and talk to people, etc. I think people would be interested in that.
Šabanović:
What do you think are some of the limitations, either technical or actually kind of within the human psychology of being present in a remote space?
Goldberg:
Well, because being present is an incredibly rich experience, right? You’re embodied, immersed in a physical environment. There's so many things that you consciously or unconsciously are part of that. So the sound of the fan that we're hearing right now that you'll probably get rid of with room tone afterward. And the fact is that there's a million things that I won't even pay attention to but if they would suddenly change, if something would scurry across the floor, I would notice. And so this idea that you're immersed in this kind of really complex thing, your brain is only giving the tip of the iceberg but your body is being sensitive to so many of the other things. So I think that's going to be very hard to reproduce. I'm not one of these sci-fi fanatics who believes that this is going to take over and it's going to be better than being there, I still, maybe call me old fashioned but I still want to go to that party if I can but if I can't and I have to be home babysitting or something, it would be great if I could have some approximation to reality. Well you mentioned technical, I would love to talk a little bit about some of the other – just going back to the day job part of this robotics work because I've been very active in IEEE and it's been a great organization for my work. I mean I've published almost all my work within IEEE. My students and I got involved in these questions about manufacturing, assembly lines, part feeding which is orienting problem and fixturing which is how do you rigidly hold parts. And one of the things we did in almost every case was abstracted the problem into – we made certain assumptions so that we could provide a nice formal mathematical model of what we were doing and then we could start to prove theorems and form… rigorously analyze the context. So we had some results in part feeding, we generalized the results from my dissertation to curved parts, algebraic parts, parts with friction, conveyor belts, so we had a number of things there. Also manipulating very tiny parts, so part of the micro scale using vibration. And then we got interested in this fixturing problem which is kind of a dual problem which is how do you now hold apart rigidly using very simple primitives. And so we had an algorithm that lets you configure fixtures, you draw the part and the system will automatically compute the fixtures. In both those cases we now have projects online, applets in Java, so you can go in and draw your own part and then it will generate the illustration, actually animate for you. And I don't know if you can edit something like that in later on the video but it's very fun to see it on – I can show you on a computer. So I'm really interested in the idea of how the network can also be used as a way to illustrate research and share results and let people – I was saying to a friend of mine recently that I did magic, I wasn't very good but I did magic shows when I was a little kid and part of that I think is that I'm aware of the many different ways that magicians work and tricks and how perception is so malleable and that in a lot of ways I really think that research – when you show a video of a research project, it's so easy to essentially show only the good case. And I always want to say, "I want to try it myself, let me try that, see how good it will really stand up." And a video won't let you do that, but if you have a website where it's interactive, you can go in and somebody can try their own version and that's much more compelling, so we did these websites. And then I've gotten interested in the last five years in medical robotics and that's been very exciting because here the environment is very uncertain. And it's bad in a factory but in a medical environment, you have just a lot more uncertainty, that you have very good imaging often and actually imaging is getting better and better with MRI and very sophisticated CT scanning but the body is very deformable and changes. So as soon as you come out of the MRI machine, everything you thought where the tumor was has shifted was has shifted slightly. And now you want to get radiation say, is a really interesting challenge is how do you account for that. So we've been applying techniques from robot motion planning that are actually are relevant for human surgeons as well as robots. But the idea is how can you try and model and anticipate deformations and errors and then plan paths for the say needles that can go around organs and deliver seeds to the desired location. So that's been really exciting. We have a project with Johns Hopkins and NIH on that and we actually just got a patent on something we call steerable needles which is a way of basically controlling a needle from outside the body by steering it at the base.
Founding the IEEE Transactions on Automation Science and Engineering
Šabanović:
So who are you working within NIH or at Johns Hopkins?
Goldberg:
Allison Okamura who would also be great, Russ Taylor, Greg Chirikjian and Noah Cowan. And we also work with UC San Francisco with medical physicians and then I'm also just started to work with a robotics expert Jacob Rosen at UC Santa Cruz and a medical roboticist, Doug Boyd who's at UC Davis. Almost all these projects are collaborative and you probably have heard this in the interviews but one of the other great things about this field is that it really is about people working together and because robotics is so cross disciplinary, you need to have computer scientists and mechanical engineers and sometimes materials experts and many different backgrounds. And what often happens is, that becomes so second nature that we don't think of ourselves as, "I'm an electrical engineer." I'm a roboticist which means that I'm comfortable doing things and I'm comfortable talking to mechanical engineers and computer scientists and industrial engineers. So I think that's an interesting aspect of that field. But it seems to be really a trend now, a scene here at Berkeley and I think at a lot of universities of cross-disciplinarity, people are really feeling much more comfortable working outside their area of specialty and working with someone who can complement them in interesting ways. One other thing I'll just mention is, another IEEE story is that I got involved in the – around 2002, the President of the Robotics and Automation Society, Dick Volz emailed me and said, "We are having a challenge that there is not enough – the Robotics and Automation Society is to heavy on the robotics and we don't have enough of the automation. So how can we get more of this here?" And he said, "We have a hunch that maybe if we had a second journal, like our journal is very theoretical but it's almost all robotics work. How can we attract the industrial folks to get involved?" He said, "Maybe we'll have a second journal that's just about industrial." And he said, "I want you to head a committee to investigate this." So not knowing that it was going to end up taking up three years of my life, I took that on and started basically getting a committee – we did almost everything on the Internet but we had thousands of emails to basically try and do a real study of what was out there and what was going on. And one of the things we came to pretty early was we didn't want to have two classes of journal where one was going to be like the theory journal and one was the practice journal because I thought, nobody would want to publish in the practice journal, it's just not going to be as sophisticated. But instead we said, "Let's have a robotics journal that does theory and applications and an automation journal that does theory and applications and they would both be the very high quality archival journal, rigorous." And that idea – so we worked on articulating that and telling that story for about two years and then we proposed it and it was voted unanimously by the Society and then went up to the IEEE and they approved it. And so in 2004, the transactions on Automation Science and Engineering started publishing. And so now it's in its about sixth year and it's just tremendous, it's now as – in fact, it was interesting, I think it was last year, it was bigger than the robotics transaction and has all the rigor and all the quality of the robotics but is much more dominated toward practical industrial problems.
Current Applications for Automation
Asaro:
What do you see as the biggest applications for automation right now?
Goldberg:
Oh, actually it's a great point because one of the things that – another aspect of this was we didn't want it to be only factories, I mean factories are the biggest place where people think about automation, they think about assembly lines and automated systems and that's still a huge area. But we wanted to say, let's not limit automation to factories. So there's so many other applications, example is the Fast Lane system for traffic, right, that's an automation system. And we started thinking about automation for all kinds of transportation problems, for areas of healthcare, not only for robotic surgery but robots that could transport food trays and prescription drugs around the hospital, that's automation. A robot that would monitor by the bedside in much more sophisticated ways than it's being done now, that's automation. Robots that would monitor patients in a home or senior citizens and if they fell or something like that could tell. Security is a huge area, so in almost every kind of environment there's security needs for systems that can monitor the environment, not just passively cameras, but actively detect when something unusual is going on or maybe send a robot out that would be able to check around if they saw something suspicious. So there's that, I mean systems in even games, you can imagine aspects of automation. So the way we characterized the difference between automation and robotics was that robotics is really where you're trying to proof of concept, you're trying to put something in an environment that there's a super high degree of uncertainty. And then automation is where you’ve understood the environment pretty well and now what you're trying to do is make something that's very, very robust and reliable and cost effective. So the criteria are different. The problems may be similar but their kind of approaches oftentimes look different.
Goldberg:
The process of working with IEEE, I've spent a lot of time in meetings and learning the policies and how a journal is born. I mean there are 50 journals, 50 societies; I think probably each has a journal, a technical journal and then a magazine. So IEEE is this huge publishing behemoth and I've visited, I'm sure you guys have been to Piscataway, New Jersey, have you been there?
Šabanović:
Mm-mm.
Goldberg:
Oh…
Asaro:
<inaudible>.
Šabanović:
Have you?
Asaro:
I've been through Piscataway when <inaudible>.
Goldberg:
Oh, yeah, well it's a Mecca, you know, it's like this huge operation, it's like a university campus, lots of buildings, all kinds of offices. And that's where they actually put together and assemble all the technical publications. So it's a really huge operation and we got to see how these decisions get made and how carefully it was – when we first put the idea of the journal out, other societies, we had to make sure we weren't on their turf and had to negotiate and all the levels of approval and financing and how we launched it. We had to develop our own kind of posters and brochures. Anyway, so it was really fascinating. But I have a lot of respect because the quality is extremely high and there's emphasis on rigor and it requires huge amounts of time. The other thing about IEEE is how much dedicated volunteer time goes into it. People around the world, as we're having this conversation, thousands of hours are put in, man hours, person hours are put in doing just reviewing, reading, writing, organizing. I mean that's one great thing about the organization because I've had the chance to work with some incredible researchers who are not only great minds or great theoreticians and engineers but they're also extremely generous with their time and organizing and planning and taking care of every detail down to, "Where is the boxed lunch going to be the for the banquet?" and, "Where's parking going to be?" There's a million of those kind of things. And people don't get paid, they're doing it all for the good of the society.
Robotics and Automation
Šabanović:
So I think from what I understand of your work, would you say that most of it is in robotics rather than automation?
Goldberg:
No, it really straddles both because all the work I was describing with the part feeding and fixturing and those things are automation. And the robotics, I would say the telerobotics is in the category and then also the medical robotics is kind of more on the robotics, yeah, I would definitely say it would be more on the robotics side.
Šabanović:
Just so that we kind of have a list of these, what are all the robotic platforms that you've worked with for the different robotics projects? So we had the Telegarden, we came to medical robotics, do we have stuff after that or have missed anything in between?
Goldberg:
No, I guess – I mean I've worked with a number of industrial robots, like this Adept arm and those systems are always very impressive because they're so well engineered and they have to work round the clock for years at a time, so I really developed a respect for how those systems are made. And Brian Carlisle in fact, the president is also another great part of this whole robotics field. And then we built a number of these kind of systems that were – oh, I guess what I didn't mention was we've been building robot systems for monitoring the environment. So we've been using robotic cameras that have pan, tilt and zoom capability and putting them into environments to monitor birds and wildlife. Where in one case there was an issue about global warming and they were not sure if the bird population has shifted because some tropical birds have moved north. And there was anecdotal evidence but there wasn't field documentation of this. And it's very expensive because it's a remote area of Texas, so we put a robotic camera out there and then we invited people to come in and operate it and they could just pan, tilt, zoom in on things and take pictures. Well, it's amazing that people have spent thousands and thousands of hours on a system, and documented huge amounts of material. We have photographs of hummingbirds, not to mention at night raccoons and people are on this thing all the time and they documented and they found three of these extremely rare birds that were not known in this area 20 years ago. So it's confirmed some of that hypotheses about climate change in the area. And those systems, again, we've often – in almost all cases, we've used off the shelf components but then we build all the software and the interfaces.
Šabanović:
And when did you do that?
Goldberg:
The bird project or I should say, we called it CONE, Collaborative Observatories for Natural Environments. And that was 2003 to 2008 or so. And in the course of that, while we were doing it, we heard about this ivory-billed woodpecker. Do you know this story about the bird that was presumed extinct, it's this grand bird that was – at one point they thought that it might be the national bird of the United States, it's a beautiful giant woodpecker and hast been seen since 1940. And then there was a sighting by a group from Cornell and it made the front page of the New York Times. And the thing is that bird watchers – are you guys birdwatchers? No. In fact when I ask around California, nobody and New York, nobody but in the middle there's 40 million birdwatchers, it's a huge activity and this is U.S. alone. So it turned out that everybody, if you're a birdwatcher knows about the ivory-billed woodpecker, it's like the Holy Grail. And so they had spotted one and it made this huge deal. Well the problem was they couldn't, they didn't have any good footage, they had one little piece of video. And so we thought, this is a perfect application for our system. So we went to Arkansas, we rode out with these experts into the back woods and they showed us what they were doing with their mosquitoes and alligators and it was a terrible environment and they're trying to monitor. So we built, Dez Song did most of this work, he's now at Texas A&M and designed a robotics system that would monitor and the key idea was that whenever something flew by that was big enough and basically had somewhat of a signature, it would take a video clip. And then we would, actually it was a collaborator who would row out on a rowboat a couple of times a month and pull out the disk drive and put in a new one and then we'd go back and analyze the images. And we found a number of woodpeckers, a pileated woodpecker which is very similar to the one we're looking for but much more common. So that didn't make any front page headlines. And then lots of other birds but we did not get any footage of the ivory-billed woodpecker, which we later did some back of the envelope calculations and it was not surprising that even if there were a couple out there that the odds of us catching, you know, actually getting it were pretty remote. But it was a really interesting experience for us to develop a system that could really work in a really remote hostile environment like that.
Work with Robot Learning
Asaro:
So from a theoretical perspective it sounds like a lot of your work is based on mathematical models. Were there other sources of inspiration intellectually or theoretically to your work?
Goldberg:
Well, the one thing I'm getting very interested in right now is robot learning and statistical robot learning. So we hired somewhere here, Peter Abbeel who's a young researcher who did his dissertation at Stanford on controlling helicopters and not just controlling to hover and sort of move up and down but he wanted them to do acrobatics, radio controlled helicopters. And you know there's this whole sub genre of people who do these incredible tricks. And it turns out there's only like a few dozen people in the world who can do these really crazy things like have it hover upside down. And so he wanted to automate that and so the way he did it was he said, "We have no idea how to program this, it's too complicated. What we'll do is we'll observe the human expert and then we'll try to infer statistically what would be the idealized control that the human expert is doing." The human can't even tell us, right, they feel it instinctively. And Peter applied these beautiful techniques from control theory, so calm and smoothing and expectation maximization. And he was able to show that you could observe a human doing the trick ten times and every time was slightly different but he could align them all using a technique from speech recognition called dynamic time warping and then he could use calm and smoothing to basically infer a smooth trajectory. Then he could execute that and he could autonomously repeat these incredible maneuvers and the videos of this are fantastic. Anyway he joined the faculty two years ago and we started working together on medical robotics, could you apply this to the operating room. So we have some results, having a surgeon perform very simple – you're not going to do the whole gallbladder operation but a suturing step, sub tasks we call them so that an expert would do the suturing with a robotic system like the da Vinci robot and then the system would observe that and record that and then do this time alignment and smoothing to basically come up with an idealized trajectory. So this is a whole new area for me, I don't have the background in it but I'm learning from Peter and it's extremely exciting, we have a number of new results in the past two years on this, what we call super human performance.
Asaro:
Were you ever influenced by the writings of <inaudible>?
Goldberg:
I've always been interested in the AI and some of the work on intelligence, chess playing robots and that work in particular; I would say I was somewhat of a skeptic. Just by nature I felt that to some degree, to me the mind is it's amazing that it's still as mysterious as it is. I mean we understand about quasars and black holes and things that are a million light years away but we don't really know how something right here works. And it's amazing because we can analyze it with scanning electron microscopes, we can look at it from every different direction but we don't know how it works. And I actually find that extremely satisfying <laughs> that the mystery exists. So I was not happy when Deep Blue beat Kasparov because I was the one who thought, those masters, there's something they have that a computer just can't do but it turned out I was wrong, a computer can do – brute force is really powerful for a lot of problems but not for everything. So I think that there's going to be many avenues of creativity, of deep understanding that computers are not going to be able to do in the near future and the same for robotics. And so I'm very interested in this idea, which problems are really resistive to the techniques, to modern science.
Šabanović:
So what would you say those would be?
Goldberg:
Well, the ability to – and I'm just realizing it's…
Šabanović:
Yeah, it's five minutes ahead or something.
Goldberg:
Is it? Okay.
Šabanović:
If mine isn't wrong.
The Future of Robotics and Technology
Goldberg:
You mean things that robots are not going to be able to do in the near future?
Šabanović:
Yeah, like what are the limits?
Goldberg:
You know, there are so many things where robots are, and I have to be careful here because I know my colleagues will probably not like me to say but I think that there are many different areas where robots will be able to perform certain tasks, but I think that all of the nuance, the kind of gracefulness and subtlety that humans have, I think we're very far from seeing that. I think you can fake it, you can sort of have a robot that looks kind of human like for a little while but I think that the idea that really having a robot that could really keep you company, I think that's pretty far in the distance. At the same time things like a robot being able to change a diaper, right, I mean that would be – I have two kids, I would love to have one of those. Actually I think the way it will be is it will probably, when that gets solved and I'll say when because I do think it will, it'll be because we'll change the diaper completely, it won't look like what our diapers do now, or maybe we'll change the baby, I don't know. But the idea is that they'll be something just like – I think what we'll do is we'll adapt to the world of robots. So people say, "I want a robot to load the dishwasher after a dinner party." The answer is if you're willing to have plates that are square heavy plastic with markings and handles and everything else, that's going to help, you can get a robot to do that. Maybe people will be happy with that in the next decade that our house will start to be more robot compatible. And that I believe, it will have sensor, markers around and things like that to help the robot navigate, so it won't have to come in and just deal with the crazy clutter of the average household but the household will sort of meet the robot halfway.
Šabanović:
Do see that there's any dangers in terms of, you know, that humans are so inherently adaptable and potentially willing to adapt all kinds of things to the limitations of robotics?
Goldberg:
Oh, well don't get me started. <laughter>
Goldberg:
So this is my whole other…
Šabanović:
Sorry about that.
Goldberg:
Yeah. No, I think that technology is this really powerful force. This is what I've been talking with Dreyfus about is that technology is this extremely compelling thing, it's not just about the mechanisms and the gadgets and the devices because those are fascinating in themselves but it's a mentality, a mode of being in Dreyfus' terms that it really is taking over and is very, very powerful, it's not something that we can even be conscious of, it's sort of, we're immersed in it. And it's something that makes – we want to make the world more and more available so that transportation, from the railroad to the jet to telepresence are all ways of doing that. But so many other things where, the World Wide Web and digital radio and all these are – the video camera, right, these are all ways of making the world available. And what I do worry about is this idea that we'll start to turn it on ourselves, that we'll see ourselves – we'll want ourselves to be more and more available. And I was talking to a friend about this and he said, "Do you think it's some sort of Big Brother, or is this going to be imposed on us?" And he said, "Maybe is it more of a seduction, that we'll want it, because it's very, very compelling, it's very appealing?" And I think he's right, I think that that's the way it will happen is that we want to be available, there's something very, very irresistible about Facebook and Twitter and all of those things, we want it because we get this – my wife is a filmmaker and she talks about the oxytocin effect that, you know, you give a hug, you get this chemical boost and it happens when you have a true meaningful communication with somebody. And so I think people are really – we're very prone to that. And the danger of course is it can take over and we don't know how to resist it and so we lose that kind of ability to have privacy and sort of internal moments and one on one connections. That I worry, are we going to sustain that in a world where the technology becomes more and more compelling? I hope so.
Šabanović:
If you can <inaudible> just a few more minutes, I was just curious, just to end, kind of wrap up, two things, one, what are your kind of goals for the future in terms of your research and then also, where do you think robotics is going in the future, looking forward?
Goldberg:
Ah, it's good. Those are hard questions. I'm very interested in – right now I'm starting to be interested in social media and so we have a system that we call Opinion Space that's a way for people to interact and have conversations over a distance. Now it doesn't look like robotics but it involves geometry and a way of visualizing this map. So you can see everybody in a conversation at a glance and the most insightful contributions get highlighted and the least insightful get pushed back. So it's a way of sort of effectively scaling up conversations to large numbers of people. So we've been working with the State Department on a project, also now with a number of companies. And so I'm really interested in that direction, I mean how can we take social media and visualize it. And was the second question about the field in general? It's interesting, the big thing that everybody's talking about, you may have heard is Robotics 2.0. So the analogy here is that just as Web 1.0 was about a one way flow of information, read only information, Web 2.0 is when it suddenly became a two way dialogue, people are participating. So when you come to a website, you expect to be able to provide feedback or contributions to it. And so there's comment lists and all kinds of things like that, collaborative filtering. And something is happening in robotics right now that we're just labeling Robotics 2.0 and there's a number of people I'm sure interviewing like Henrik Christensen involved. But the White House is going to make announcement sometime in the next month or two about this white paper that we've been working on and the idea is that Robotics 1.0 which is up the first 50 years is really robots replacing humans and Robotics 2.0 is about robots collaborating with humans. So the idea being that it's much more human centric, we're realizing that the human is very much a part of this system so robots have to be designed very differently than when they're working autonomously, they have to be sensitive to people's presence, they have to be safe. Another thing, because I was directing the Center for New Media here, so this whole thing about what is a medium has been a fascinating question. So I think robots are media, absolutely. Thanks you guys.
Šabanović:
Thank you.
Asaro:
Thank you.
Šabanović:
Thank you so much for making time for this.
Goldberg:
Absolutely.
