Oral-History:John Hollerbach
John Hollerbach
IEEE Life Fellow (IEEE Fellow, 1996), John Hollerbach, was born in Marktheidenfeld, Germany to Hungarian-German refugees. He lived in Germany for five years before moving to Detroit. He received a B.S. in Chemistry and M.S in Mathematics from the University of Michigan in 1968 and 1969, respectively. He worked briefly as chemist for IBM (1969-1971) while taking artificial intelligence and computer science courses at Syracuse University. He later applied to MIT, where he worked as a Research Assistant in the Artificial Intelligence Lab and completed a S.M. and Ph.D. in Electrical Engineering and Computer Science in 1975 and 1978, respectively. Following graduation, Hollerbach continued at the AI Lab as a Research Scientist of the Department of Psychology. He went on to serve as Assistant Professor (MIT: Department of Psychology, 1982-1985), Associate Professor (MIT: Department of Brain and Cognitive Sciences, 1985-1989), NSERC/CIAR Professor of Robotics (McGill University: Department of Mechanical Engineering and Biomedical Engineering, 1989-1994), and finally Professor of the School of Computing and Research Professor of Mechanical Engineering at the University of Utah (1994-present).
His research involves robotics, virtual environments, human motor control, and haptic interfaces. In 1984 he was awarded the Presidential Young Investigator Award from the National Science Foundation, and was named Fellow of the Canadian Institute for Advanced Research and IEEE Fellow in 1988 and 1996, respectively. He later received a Distinguished Service Award from the IEEE RAS in 2012.
In this interview, Hollerbach discusses his pursuit of, and career in, robotics. Outlining his career at the AI Lab, he describes the research environment, and the interactions and collaborations of fellow researchers. Reflecting on the evolution of robotics and his involvement in the Year of the Robot and other robotics projects, he later comments on the current state and challenges of robotics and its future development and applications.
About the Interview
JOHN HOLLERBACH: An Interview Conducted by Peter Asaro and Selma Sabanovic, IEEE History Center, 16 May 2012.
Interview #722 for Indiana University and IEEE History Center, The Institute of Electrical and Electronics Engineers Inc.
Copyright Statement
This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to Indiana University and to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.
Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, IEEE History Center, 445 Hoes Lane, Piscataway, NJ 08854 USA or ieee-history@ieee.org. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user. Inquiries concerning the original video recording should be sent to Professor Selma Sabanovic, selmas@indiana.edu.
It is recommended that this oral history be cited as follows:
John Hollerbach, an oral history conducted in 2012 by Peter Asaro and Selma Sabanovic, Indiana University, Bloomington Indiana, for Indiana University and the IEEE.
Interview
INTERVIEWEE: John Hollerbach
INTERVIEWER: Peter Asaro and Selma Sabanovic
DATE: 16 May 2012
PLACE: Utah, United States
Early Life and Education
Peter Asaro:
And so, what we're going to do is just kind of get your life story a little bit. So, we'll start just by asking where you were born and where you grew up.
John Hollerbach:
Right, well I was born in Germany, Marktheidenfeld a mine which is near Frankfurt. I'm the first of the baby boomers you might say. My parents were refugees from Hungary, from Hungarian Germans, and they met in a displaced persons camp and got married there. And so, I was born there. We were very poor. We had to live in a priest's attic for five years and then we eventually moved to the U.S. as Hungarian refugees, actually, even though we're ethnically German. So, I went to college and eventually found my way to MIT and decided I wanted to pursue robotics.
Peter Asaro:
Where did you move to in the States?
John Hollerbach:
Moved to Detroit, moved to the Motor City because that's where the jobs were at the time.
Peter Asaro:
Yeah.
John Hollerbach:
Maybe still some jobs there. <laughs>
Peter Asaro:
And so, you knew when you went to undergrad that you wanted to do engineering.
John Hollerbach: I didn't until I started chemistry and at that time the chemistry field is pretty exciting now, all these novel materials and nano things and so on, but at that time it seemed pretty staid. It didn't look at all to be an attractive career path. You know, you had to work with Dow Chemical or something not very exciting and at that time computers were coming on. We're talking about big mainframes, they were still coming on, somehow that seemed much more exciting to me. And so, even though I got a degree in chemistry from the University of Michigan I started to transition over to Computer Science. So, I started taking computer science courses and stayed an extra year to get a Master's Degree. You couldn't get a Master's Degree in Computer Science at the time; it had to be in Mathematics. And so, I actually worked at IBM for a couple of years as a chemist, but I took a bunch of courses in artificial intelligence and computer science as part of an IBM program with Syracuse University. Then, I applied to MIT; I got in, eventually hooked up with Pat Winston in the Artificial Intelligence Laboratory. Like I said, my main interest was robotics. My Master's Degree was in Computer Vision and my Ph.D. was in robotics. Actually, it had a biological angle to what I'd do. I had to model human handwriting and I got a robot to write like my model says a human writes. My Ph.D. advisor was David Marr; you might've heard about him. In fact, I was his first student.
Peter Asaro:
Wow.
John Hollerbach:
People think of him as a computer vision person, but actually he was initially known for his theory of the cerebellum, which was a perceptron like model. If you're interested in neuro meds you might know about perceptrons and things like that. So, he had a perceptron model of the cerebellum and as you know the cerebellum is primarily involved in motor control. And so, I was interested in the acquisition of fine motor skills, and so that's why I studied handwriting and the robotics part was to implement the model. So, I remember I had to get special permission to have David Marr be my supervisor because he was not yet a faculty member at the time. He was a research scientist so I had to get special dispensation so I wasn't the first student who graduated from him; I was the second. Shimon Ullman was the first, but I was technically his first graduate student.
Peter Asaro:
What was he like to work with?
John Hollerbach:
Oh, fantastic. He's just so motivating. He just would get you so enthused.
Peter Asaro:
Great, and who did you work with for your Master's?
John Hollerbach:
Pat Winston was my Master's thesis supervisor. I did something on modeling solid objects, describing solid objects to <inaudible> Greek vases.
Peter Asaro:
What years?
John Hollerbach:
That was `72 to `75, was the Master's and then `75 to `78 was Ph.D.. I continued along as a member of the artificial intelligence lab, but I actually joined the Department of Psychology, which became the Department of Cognitive and Neuroscience because of my interest in motor control. So, I still was interested in theories of human movement control, but also how you might apply these models or adapt models from robotics to understand humans. So, that's been kind of a consistent theme throughout my career. In fact, these days I'm going a bit more back to them, the logical side because my interest is in medical robotics, so I'm pursuing that right now.
Career Experiences
Peter Asaro:
And so, at that time was there a lot of control theory being used to try to understand human motion control?
John Hollerbach:
Yeah, there is a lot of inappropriate control theory being applied, but I think robotics is starting to make an impact again because of the machine learning stuff. So, I would say the application of control theory is very limited to understanding humans because we're not built like typical control systems, whereas the machine learning kind of stuff seems much more appropriate. You know, you can argue what flavor of machine learning is appropriate, but it seems to be making an impact. And also, the robots are more complicated. You've got humanoid robots and the tasks that robots are trying to achieve are much more like human tasks and so I think a lot of fields have been invigorated by getting away from the assembly line, not just service robots, but robots in the home. So, humanoid robots, robots that you have to deal with human-like environments have to have human-like capabilities and they can't all be programmed to do everything. They have to learn themselves, so I think we're getting much more strong ties to the biological side, which were absent for awhile.
Peter Asaro:
And, at that time, were problems about vision and problems about motor control separated or were there people who were trying to integrate the two in a single problem solving?
John Hollerbach:
Initially yes, but for a while the vision community drifted away so there was a separation between the computer vision community and the robotics community. It mostly had to do with the funding sources. Robotics has never been that well-founded until maybe recently, whereas computer vision also includes medical image processing, and I think that computer graphics, computer animation, you know the entertainment field those had a lot more dollars to offer to people and so I think that's a major reason why computer vision drifted away from robotics, but now we're starting to see the reintegration and so there's a strong reintegration going on now.
Peter Asaro:
You mentioned funding, so over the arc of your career where has the bulk of your funding come from?
John Hollerbach:
Well, it started out being military funding. So, at the AI Lab we had the luxury of having big grants from DARPA, ONR, and allowed us to function with relative freedom. So, we were lucky to have that kind of funding available. Also, the System Development Foundation kicked in, like I said. But, it was feast and famine, a lot of famine, not a lot of support for the years. I mean NSF has been kind of a steady growth with blips here and there, this or that program, but we never had a lot of money in the States for robotics, not compared to Europe, say now. We still don't have a lot of money compared to Europe. I mean the National Robotics Initiative is like 21 million a year or something like that and in Europe they're talking about a billion dollar program in robotics, so the scale is different.
Peter Asaro:
In terms of the AI Lab itself, what portion of the research going on was robotics and who else there was working on robotics?
John Hollerbach:
Well, I think the AI Lab was basically run by the robotics and computer vision people. We didn't really trust the other people <laughs> and natural language processing what have you. We didn't trust them so we basically ran the place. The faculty members, early faculty members, Pat Winston wasn't really a robotics person. David Marr was the computer vision person. Berthold Horn was kind of a steady influence throughout there. Tomás Lozano-Pérez became a faculty member. I was briefly a faculty member there for awhile. Graduate students there went on to play prominent roles. In fact, Matthew Mason, Marc Raibert, and I all shared an office. Eric Grimson and I shared an apartment. So, it was a small group of graduate students who I think went on and became important contributors to the field. It's interesting.
Peter Asaro:
And then, so you were interactive with them pretty heavily, did you collaborate on projects as well?
John Hollerbach:
No, I think we mostly helped each other. The way MIT was then and maybe still is now is it was graduate students were pretty much self-directed. In fact, I had dinner with Mike Brady last year or at the time of the Symposium and he said the thing that most astounded him about MIT was that; that you could have killed off all the faculty and it wouldn't have made any difference. <laughs> So, David Marr was an inspiring figure, but the first time he saw my thesis was when I handed it to him. So, the students helped each other a lot, right, and there was the expectation that you would take charge.
Year of the Robot
Peter Asaro:
Can you tell me about the Year of the Robot?
John Hollerbach:
The Year of the Robot was made possible by the System Development Foundation and also the Office of Naval Research, so the 1970s was not a great time for robotics worldwide. It was not the discipline and the realm of applications that we have today. The rather narrow field, not many people, and not that serious of a scientific discipline. In fact, there was a period of time in the 1970s where DARPA wouldn't even let us use the name robots, robotics, in proposals or on office doors. So, it's had its ups and downs. And so, there's a feeling by people, not just at MIT, but around the world the top robotics people that we had to make this into a more serious discipline and things needed to be jumpstarted. And so, in 1981-82, we started something called the Year of the Robot Program, funded by these two sources, and the Year of the Robot Program had several goals. One was to start a journal, a serious scientific journal. There had not been a journal dedicated to robotics at the time and so the International Journal of Robotics Research was started in 1983. IEEE Journal of Robotics and Automation was started in 1984, so it came along shortly afterwards. A lot of the same people were involved so it's not like warring camps or anything. The second thing was to have a serious scientific conference and so we started the International Symposium of Robotics Research. Right now, it doesn't seem so special. You can go to a robotics conference every week sometimes two a week if you wanted, but back then there was nothing practically to speak of. There was something called the International Symposium of Industrial Robotics, which in the 1970s was the main large robotics conference worldwide. It became more and more applied and less interested in research so it became just industrial robotics, and so there was a feeling that there was no real venue to present scientific results in robotics. So, that's why IS, our National Symposium of Robotics Research was started, again by the same group of people who started IJRR. It was initially an annual conference, then became a biannual conference and I think people have lost track of the seminal role that conference played back then. Now, it doesn't seem so special, but people are always surprised, especially the younger people, when the history is described to them and they understand what role it played back then. They're appreciative. Other aspects of the Year of the Robot was to write a source book of robotics so we wrote something called "Robot Motion: Planning and Control." The authors were Michael Brady, Matthew Mason, Tomás Lozano-Pérez, Timothy Johnson, and myself, and we were all kind of section editors. And, other things we promised to do was to build a robot hand, so we built the Utah/MIT Hand. I was at that time the MIT part, and the Utah/MIT Hand, Stephen Jacobsen at Utah was really the mechanical engineer who built it, and because of our association that's how I eventually found myself up at Utah to work with him. Other aspects were a Visitor Program where we brought many well-known researchers to MIT for extended periods of time. We also brought in some graduate students, graduate student programs. In fact, one of the graduate students we brought in was Richard Voyles who's now the program manager at NSF in Robotics. He was one of our student visitors back then. So, I would have to say we were successful. You know, we did kick start robotics and make it a serious discipline having its own journals, conferences, serious directions.
Peter Asaro:
Why did you choose to do a hand and what were some of the challenges?
John Hollerbach:
Well, the hand is always the epitome of dexterity and manipulation, which we have yet to still realize. It's still an open problem, but that's basically it. I mean our hands are an important part of the advance of human intelligence. You know, it's not just solving chess problems. It's intelligently manipulating things with our hands and so there's a lot that goes into it. So, it's the ultimate control of movement.
Peter Asaro:
And, you also laid out some of the other major problems of robotics back then.
John Hollerbach:
Yeah, they haven't changed much. I mean it's easy to say what the problems are. It's much harder to solve them. You know, there are still a lot of problems like general object recognition capabilities. So, you have this thing called semantic perception. Well, what they mean is what is that object? And so, these robots appear in unrestricted environments like the home; they have to be able to recognize all kinds of cups and all kinds of situations, cluttered, uncluttered and so on. And so, a big part of the robot in the home would be how does it recognize classes of objects so it knows what to do with them. So, before you can manipulate them you have to figure out what it is that you want to do with them, what they are, what you want to do with them. So, that's why semantic perception is such a hot topic again.
Peter Asaro:
In terms of theories of that kind of representation, did you find the ecological data of perception useful or were you more following the analytic composition?
John Hollerbach:
I was doing analytic composition, but I can appreciate some of the ecological approaches. So, it's more of a functional perspective. So, I think there's some validity to the ecological approaches.
McGill University
Peter Asaro:
And so, what year did you wind up at Utah?
John Hollerbach:
I ended up in Utah in 1994. I left in 1989. I was offered a Chair at McGill University and so I accepted the chair. It was called the NSERC/CIAR Industrial Chair in Robotics. So, I also had a Junior Chair that I could hire in to, so I hired Martin Buehler as my Junior Chair and the person I worked there with, mainly we formed the joint laboratory was Ian Hunter. Ian Hunter is now at MIT.
Peter Asaro:
What were some of the projects you worked on while you were at McGill?
John Hollerbach:
I continued to work on a lot of work until the operation, both the hands and arms. I started doing more fundamental actuator work because of Ian Hunter's interest. So, novel actuator materials, new motor designs, which continued to be an interest and are still a problem. I mean the whole design space for robotics is still a big problem. I mean computers have gotten faster, but motors haven't gotten much stronger or cheaper, but still a big limitation of robotics is the hardware. But, we might start seeing some advances there before long.
Peter Asaro:
What were some of the issues with the operation, were you also looking for the haptic end of it?
John Hollerbach:
Yeah, the haptic end of it is still a big problem primarily because designing good robot tactile senses continues to be a problem. So, all these humanoid robots, they're basically just walking machines. So, the ability to use their arms is very limited and the ability to use their hands is practically nonexistent, so they're basically just fancy walking machines. And so, the design of dexterous hands continues to be a problem, especially if you want to make them affordable. And still, we haven't come up with good contact sensing. So, it's my opinion that the lack of good contact sensing has been a key limitation in robot dexterity. That's why we don't have robot hands doing great things because they can't feel anything. So, that continues to be a problem and it's becoming in the forefront of humanoid robots because that lack is going to become more acute. So, you know, people are trying to do stuff to continue to try to develop tactile sensors and so I encourage that. Tactile sensing is not just on fingertips or gripper tips, but all over the body in terms of, if no other thing safety, because you need robots around humans and so that's an issue making them mechanically and control-wise friendly towards human interaction so they don't clobber us. And, their efforts to build reduced hands, hands which are not quite as complicated as the human hand, but capture significant parts of the functionality. So, there's some of that going on as well.
Peter Asaro:
Were you doing similar projects at Utah?
John Hollerbach:
Yeah, I was, I was. I've continued along that line and more recently focused a little bit more on medical robotics. So, the main project I'm working on now is working with spinal cord patients, people who have partial spinal cord injuries. We're gaining the ability to walk, but it's not a full ability, so they reach a plateau of walking ability and their walking is fragile. So, a gust of wind could knock them down. And so, we think the problem is that there's a gap in rehabilitation between what they get in the clinic versus the requirements of daily life. And so, we have a platform called the TreadPort, which is about as close as you get to realistic walking in a virtual environment and we think fills the gap. And so, we've done a pilot study with patients and they immediately walk better in our system. So, that's been quite encouraging because it's got them over the plateau and we think we can do a better job of preparing them for the real world.
Peter Asaro:
Who would've been some of your collaborators there?
John Hollerbach:
The collaborators are mostly the younger people you may not know of. You know, we're building up robotics at Utah so Jake Abbott, Mark Minor, Steve Mascaro, they're younger people.
Robotics and Automation Systems Community
Peter Asaro:
Great! And just to kind of jump back, so how did you become involved in robotics and automation systems?
John Hollerbach:
Well, it was a small community back then. You know, there were not that many people, now there are a lot of people so I remember ICRA 1984, in Atlanta. There were two parallel sessions, one in vision and one in manipulation, a couple of hundred participants. And so, I was involved, session chair and so on. And then, in 1985, we went to four or five parallel sessions. This was in St. Louis, and so in `86 went to San Francisco, went to six parallel sessions, so it started growing pretty fast. In 1989, I was Program Chair for ICRA, in Scottsdale and so I've had constant service throughout. So, you probably recognize that we all wear many different hats and we're part of this organization, part of that organization, you might think it's exclusive, but it's not.
Peter Asaro:
Can you tell me a bit more about the IJRR and setting up that journal and what it meant for the field?
John Hollerbach:
Yeah, I think it had a significant effect on the field because before then there was no journal dedicated to robotics. So, if you wanted to publish a paper in robotics it was in another journal, which had a broader scope than that. It certainly was not primarily robotics and so people would publish in IEEE Systems, Man, and Cybernetics or ASME Journal of Dynamic Systems, Measurement, and Controls, something which is not robotics. And so, IJRR was the first scientific journal in robotics. It came out in 1983. It was part of this Year of the Robot initiative and so the same people who started IJRR also started ISRR and were generally responsible for this big attempt to bootstrap robotics. And so, Lou Paul and Mike Brady were the inaugural editors. It was published by MIT Press and so Mike stayed on as editor until 1999. Lou Paul stepped off as co-editor in 1986. Tomás Lozano-Pérez briefly was an editor, but then Mike stayed on. In 1997, MIT Press sold IJRR to Sage Publishers. They were having financial difficulties and so they had to sell some of their assets. And so, Sage, in 1998, doubled the number of issues from six to 12 to clear a backlog. In 2000, I stepped in, Mike stepped down. I'm still the editor today. We increased the number to 14 issues per year and I'd say it's safe to say it's one of the two top journals in robotics along with TRO. We're overall friends and so I was just at an editors' lunch where we had Seth Hutchinson of TRO, Peter Corke of RAM, Robotics Automation Magazine, Ken Goldberg of Transactions on Automation Science and Engineering. Gaurav Sukhatme, Editor of Autonomous Robots, and Sanjiv Singh, Editor of Journal of Field Robotics; however, it's much more incestuous than you might think. Most of these people used to work for me and so you might say I trained them. So, Peter Corke, Seth Hutchinson, Sanjiv Singh, were all on my board and then left. But, it gets worse than that because my boss at Sage is Wendy Truran, and Seth Hutchinson married her last year <laughs>. So, you can see we're a close-knit community. We compete, but we're friendly and have community spirit.
Peter Asaro:
So, you think there was something about the early `80s in terms of something I'm interested in, which is interdisciplinary where people started to think I'm not just a control theorist or computer scientist or an electrical engineer, but I'm a roboticist or there is this field of robotics that's just...
John Hollerbach:
I think that feeling was always there early on. I think early on, so I think the discipline boundaries are sometimes imposed by academic rigor, but I think the view was always that robotics was its own field and you're starting to see that finally in terms of separate academic programs. So, at Utah, we have the second robotics program in the U.S. at the graduate level. CMU was the first and certainly the pacesetter in terms of the robotics institute. It's actually a department called the Robotics Institute, but it's actually a department. So, they have a graduate program in terms of required courses and so on. And so, you actually get a Master's or Ph.D. in robotics from CMU. At Utah, we don't have a degree in robotics per se, you stay with your discipline so you get your Ph.D. in Computer Science or Mechanical Engineering, but your program of study is robotics. So, basically we act as a virtual department. So, you might say it's an overlay under formal departmental structure where we do our own admissions, their own fund raising. Anyone can supervise a student from either department, so we do our own curriculum of study. We offer our own courses, so it's a way of getting around academic rigorous boundaries of departments.
Mentees and Collaborations
Peter Asaro:
So, you mentioned a few of the people that you mentored, but maybe you want to expand that list. Are there other people who are very active in robotics who have been students of yours?
John Hollerbach:
Yeah, Ronald Fearing would be one.
Peter Asaro:
Any that you want to <inaudible>.
John Hollerbach:
Well, I was the, you might say shadow advisor for some people, Christopher Atkeson.
Peter Asaro:
We interviewed him too. You're just jumping in, do you have some questions you want to ask him.
Selma Sabanovic:
I’m going to follow because I don’t know what you already talked about. Did you talk about the Sarcos Arm? Did you talk about the Utah/MIT – the –
John Hollerbach:
Utah/MIT Hand. Yeah. Well the Sarcos Arm, I didn’t play a role in. But yeah. The Utah/MIT Hand was part of the Year of the Robot initiative. And I was explaining that dexterity problem is still a huge problem for robotics.
Peter Asaro:
I guess in terms of other people that you’ve collaborated with over the years, who stands out? Who are some of the stronger collaborations that you’ve had?
John Hollerbach:
Well, with Ian Hunter. You’ll be hearing a lot more of him in the future. He’s at MIT.
Peter Asaro:
And this was the actuator work?
John Hollerbach:
Yes, actuator work which he continues these days. But that really is still a limitation to robotics. The computers have gotten much more powerful. And that’s been really the main reason robotics has progressed I think is better computer power. Of course people develop better theories and so on. But the computers enable you to do more and more, especially the machine-learning stuff which is very compute intensive, and the computer vision stuff, which is very compute intensive. But the hardware’s still a big limitation. So, we have all the limitations that other people have in terms of batteries and electric motors. And so, we need the same revolution everyone else does in terms of improvements to make better robots. Otherwise, they’ll never be like us. They’ll never have as much power in such a compact package and such dexterity and sensing and so on. I mean humans are just way ahead. But I think progress is coming maybe faster than the people think.
Selma Sabanovic:
And so, what is the – I don’t know if you talked about Ian Hunter.
John Hollerbach:
Yeah.
Selma Sabanovic:
Oh, you did. Okay. So, I’m not going to go into what is the kind of stuff that he’s developing that you think is going to push –
John Hollerbach:
I can’t say because I – he started a spin-off company after I signed a non-disclosure agreement. But it’s going to be significant. And it’s going to be soon <laughs>.
Selma Sabanovic:
You’re not the only one who’s given us this spiel.
John Hollerbach:
I would buy stock if he had any stock to offer.
Selma Sabanovic:
Did you guys talk about DB?
John Hollerbach:
DB I didn’t have anything to do with.
Selma Sabanovic:
Okay. Oh, I thought I saw that on your –
John Hollerbach:
No.
Selma Sabanovic:
Website.
John Hollerbach:
DB.
Selma Sabanovic:
The Dynamic Brain.
John Hollerbach:
No, never had anything to do with that. That’s Kawato’s stuff right?
Selma Sabanovic:
Right, right.
John Hollerbach:
No, I didn’t have anything to do with that.
Selma Sabanovic:
But it was develop – I know that it was built by Sarcos.
John Hollerbach:
Oh, sure. That’s just one of his humanoid robots.
Selma Sabanovic:
Right.
John Hollerbach:
Right. So, Steve did build a bunch of different kinds of humanoid robots initially for Disney, but later on for other places, particularly amusement, casinos.
Psychological Influence
Selma Sabanovic:
One thing that I was curious about in your CV, and if you’ve covered this already, I’ll just watch the tape, but was the fact that you were in the psychology department for a while.
John Hollerbach:
Yes, I was explaining that. I always had an interest in human skill acquisition. And I was explaining that I was David Marr’s first graduate student. And people think of him as a computer vision person, but he was actually known for theory of the cerebellum. And you probably know that cerebellum is involved with our rhetoric skill set, not the only center for sure, but a major center in our skill set. So, I was very interested in that theory from the standpoint of human skills. And so, I did my Ph.D. in a model of human handwriting, which was implemented on a robot. So, I used to have the robot write good luck. And then I would give them out to people as a stroke of good luck <laughs>.
Selma Sabanovic:
That’s sweet. Were you motivated at all by the – so, the early automata, they all had these – some of them were writing?
John Hollerbach:
No. Not at all, not at all. It’s just – it was a complex motor skill.
Peter Asaro:
And do you see that biological models and neurological models still have a lot to teach robotics?
John Hollerbach:
It remains to be seen. I think the connection is less strong than maybe I initially did. Just the hardware is so different, so to speak, huge differences in hardware. I mean computers are not at all like our brains. And motors are not at all like our muscles. And there’s nothing like human contact sensing. So, the structures are just so different that you have to wonder to what extent they operate by the same principles, maybe high level principles yes. We talked about machine-learning. But in terms of the details, I’m less certain about that. There might be more than one way to implement intelligence. Yeah. So, people have tried for a long time. And I did. That’s why I was back in the psychology department back then. But I think if you directly attack the problem from that standpoint, you’re likely to be disappointed these days. You’re better off just doing one or the other, either studying humans or building better robots, but – no, not necessarily building better robots based upon humans. I should make caveats there because you can do a lot with say gecco kind of robots or you know insect flying robots. So, I think there can be things learned there. But I also think it’s a bit limited.
Peter Asaro:
What’s up with the stuff, say, visual flow?
John Hollerbach:
Sure, that’s been going on for a long time.
Advice to Young People
Peter Asaro:
In terms of what you’d recommend to young people who would be interested in a career in robotics, what kind of advice would you –?
John Hollerbach:
Well, it’s much easier now than ever. I would encourage it because I think the field is just exploding. For me, it’s very encouraging because I’ve been the dark ages of robotics before. I’ve been in situations where I told you we couldn’t even use the word robotics at MIT. We were not allowed to use the word robotics by our sponsors. And I’ve seen booms and busts. I’ve seen people over promising. Then you get the poisoned water hole syndrome where you over promise, and so you can’t get any money because people are disappointed with you. They’re not about to give you any more money. But I think now we’re serious. Things are really working from a multiple standpoint. So, I think there’s a general feeling that finally we’re here. Things are really working now. And applications are really appearing. And they make economic sense. And there’s a need for them. So, it’s just exploding. It’s great. I think it’s a great time to be in robotics, so many spin-off companies happening. It’s really encouraging.
Future Applications
Selma Sabanovic:
What are the some of the future directions that you see robotics going in, particularly perhaps the bio-inspired, or cognitive robotics?
John Hollerbach:
Well, I view the medical area as being a huge application area. That’s the one I’m gravitating more. I mean I can talk about other areas, but I don’t necessarily work in those areas. But I think the medical area is a big one. And there’s lots of reasons for it, better care, cheaper care, aging population, less people to help out. So, there’s all kinds of reasons for it. I was describing some of the work I’m doing in rehabilitation robotics. So, I think there’s a lot of opportunity to make people’s lives better as we age. And so, I think that’s a huge application area.
Peter Asaro:
Have you looked at prosthetics in addition to the rehabilitation?
John Hollerbach:
In a way. So, for spinal cord patients, there are people who work on these powered exoskeletons and so on. But I’m not so much a fan of that anymore because I think there’s a better possibility to make the spinal cords work and have people use their own muscles rather than an external device. So, there’ve been some recent advances which are very significant in terms of getting people with spinal cord breakage to walk. There’s something called epidural spinal stimulation, which is an emerging technique, which is really, really promising. And so, instead of putting electrodes in the muscle nerves, which some people do, or even worse, outside – outside doesn’t really work. You have to actually get it at the nerves inside or embed electrodes in the spinal cord. But the epidural spinal stimulation does none of that. What it does is it has an electrode array outside the dura. The dura is the sheath around that protects the spinal cord. You also have dura in the brain, right? There’s fluid inside. So, it turns out that, a lot of the – our spinal cords have a lot of coordination circuitry in them. It’s not like it’s just all cortical. It’s actually very little cortical, even maybe very little in terms of the cerebellum. But our spinal cord has tremendous capabilities built into them probably due to evolution. And so, the circuitry still exists there for walking, basically. And all that’s been done is you’ve lost your command centers to initiate the walking. And so, what the epidural spinal stimulation does is kind of substitute for the command centers and say, “Start walking,” or “This is your stand up command.” But it has all the coordination there to do that. And you don’t need detailed muscle stimulation because you already have that. And so, this is very promising because it does more than just that. It also seems to improve autonomic functions, so incontinence. And it even improves sexual function. So, this is tremendous. You might still need robotic training to get you to the point of being more independent, maybe even need some kind of lightweight robotic brace or pacemaker or something, but not these heavy robotic things. I think that’s looking less and less attractive.
Selma Sabanovic:
Well, it seems that in a lot of robotic application areas, it’s not so much the robotics that’s being forwarded, rather robotic technology kind of incorporated into existing –
John Hollerbach:
Well, that’s exactly right. So, this is kind of the notion of ubiquitous robotics or pervasive robotics. So, I think – I view robotics as being a kind of core discipline. I’ve heard someone describe roboticists as universal engineers. And I actually subscribe to that notion. I think insofar as anyone can be considered a universal engineer, it’s us because we do so many different things. To make a robot work just covers all aspects of technology. And I think relative to other people, robotics people are fairly fearless about tackling problems. They’re not easily – they don’t say it’s not my discipline. They just go out and do it. If they haven’t done it before, they’ll learn about it. And so, I think it’s really great about the field. That also explains why we’re expanding so rapidly. You’ll find people dabbling in this and this, and they have no relationship. And it’s all the same person. That’s very common in robotics.
Contributions and Challenges
Peter Asaro:
If you had to, how would you define robotics?
John Hollerbach:
It’s the study of movement. It’s realization and control.
Selma Sabanovic:
That’s always an interesting question.
John Hollerbach:
<laughs> No, see it also covers non-devices, right? It covers computer animation, which robots just play a role there. I still think a roboticist can help understand human motion.
Peter Asaro:
What do you consider to be your first robot that you built?
John Hollerbach:
Oh I built a tendon-driven shoulder, but before that, I used an early version of the PUMA robot do the handwriting, the so-called Vicarm. It was like a pint-sized version of the – of PUMA robots. We brought in Victor Scheinman around 1974, I think, something like that, to the AI lab. There was a competition between him and a guy named Carl Flatau to build a small robot arm for the AI lab. We liked Victor Scheinman’s design. So, we built the Vicarm, which eventually became the PUMA robot.
Selma Sabanovic:
How do you spell Flatau?
John Hollerbach:
F-L-A-T-A-U.
Selma Sabanovic:
Okay, that’s what I thought it was.
John Hollerbach:
I forgot his first name at the moment.
Peter Asaro:
Tell me about this tendon shoulder.
John Hollerbach:
Yeah it was again an attempt to mimic biology and learn by controlling a similar robot device. So, it had antagonistic, wrapping tendons. We tried to do the same with the Utah/MIT Hand. We tried to learn from the biological principles in the design of the hand. At the time, it was probably expensive for what people had envisaged. I just think people are getting back to that kind of design. Like at DLR, they're now building a hand in which many ways borrows from that old design.
Selma Sabanovic:
What were some of your challenges, both conceptual and technical, implementing the hand?
John Hollerbach:
Well, just the human tendons are incredible, just the low friction of the human tendon system. There's no real fluids and sheath, practically no friction. We tried to build a robot with that kind of design principle, and the joints just froze. Things didn't slide. They didn't move. And so, we were going to original do sliding surfaces. I mean there's still people thinking about it like Yoky Matsuoka. So, we were initially going to do sliding surfaces. You know, even in our human hands, we have tendons sliding over each other. We tried to do that, and we got a perfect break. And so, we couldn't duplicate biology. So, we ended up just using pulleys, so a complicated pulley system.
Selma Sabanovic:
So, it's kind of inspired by biology, but then solved by engineering.
John Hollerbach:
Solved by engineering because we gave up.
Peter Asaro:
Do you see the materials available now, things like Kevlar and stuff like that, as really changing?
John Hollerbach:
That's what we used back then. We used spectral fibers. And I don't think it's any better really. And we used pneumatic actuators back then. And fluidic actuators have a lot to recommend them. They have a lot to not recommend them. So, it would be better to have electric motors, but we all know what their problems are.
Selma Sabanovic:
So, have there been any improvements that would make some of this biologically inspired work a little bit more feasible?
John Hollerbach:
No. I think still you'd think with MEMS technology and so on, we'd get better tactile sensors. But I haven't seen them yet. Maybe they'regetting better. So, I was saying that I think the lack of good contact sensing has been a huge limitation for robotics and is going to be a huge limitation for having robots in the home because they've got to be able to feel if for no other reason than safety of people around them. So, robots have to be built in such a way as not to injure people, let alone to guide dexterity.
ISRR and Other Societies
Peter Asaro:
Just to jump back a bit, to go back to the first ISRR meeting, what was the mood like there? And what kinds of –
John Hollerbach:
Well, I actually missed the first one because I was in a conference in Australia. So, I ended up going to the second one. But I helped design the conference. So, it's based upon my recommendation that it followed the script of a Gordon Conference, so called Gordon Conference, which were biological conferences held mostly in the New Hampshire area. So, I had been going to a Gordon-like conference for a while. It's called the Engineering Foundation Conference in Biomechanics and Neural Control of Movement. And so, this was a very popular conference in the 1970's, which brought together engineers, roboticists, neuroscience people to – so we were discussing about this. So, people have been trying to do this for a long time, to apply engineering models to understand biology, and vice versa to have biology inspire engineering. So, that's been going on for a very long time, almost from day one I would say. So, in that sense there's nothing new under the sun in terms of people's motivations. They always had biology as an initial motivation. So, those conferences were structured so that you would get together in a resort for four days. So, there was no distractions around. You would have morning sessions. Then you would have the afternoon off with some sponsored activity. Then you would have a late afternoon session, then an evening session with lots of drinking. So, that was the idea. And the sessions were- the activities were structured to be meaningful. For example, when I ran one, I think in 1979, I was one of the co-chairs. I brought in one of the coaches for the U.S. swimming team who did some of the scientific studies of swimming propulsion. And so, we all went to a lake. And people would swim by, and he'd correct their technique. That was kind of fun. So, it was inspired after that. So, the first conference was like that. It’s tended to get more and more business like I’m sorry to say. But it’s still has the tradition of having an excursion. It still has the tradition of being at a resort. It still has the tradition of full attendance the whole time. You just don’t pop in and out. You’re expected to stay. Otherwise, you’ll never be invited back. And it also has a strong tradition of discussion. So, I think it still has a role. It’s not just presenting your most recent results. And typically, the results are not presented by graduate students. They’re presented by the senior people in the field because we want there to be a serious discussion going on. Not that you can’t with graduate students, but this is the kind of interaction we wanted. So, it’s still the spirit of ISRR today, which perhaps still makes it distinct. And that’s discussing that it’s still the granddaddy conference in robotics in terms of its impact. And now, it doesn’t seem so special because we have so many conferences. It seems like we have too much. Why have ISRR? But whenever I describe people what the history is, what role it’s played and what role it might still play, they’re very appreciative. The young people particularly are very appreciative and want it to continue.
Selma Sabanovic:
And how do you get invited to ISRR because I’ve only had a chance to go to the one in Flagstaff? And I noticed that there were a lot of people who have been there obviously from the beginning.
John Hollerbach:
There’s a board. So, the board –
Selma Sabanovic:
And then there were some new, including grad students.
John Hollerbach:
Yeah, sure. So, we – half of the talks are invited. And half the talks are open submissions. And so, we do open submission possibility so we’re not just a closed boy’s network. But there’s a geographical group of what is it – I think like four people from the three major geographical areas of the world who are members of the board and who do the invitations. So, they identify people from their regions who they think are doing significant work that has not been presented there before. So, you saw the results there. So, it’s partly an old boy’s club, but I think in the good way because it’s good to have the senior people there listening to your paper if you’re a younger person. And we’re also more reflective. We have the blue sky sessions where we take a longer term view of what’s going on, what’s important, and what we should be doing.
Selma Sabanovic:
And aside from being at the conference, what kind of impacts do you see with ISRR?
John Hollerbach:
Not enough, so that’s actually a big discussion we had at board meeting two days ago about we’re not having the impact outside the conference. Sure when people are there, they have a great time. They’re getting a lot out of it. But what can we do to have more of an impact. And if we don’t, should we continue or not? So, that’s under discussion right now.
Selma Sabanovic:
Are there any comments about that that you could share with us, just what are parts of the discussion? Where would you like to have impact, perhaps?
John Hollerbach:
Well, I think we have to bring in more people. So, personally, I think we should double the size of the board and bring in a lot more people. A big problem that we have is geographical representation, especially in Asia because of the growth in countries other than Japan. And right now, we’re very Japanese oriented in terms of our board membership. And so, I think that has to be greatly expanded through the Koreans, and one or more subsets of the Chinese. We already have an Australian there, but I think we need to greatly expand the people involved is my opinion. But keep this core philosophy. I think this core philosophy is an important one.
Selma Sabanovic:
Are there other important either conferences or boards or just kind of groups, societies, that you’ve been part of that you think are both important to you in terms of your work, but also in terms of in guiding where the community goes?
John Hollerbach:
Well, the main would be RSS, Robotic and Science Systems. You might say that they’re the younger community’s version of ISRR. And you might say it was almost a reaction against the way that ISRR sometimes has been perceived to operate. So, yeah, I think that would be the main one that would be the trendsetters. Probably the second most important conference besides ICRA, I would say. So, they’re very much independent of IEEE, which I think is good. I think it’s good to have competing alternative organizations. It keeps people’s feet – it keeps people a bit honest I think if there are alternatives. And so, that’s a good one. But what’s really happening with the field is that you might say it’s fractioning just because of significant developments in multiple fields. So, you’re getting more and more special conferences, humanoid robots, robot interaction, algorithmic foundations of robots, computer vision, haptics, medical robotics. So, what’s happening is the field has become so successful and so big that it makes sense to subdivide into these smaller groups. And the challenge is to keep them within one umbrella. And so, this gets a little bit in society business here because I was, until this year, the vice president for technical activities. And so, my proposal was basically to build up these kind of mini-societies within the society, but to stay together, not to fractionate into distinct communities. Make it so that we can all work together within the same society, basically by decentralizing. That’s my idea.
Selma Sabanovic:
Do you think this kind of fractioning poses a problem in terms of robotics being something that’s very interdisciplinary?
John Hollerbach:
I think so. I think it would defeat it. I think being together on the same umbrella, or the big tent model, makes it more likely that we’ll have interdisciplinary interactions. So, I think there would be a danger of narrower visions if we split into multiple societies.
Selma Sabanovic:
Yeah because you mentioned RSS, and I know last year, they were trying to get more HRI work in. I’m just familiar with that because I’ve done HRI stuff. But I also know that HRI people feel like the technical community doesn’t really consider – doesn’t know necessarily how to read and evaluate HRI work, so I can’t get in. So, I’m just wondering if there are other parts of –
John Hollerbach:
We don’t trust them <laughs>. I think the way to look at it is the field is in a young stage.
Selma Sabanovic:
Oh, definitely.
John Hollerbach:
So, the soft and fuzzy stuff, some people can’t handle that very well. So, we’ll see. I mean everyone recognizes the importance of HRI, but when is a paper ready to be a journal paper in IJRR? Well, probably not yet. Maybe it has to evolve a bit more. But you know there’s now a new HRI journal out. One was just created.
Reflection and Final Remarks
Peter Asaro:
What do you consider your biggest contribution to the field of robotics? What is your most – short list of contributions?
John Hollerbach:
I think I played a role in getting the Year of the Robot working. So, I was in charge of Utah/MIT Hand work. I was the one who brought out the robot motion book. I was the one who primarily made that thing work. I made the suggestion about the ISRR. And I played a lot of background work. And now, I’m the editor of IJRR. So, the Year of the Robot stuff continued today, I think has been an important contribution. Certainly, I’ve made technical contributions in particularly robot calibration. And as you get older, I think you have to play a more supportive role. I mean I still do my technical stuff. I’m very excited about my rehabilitation robotics. So, you always have to keep sharp technically, I feel. So, you have to be doing something which is technical, not just be a robot administrator is my feeling. But I think, as we get older, we’re expected to help other people more, to take leadership roles, to nurture the field. And I’m growing a robotics group at Utah. We’re about ten people now. So, I’ve been growing that up, starting this robotics track program of study. I raise a lot of money for people, got an IGERT grant. For the society, I think I’ve played a significant role. I started the conference editorial board instead of having independent PCs. This new vision of technical committees as kind of these mini-societies which cover the intellectual landscape of the field is my idea. So, I think I’ve adapted to being an elder statesman, while not being yet an old fogey.
Selma Sabanovic:
Something to look forward to.
<laughter>
Selma Sabanovic:
Do we –?
Peter Asaro:
I think we’ve covered everything. Is there anything you feel like we’ve missed?
Selma Sabanovic:
Or I don’t know if you have any – you mentioned how some of this failed with your tendon mimicking. I don’t know if you’ve had any other kind of interesting failures where you learned stuff that you’d like to tell us about. Everybody likes to talk about successes, but there’s sometimes really important things that we learn from things that don’t work. So, I’m just wondering if you have some other that didn’t work, or it exploded in our face.
John Hollerbach:
Well, that was the main one. We couldn’t build a human hand – couldn’t build a robot hand like a human hand. We could not. We had to approach with what engineering design capabilities were available. So, it’s still an issue today.
Selma Sabanovic:
Anything else we missed that you’d want included?
John Hollerbach:
No, I think I’m fine.
Peter Asaro:
All right, thanks.
Selma Sabanovic:
Thank you.
Peter Asaro:
Thank you.
John Hollerbach:
Okay, sure.
