Oral-History:Robert Riener
About Robert Riener
Robert Riener was born and raised in Munich, Germany. He studied Mechanical Engineering at TU München and the University of Maryland from 1988 to 1993. He received the Dipl.-Ing. degree and Dr. degree from TU München in 1993 and 1997, respectively. In 1993, he also joined the Institute of Automatic Control Engineering, which sparked his interest in neuro-prosthetics. From 1998 to 1999, he performed post-doctoral work at the Centro di Bioingegneria, Politecnico de Milano before returning to TU München to finish his Habilitation in Biomechatronics and acquiring a professorship at Zurich in 2003. As of 2010, he has served as Full Professor for Sensory-Motor Systems at the Department of Health Sciences and Technology and Full Professor of Rehabilitation Engineering at ETH Zurich. Riener is also actively involved with the Spinal Cord Injury Center at Balgrist University Hospital.
Riener's research interests include human motor synthesis, biomechanics, virtual reality, man-machine interactions, and rehabilitation robotics and prosthetics. For his work he has received several awards and honors, including the humanTech Innovation Prize, the Swiss Technology Award, and the euRobotics Technology Transfer Awards in 2011 and 2012.
In this interview, Robert Riener discusses his work in robotics, with a focus on simulation and control, medical applications, and prosthetics. Outlining the social challenges which sparked his interest, as well as the challenges of his research, he describes the relationship between robotics and prosthetics and its potential applications for societal advancement.
About the Interview
ROBERT RIENER: An Interview Conducted by Peter Asaro, IEEE History Center, 15 July 2011.
Interview #743 for Indiana University and IEEE History Center, The Institute of Electrical and Electronics Engineers Inc.
Copyright Statement
This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to Indiana University and to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.
Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, IEEE History Center, 445 Hoes Lane, Piscataway, NJ 08854 USA or ieee-history@ieee.org. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user. Inquiries concerning the original video recording should be sent to Professor Selma Sabanovic, selmas@indiana.edu.
It is recommended that this oral history be cited as follows:
Robert Riener, an oral history conducted in 2011 by Peter Asaro, Indiana University, Bloomington Indiana, for Indiana University and the IEEE.
Interview
INTERVIEWEE: Robert Riener
INTERVIEWER: Peter Asaro
DATE: 15 July 2011
PLACE: Zurich, Switzerland
Early Life and Education
Peter Asaro:
So if you could just start by telling us where you were born and where you grew up and went to school.
Robert Riener:
Yes, I was born in Munich, Germany, where I also grew up. And I studied in Munich and did my Ph.D. in Munich, and then I went to Milan to the polytechnic to do my postdoc there. And then I came back to Munich where I did research and teaching. And 2003 I got my professorship in Zurich.
Peter Asaro:
What did you study as an undergraduate?
Robert Riener:
Mechanical engineering.
Peter Asaro:
And then for your graduate work?
Robert Riener:
That was done in electrical engineering, but the area was biomedical engineering and rehabilitation engineering. So it’s an engineering degree, general.
Peter Asaro:
And who did you study with for your graduate work?
Robert Riener:
That I studied first with some clinicians. One of them was Jochen Quintern. He was my mentor in the area of neuroprosthetics and functional electrical stimulation. Later I got additional collaborators in the area of orthopedics and I worked together with Dr. Rainer Burgkart in Munich.
Interest in Medical Applications
Peter Asaro:
What stimulated your interest in medical applications?
Robert Riener:
My interest was quite broad in research already when I was a boy and there was my dreams for jobs – I was switching between technical researcher or physician. And then when I grew up I wanted to study medicine, but then I had to first go to military and there was no – oh, you have to cut this off.
<laughter>
Robert Riener:
Because actually I was planning to go to medicine and I did not have to go to military. So – but I did not have the right test results so I could not study medicine because I needed to have more time for this, which I did not have because I did not go to military. Anyways, what I needed – but I decided it was mechanical engineering and later, then in my postdoctoral work – or on my doctoral work as a Ph.D. student I got into touch then in the area of medicine, which was the dream area as a boy.
Peter Asaro:
And why prosthetics and not say…like other kinds of biomedical instruments?
Robert Riener:
Many things – the exact details, they came by accident. So, on the one-hand side, I was interested in machines. My father is a technician for cars and I was interested in medicine and was just coincidence that there was an open position in the area of neuroprosthetics. So that’s – yeah, prosthetic developments, electrical stimulation applied to the human body. Here we also need to understand the mechanics of human body in order to control human movement. And so it fitted to my interests of mechanics.
Peter Asaro:
And what year was that that you started working in neuroprosthetics?
Robert Riener:
That was in ’93.
Peter Asaro:
And what was the sort of the state-of-the-art – what were the outstanding problems that you were concerned with at that time?
Robert Riener:
At that time, the basic technology of stimulating electrical muscles was already well-developed in the States and we knew about maybe thirty years ago, but was still a problem to control movements, to record movements with sufficient accuracy and reliability and then also to use the signals in order to control movement. So we started to control single joint movements and later we controlled standing up, sitting down and then also walking and stair climbing, together with other Ph.D. students. And we could solve this from at least from a technical point of view. It works in a lab environment, but there was still no real breakthrough because the technology is still too complex for daily use. And then there was a breakthrough, I would say, in research, in control, but not in clinical applications or daily practice.
Peter Asaro:
And what was your thesis on?
Robert Riener:
That was my thesis topic: control – simulation and control of neuroprosthetics for standing up, sitting down and also walking.
Post-doc Work
Peter Asaro:
Okay, and why did you go down to Milan to do your postdoc and who did you work with?
Robert Riener:
In Milan was a great area, not only because of la dolce vita in Italy, but also because there was a famous group – there is a famous group who did motion analysis, biomechanics, one of the most famous groups in Europe and they also did work in the area of neuroprosthetics. So I could first apply my neuroprosthetics knowledge there and continue some of my work there and the other half of the time I did some new work on motion recording and this was some – I would say some pioneering work on recording of gait climbing…, stair climbing, ascending and descending stairs. And we made – we were the first ones to record the gait patterns at climbing at different inclinations. It’s quite well cited now, this work.
Peter Asaro:
I know it’s much more difficult than just walking on a flat plane.
Robert Riener:
It’s more difficult, maybe there are more challenges in the society because elderly or some people with movement deficiencies. For them it’s a challenge to climb stairs, but maybe not to walk in at the level ground. So it’s – there are more problems with this basic knowledge which we have now. This knowledge can help to design other technologies of other groups in prosthetics, even in robotics, humanoid robotics.
Robotics and Prosthetics
Peter Asaro:
And what would you say is the kind of interface between the prosthetics and robotics, and how has your work been received in robotics?
Robert Riener:
Maybe there are at least two interesting connections of robotics to prosthetics. One is that the movement of the human body – of a normal, healthy human body, this needs some knowledge about mechanics. And the simulation, for example, of this movement and control of this movement can be quite similar in both – in the human side, on the natural side, as well as on the technical side. So some robots try to copy also the movement of the human. Or if you want to understand the movement of the human, you have to simulate it like a robot. So that’s one issue. The second issue is that some of the prosthetic components, especially when we have mechanical prosthesis, like exo-prosthetics for the arm or for the legs, then these are also robotic components with maybe several degrees of freedom of movement. If your finger moves, you need several motors to drive these movements, you need maybe also to control these movements with using sensors and controllers. And that’s also then robotics.
Peter Asaro:
And have you actually engaged in building these kinds of robotic prosthetics or exoskeletons on your work?
Robert Riener:
Well, what we did, when I came to Zurich and first in Munich I – there’s a gap so far in the story. I mean – I started also some work on robotics, using robots as a haptic interface to simulate a virtual human body. And we used this haptic interface, for example, to simulate a human leg, a human knee joint, and this helped to train orthopedic physicians doing knee joint diagnosis, for example, to detect a ruptured anterior cruciate ligament. So this was with my second mentor, Dr. Burkhardt, in Munich. So here again, a lot of knowledge in robotics. We also were successful in transferring these results to industry. We made not only a knee joint simulator, but also a delivery simulator, a birth simulator, which enabled us to simulate the delivery, the extraction of a baby, with the forceps or vacuum cup. And this is very helpful for young physicians to gain some more experience without harming or endangering a living human body. And this delivery simulator is now being sold by a company and we continue research and development on this area as well. So here I gained that was still Munich ten years ago, again, some more experience in robotics. And here then I combined this: I applied robotics to the area of rehabilitation and I started in developing an exo-skeletal device for the rehabilitation of arm movements in stroke patients. Stroke patients usually have hemiplegia besides other neurological deficits, and the hemiplegia makes the paralysis of one body half. So they cannot move properly their one arm and this robotic device supports movement. It also supports the therapist, because it can be quite time-consuming and also exhausting for therapists to lift the arm all the time. This robot is then supporting this – but that’s not a prosthetic device. That’s more considered as an orthotic device. And later we also started a new project, leg joint prosthetics, where we developed an actuated prosthesis. That’s quite rare in general. Of course, there are research groups, but normally it’s hard to buy actuated prosthesis. Most prosthesis are just passive. But our prosthesis is active with motors and energy supply. But then the question is how do you have to control it to get a functional and safe movement? Because it should be in alignment with control of the human body. And the passive device, passive prosthesis, like prosthesis for example, it’s a ballistics and the control of the intact body, which determines the movement of the prosthesis. But if you add some actuation, this would somehow know what the body wants. And so that’s a challenge to make the right interface between the natural control of the human and the artificial control of the prosthesis.
Peter Asaro:
And what are some ways that you’ve addressed that problem of control?
Robert Riener:
Pardon me?
Peter Asaro:
How have you started to address that problem of control, like what are some of the techniques and strategies that you use?
Robert Riener:
Yes. Here we orient on the real clinical needs so that’s what the physicians and the therapists tell us is very important. And so for example, in orthotic devices, which we use to support movement training, it’s a device in the clinics. They’re applied in the clinics for stationary, but also ambulant patients. Patients come, train once per day, maybe eight weeks long, and then they go home and they may have an improvement. And for these applications it’s important that we take into account what the therapists tell us. And here, for example, they told us that it’s important that the patients can get very fast into the devices to save time and to make it less exhausting also for the patients. So we – also, the mechanical interface is one issue, for example. Another important issue is that the devices do not force the patients to move according to a special trajectory or movement. So it’s important that the patients have enough freedom and that they initiate the movement and try to make the movement as much as they can, and the robotic device should just assist as much as needed. So we made special controllers, which are kind of, yeah, interactive or maybe even defensive, but they’re still also active because otherwise the patient would not move. The force of the patient, the voluntary force is too little. They still need some force otherwise the therapy maybe not – may not be sensible, but the force they have is too little. And controllers are trying to detect this little force and then support the patient just as much as needed.
Peter Asaro:
And have you subsequently worked on other kinds of neural interfaces for this or is it primarily pressure and force actuated?
Robert Riener:
Yes. Our interfaces are mainly based on force recordings and movement recordings. We – in most projects we do not need neural recordings like we get them from the brain, because the intuition of the patients – so the neural – the voluntary will of the patient, which is a neural signal, of course, is also expressed in the force which they can produce. It’s just a little force, but it’s one way how the neutral activity is represented. Not only by neural signals in the brain, it’s also going to the muscles, muscles are contracting and some little movement is starting and we are recording this little movement. However, in other projects we try to record also other things, for example, the motivation of the patient, if they’re engaged, if they’re concentrated or if they’re bored or if they’re over-challenged. And for this, to detect these activities, we need also to record some more physiological signals. For example, we’re recording heart rate and heart rate variability because both are signs for mental and physical stress. We’re also recording respiration frequency, skin temperature and galvanic skin conductance in order to figure out how much are the patients mentally involved and then adjust the movement of task to the mental load of the patient and physical load of the patient. And in another project we also try to detect intention, not by brain signals, but by signals of the autonomous nervous system, beta nervous system. So for probably – if you do a shoulder movement, some negative signals like the heart rate or respiration rate is correlated to the movement. So probably if you grasp a glass or something the respiration is somehow adjusted. It’s very clear if you have to carry a heavy object and then for sure your breathing is changing. But probably also for normal movements. And such signals can help us also to detect intentions, which we need to detect in order to drive a prosthesis, for example, or training or arm training. And this helps us to detect what the patient wants to do without needing to detect brain signals.
Peter Asaro:
And how do the patients react to these kinds of systems? What have been your experiences with their reactions?
Robert Riener:
Yes. Of course, maybe the patients we work with are not the clear representation of all patients, because people in Switzerland, they get to know about our work due to the media or maybe the Internet and only those are coming who are really motivated. But it’s quite many. Maybe it’s already fifty percent of all patients have an interest to come. So they’re very motivated ‘cause many of them are also desperate and they’re searching for help. And then we explain them the things and show them things and then most of them are still motivated. Some are maybe – are a little bit surprised that we cannot support as much help as they expect maybe. Or some, but not many, for them it’s too cumbersome to get here because they have to one hour in travel time with a car. So it can be maybe too exhausting for them. But most, maybe eighty percent of those who come the first time are then coming more often and then they are excited. Because, first, they have – these are fancy devices. We apply special – not only movement, but also tasks, which are represented by an audiovisual display by virtual reality scenarios. So we are somehow provoking their game instinct and they like – they start to like playing these games. Also, these are old subjects, mainly. Stroke patients are elderly, older subjects. They really love it. They start to play like little schoolboys.
Peter Asaro:
So what kinds of tasks do you have them perform within those prosthetics? What are they trying to do <inaudible>
Robert Riener:
Yes. In the training task with the orthotic devices, there are different tasks. For example, for the upper extremity movement with our so-called arming robot, we do first some games, whether you see, for example, a ball and the task is to catch the ball before it’s falling down. And they’re moving a bar by moving their hand, like they’re moving a joystick and then they can catch this ball. Then we also – first, it’s important that the patient, that there’s a transfer to daily life so that they not only increase strength and range of motion, but also learn special movements. Important movements for daily life. We call these activities of daily living. And they are then training these activities which are programmed in that virtual reality so they can virtually cut vegetables or meat, virtually put them into a sauce pan, switch on the oven virtually, prepare the table, virtually eat, clean the table, go to the ticket machine, buy a virtual ticket to take a tram and so on. And so all these tasks are virtually represented so that we have a quite large variety of different movement. And it’s also enjoyable for the patients. So that’s upper extremity training. In the lower extremity training people are able to walk through virtual landscapes. So we had a European project where people can walk over canyon. If you detected the patients are still de-motivated or even bored then we can make it more exciting by making the canyon deeper or making the bridge smaller or removing slowly the handrails, changing the weather, producing the thunder and lightning to wake up the patients. So we have difference enough, which we adjust to individual patients to always motivate them, engage them, but not over-stress them, of course.
Birth Simulator
Peter Asaro:
What were some of the challenges of building the birth simulator?
Robert Riener:
Yes, one challenge was that, for example, we needed to know how large are the forces you need in order to extract the baby when you use the forceps. And the head is still large in relation to the pelvis, then the birth process can be accelerated and there’s a danger that too little oxygen goes to the baby’s brain and they’re gonna get injured. And in this case, forceps or vacuum cup has to be used. And there was actually no data about the forceps, which you need to get – to extract the baby, in the literature. So we had – yeah, to do our own measurements with experts to get to know these forces, for example, and it turned out that there’s a large gap in literature, which is maybe around <inaudible> towards and afterwards this topic was kind of not popular. It’s maybe too emotional, too indiscreet. And so part of the data was from the last – you know, from 1900 this time, got some force data. And other data we did get through our own measurements.
Peter Asaro:
And once you had that you were able to build a mechanism that would simulate those forces effectively?
Robert Riener:
Yes. We made a robotic device where we took the first version, the baby head connected to some actuators and force sensors. And now the force and the action of the trainee can be measured and the baby head, which is integrated into our pelvis and into our mother body phantom, the movement of the head is then simulated in that way as it would move in a real case, depending on the forces, which you apply onto the baby head.
Prosthetics in Zurich
Peter Asaro:
And when you first came to Zurich was there a lot of interest in prosthetics and the tradition here for doing that kind of work?
Robert Riener:
Yeah, definitely because here there was already some pioneering work done before on gait robotics and gait training, so, for example, the neurologist Volker Dietz did develop a lot in neuroplasticity, neurophysiology and gait – control of gait from a medical point of view. Then later Gery Colombo was here, an engineer, who then developed the first robotic device called the Lokomat. That’s together with Dietz and this was a start, maybe around ’96 or so, with robotics. And I came then 2003 and then together with these people we increased the size in the research in this area. So in the meantime there’s another professor, we have together about fifty people, staff, working in this area only from robotic side, including prosthetics, virtual reality sensing, learning of movements also in sports.
Peter Asaro:
And you are actively applying that, results of your research, to the development of new locomotive robotic systems?
Robert Riener:
In principle we are transferring the results of research into industrial applications. At the moment concerning the gait, only with this company, Hocoma, who is – actually, this guy I mentioned, Gery Colombo, who developed the first gait robotic device, Lokomat, founded a company, Hocoma, and it’s very, very successful company. And with them we developed now new Lokomat, for example, and new features. We could develop also many new features, many new controllers. Also, virtual reality we develop things together. So these things are then transferred into products and therefore also clinical applications.
Peter Asaro:
What would be the kinds of customers and applications for Lokomat, this new version?
Robert Riener:
Say again?
Peter Asaro:
Who would be the customers and what would be the applications?
Robert Riener:
Yeah. The customers are usually clinics, research clinics, but also general rehabilitation clinics, private clinics who have a little bit more money because the devices are not so cheap. Even private persons who want to do some training at home really in the States. There’s some private patients. But it’s mainly clinical.
Future Applications and Challenges
Peter Asaro:
Yeah. And what do you see as a big future applications and future challenges to prosthetics and rehabilitation robotics?
Robert Riener:
Yes, I think important challenges are to make mobile devices, which can be – which are portable, which can be taken at home, so that people can use them at home, still for rehabilitation on the one-hand side, to improve their state, but also for assistance. If the state cannot be improved anymore, then these devices can also support the daily life. But it’s a challenge because to have a real functional benefit most of the devices need some actuation and if they’re mobile on the street for walking functions for example, then it’s a challenge to make them light-weight and also to get enough energy supply for all day long, for example. That’s also a problem with the prosthetic devices. So these are the challenges to make them portable, with energy supply and also from a control point of view to make them interactive and to detect what the user wants to do – patient intention detection and apply the right support depending on the patient’s intentions.
Advice to Young People
Peter Asaro:
And what would be the – I know you’re short on time. I’m trying to wrap up. So what would be your advice to young people who are interested in pursuing a career in robotics or in prosthetics and rehabilitation robotics?
Robert Riener:
A scientific career or career in a company?
Peter Asaro:
Either one.
Robert Riener:
Yes. Yeah, first of all, they should not look at the market chances, they should study what they want, what their intrinsic feeling tells them, what is motivating them, because then they will always find a niche or open a new niche. They should try to get – I mean I work not very introduced to this very field where I need engineering, some computer science, medicine, biology, movement, but it’s important that they have some basics, that they are an expert in a special basic area of engineering. They should not be broad at the beginning. They should get into depth and later enlarge the area by looking into other areas.
Collaborations and Students
Peter Asaro:
And who have been some of your graduate students and where have they gone on to work and what kind of things are they doing now?
Robert Riener:
Two of them got professorships already now. One, first in States, and then in Switzerland. And the other one now in Abu Dhabi, the new international university. And others got good positions at industry, rehabilitation industry or also other industries. But all of them do work in development or research in the industries.
Peter Asaro:
Great and any other collaborators or collaborations you want to talk about that you’ve had over the years?
Robert Riener:
It’s – yeah. We have a lot of collaborations through the clinical partners where we do our clinical trials, even multi-centric clinical trials we have through some companies, but we have mainly through this company, Hocoma, who makes the Lokomat and other devices. But also smaller companies for some detailed, for example, force sensor applications and developments. I have some contacts at KUKA, who’s a large robotic company in Germany for industrial robotics. Yes.
Peter Asaro:
Good.
Robert Riener:
Okay?
Final Remarks
Peter Asaro:
Do you need to go?
Robert Riener:
Yeah.
Peter Asaro:
Yeah.
Robert Riener:
I have to finish.
Peter Asaro:
Anything else you’d like to add?
Robert Riener:
No, I think I placed all scientific areas in this talk now.
Peter Asaro:
Good, okay.
Robert Riener:
Okay?
Peter Asaro:
Thank you very much.
Robert Riener:
Yeah.
