Oral-History:Chung-Chieh Jay Kuo
About Chung-Chieh Jay Kuo
Dr. C.-C. Jay Kuo was born in 1957. C.-C. stands for Chung-Chieh. He received his B.S. degree from the National Taiwan University, Taipei, in 1980 and the M.S. and Ph.D. degrees from the Massachusetts Institute of Technology, Cambridge, in 1985 and 1987, respectively, all in Electrical Engineering. From October 1987 to December 1988, he was Computational and Applied Mathematics Research Assistant Professor in the Department of Mathematics at the University of California, Los Angeles. He joined the University of Southern California (USC) in January 1989. He is Director of the Multimedia Communication Laboratory, Holder of the William H. Hogue Professorship in Electrical and Computer Engineering, and Distinguished Professor in Electrical and Computer Engineering and Computer Science at USC.
His research focuses on multimedia compression, coding, and processing. But since 2014, Kuo has begun expanding his research to explainable machine learning. Dr. Kuo received the IEEE Computer Society Edward J. McCluskey Technical Achievement Award in 2019 for his “outstanding contributions to multimedia computing technologies and their applications.” He is the recipient of the IEEE Computer Society Taylor L. Booth Education Award in 2016 for his “excellence as an inspiring educator and for distinguished contributions to multimedia education with impact on academic and industry realms.” He has graduated 156 PhD students as of January 2021. He has also received the Pan Wen-Yuan Outstanding Research Award from the Pan Wen-Yuan Foundation in 2011, the IEEE Circuits and Systems Society John Choma Education Award in 2016, the IEEE Signal Processing Society Claude Shannon-Harry Nyquist Technical Achievement Award in 2019, the IEEE Computer Society TCMC Impact Award in 2020, and the 72nd annual Technology and Engineering Emmy Award in 2021, among many other awards. Dr. Kuo is a Fellow of NAI, IEEE, AAAS and SPIE. In this interview, Dr. Kuo talked about his education in Taiwan, his time at MIT, his research on multimedia compression and deep learning at USC, and his teaching philosophy.
About the Interview
An Interview Conducted by Honghong Tinn, Sep 21, 2020, via a Zoom Meeting.
Copyright Statement
This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to the IEEE Computer Society. No part of the manuscript may be quoted for publication without the written permission of the IEEE Computer Society.
It is recommended that this oral history be cited as follows:
Chung-Chieh Jay Kuo, an oral history conducted in 2020 by Honghong Tinn, IEEE Computer Society
Interview
Interviewee: Chung-Chieh Jay Kuo
Interviewer: Honghong Tinn
Date: Sep 21, 2020
Place: Zoom meeting
Tinn:
Thank you so much, Dr. Kuo. My name is Honghong Tinn. I am now conducting an oral history interview with Dr. Kuo for the IEEE computer society history committee. I really appreciate your taking the time to meet me. The first question I have is about your family background. Would you kindly let me know a little bit about where and when you were born?
Kuo:
I was born in Hsinchu, Taiwan, in 1957. Hsinchu, you know, is a high-tech center in Taiwan today. But when I was a kid, it's not like that. There was no Hsinchu Science Park yet. There were only two famous universities there, National Chiao Tung University and National Tsing Hua University, when I was a kid.
Tinn:
Can you talk a little bit about your childhood? Was there anything particular that led you to study computer science?
Kuo:
Both of my parents were elementary school teachers. They taught kids in the elementary school. So, [they made sure] all development [of mine in my childhood] was very balanced. I did well in almost every subject, such as math, other natural sciences, social sciences, whatever, and biology. My parents’ expectation for me was to become a doctor, medical doctor. I didn't fight with them. I said, well, okay, I will follow that [suggestion]. But when I was in the high school, I found that I didn't like the medical doctor career. First, it's very stressful. If you want to be a good doctor, you should care about your patients. But, when you get involved emotionally, it might be tough. On the other hand, if you isolate yourself from the patients and just be professional, I didn’t want to be that type of doctor, either. So, I probably should not be a doctor and go to another field such as science and engineering. That’s really a decision from my high school time. Generally, my study was smooth. I went through all competitions [i.e., entrance exams for high schools and universities in Taiwan] stage by stage. There was nothing special.
Tinn:
Where did you go to high school?
Kuo:
We lived in Hsinchu until I was 11 or 12. I finished my elementary school in Hsinchu. Then, my family moved to Taipei. I studied in Tsai-Hsing Junior High School (Tsai-Hsing Zhongxue) for three years, Taipei Municipal Jianguo High School (Jianguo Zhongxue) for another three years and, then, National Taiwan University. I studied in all top schools. In Hsinchu, my elementary school is called Hsinchu Shizhuan Fuxiao (or Chushi Fuxiao, the elementary school of Hsinchu teachers’ college). That's also a top elementary school. So, I was always [enrolled] in this kind of elite schools. I was very fortunate.
Tinn: Can you say a bit about your time or your experience in college? Were you a major at the electrical engineering department at National Taiwan University?
Kuo:
Yes. I like to be in small schools. I like a private school setting or a small school setting. In Cushi Fuxiao, there was only three classes [of new students (about 150 students) admitted] each year. So, it’s very small and people know each other well. I like that. Tsai-Hsing is a private school. Also, we know each other very well. I also feel I enjoyed [being there]. But when I started studying at Jianguo Zhongxue, I think it’s huge. No one knows everyone in the 26 classes of one year. Right? So, I really didn't have a lot of memory about Jianguo Zhongxue. Everyone was doing their own business. Free spirits. For National Taiwan University, it is the same thing. It’s just a big university and you can do whatever you want. You can say freedom is a certain type of culture. But for engineering students actually…. the freedom is not that important. We [engineering students] really need good examples. We need good role models.
Tinn:
Sure. I understand. Why did you decide to go to MIT?
Kuo:
I am very thankful for the opportunities given to me. When I was a sophomore and junior at National Taiwan University, I liked physics and took a few physics-related courses such as quantum mechanics and solid state electronics. They were at a theoretical level. Math was also attractive. Yet, I didn't realize until my senior year that the EE people would spend long time in the lab focusing on the manufacturing pipeline, monitoring the equipment, and paying attention to every tiny step since these steps are critical to the yield. They were however not interesting to me. Since most of my courses focused on physics, applied physics, and semiconductors, I would have a better chance to get admitted to a graduate school in its solid state electronics program. Yet, I did not like it. I was unsure on what I should do next. After graduation from NTU, I had a two-year military service in a Naval base in Penghu (the Pescadores). The workload was light. There was a senior colleague, one year older than I, in the same unit. He planned to apply for several graduate schools in US, including the MIT. He had an MIT catalog. I spent a lot of time reading the catalog. Then, I decided to apply for the MIT in two areas: solid state electronics or digital systems such as communications, control, signal processing. For all other universities, I just applied for the program in semiconductors. Luckily, I got admitted to MIT and decided to change my area from semiconductors to digital systems.
Tinn:
Can you say more about your time at MIT? Who did you work with and the lab you stayed in at MIT?
Kuo:
I joined the Laboratory for Information and Decision Systems (LIDS) at MIT in Fall of 1982. I talked to quite a few professors and met Professor Bernard Levy. He was an Assistant Professor at that time. He was kind and very willing to talk to me. Eventually, I decided to work with him. Professor Levy was my MS and PhD thesis advisor. I had an MS co-advisor, Professor Bruce Musicus. I had another PhD co-advisor, Professor John Tsitsiklis. The first topic that I worked on was solving partial differential equations using parallel algorithms and architectures. I struggled a lot in the first nine months. After that, I got some idea and spent another three months to come out with something myself. Then, I won the trust from my advisors to do research on my own. MIT is a private university, and it pays a lot of attention to students. There is close interaction between students and faculty. It reminded me of my previous experience at small schools, where members have close relationships with each other. That's what I like. Education is about humans to humans interaction.
Tinn:
Was your research at MIT related to multimedia, data compression?
Kuo:
No. Multimedia came out in late eighties and early nineties. When I studied at MIT, there was no multimedia at that time. I began with digital signal processing, digital image processing, numerical analysis, and parallel computation. Basically, they are all mathematical tools. Yet, they are the foundation. If you have good tools, many research fields could be just applications.
Tinn:
Can you talk a bit about the lab, LIDS? Was it a big lab and how did colleagues interact each other, in addition to advisors?
Kuo:
LIDS was a big a lab. It had around 15 professors at that time. Okay. People worked on a wide range of topics such as control, communications, signal processing, etc. My research was on parallel processing, which was quite unique in LIDS. Generally, the environment was very friendly, and people were willing to talk about their research. Furthermore, I interacted a lot with Professor Nick Trefethen of the Math Department, and learned a lot from him. Overall, MIT has a very nice culture. It encourages people shared different ideas.
Tinn:
I recalled that the first job you got after graduation was at department of mathematics at UCLA.
Kuo:
Right. I graduated from MIT in 1987 summer and started to apply for jobs in 1986. I got an offer from Professor Tony Chan in the Math Department of UCLA. It was a Research Assistant Professor position for three years. I would have to find another job after that. At the same time, I received a tenure-tracked Assistant Professor offer from an electrical engineering department of a famous university but gave it up due to my passion for mathematics. When I look back, it was a bold but good decision. I joined UCLA and planned to stay there for three years. But something interesting came up in the April of 1988. I got a phone call from Professor Sandy Sawchuk at the USC, who was the Director of the USC Signal and Image Processing Institute (SIPI) at that time. He asked me, “Do you want to come to USC for an interview?” I said, “I didn't apply for a job there, and I plan to stay at UCLA for three years.” But then he said, “But you applied before [in 1986].” I did apply for USC in 1986 but didn't get any response. Professor Sawchuk said, “well, you know, you're local. Why don’t you come to USC for an interview?” I had the interview, got the offer and joined the USC in 1989 January. Once I joined the USC, my colleagues told me, “You are from applied math, you can work on any research problem.” Although this is a compliment, it is true that, if you are good at math, you can do many things in engineering.
Tinn:
Can you talk a little bit about how you started your lab there, at USC?
Kuo:
I chose an applied math topic - how to solve the Toeplitz system of equations effectively -in the beginning since my first PhD student had a strong applied math background. I also worked with other PhD students on wavelet signal processing and computer vision. I tried exploring all possibilities. But, gradually, I focused on image and video compression, which was my main research area for 30 years at USC. Sometimes, I adapt myself to students coming from different backgrounds. As long as we have math as the same language, I can accommodate. Furthermore, I'm willing to learn new subjects. To me, they are all applications of math. Nowadays, there are a lot of new hot topics such as machine learning, big data, neural networks, artificial intelligence. Again, they are built upon math. I'm not afraid of going to new fields, and always try to use math as the main tool to address the challenges. So, deep in my heart, I feel I'm an applied mathematician. During the training of PhD program, I felt that I’m an applied mathematician.
Tinn:
Let’s talk a little bit about the award you received in 2019: The IEEE Computer Society Edward J McCloskey Technical Achievement Award. And you just reviewed a little bit several topics that your lab has covered in the past years. I wonder if you could say more on the research fields, the research achievements that led you to be awarded with this really prestigious honor?
Kuo:
I need to say thanks to the Computer Society for this prestigious award. There is a multimedia technical committee in the Computer Society - multimedia technical committee (MMTC). I was involved a little in the committee but not heavily involved. It indicates that the award was merit-based. My major field is in multimedia computing. Multimedia includes image, video, audio, speech, etc. My main work lies in multimedia compression. For example, you can watch Netflix video over the Internet, which involves video compression. You need video compression for video conferencing. Thus, compression is a very basic infrastructure in our society. There are several generations of video coding standards. I contributed to some of them myself and some of my former students played an important role in the standardization activities. Another related topic is the measurer of perceived video quality. A famous video quality assessment tool is called VMAF (Video Multimethod Assessment Fusion). VMAF is used by Netflix not only for video quality assessment but also for video encoding optimization. VMAF contributes to high quality streaming video from Netflix as well as other video streaming service providers. Netflix and I received a Technology and Engineering Emmy Award in 2021 because of this contribution.
Tinn:
Would you mind saying more about fast motion search, because this was specifically mentioned in the award citation, and also H.264 rate control and perceptual coding. Could you kindly elaborate a little bit more on those fields?
Kuo:
Two adjacent video frames have a lot of similarity. It is desirable to use a frame to predict what’s next. After prediction, the differences, which are called residuals, become small so that you need fewer bits to encode residuals. So, how do you do the prediction? Usually, it’s a block-based prediction. For a block in the current frame, you would like to find a similar block in the previous frame. That's called motion search. It is computationally expensive in video coding. How to speed up motion search was a hot topic in the ‘90s. We were the first in pointing out that motion vectors have spatial and temporal correlations, and this property can be exploited to speed up motion search by two orders of magnitude. The PhD student who worked with me on this topic was Junavit Chalidabhongse from Thailand. He went to the Harvard Law School to get his Law degree later, and now a Professor of Law at Thammasat University in Thailand. As to rate control, it is a technique how you allocate the coding bits to different units of a video stream such as headers, motion vectors, prediction residuals, etc. If it is done properly, one can reduce the number of coding bits yet offering the same quality. It is one of the most important modules in a video encoder. The rate control of H.264 video is more challenging than that of MPEG 1 and 2. We proposed an elegant solution to this problem and it has a great impact to the field.
Tinn:
How about perceptual coding?
Kuo:
We can exploit human perception to reduce the bit rate. Traditional video quality is measured by the mean-squared-errors (MSE) or the peak-signal-to-noise-ratio (PSNR). It is mathematically defined, yet it is not correlated well with human visual experience. The video community has strived for better video quality metrics for decades. VMAF is now the de facto metric of perceptual video quality. Then, we can optimize a video encoder based on VMAF rather than MSE or PSNR, which is perceptual coding.
Tinn:
Do you hold patents in those fields?
Kuo:
I have patents on H.264 rate control but not for fast motion search, VMAF and perceptual coding. Fast motion search was a fundamental research, and we would like to maximize its impact. We collaborated with Netflix in developing VMAF and perceptual coding. Netflix did not file patents. They want the developed technology to be open source for the same reason [i.e. maximizing its impact].
Tinn:
I have a related question. How did these standards get achieved?
Kuo:
Basically, they are conducted within a standardization organization such as ISO (International Standard Office) and ITU (the International Telecommunication Union). Sometimes, ITU and ISO may have a joint standardization activity. If a person would like to initiate a standardization activity, he or she can send a proposal to one of these organizations. If it is approved, you can call for contributions, compare technologies and finalize the standard documents. For the video coding standards, you can see many countries participating. Each country has their delegates, or teams. They can submit proposals, compete, and get evaluated. Eventually, they choose a set of proposals and merge them into the final one. If a company uses the standard, they have to pay the licensing fee. However, there is a new trend. Large companies such as Netflix, Google and Facebook do have a lot of video contents. The licensing fee model is not attractive to them. They would like to develop a new standard to compete with ISO and ITU, which is royalty free.
Tinn:
That's really interesting. My next question is actually more about your research career. Because you are interested in and made a lot of achievements in different sub-fields, I am curious about the technical difficulties you encountered in some of these fields and how you made breakthroughs in those fields.
Kuo:
Technical difficulties are not the biggest barrier to me. If I am given enough time, I can overcome them. I think the most difficult one is the belief that it will work. I can offer an example about deep learning. Its performance is very good, and everyone just uses it. But, there is no clear explanation even today. We need to find a way to explain it, especially on the function of nonlinear activation. I challenged my students in 2014, 2015, but could not get anyone to work on this fundamental problem. I had no choice but worked on it by myself. Eventually, I wrote and submitted a single-authored paper on the explanation of nonlinear activation. I couldn't tell it to my PhD students because they would all laugh at me if the paper got rejected. The paper got rejected twice. For each time, the rejection had only a couple of words: “Not interesting,” “This is well-known.” I wondered whether the reviewer even read my paper. I was so disappointed. At the end, the paper was accepted by a journal and got the best paper award. This kind of difficulty is what I think the most challenging one. It's not about the technical content but the mental strength.
Tinn:
Thank you for sharing with me about this. When did you start working on the deep learning field?
Kuo:
I started from 2014. This research challenge makes me feel young. If I don't have any challenge, I may feel I'm old. Right? Through efforts in the last 6-7 years, I feel that we are on the right track. Besides interpretability, I would like to emphasize another aspect, i.e., green computing. Substantiality has become a main theme of science and technology in recent years. In the area of artificial intelligence and machine learning, it is urgent to explore a novel green machine learning technology, which is competitive with deep learning in performance yet with significantly lower power consumption in training and inference.
Tinn:
Sure. If you still have some time, I will ask about the education award from IEEE Computer Society Taylor L. Booth Award you received in 2016. In our conversation, you have already provided some thoughts about how to run a lab. And I wonder if you can say more about your philosophy about educating PhD students. I hope you don't find this question a little bit repetitive because you have been talking about your students all the time in this interview.
Kuo:
First, you should give students clear leads. Don't let them just search around to find their own research questions. Most people, young researchers, have no direction. Usually, they spend a lot of time finding a research topic. So, in the beginning, you should interact with students frequently and monitor them until they are ready. At the end of their PhD stage, you want to give them more room for independence. I have many ideas to guide students and speed up the beginning stage. Once they are on the track, I can be hands-off. My PhD students are mostly employed by top companies such as Facebook, Google, Apple, in recent years. They're very competitive.
Tinn:
I have one more question about the role of industry in academia, and especially its role in your field. I was wondering if you can elaborate a little bit about the historical change of it, from the beginning of your career to now. Do you think the role of industry has been changing over time during the past 30 years?
Kuo:
There are different industry sectors. The dominant industry was telecom industry, including companies such as AT&T, Bell labs 30 years ago. Next, the PC industry such as IBM, Microsoft, Intel, and now the Internet companies such as Facebook, Google, Apple, Netflix, Zoom, etc. The IT industry keeps evolving. In the multimedia field, academic research was very active in the ‘80s, ‘90s and ‘00s. We see more industry-led research nowadays. Facebook, Google and Apple can spend a lot of money doing multimedia research. If a lab is closely working with industry, the students tend to go to industry. Besides, the pay is much higher. What NSF wants to fund is for some topics for the long term, such as biotechnology, the industry is still at its infancy. We don't see many big bio or biotech companies yet. Some biotech companies could be the early IBM. The bio industry may take another 10 or 20 years to grow. If a professor works on more theoretical topics, the chance for his or her students to take university jobs is higher. A theoretical person can do applied research later. But once you focus on a very applied topic, it's more difficulty to go back to do theory. My lab is not a theoretical one. Actually, I want to do more theory. But my student wants to do applied. But we try to balance the two.
Tinn:
That's really interesting! So, here is my last question. You were the editor in chief for the Journal of Visual Communication and Image Representation. You were the editor in chief from 1997 to 2011. And I know you have served on many journals and obviously this was the longest among all journals that you have served before. Would you like to talk a bit about it?
Kuo:
The previous editor-in-chief (EIC) of the journal, Dr. Russel Hsing, was a good friend of mine. He used to work on image processing, but moved to another field in early 90s. So, he invited me to succeed him to be the EIC. I used this opportunity to know quite a few friends. It is a very good journal covering multimedia computing, communication, and compression. Among non-IEEE journals, it is probably one of the best.
Tinn:
Thank you so much. I really appreciate your time.
