It was purely by chance. After getting my engineering diploma at the Technical University of Hamburg-Harburg I wanted to go abroad. I asked around where I might go, and someone suggested UC Santa Barbara. When I went there, I started working with a scientist, whose name was T. George Campbell, who got me doing this video coding. That was back in 1995. I stayed for half a year, went back to Germany to do my Ph.D. at the University of Erlangen-Nuremberg, and stayed in the field. And it turned out to work for me. Now I am 37 years old and still active in this field.

I am currently heading a research group at the Fraunhofer HHI in Berlin. The group consists of 18 researchers including myself. We are working in the area of video coding, multimedia transmission, and computer graphics coding. I am also teaching a class at the Technical University of Berlin on video coding. Our biggest achievements so far have been our contributions to the H.264/AVC standard. We have made a number of technical contributions, and I was one of the co-chairmen and the principal editor of the standard.

Standards are essential in communications. You can think of the standard as a format that you use to represent data—in our case, video data. With the standard in place, one manufacturer can do the encoding and another one can do the decoding and they’re both working with the same format. A video system that does compression will contain an encoder and a decoder. Both together are called a codec. The standards actually specify the decoder only; they specify how the input to the decoder transforms into the final video signals. This has great practical impact. Once you have the standard, people can rely on this format to make chips and hardware. It allows you to have the kind of massive use of video we have today.

The first big video coding standard that went to mass market was called H.262/MPEG-2 Video. H.262/MPEG-2 Video is the basis of all digital television. The DVD, for instance, is based on H.262/MPEG-2 Video. And now this new standard, H.264/AVC is gradually replacing or augmenting H.262/MPEG-2 Video. What that means is that typically the existing H.262/MPEG-2 Video deployments often stay the same as they’re very cost efficient and do work. Your DVD player is not going to change. But the new video applications that are coming out—the video iPod, the Playstation portable, a lot of internet video like YouTube, HDTV, and the Blu-ray Disc and HD DVD—all use the new standard H.264/AVC.

There are others, but this is the most frequently cited. One reason is probably because all co-authors were very active in the development of the H.264/AVC standard. Among the authors were Gary Sullivan and Ajay Luthra—we were all co-chairmen of the standard committee. Gary Sullivan and I also did most of the editing work (300 pages of standardization text). And also, all four co-authors (Gisle Bjontegaard is the fourth) contributed technically, with Gisle providing the first model for the standard in 1999.

Bringing it all together within a reasonable time frame, so it could then be adopted by industry. We held 16 meetings over the course of four years. They were large and long and we had to reach a consensus each time. So bringing the people together and the technology and getting consensus within a time constraint was the challenge.

Once you reach consensus among people coming to the meetings, the approval is more a formal process. That’s not the tough part. The tough part is to get an agreement among the people in the room, who themselves are involved in the different technologies worth considering. They are the ones who really want you to do certain things. Then there are other people in the meetings who don’t think some particular way of doing things is a good idea and they want you to do something else. You have to build a consensus, find a way to bring everybody together, and put all the technology together so the whole thing does what it’s supposed to do, so it works for large-scale applications and is still competitive in terms of its efficiency.

Against previous standards and also against proprietary solutions.

I would have changed a few things that turned out later to be unexpectedly difficult to do from a technical perspective. I would have also, maybe, taken a little more time. Not much, but a little. Although that is already a difficult judgment call. The one thing you always have to remember is that there will always be a few things that you technically could have done better. But if we had done those things differently, at some point we’d find out that we had made yet other mistakes.

Yes. One is the multiple reference picture work (Wiegand T, Zhang X, Girod B, "Long-Term Memory Motion-Compensated Prediction," IEEE Trans. Circ. Syst. Vid. T. 9[1]:70-84, February 1999), which was the subject of my Ph.D. thesis. Another one is called Context Adaptive Binary Arithmetic Coding, or CABAC (Marpe D, Schwartz H, Wiegand T, "Context-based Adaptive Binary Arithmetic Coding in the H.264/AVC video compression standard," IEEE Trans. Circ. Syst. Vid. T. 13[7]:620-636, July 2003). Another one is the rate-constrained video encoder optimization, which was the subject of my second most-cited paper (Sullivan GJ and Wiegand T, "Rate-distortion optimization for video compression," IEEE Signal Process. Mag. 15[6]: 74-90, November 1998).

So far we’ve been working on extensions of the standard towards scalable video coding (SVC) for improved transmission over mobile networks and over the Internet. SVC may change the way that video is being transmitted over these modern networks. It is already changing the way in which video conferencing is done.

The next big standard will take some time because it has to be justified by another big technical advantage. The big point about H.264/AVC, why it was successful, is that it gave you the same video quality as the previous standard—H.262/MPEG-2 Video—at half the transmission bit rate. When we come out with another standard, we want to get the same quality as H.264/AVC for half again the bit rate. That’s the challenge and the technology to live up to it is not available currently. Maybe by 2011 or 2012, we’ll see the next standard. We’re working on the technology now.

Oh yes, they got heated. The meetings took nine days, with 14 hours’ work a day, and we had four per year. Sometimes we would go until four in the morning with these discussions. It would have been surprising if they didn’t get heated. But there was, in my recollection, not a single point in time when the people that were in the meetings didn’t think, "This is going to be the next big thing and so we have to make it work." I think that was also the biggest winning factor for this project. Once people feel like that, then they’re motivated; they work hard, and you get the results you want.

When you look at the attendance at the meetings, then you’ll see that there were up to 150 people sitting in the room at any one time. If you look at the number of people that actively contributed, I would think there were maybe a dozen, probably less, that actively contributed. Out of that dozen, maybe four or five were the main contributors to the standard.

That is often difficult to explain. Let me tell you about my first attempt to describe my work to my grandmother. I did explain to her the concept for trading off quality versus bit rate for video. I guess I did a poor job of conveying it as she says, "You do all the work and finally the pictures look worse than before."

But let me go back to the subject. Together with my excellent team, we are trying to provide innovations to the field of visual communication based on the research we do. We keep on constantly working on the subject, including new, exciting areas like 3DTV and mobile TV, publish papers, and contribute to standardization. The Fraunhofer society and the Heinrich Hertz Institute is a very supportive environment for that. And we have only just started.

The thing to say about our largest contribution, which is a good portion of H.264/AVC, is that it enabled a number of new video applications—HDTV, HD DVD, Blu-ray Disc, video iPods, and iPhones. It enables these types of applications to be possible. And by enabling, I don’t just mean in the context of making them technically easier to do. Within the business framework, you can still do HDTV with H.262/MPEG-2 Video, for instance, but it would take up more than twice the bit rate. For HDTV satellite transmissions, it would mean you have to do only two channels in one satellite transponder, instead of five as you can with H.264/AVC. For satellite transmission, H.264/AVC brings the cost down and enables it in that way. It makes it easy to sell it to the consumer. So what we do also enable these new technologies to be available at a price that people are willing to pay.