The Making of Football's Yellow First-and-Ten Line

This article was initially published in Today's Engineer on June 2010

What would televised football be without the yellow “First and Ten” line? This graphic enhancement provides the viewer at home with an immediate visual appreciation of where the offense has to take the ball to make another first-down. For the spectators in the football stadium watching the game, there is no yellow line on the field. No matter where it’s placed, for the television audience, the yellow line appears to be an integral part of the playing field like any of the white yardage lines. When players fall on the yellow line their bodies cover it. Today, football fans at home rarely question how this yellow line appears on their television screens. To a whole generation of young television viewers, the yellow “First-and-Ten” line is as natural to the game as the green color of the football field.

In 1997, when ESPN first aired the First-and-Ten Yellow Line, amazement and wonder were the universal reactions to this new broadcasting technology. Sports journalists were baffled. How was this line put on the field? All sorts of fanciful speculation filled the air after the first broadcast of the First-and-Ten Yellow line. “Is there a guy running out there with a vacuum and chalk?” “Could it be done with laser beams?” The actual details of the innovation were as incredible as the speculation. Sophisticated modeling based on precise measurements, ingenious real-time image processing, and a truck load of workstations made the First-and-Ten Yellow Line look as natural to the game as the turf itself. Behind it all was a small start-up company whose technical team was composed of an aeronautical engineer, mathematician, broadcast engineer, software engineer and a couple of electrical engineers. The new company was Sportvision and its president was an IEEE member.

The story of the First-and-Ten in NFL football starts with an NHL hockey story. In 1996, a team of engineers at Rupert Murdoch’s NewsCorp had pioneered a system that tracked a hockey puck and highlighted the puck’s motion, in real-time, during the live broadcast of NHL games. Officially, the puck tracking system was called FoxTrax, but the popular name became “Glow Puck.” First aired in the 1996 NHL All-Star Game, FoxTrax was developed for the Fox Sports Network, which was part of the NewsCorp media empire. The story of FoxTrax can be found in the September 2009 issue of Today’s Engineer. Exhilarated by their success, the engineering team behind the Glow Puck wanted to develop other sports broadcasting applications for Fox Sports. But with a downturn in NewsCorp’s business, such opportunities were not forthcoming. So with Murdoch’s blessings, the engineering team behind the Glow Puck left NewsCorp to start Sportvision, with Stan Honey as the president. In return for exclusive licensing rights to prior patents, NewsCorp obtained an equity share in Sportvision. Once NewsCorp was onboard, finding other investors proved relatively easy.

Sportvision now faced the challenge of coming up with a first product. The first effort was Air FX. The idea was to enhance the “color commentary” in NBA basketball by providing a graphical system that tracked, measured and displayed the players’ jumping abilities. But Air FX did not have the network appeal that Sportvision had hoped for. While work on Air FX was still ongoing, another product idea started to crystallize: to make the first-down line in football visible to television viewers. Adding graphical enhancement to video streams was not new to football. John Madden, a well know football commentator, had popularized the use of the “telestrator” in televised games. The telestrator allowed one to superimpose free-hand sketches on a video image. While useful in analyzing replays, telestrator sketches could not be used during the actual televised play because they obscured parts of the underlying video image. The Sportvision idea for a first-down line was radically different from the telestrator technology. Could a first-and-ten line be superimposed continuously on the broadcast image so that, wherever the line was place, it would look like a natural part of the field itself? The goal was to enhance the viewing experience without detracting the viewers’ attention away from the flow of the game.

The idea was very attractive, but there was one very imposing technical obstacle. Could the “keying” problem be solved? “Keying”, or more correctly, “chroma keying” was, and still is, the standard technique used in broadcast applications like weather forecasts where the meteorologist stands in front of large map. In reality, the meteorologist is in the studio standing in front of blue (sometimes green) screen. The map is superimposed on the blue screen by a simple rule: replace all the blue with the corresponding map image but do not replace any color that is not blue, i.e. the meteorologist. For this technique to work well, the meteorologist’s wardrobe has to be carefully chosen so as to not be confused with the blue color of the background screen. Proper studio lighting also guarantees the effectiveness of chroma keying. Keying thus works well in the very controlled environment of the studio. Keying for a First-and-Ten line would be in the uncontrollable environment of a large football field with considerable variability in the background colors: wet & dry grass, natural turf artificial grass, mud and dirt, variable lighting, sunlight and shade, and even snow. The great color variety of team uniforms also raised issues. Could they be easily picked out from various colors in field environment? There were serious doubts within Sportvision. Could the available processing power, at the time, be able to sample the pixels in time to figure out when to draw or not draw the First-and-Ten line?

J.R. Gloudemans provided the proof-of-concept that the keying problem could be solved. Glaudemans, who had worked for Shoreline Studios, joined Sportvision very soon after its creation. While working on the Air FX project, he heard that others within the company were brainstorming the “First-and-Ten line” idea. The chroma-keying challenge immediately piqued Gloudemans’s interest. He convinced his immediate superior, Marv White, to let him work on the problem quietly, under the company radar. Neither he nor White told the others in the company that he was experimenting with chroma keying and not working on Air FX. He took a video clip from football game and started to examine the keying problem within various color spaces. Glaudemans discovered that YCbCr was the best set of color spaces for keying in the First-and-Ten line. After a couple of weeks of experimenting, Gloudemans had shown that they keying problem could be solved with the available processing technology. There would still be a lot of refinement work to do on the keying technology, but the door was now open to develop the First-and-Ten line.

Sportvision took the idea to the Jed Drake at ESPN. As head of Event Production, Drake is responsible for all events that ESPN televises outside the studio, which is essentially all the live sporting events. The idea intrigued Drake. Before ESPN could enter into any agreement with Sportvision, the NFL had to approve the idea. After seeing a demo tape, NFL executives enthusiastically endorsed the idea but with one caveat. The First-and-Ten line could not be shown during replays. NFL officials did not want this technology being used in any way to second-guess the referees. With the NFL on board, ESPN negotiated a one-year exclusive with Sport Vision for the 1997 football season. For Drake it was a big gamble. He could not be sure how the viewing audience and sports writers would react. Would they accuse ESPN of ruining the game?

After ESPN signed the contract, Sportvision now had to design and produce a system that was reliable enough to be inserted into an ESPN live broadcast. A lot of hard engineering work was still needed. Once the proof-of-concept was shown, Glaudemans redesigned the keying component from the ground up to handle every possible lighting and color environment on the field in real time. The keying component of the system ensured that the color of the First-and-Ten appeared to be part of the field. But sophisticated modeling and calibrations of the cameras and playing field were still needed to make the line appear to be a natural part of the field.

Camera lenses introduce subtle distortion as you go to the edges of the field of view. So the white yardage lines are not really straight. If the First-and-Ten line graphic were imposed on the video image as an exact straight line, the human eye would catch the discrepancy with the white yardage line. So, the First-and-Ten line had to be distorted to match the white lines. The adjustment became even more complex because a television camera lens’s distortion changes as a function of zoom and focus. As it zooms and focuses the lens elements move. A TV lens will go from 10 percent pincushion to 10 percent barrel all the way back to 10 percent pincushion. The lens had to be retrofitted with sensors to measure the movement of the lens. Tables were created to convert sensor data to line distortion data. The shape of the field’s surface also had to be accurately modeled in order to bend the First-and-Ten line to compensate for the surface shape. In rainy areas, fields had various dome shapes to help run-off and drainage. In dry areas, the surfaces of fields were flatter. All these surfaces had to be precisely surveyed by laser techniques. The insertion the First-and-Ten line into the video image also had to take into account the continually changing perspective of each camera. The exact position of each camera in three-dimensional space, which was fixed, had to be measured. Sensors were also put on the camera to measure pan, tilt and zoom. All the cameras had to be “synchronized” to ensure that computer knew, at any given instant, which specific camera was on-air before it inserted the First-and-Ten line. All this processing had to happen in real-time. The Sportvision team had to push the available computational power to the limit. A separate truck, filled with workstations and image processing hardware, would accompany the regular broadcast truck.

Jed Drake was responsible for the actual color of the line. Originally, for technical simplicity, Sportvision was using reddish-orange for the color of the First-and-Ten. But Drake wasn’t happy with it. One day, he sat down with Glaudemans and they started to experiment with different colors. Drake would say “a bit more towards gold, a little bit more towards green,” and Glaudemans would make the changes. Finally Drake zeroed in on the color that he liked — the yellow color that is still used today.

The First-and-Ten line went live for the 1997 Fall football season. The following year, ESPN won an Emmy in Sports Broadcasting for the First-and-Ten.

More technical information on the First-and-Ten Line can be found in Method and Apparatus for Adding a Graphic Indication of a First Down to a Live Video of a Football Game, U.S. Patent #6,141,060 filed on 5 March 1999, issued on 31 October 2000.

See Also

Recollections of the development of the FoxTrax hockey puck tracking system

Sportvision

Tracking the Ice Hockey Puck - FoxTrax (Glow Puck)

My Recollections: Development of Football's Virtual First Down Line