Integrating HDR with HEVC video encoding

Notice that some material from this post appeared originally in the article, “Integrating HEVC Video Compression with a High Dynamic Range Video Pipeline,” by Raul Diaz, Sam Blinstein, and Sheng Qu, SMPTE Motion Imaging Journal, 125 (1): 14-21, January/February 2016.

You can download the original paper here.

Advances in video capture and display are moving the state of the art beyond higher resolution to include high dynamic range or HDR.  Improvements in video resolution are now informed by advances in the area of color gamut and dynamic range.  New displays shipping today are capable of reproducing a much wider range of colors and brightness levels than can be represented by the video content being produced for streaming, download, digital broadcast, and even Blu-ray.  

By combining technical advances such as higher spatial resolution, larger temporal resolution, and higher dynamic range, distributors looking for differentiated offerings will be able to provide an improved video experience with the existing workflow and technology that is available today.  Content providers are taking notice of technology advances to deliver high-resolution content to viewers, and the newest video compression standard HEVC enables them to compress high-resolution digital video more efficiently for broadcast, mobile and Internet delivery enable exciting experiences such as HDR.

Higher frame rates are now widely supported in modern video compression and delivery standards, yet content producers and distributors largely use workflows and grading protocols that target the dynamic range standards set by technology from the early and mid 20th century. With modern cinema distribution moving to digital, and given the wholesale replacement of cathode ray tube displays by flat panels, the technology is now available to display and deliver a much wider dynamic and color range viewing experience. As a result, an opportunity exists to demonstrate high resolution, high frame rate, and high dynamic range in a single viewing environment.

Is your codec HDR ready?

The question then, is will the codec you are using be able to support HDR?  Native HDR support can be found in HEVC where extensions and new features were added to the standard in July 2014, many of which specifically address high dynamic range. Color profiles previously limited to sample sizes of 8 and 10 bits (named Main and Main10) were expanded. And because chroma subsampling was limited to 4:2:0 this created a significant restriction of chroma relative to luminance by a factor of 4 to 1, but the new HEVC range extensions can now support bit depths of 12, 14, and even 16 bits, along with lower subsampling of chroma to 2:1 (4:2:2) and 1:1 (4:4:4).

SEI messages have also been added to accommodate signal conversions from one color space to another and to facilitate conversions across varying dynamic ranges. With these tools, an end-to-end system for any arbitrary dynamic range and color gamut can be built, and transformations between disparate dynamic range systems implemented.

However, a challenging element is the need to address the widely varying dynamic range of display devices, including legacy systems. Efforts have been underway to develop a solution that can support the future while providing playback compatibility with legacy devices from various organizations and companies, one of which is Dolby Laboratories, who has developed a backwards compatible approach for HDR.

Dolby Vision

One example of this framework is Dolby Vision, the result of nearly a decade of research and development in dynamic range and color gamut scalability. Dolby Vision HDR exists in two flavors: dual layer and single layer.

The original architecture was a dual layer HDR schema, designed as a combination of a base layer containing video data constrained to current specifications, with an optional enhancement layer carrying a supplementary signal along with the Dolby Vision metadata.  This architecture implements a dual layer HDR workflow where the secondary data is an extension of a traditional signal format that provides backwards compatibility with legacy SDR devices (TVs, Set-top-boxes).  But it has a disadvantage of requiring two decoders for the client device to render video.

Single Layer Dolby Vision HDR was introduced later as an alternative to the original approach and to address competitive single layer technologies. It uses the similar metadata as a single layer, but has only one HDR video stream multiplexed with metadata. With this approach, Dolby loses compatibility with legacy devices, but the upside is that it is highly cost-effective for new deployments and operators as some consumer playback devices and TV’s can be upgraded to support single layer Dolby Vision HDR after they were originally sold.

When HDR and wide color gamut is supported on a given device, the two layers are simultaneously decoded and combined using metadata coefficients to present a highly engaging and compelling HDR viewing experience. On legacy equipment, the enhancement layer and metadata are ignored (or not transmitted) and the traditional viewing experience is unaffected. In this way, the dual layer HDR system offers multiple touch-points in the video pipeline to transition without the need for strict synchronization across multiple technologies. In contrast, single Layer HDR requires a hardware and software upgrade to the display device which is not always possible or easily achieved.

As HDR capable reference monitors become more cost effective and available, creative directors will be in a better position to review and master content in the higher range that the video was captured. Such a shift will preserve a much wider range of luminance and color, and invert the mastering stage bottleneck of grading to the lowest common denominator.

Naturally, a dual layer HDR architecture adds a certain complexity to the distribution of video content where both the encoding and decoding stages must handle the secondary layer Dolby requires for HDR. Also, color volume considerations of traditional SDR signals in the new HDR framework requires more than the traditional 8-bit signal.

By using a modular workflow in which augmented encoding and decoding systems can be integrated separately, introducing auxiliary metadata paths to support higher bit-depth requirements of a dual layer HDR system can leverage new video compression standards while simultaneously offering backwards compatibility with legacy equipment and signals.

From experience in production environments, offline encoded 4K HEVC content requires a bitrate between 10 and 15 Mbps to generate quality that is in line with viewer expectations. AVC has multiple inherent limitations relative to HEVC. These limitations make it particularly difficult for AVC to achieve acceptable quality levels at bitrates that can be streamed and delivered effectively to most viewers, particularly at 4K content bitrates.

Dolby Vision adds approximately 20% to 30% additional bitrate load for the higher dynamic range data, making the challenges even more severe for AVC. For 4K content combined with high dynamic range information, the HEVC video compression standard presents the optimum solution to generate acceptable video quality at the bitrates that are needed.

For traditional HD content, AVC video compression has been used successfully for many years for broadcast, the Internet, and mobile video delivery. By using HEVC video compression, this HD content can be compressed at least 30% more efficiently than with AVC video compression.  For high dynamic range HD content encoded with a dual layer HDR system, HEVC video compression can generate a bitrate that is equal to or less than the bitrate needed for a non-HDR HD bitstream encoded using AVC.

Consequently, any video delivery system that can deliver non-HDR HD content today with AVC can also distribute HDR HD content using HEVC video compression without altering the delivery infrastructure.

HEVC offers two primary advantages for HDR content delivery:

  1. An effective and viable method to deliver 4K content,
  2. Bandwidth compatibility with existing video delivery infrastructure to deliver HDR HD content.

If you have more questions about HEVC and encoding for next-generation high dynamic range solutions such as Dolby Vision, email info@beamr.com to learn how Beamr supports some of the largest streaming distributors in the world to deliver high-quality 4K UHD streams with Dolby Vision.


Share