Video Codecs: Everything You Need To Know About

During the pandemic when most of the world was under lockdown and more people were online than ever, major video streaming platforms took a collective decision to limit video content to 480p to relieve strain on global mobile networks. Did you know YouTube (35%) and Netflix (15%) account for 50% of all global mobile internet traffic?  The report also suggests 79% of global mobile data traffic will be video by 2022. With this growth rate, video codecs will play an even bigger role in the years to come.

Statista provides information on the level of global mobile video traffic from 2017 to 2022.

In 2018, global mobile video traffic amounted to 12,051 petabytes per month and is expected to multiply to 60,889 PB per month in 2022. Mobile traffic accounts for only 60% of total traffic. It is safe to predict video content consumption will reach 100, 000 PB/month in 2022.

What is a Video Codec?

A video codec compresses a raw video file so that it can be easily streamed online. It consists of an encoder and a decoder. An encoder compresses an analog video file, whereas a decoder decompresses the video file at the device to prepare the video for playback. 

Codecs essentially work in the way that they apply algorithms to compress the video file into the relevant container format. This makes storage and transfer of the file easier as the compressed file is of significantly low size than the raw format. When the video file is at the targeted device, codec then the video file is decompressed, so that it can be viewed by the user. 

Why Do You Need A Video Codec?

Raw video and audio files are very high in size. This makes it difficult to stream over the internet, as it will end up taking a lot of bandwidth and storage in order to play a video. Without a codec, it’ll be difficult to stream videos online. As for a full HD file, i.e. 1080p file, the bandwidth consumption can come up to 7GB/ minute. It’ll take forever to stream a video even with a decent network. 

Video codecs make it much easier to stream the content online. As they drastically decrease the video file size and optimize it for online streaming. Which in turn gives a smooth and high video frame rate. 

How Do Video Codecs Work?

For any video codec to work it needs to compress the frames. There are two types of frame compressions – inter-frame and intra-frame compression. In an intra-frame compression, each frame is compressed independently of the adjacent frames. It is therefore essentially image-compression applied to video.

As discussed above, video codecs essentially compress the video file. They do it by compressing the individual frames in the video. This type of compression can be further divided into two parts. They are intra-frame compression and inter-frame compression. 

With intra-frame compression, every single frame is compressed individually. It can also be stated as image compression applied to each and every frame of the video. 

Inter-frame compression identifies all the static parts in the frames and uses them again in the consequent frames. For example, if there’s a video of a single person speaking in the stationary background, it’ll use all the pixels which are stationary in the next frames, while only adding the moving pixels from the subsequent frames.  It is much more efficient than intra-frame, this is why most of the video codecs are optimized to identify redundancies across frames.

The Relevance Of Video Codec In The Post-Streaming World

Every time we binge a series on Netflix, make a Facetime call on our iPhones, or share a video on Telegram, we are streaming video from one computing device to another. It could be from the Netflix data center to your laptop or your iPhone to your manager’s iPad via Apple data centers.

Since bandwidth is limited from server to users and there is only so much traffic a data center can handle, especially if it is a colocation data center, streaming video is all about delivering the best quality videos to users in the smallest of bytes The question is how can you compress a video without losing its quality?

The answer is video codecs.  Video Codecs are the reason we could stream 1.336 billion minutes of Mandalorian in 4K in a matter of weeks.

Netflix, Apple, Telegram, Disney Plus all employ a bunch of video codecs to ensure smooth delivery of video content from their data centers to your device.

What’s The Difference Between Video Codecs and Containers

It can be pretty easy to confuse between the two, as a beginner to the topic.

Video container is essentially a package which contains the metadata for the file, along with video and audio codec. 

A single video container can contain different types of video codec. They are more or less, a storage unit. What they essentially do is, open and allow the codecs to work on the video. But for this to work, a codec should essentially be supported by the device used to stream the video. 

Not all devices support all types of video containers and codecs. This is why it is generally advised to encode video in different formats. Based on what devices you want your video to be streamed. 

What Are Some Of The Best Video Codecs for Streaming?

The video codec, you should eventually go for, essentially depends on what devices you want your videos to be streamed on. Here are some popular video codecs out there. 

• H.264/AVC
• H.265/HEVC
• AV1
• VP9

H.264/AVC

H.264/AVC is the most widely used video codec for video streaming. It is supported by every single device out there. At a similar file size, it gives much better video quality than the video codecs before it. 

H.265 or HVEC

H.265 is better than H.264 in terms that it gives half the bitrate than it and is good for playback high-resolution videos. But it has a downside, H.265 uses about three times more resources than H.264. H.265, the successor to near-universal codec H.264, HEVC’s loyalty model is too complicated. Hence, major tech companies are giving it a pass to circumvent the loyalty issues

Indeed, there are some issues to iron out. For example, if we talk about Apple’s beloved codec HEVC, or and working together to build a loyalty-free, open codec to replace H264 and VP8 for 4K streaming.

AV1

AV1 is coming as an immense success with many device makers, smartphone platforms, desktop operating systems, and browsers supporting it. The Alliance for Open Media (AOMedia), which counts Apple, Google, Amazon, Facebook, Mozilla, and Microsoft as its members, sees AV1 as a successor to VP9 and a royalty-free alternative to HEVC. AV1 uses elements from Daala, Thor, and VP10, three preceding open video codecs for backward compatibility.

VP9

VP9 is a free video code developed by Google. It is supported by all of the Google products, i.e.Android, Chrome, and YouTube. In terms of performance, it provides better video quality at the same bitrate than H.265. But, it is not supported by Apple devices. 

Uncompressed RAW and Consumer Codecs

Outside a video codec, a video is uncompressed, unprocessed, lossless, and enormous in size.

 

A raw, uncompressed video is perfect for storing and editing purposes but transmitting such a large video over a network is impractical.

Codecs compress a video before sending it over a network through a process of encoding. Each video codec has a different method for video encoding and decoding an uncompressed RAW video. Video decoders for common consumer video codecs are generally baked into operating systems such as Windows, iOS, Android, etc. although that might not be the case all the time.

Mac and iPhone users were unable to stream YouTube in 4K until recently because Apple wouldn’t add a VP9 decoder to its platforms. The codec feud between Google and Apple remained unresolved until the release of macOS Big Sur and iOS 14 and came at the expense of millions of users’ experience.

Compressed RAW vs Uncompressed RAW

A raw uncompressed video is compressed using one of the available video codecs before transmitting to the network and making it available across the users.

The noir backlit looks are some of the signature elements of the often very violent period drama and have helped make the BBC series an international hit.

Uncompressed RAW Video Encoding

The first four seasons of Peaky Blinder were shot on an ARRI camera in ARRIRAW format. ARRIRAW, unlike Apple’s ProRes RAW or Google’s VP9, is not a video codec. Rather, it is a format that literally returns uncompressed, unencrypted, and uncompromised sensor data. In a conventional sense, ARRIRAW can be considered a digital version of the camera negative. Most of our favorite movies including the Snyder Cut of 2017 Justice league were shot on an ARRI camera in ARRIRAW format.

Compressed RAW Video Transcoding

For later seasons, however, cinematographer SI Bell shifted to Red cameras and production went into Red’s proprietary Redcode RAW format. Redcode RAW is a professional-grade, intermediate CODEC, which is intended for post-production use. Redcode RAW along with Apple’s ProRes RAW, Blackmagic RAW, Hasselblad RAW are some of the examples of compressed RAW codecs. In fact, pretty much every major camera-maker and video-editing software company has a proprietary RAW format to its credit.

The Tradeoff Between Professional and Consumer Video Codecs

Professional codecs, while doing a great job for post-production, aren’t intended for end-users consumption. Professional codes need specialized software to run, and the encoded video is comparatively huge in size to transmit over a network, making them impractical for streaming partners or P2P transfers. Therefore, videos in professional video codecs are transcoded to consumer codes such as H264 or MPEG4 before transmission.

When Warner Bros picked HBO Max to release the Snyder’s Cut of Justice League, the team at HBO had to encode the movie from ARRIRAW to tens of consumer codecs to support every device and browser HBO Max is available on, including Apple iPhones and Google Chrome.

To give you an idea, an HD movie in uncompressed RAW format could go more than a Terabyte in size.

It was a huge task for the team to retain the video quality of the theatric masterpiece in 4:3 aspect ratio on our small screens. Consumer video codecs have a considerably higher compression ratio than professional codecs, but they are essential for consistent delivery of video content over streaming and P2P channels.

Compressed RAW codec is smaller in size and faster to transcode

However, for the fifth season of Peaky Blinders, Netflix did not have to encode the uncompressed video since it was shot in Redcode RAW, which is a compressed RAW codec itself. Rather the team just had to transcode the video from a professional video codec to various consumer codecs such as H265, MPEG4 upon user requests. Video transcoding is a simpler and less time-consuming job than encoding. Therefore, video content intended for small screens is shot in one of the professional RAW codecs rather than uncompressed RAW.

The Anatomy Of Video Transcoding From Server To Client

Every time you’re converting a movie from H264 to MPEG4 or exporting a Hasselblad RAW footage to HEVC, you’re running a transcoding job. In a post-streaming world, however, transcoding is more about compressing video files till the point they start showing a significant quality loss. A smaller video file is easier on streaming platforms’ data centers and end-users limited bandwidth. If you want to know more about, what is transcoding, you can visit the blog linked.

The various parameters to video transcoding

While video compression is a trade-off between video bitrate (or size of the video) and video quality, there are many more parameters to consider: the resolution, bit-depth, and chroma-subsampling, encoding time, hardware, etc.

Bitstream and transmission of video content

A compressed video is transmitted in bitstreams over a network. Bitstream for each video codec is different in format and can only be decoded by a designated decoder.

The job of the decoder is to reproduce the video at the receiver’s end.

Quality loss is inevitable!

Since algorithms dictating video codecs compress the video before transmission, some quality loss is evident at the receiver’s end. However, video quality loss from modern coding algorithms is insignificant to average users.

The Lifecycle Of A Video: From production houses, streaming platforms to your living room

Most of the production-grade video contents start life as a series of compressed or uncompressed RAW clips. These clips are merged to create the end content during post-production. The end-content is then transferred to the data center of the streaming partner.

User-agent detection

Now every time a user requests to stream that video on an app or browser, the application server will run a script to gather metadata on his device, application, or browser. The video metadata contains valuable information about the user’s device: user-agent, version, operating system, screen resolution, supported codecs, etc. Based on the information, the streaming server can determine the best video codec for content delivery.

For example, if the user requests Mulan in 4K on his latest Chromebook, Disney Plus will probably deliver the content in the VP9 codec.

Identification of the best codec for content delivery

Once the best video codec is determined for content delivery over the streaming channel, the server will compress the RAW video in bitstreams and transmit the bitstreams to the user over a secure HTTP connection. The format of a bitstream varies from one video codec to another.

Reproduction of the video at the receiver’s end

Once the streaming server establishes a link with the user’s device, the codec decoder on the user’s device will start reproducing the video from the bitstreams. This how a video travels from video production houses to your living room.

Conclusion

Today, far more people than ever rely on their devices for entertainment, socializing, learning, communication, and work. Without codecs, a seamlessly connected world that we live in and take for granted would be a distant dream in a post-apocalyptic world. The idea that we can call a Zoom meeting and connect to all our colleagues at the same time in a matter of seconds regardless of the device we are on sounds surreal unless we consider codecs.

Codecs are, literally, the engines driving modern web applications. Ideally, we need a universal video codec that constantly improves to fulfill our insatiable demand for running video-centric applications. Learn more about encoding, transcoding, and metadata in the blogs linked.

FAQs

• What are the most popular video codecs?
  • The most popular video codecs include H.264 (AVC), H.265 (HEVC), VP9, and AV1. H.264 is widely used for its balance of compression and quality, H.265 offers improved compression rates, VP9 is an open-source alternative supporting high-quality video, and AV1 aims to provide high compression efficiency with royalty-free licensing.
• Why is H.265 considered better than H.264?
  • H.265, or High Efficiency Video Coding (HEVC), offers up to 50% better data compression at the same level of video quality, or significantly improved video quality at the same bit rate as H.264. This makes it more efficient for streaming high-definition video and for use in areas with limited bandwidth.
• What is the significance of AV1?
  • AV1 is a cutting-edge video codec developed by the Alliance for Open Media. It is designed to be more efficient than its predecessors, allowing for better compression and improved quality at lower bitrates. Its open-source, royalty-free nature makes it an attractive option for web video and streaming platforms aiming to reduce bandwidth usage and improve accessibility.
• Can different video codecs affect playback compatibility?
  • Yes, the choice of video codec can affect playback compatibility across various devices and browsers. Older devices and browsers may not support newer codecs like H.265 or AV1, leading to playback issues. Ensuring compatibility often involves encoding videos in multiple codecs or using fallback options for broader support.