What is a Video Codec?

Photo by veeterzy on Unsplash

Unclear abbreviations

AVC, VP8, VP9, HEVC and modern AV1, VVC — what are all these? And why so many of them?

  • MPEG LA royalty: AVC ->HEVC -> VVC
  • Open source and royalty free: VP8 -> VP9 -> AV1
  • Current popular codecs are AVC (H.264) and VP9 (used in all popular streaming services, such as YouTube)
  • HEVC (H.265) is a good high-efficiency codec released in 2013. In spite of its efficiency, HEVC never really took off in popularity because of its failed licensing policy.
  • AV1 is a modern and totally free codec. The first stable release was on the 28th f March, 2018. It is actively integrated in services like YouTube, Netflix, etc.
  • VVC (H.266) is the successor to HEVC. It is the most efficient codec on the current market, and is expected to be finalised and approved in July 2020.

Why Codecs Were Developed

First of all, we have to understand why we need codecs:

  • Length of the film: 2 hours
  • Frames per second: 30
  • Resolution: 1920x1080
  • Color format: RGB

Problem Identified. How Can It Be Fixed?

Ok, I hope you see the problem: raw streams are very, very big. And they will only get bigger thanks to increased resolutions, frame rates, length, effects (like HDR) and so on.

  • lossless
  • lossy

Color formats

Bright/luma encoding (.yuv) is used in most cases instead of RGB format.

Lossless compression

Let’s emulate lossless compression (and it is used — see VLC) and see how good an outcome we can get:

  • Raw video file with YUV color format: vicue_test_1920x1080_420_8_500.yuv
  • Size: 1,44 GB (1 555 200 000 bytes)
  • Number of frames: 500
  • Format: 7z
  • Level of compression: Medium
  • Method of compression: LZMA2
  • Dictionary size: 16 MB
  • Size of word: 32
  • vicue_test_1920x1080_420_8_500.7z
  • Size: 301 MB (316 626 856 bytes)
  • The compression rate is almost 5:1. Not ideal…

Lossy Compression: Reference Test

How well can modern encoders compress the video file?

ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r 25 -i vicue_test_1920x1080_420_8_500.yuv -c:v libx265 output.mp4
  • output.mp4
  • Size: 12,5 MB(13 168 899 bytes)
  • Compression ratio 118:1!
  • Raw log of ffmpeg:
Input #0, rawvideo, from 'vicue_test_1920x1080_420_8_500.yuv':
Duration: 00:00:20.00, start: 0.000000, bitrate: 622080 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, 622080 kb/s, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (libx265))
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.0_Au+7-cb3e172a5f51
x265 [info]: build info [Windows][GCC 8.2.1][64 bit] 8bit+10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 3 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 3 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-28.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 rskip signhide tmvp strong-intra-smoothing
x265 [info]: tools: lslices=6 deblock sao
Output #0, mp4, to 'output.mp4':
Metadata:
encoder : Lavf58.26.101
Stream #0:0: Video: hevc (libx265) (hev1 / 0x31766568), yuv420p, 1920x1080, q=2-31, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc58.47.102 libx265
frame= 500 fps=5.4 q=-0.0 Lsize= 12860kB time=00:00:19.88 bitrate=5299.3kbits/s speed=0.217x
video:12854kB audio:0kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 0.049566%
x265 [info]: frame I: 4, Avg QP:27.84 kb/s: 6871.70
x265 [info]: frame P: 456, Avg QP:30.31 kb/s: 5370.48
x265 [info]: frame B: 40, Avg QP:33.20 kb/s: 3891.26
x265 [info]: Weighted P-Frames: Y:4.8% UV:4.2%
x265 [info]: consecutive B-frames: 91.7% 7.8% 0.4% 0.0% 0.0%
encoded 500 frames in 91.77s (5.45 fps), 5264.15 kb/s, Avg QP:30.52
Pic.1: Screenshot of frame from original YUV
Pic.2: Screenshot of frame from HEVC stream

Lossy compression: explanation

‘If we use lossy compression we lose the quality.’ While this is true, there are some technologies and techniques which can help us achieve an almost (or completely) unnoticeable loss in quality.

  • Similar elements in a frame can be ‘inherited’ from another place in the same frame (called intra prediction)
  • The same concept of ‘inheriting’, but taking place between different frames (inter prediction)
  • Quantization

Process of encoding

So next we can look at this diagram, which shows the high-level encoding process:

Pic.3: High-level view of encoding process of abstract encoder
Pic.4: Representation of ME in encoded streams, visualized by the VQ Analyzer 4.2.0 (av1 stream)
Pic.5: Representation of intra prediction in encoded streams, visualized by the VQ Analyzer 4.2.0 (vvc/h.266 stream)
  • quantized_coefficient — vector of coefficients after quantization
  • coefficents— vector of coefficients after DCT
  • q_step— vector of constants (or dynamical values in modern codecs) which are part of codec specification.

Conclusion

As we can see, codecs are complex software that use a wide range of technologies (a modern codec like VVC even uses some elements gained from the world of machine learning). Their development is always ongoing, thanks to a wide range of areas for improvement and evolving techniques. Finally, codec evolution is continuously pushed by the market with its increasing appetites like higher resolutions (fullhd-4k-8k-16k), 360 video, higher frame rates (30–60) and special effects (like HDR) but in the same time with smaller bandwidth cost.

Recommended literature

  • Iain E. Richardson — H264 (2nd edition)

Special thanks

  • Nikolai Shostak
  • Vitaliy Chemezov
  • Alexey Fadeev

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store