What is a Video Codec?

Photo by veeterzy on Unsplash

Unclear abbreviations

  • MPEG LA royalty: AVC ->HEVC -> VVC
  • Open source and royalty free: VP8 -> VP9 -> AV1
  • Current popular codecs are AVC (H.264) and VP9 (used in all popular streaming services, such as YouTube)
  • HEVC (H.265) is a good high-efficiency codec released in 2013. In spite of its efficiency, HEVC never really took off in popularity because of its failed licensing policy.
  • AV1 is a modern and totally free codec. The first stable release was on the 28th f March, 2018. It is actively integrated in services like YouTube, Netflix, etc.
  • VVC (H.266) is the successor to HEVC. It is the most efficient codec on the current market, and is expected to be finalised and approved in July 2020.

Why Codecs Were Developed

  • Length of the film: 2 hours
  • Frames per second: 30
  • Resolution: 1920x1080
  • Color format: RGB

Problem Identified. How Can It Be Fixed?

  • lossless
  • lossy

Color formats

Lossless compression

  • Raw video file with YUV color format: vicue_test_1920x1080_420_8_500.yuv
  • Size: 1,44 GB (1 555 200 000 bytes)
  • Number of frames: 500
  • Format: 7z
  • Level of compression: Medium
  • Method of compression: LZMA2
  • Dictionary size: 16 MB
  • Size of word: 32
  • vicue_test_1920x1080_420_8_500.7z
  • Size: 301 MB (316 626 856 bytes)
  • The compression rate is almost 5:1. Not ideal…

Lossy Compression: Reference Test

ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r 25 -i vicue_test_1920x1080_420_8_500.yuv -c:v libx265 output.mp4
  • output.mp4
  • Size: 12,5 MB(13 168 899 bytes)
  • Compression ratio 118:1!
  • Raw log of ffmpeg:
Input #0, rawvideo, from 'vicue_test_1920x1080_420_8_500.yuv':
Duration: 00:00:20.00, start: 0.000000, bitrate: 622080 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, 622080 kb/s, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (libx265))
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.0_Au+7-cb3e172a5f51
x265 [info]: build info [Windows][GCC 8.2.1][64 bit] 8bit+10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 3 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 3 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-28.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 rskip signhide tmvp strong-intra-smoothing
x265 [info]: tools: lslices=6 deblock sao
Output #0, mp4, to 'output.mp4':
Metadata:
encoder : Lavf58.26.101
Stream #0:0: Video: hevc (libx265) (hev1 / 0x31766568), yuv420p, 1920x1080, q=2-31, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc58.47.102 libx265
frame= 500 fps=5.4 q=-0.0 Lsize= 12860kB time=00:00:19.88 bitrate=5299.3kbits/s speed=0.217x
video:12854kB audio:0kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 0.049566%
x265 [info]: frame I: 4, Avg QP:27.84 kb/s: 6871.70
x265 [info]: frame P: 456, Avg QP:30.31 kb/s: 5370.48
x265 [info]: frame B: 40, Avg QP:33.20 kb/s: 3891.26
x265 [info]: Weighted P-Frames: Y:4.8% UV:4.2%
x265 [info]: consecutive B-frames: 91.7% 7.8% 0.4% 0.0% 0.0%
encoded 500 frames in 91.77s (5.45 fps), 5264.15 kb/s, Avg QP:30.52
Pic.1: Screenshot of frame from original YUV
Pic.2: Screenshot of frame from HEVC stream

Lossy compression: explanation

  • Similar elements in a frame can be ‘inherited’ from another place in the same frame (called intra prediction)
  • The same concept of ‘inheriting’, but taking place between different frames (inter prediction)
  • Quantization

Process of encoding

Pic.3: High-level view of encoding process of abstract encoder
Pic.4: Representation of ME in encoded streams, visualized by the VQ Analyzer 4.2.0 (av1 stream)
Pic.5: Representation of intra prediction in encoded streams, visualized by the VQ Analyzer 4.2.0 (vvc/h.266 stream)
  • quantized_coefficient — vector of coefficients after quantization
  • coefficents— vector of coefficients after DCT
  • q_step— vector of constants (or dynamical values in modern codecs) which are part of codec specification.

Conclusion

Recommended literature

  • Iain E. Richardson — H264 (2nd edition)

Special thanks

  • Nikolai Shostak
  • Vitaliy Chemezov
  • Alexey Fadeev

Software engineer at ViCueSoft.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Simple queries against Oracle Cloud NoSQL Database

Agile and Scrum

How We Used Redux on Backend and Got Offline-First Mobile App as a Result

Spark partitioning: the fine print

GitOps applied to Kubernetes environments

How to contribute to Grafana as junior dev 📊

Key Aspects to Consider before building Native or Hybrid Mobile Apps | Perfomatix | Full Stack…

Design Patterns — Memento Pattern

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alexander Dydychkin

Alexander Dydychkin

Software engineer at ViCueSoft.com

More from Medium

Open Letter to the Forest Service

A Letter to Dr. Huey Percy Newton

Jocelyn Sage Mitchell Discusses the Varieties of Democracy — IssueWire

GlobeMed Remembers Dr. Paul Farmer

Paul Farmer stands between two GlobeMed students. He is wearing a long-sleeve, light blue button up shirt with a stethoscope slung around his neck. To the left is a student who is taller than Paul, has some scruffy facial hair, and is wearing a short-sleeve, gray button up. To the right is a student who is shorter than Paul and is wearing a blue patterned top with a square neckline. They are all smiling at the camera, with a rural landscape in the background.