Breaking VMAF and VMAF NEG, or Why comparing codecs in coming years will be considerably harder

For the past 18 years Dmitry and his lab colleagues have been conducting an annual comparison of video codecs. The last one involved more than 20 codecs, the size of the report is more than 200 pages, and there are more than 20,000 graphs in the appendix.

In the comparison, the struggle of standards is clearly visible. In particular, a sharp rise in the number of participants occurred first with the advent of H.265/HEVC codecs, and then with the advent of AV1.

The peculiarity of this comparison is that developers provide codecs and settings themselves, so if a codec is potentially weak - it is removed from participation. In 2021 6 new VVC-codecs participated, also new "Chinese" standard AVS3 appeared, that is the situation becomes hotter and hotter.

At the same time video quality metrics are actively developing, including Netflix's VMAF. This metric is indeed better in terms of correlation. But as research has shown, it can be "cracked", i.e. it can improve results by preprocessing without codec improvement.