We often hear discussions about users’ quality of experience (QOE) relative to video. The term usually refers to subjective judgements about videos and audios that are played on a device. However, I believe that IT network administrators and AV professionals see the issue of video quality in different terms. Let’s explore this.
The quality of the video that is played depends primarily on two sets of characteristics. The first set includes the quality of the capture, the effectiveness of the encoder and decoder, and the quality of the playout device. The second set is related to the ability of the network to deliver the video, that is, the video transport. The first set tends to be the collection on which AV professionals focus. The second, video transport, is generally what network staff members will try to assess.
If we look at the first set, we find that there are tests and metrics that are subjective or objective. Objective measures aim at predicting the quality of a user’s experience. These metrics are also separated into full-reference, reduced-reference, or no-reference. These measurements are determined by whether they compare delivered video frames to: full frames from the source, a reduced set of data that describes each source frame, or no information from the source frames. Tests in the objective quality set include PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index measure). Both of these available in the widely used product VQMT (Video Quality Measurement Tool), created by Moscow State University. Each technique is based on standard statistical measures of the difference in data sets between the source and destination frames. Techniques such as these are widely used in all sciences.
AV professionals also use subjective tests. These generally depend on gathering the opinions about a video or set of videos from a group of individuals. These opinions are generally quantified by assigning a rank from one to five. The average from the group becomes the MOS (mean opinion score). Research has been conducted to determine whether methods of selection of the group, conditions of the testing environment and other factors affect the MOS.
On the other hand, members of the IT staff would likely evaluate the efficiency of the network delivery. They would focus on packet loss, network latency and jitter. One measure, MDI (Media Delivery Index) has been developed and promoted by Ineoquest. It is composed of two measurements, one for loss and one for delay. Originally used for VoIP, it is now applied to video streams such as IPTV and video conferencing. Its use with ABR (adaptive bit rate) video is questionable. MDI is specified in RFC 4445.
A second measurement technique is licensed and sold by Telchemy. The product VQMon gives a MOS score by analyzing video and audio quality and buffer conditions. It can be used for a wide variety of video forms including ABR, IPTV, video conferencing, and video delivered using Google’s QUIC (Quick UDP Internet Connections).
A problem was pointed out in a research paper by Bampis, Li, and Aaron, and was published by Netflix. They argue that most of the techniques I have mentioned, especially in the objective category, are inadequate for assessing the quality of Netflix. By extension, this seems to imply that they are inappropriate for other forms of ABR video. They argue that the wide variety of content in a service like Netflix and the bursty nature of HTTP/TCP delivery make the techniques ineffective.
If the IT industry and the AV industry continue to see video quality from two different points of view, we will be kept from any uniform measure of QOE. It seems we need a merging of assessment techniques from the network and video engineers to develop one universal measurement.
