How Netflix Uses AI and ML to Determine Which Thumbnails They Use?

Reshma M
4 min readNov 17, 2021

If you have subscribed to Netflix, you have probably noticed the thumbnails which appear next to each content piece you get recommended. People are far more inclined to watch something if the thumbnail depicts elements they are interested in.

The Netflix Artificial Intelligence algorithms focus on these aspects as well. After all, the more you watch, the more likely you will renew your subscription. Furthermore, being the industry leader among the most successful streaming services, they enjoy some extra time to figure out strategies to keep users engaged with the platform.

Why Do Netflix Thumbnails Optimisation Play a Crucial Role in User Engagement?

In general, we know that if you don’t grab an individual’s attention within 90 seconds, they’ll lose interest and move on to something else. It is exactly the case with any Netflix user. Such unsuccessful sessions may occur when the platform fails to attract the user with relevant content. Also, it could be the case when they do show the appropriate content but don’t provide a strong rationale as to why their subscribers should consume it.

Therefore, Netflix’s most persuasive weapon for influencing a viewer’s choice is thumbnail artwork. Since a user will only look at one for 1.8 seconds, the service invests heavily in optimising the Netflix recommendation algorithm.

How Does Netflix Leverage AI and ML to Achieve Magnificent User Engagement?

Here are some prominent strategies and techniques harnessed by the streaming services giant to entice an enormous consumer base-

1. Consumer research on the impact of thumbnails-

According to a study conducted by Netflix in 2014, thumbnail artwork came out as the most significant influencer to induce users to consume content. Also, it found that 82 per cent of browsing time gets spent on a thumbnail.

As a result, Netflix employs a sophisticated thumbnail selection procedure for each of its 200 million+ subscribers. The method is known as Aesthetic Visual Analysis (AVA), which commences by extracting all the frames from video content. For example, one hour of “Stranger Things” has 86,000 frames.

2. Deploys Frame Annotation process-

In this process, each frame is tagged or labelled with metadata that indicates the key variables about the content piece. These variables include-

  • Saliency
  • Frame number
  • Brightness / Contrast
  • Nudity probability
  • Face/skin tone

The following characteristics are then used to grade the frames-

  • Visualisation (brightness, contrast, colour, motion blur)
  • Contextual (detection of faces / shot angle)
  • Composition (photography principles like “rule of thirds”, symmetry, depth of field, etc.)

3. Emphasis on localisation-

Regional differences still exist, according to Netflix, and are crucial for particular titles and graphics. While excellent stories transcend borders, they discovered that how each narrative is presented in different regions impacts how quickly users discover that piece through artwork. For example, the ideal thumbnails for the show “Sense8” featured the characteristics that were most appealing to that country.

4. Focus on the portrayal of the cast of the content-

Netflix discovered these facts when displaying the content’s cast and characters on thumbnails-

  • Villainous characters on thumbnails attract more viewership.
  • When there are more than three characters in a thumbnail, it significantly underperforms. For the premiere of “Orange Is The New Black,” Netflix used this knowledge.

5. Netflix AI in action-

Netflix applies Machine Learning techniques to recommend thumbnails to its user base based on their recent watch history. Also, Netflix performs consistent A/B testing of thumbnails before displaying them to the subscribers.

Netflix’s A/B testing methodology emphasises progressive development, data-driven decision-making and failing fast. While investigating a complicated topic like image selections, they attempt to prove the hypothesis in gradual, incremental steps with escalating precision and intricacy. For example, the analysis of thumbnails for the film “The Short Game”.

6. Image analysis and ranking for depicting emotions-

Faces exhibiting complicated emotions outperform stoic or benign ones, according to Netflix. People are more likely to watch a tale if they see a wide array of feelings and emotions. Thus, image analysis and ranking become crucial in finding the traits that make a thumbnail more likely to be clicked. Some vital traits include-

  • Expressive faces
  • Main characters
  • Brightness

Netflix analysis revealed that the bottom right frame was the “winner” in the “Unbreakable Kimmy Schmidt” example.

Conclusion

Imagery is an effective instrument because of its ability to affect individuals in a variety of ways. Netflix has worked diligently over the last several years to figure out how a comprehensive alliance of Artificial Intelligence technology, consumer research and creativity helps users discover the right set of content they’ll appreciate faster and have a richer Netflix experience.

Picking the perfect thumbnail artwork has made a tremendous difference in the Netflix product experience. While the findings of the Netflix studies were frequently astounding, it is apparent that images have the power to impact people and when used precisely, Netflix thumbnails may help users locate narratives they would enjoy considerably.

--

--