We’ve all played around with Snapchat’s filters before – from the popular dog filter to the flower crown to the puking rainbow, Snapchat has gained immense popularity amongst teens and young adults due to these features. What’s more interesting is the technical side behind such technology.
Vox investigates how Snapchat filters accessorise or distort faces in real time. These filters are known as “lenses”, sold by Looksery, a Ukrainian startup. And in 2015, Snapchat bought the software for about $150 million.
Computer Vision
The technology is called computer vision, more specifically, image processing. It collects pixelation data by breaking down the image into tiny sections i.e. coordinates to identify objects and thus gauge each feature on your face. The 3D space is recognised by assigning values to each section: a number between 0 and 255 whereby 0 means black and 255 means white. For example, it collects the light and dark areas to determine the position of your nose. This is possible because of our facial features – the sides of the nose are darker than the nose bridge. And so, due to image processing, your face now becomes a series of binary numbers. Snapchat then determines if the camera is capturing a human face or not with the Viola-Jones algorithm.
Image Source: open frameworks.cc
Facial landmarks study the data of millions of faces to understand how the human face is structured. Essentially, they help filters locate each facial feature. Once coordinates are plotted for your eyebrows, eyes, nose, and mouth, a computerised 3D mask is placed on top of the virtual face, forming the filter – a process known as active shape modelling. As you move your phone, the filter traces the coordinates so that it works in real time. Other uses of computer vision include how Facebook identifies people in photos and how self-driving cars avoid other vehicles and pedestrians.
In 2016, Snapchat acquired Seene, a 3D photo app maker that captures 3D models from your camera. This has since brought on new selfie filters to improve the augmented reality selfie experience. Due to the popularity and increasing rates of user engagement, filters will continue to be a fundamental Snapchat feature.
Sources
Written by Nichapatr (Petch) Lomtakul