news

The end of privacy ‘Meduza’ takes a hard look at FindFace and the looming prospect of total surveillance

Source: Meduza

The FindFace mobile app, which makes it possible to find a random person's social-media page on Vkontakte after taking a photo of them in the street, has made news headlines for its use in experimental art projects and bullying women who appear in pornography. Meduza special correspondent Daniil Turovsky sat down with the authors of the “FaceN” technology that powers FindFace, and met with some of the technology's clients. It turns out that FindFace is just the beginning, and the underlying algorithm is based on a neural network that can actually help to identify any person in any photo or video, creating unlimited opportunities for an almost undetectable system of total surveillance. And Moscow's city officials, along with law-enforcement agencies throughout the country, are now expediting plans to put FaceN's technology to use.

The man took a cottage-cheese pastry from the pack, hit the play button on his music player, and opened a folder containing about a 100 gigabytes of porn.

An hour earlier, Vadim Gusygin (this name has been changed), a system administrator living outside Moscow, had gone home after a short day at work. It was a hot day in June. On his way home, Gusygin added a few hundred rubles to his QIWI e-wallet. He had been planning to tidy up his room for a long time, but then something more important turned up.

For the previous two weeks, Vadim had been waiting for Friday, June 10, to come. He couldn't stop thinking about what had happened earlier. Sometimes, he was overcome by gits of uncontrollable rage.

Earlier in June, a stranger on the social network Vkontakte approached one of his female relatives and informed her that he had a video showing her using the bathroom at a Moscow coffee shop. The stranger identified the victim with the help of FindFace, a Russian dating service that can find anyone on Vkontakte with just a single photo. Vadim was furious not only about the peeping and the invasion of privacy, but also because his relative had refused to heed the advice he'd been repeating for years: never register on social media using your real name. He'd managed to convince everyone in his family, except this young woman.

Vadim wanted to find out who made the video and why. Having scanned a number of forums, he found a few communities of people who deal in hidden-camera footage recorded in public bathrooms. He downloaded hundreds of such videos to his computer. Almost at once he decided to identify other girls in the videos and inform them about the footage. Vadim would speak to them and find out when and where the footage was recorded. To identify the girls, he used the same FindFace service as the troll who had tormented his relative.

Once home, Vadim paid for a premium FindFace account, in order to get up to 300 face searches per month. He launched an application that aggregated all the screenshots from the video archive and configured it to display three images at a time, to speed up the work. Many screenshots didn't have faces, but a few minutes later Vadim came across a photo of a girl who was looking directly into the camera. He saved it and uploaded it to FindFace. The service identified the girl as Polina Anisimova from Moscow.

On June 12, he sent her a personal message on Vkontakte, attaching a screenshot with a comment, "Excuse me, is this you in the photo?" Polina read the message a few hours later, at night. "So what?" asked she. "If it's really you, you might be in trouble," Vadim answered, summarizing the situation for her. At first, she didn't take it seriously, but soon she found herself in a video on one of the hidden-camera online forums. The next day, she wrote a post about the incident on her Facebook page.

Vadim continued digging through the archives and analyzing screenshots. He spent about 30 hours at his computer, looking through images secretly recorded in public bathrooms. As he says, during his "research" he felt like a bit of a crazy person, sometimes worrying about the "awkwardness and irrelevance" of what he was doing. But he was still confident that the victims deserved to know what had happened. Most girls never answered his messages, but a few admitted that it was them in the screenshots. According to Polina Anisimova, Vadim put her in touch with other victims and now they are planning to file a police report with federal investigators. (As for Vadim's relative, she works as an official in Moscow's city government. She didn't sign the petition for fear of losing her job.) Other women reacted to Vadim's messages rather aggressively. One of them replied, "Yeah, same face, so fucking what? I'm not into this crap, unlike you, you sicko."

Identifying people using random photos and image archives accumulated over decades of Internet activity (the Internet Archive alone had stored 15 petabytes of data by May 2014, and there are other such resources online) is one of the many possible applications of facial recognition algorithms, which are likely to spread ubiquitously in the near future: from dating apps to CCTV cameras, allowing security services to track specific people. Facial recognition algorithms will enable almost anyone (including the state authorities) to peep into the lives of passers-by and anyone in whom they've a special interest. It may well be that a small Russian start-up of fewer than ten people will pioneer and lead this field.

A technology creator

In the mid-2000s, you could often meet a tall and smiley schoolboy on a weekday train from the suburban town of Troitsk to and from Moscow. The kid's name was Artem Kukharenko. The commute took him about 90 minutes one way—past the Vnukovo airport and Mosrentgen factory, and across the Moscow Ring Road. Still just a boy, he was already an active programmer. He had been member of IT clubs for schoolchildren since the fifth grade, and for eighth grade he entered school 1543 in southwest Moscow, which is considered to be one of the city's best math schools. On vacations, Kukharenko would go to IT summer schools, where he learned algorithmic programming, data structures, and analysis methods. During the academic year, he participated in contests, and in 2006, he won the all-Russia Information Technologies Olympiad.

After school, Kukharenko entered Moscow State University, joining the department of Computational Mathematics and Cybernetics, without even considering other options. In his second year, he took Anton Konushin's special course on "Introduction in Computer Vision." At the end of the year, Konushin invited his most motivated students to take a few additional tests and sit for interviews. This is how Kukharenko joined the department laboratory of computer graphics and multimedia. The laboratory carried out experiments on machine learning and neural networks (a set of algorithms you can instruct to solve tasks; a neural network can have multiple levels, with every level combining the attributes of previous ones, creating more and more sophisticated combinations, which are often unforeseen even by the authors of the initial algorithms).

As his fourth year drew nearer, following the head of the laboratory's lead, Kukharenko focused his attention on a new and underdeveloped area: facial recognition. In 2012, he co-authored a paper with Konushin on photo-based gender recognition.

Kukharenko's post-graduate profile on the laboratory's website lists machine learning, deep neural networks, and computer vision as his primary academic interests. "I finished school 1543, too, so I knew they don't goof around there; their guys are solid," Konushin recalls today. "Artem showed a profound interest in the subject. His academic project and his thesis were all dedicated to facial recognition. This is an example of a student really putting his knowledge to use."

Having graduated, Kukharenko abandoned facial recognition for three years, focusing on neural networks and machine learning. He traveled and even lived in Argentina for a while, working on a freelance basis for the laboratory of Purdue University, Indiana. American researchers hired Kukharenko to write algorithms that enabled them to classify objects on video. The initial idea was to use such CPUs with a neural network in car auto-pilot systems for automatic classification of objects on the road, such as buildings, pedestrians, and road signs. Upon returning to Russia, Kukharenko joined the Russian division of Samsung, where he continued working on neural networks.

During the lengthy New-Year holidays in early 2015, Kukharenko had a lot of free time, and he took another shot at his university work—just for fun. Together with his girlfriend, they mapped dogs of the same breed as theirs using 150 photos. Then Kukharenko uploaded the mapped images to his learning neural network, which automatically classified the dogs.

He wrote an application for Android that determined a dog's breed using a single photo, naming it “Magic Dog.” The app never gained much popularity (though Microsoft launched something similar a year later), with only about 10,000 downloads. User feedback for Magic Dog was also far from enthusiastic: "I upload a cat's photo, and the app tells me it's a dog," or "My mutt turned out to be an Australian Kelpie." Among the comments, there was a suggestion: "Extend the app to identify people, as well. It's sometimes hard to tell the species by the looks without a special app. At first sight, he's a real bunny rabbit, but then he turns out to be a swine."

Last spring, Kukharenko decided to show his app to investors and contacted the Russian venture fund “Typhon Digital Development” (with an estimated $10 million in assets) through an acquaintance of his. The fund's three managing partners were graduates of the MSU Department of Journalism and ex-employees of media outlets NTV and Izvestia. The fund's motto, found under the logo on its website, reads, "It has been said that something as small as the flutter of a butterfly's wing can ultimately cause a typhoon halfway around the world." The list of the fund's projects includes “Raketa,” a developer of browser-based mobile games, and the mobile advertisement agency “Add In App.”

At his first meetings with the investors, Kukharenko referred to Silicon Valley's experience, to convince them that neural networks and facial recognition are the future. Over the last few years, Internet giants such as Google, Facebook, and Apple have been buying dozens of projects related to facial recognition and neural networks: Deepmind, MSQRD, Face.com, and many more. Facebook has been testing facial recognition within its environment, suggesting friends to tag in photos. According to Bloomberg's estimates, the facial-recognition market could hit the $6.2-billion threshold by 2020.

The negotiations resulted in the founding of a company that was eventually called “N-Tech.Lab”; Kukharenko currently owns one fourth of its shares. Though the developer suggested dozens of tasks that could be solved with the help of neural networks, the company decided to focus on facial recognition.

Kukharenko's company got its first investment cash within a couple of months, and then he quit Samsung, taking one of his fellow programmers with him. He met another developer in a Vkontakte community dedicated to neural networks, simply by browsing through comments.

In May 2015, the three of them moved into an office at an unspectacular business center not far from Tishinskaya Square in Moscow. N-Tech.Lab occupies the entire floor, but there is almost no furniture in the office. The company spent its first investment money on four servers, each costing a few million rubles; three of them were installed right under the developers' desks and one more was placed in a separate cooled room. Kukharenko shows us around with a proud smile, saying, "Google uses a thousand servers for similar purposes, and we only have four."

How it works 

Once Kukharenko already knew how to write the algorithm, the servers were the only missing element. Neural networks are extremely demanding in terms of hardware, as they need a lot of data for learning and a high computation capacity.

It took the programmers a few months to fine tune the network architecture and configure it. Unfortunately, Kukharenko can't go into detail, as it is commercially sensitive information. In essence, the entire algorithm, which the team has titled “FaceN,” is the created neural network that is capable of learning and determining distinguishing features for personal identification: eye size, eyebrow thickness, lip shape, and so on.

"We train the neural network with millions of mapped photographs," explains Kukharenko. "In a semi-automatic mode, people in the photos are identified as John, or Jack, or Stephen. Then the network learns by itself, trying to extract the vectors of features that would solve the task." The neural network determines attributes, assigns them certain importance, and builds interconnections between them. According to Kukharenko, the neural network generates about 80 numbers to describe the information about a face. Even the N-tech.Lab specialists are unaware of what many of them mean. Having "calculated" the features, the neural network can apply them to other photos, as well.

In September 2015, Kukharenko learned that the University of Washington was holding a world championship on facial recognition and decided to present his solution there. The algorithm created by his team beat Google's product in one of the tests, recognizing 73.3 percent of required faces from 100,000 photographs of people from the same age group (the precision of Google's “Facenet v8” was only 70.5-percent accurate). However, in the database experiment, N-tech.Lab's result was only 52 percent (compared to Google's 74.5 percent).

By late June 2016, the competition's organizers announced the creation of a 500,000-person database with each person presented as a set of photographs from different angles and at different ages. The database will be available to facial-recognition companies for the training of their neural networks. It will be launched for public access at the end of summer.

How they're using it

Kukharenko wears shorts and a T-shirt to work. He looks a bit like Edward Snowden, but his views on privacy don't exactly reflect those of the world's most famous opponent of surveillance and traffic interception. "As a common man, I value safety over privacy," admits Kukharenko.

After winning the American contest, N-Tech.Lab was flooded with offers to buy the algorithm. Potential buyers varied in scale and profile, from Chinese casinos to the Turkish border service, which it turns out is rather preoccupied with identifying the people crossing the border. An Australian startup considered using FaceN in a fun park app: if a visitor were photographed at the entrance, they could later receive professional photos of them while at the park, for instance, riding a roller coaster or eating a meal.

The startup team was also approached by intelligence services, both Russian and foreign. They contacted the company through "integrators"—companies that install software on classified objects. (Kukharenko refused to provide any more details.)

In May 2016, N-Tech.Lab concluded an agreement with Moscow's city government to test the facial-recognition service on CCTV camera footage. Moscow runs over 100,000 such devices: 98,000 on the stairs of residential buildings, 20,000 in courtyards, and many more in the streets and city squares, on the roads, and on public transport. All the images are accumulated at the Unified Computing Center of the Information Technologies Department.

"People who pass by the cameras are verified against the connected database of criminals or missing persons," Kukharenko explains. "If the system signals a high level of likeness, a warning is sent to a police officer near the location."

Kukharenko says there is no such system in any other city of the world. When the trial period is finished, N-Tech.Lab will install its system on a closed circuit of the city's video surveillance network. The algorithm will be able not only to check people against the database of criminals, but also pick specific individuals in any part of the city and find their pages on social networks, which almost always contain valuable information about a person's life.

Secret services will be able to use Face.N to identify demonstrators at opposition protests and even to analyze old footage from a violent rally at Bolotnaya Square on May 6, 2012, in order to find new suspects and their pages on Vkontakte.

Facial recognition systems will make it impossible for people to move around in Moscow unnoticed. Even if you leave your phone at home, they won't have any problem tracking you on CCTV footage using cameras, which installed on almost every building in the city.

Moscow's authorities have more than once tried to reform its video-surveillance system. In 2013, they tested a dedicated TV channel that enabled ordinary citizens to monitor video feeds from certain halls and public spaces. The area chosen for the experiment was Konkovo. At the time, the deputy head of the Information Technologies Department said, "Cameras are not a threat to personal freedom. Big brother, small brother... Cameras are an element that can make urban environment a safer place." 

Around the same time, an activist calling himself “the Russian Cat” published his thoughts on the "tactics and strategy of revolutionizing the struggle," by which he meant destroying cameras with paint, glue, plastic bags, sledgehammers, firearms, or crossbows. Less radical citizens use special hairstyles and masks to conceal their identity from cameras.

"A neural network's performance depends on how similar the targeted images are to its training data set," explains the MSU researcher Anton Konushin. "That is to say, a neural network will not be very successful at identifying people from camera footage if it was trained on Vkontakte photos. But it can be retrained."

In June 2016, N-Tech.Lab sold its technology to the Alfa Future People electronic music festival. The festival will take place on July 22-24 in the vicinity of Nizhny Novgorod; last year the event gathered an audience of about 40,000. The FaceN-based application will transmit a visitor's selfie to a robot that will find other photos of the user in the festival's database.

Kukharenko mentions that a number of retail chains have showed an interest in his algorithm, and he's signed multiple nondisclosure agreements. "Visitors' photos will function as cookies referring to the identification and storage of user settings]," he says. "In other words, loyalty cards will become obsolete. As soon as you walk into a store, the staff will already know what you bought last time, thanks to the camera's footage and our technology."

Another possible application is online re-targeting based on CCTV footage from stores. For instance, if you walk into a supermarket, and examine a coffee maker but do not buy it, in a few days you could be shown an Internet ad of the same item with a discount or receive a personal message on social media with relevant information about such products.

Kukharenko also presumes that FaceN will be useful during the 2018 FIFA World Cup, which will be held in Russia. "We could equip pass systems and track blacklisted soccer fans," he points out.

The legal basis for such practices is ambiguous. Damir Gaynutdinov, a lawyer from the Agora law firm, argues that "all such activities fall under personal data processing, according to Russian law." He added that Roskomnadzor (the Federal Supervision Agency for Information Technologies and Communications) regards any information about people as personal data: the agency supported musician Valery Syutkin in his legal action against Lurkmore, a satirical Russian-language version of Wikipedia. At the same time, when users register on social networks, they consent to the processing of their personal data. For instance, Article 5.8 of Vkontakte's user agreement says the network's administrators have the right to transmit such information to third-party developers of services and applications.

"Our technology will change everything," Kukharenko claims. "People will adjust their behavior. They will think twice about posting certain information about themselves." N-Tech.Lab's investor Alexander Kabakov tends to agree with him. "It is a warning of some kind: guys, be more careful with what you put online," he told Bloomberg.

FindFace and the end of privacy

In January 2016, a month after the developers won the competition, Kukharenko was approached by Maxim Perlin of the Blacklight advertising agency. They had met a few months earlier, when Perlin was considering ideas for a new type of billboards.

"I was overwhelmed by patriotic feelings," Perlin recalls. "How is it that nobody is talking about these guys? They work in a cubicle, and they've won a world-class contest."

Perlin and Kukharenko arranged a meeting. Perlin asked why N-Tech.Lab wasn't making any attempts to bring the algorithm to the mass market. Kukharenko told him that negotiations with law enforcement agencies were underway, and Perlin suggested creating a search application for Vkontakte, as the social network uses an open-code technology that enables the extraction of users' photos. Kukharenko wasn't enormously interested in the idea, but he agreed to sell the algorithm. According to Perlin, the contract included "annual payments of moderate size."

Perlin hired a team of two developers and a manager, giving them space on the agency's premises. He went for a straightforward application of the algorithm. "You're walking down the street, you see a person, you take a picture of them, you upload it to the application and find his profile," he explains. "That's why we chose to make a dating app."

N-Tech.Lab provided the search engine for the application, while Perlin's team worked on the mobile application and the Web interface. They called the service “FindFace.”

Kukharenko says the app works similarly to Yandex or Google search: it has indexed each of the 3 million photos extracted from Vkontakte, accumulating a collection of "molds," or combinations of identified features. After a user uploads a photo to the application, it is analyzed and compared to the "molds."

Before the launch, Perlin and his team tested it on the movie “The Barber of Siberia,” and on screenshots from Russian pornpgraphic films and photos from prostitute websites. They were studying the film's extras, looking for matches. They found [the Vkontakte account of] one of the white-wigged men in a crowd scene, for example. They matched the actor to a photo of him at his dacha, holding a can of beer.

FindFace was launched on February 18, both as a website and a mobile application. Perlin commented on the launch in a post on Facebook: "This service really breaks all the stereotypes and brings anonymity to an end. Seeing a pretty girl in a club, you can take a picture of her and find her Vkontakte profile in no time, to learn her name and her interests, and even send her a personal message."

According to Perlin, by the end of June 2016, the application has had over 1.3 million downloads, and the Web interface has performed about 300,000 searches. The technology, he says, successfully identifies about 70 percent of the people using Vkontakte.

The project has paid off on its initial investment and is now generating profits. The mobile app is free, but there are several options to make in-app purchases: for 149 rubles per month ($2.50) you get up to 75 searches (instead of the 30 free searches), and for 459 rubles ($7) a month, you can hide your own profile from FindFace's search results. Perlin claims that each day brings in about 100 premium account subscriptions. Their plans include adding search capabilities across other social networks. Until now, it hasn't been possible, because Asian social networks, Facebook, and Instagram prohibit third-party developers from extracting images. Perlin says this is because Facebook is "upset with a bunch of Russian guys creating a better technology than theirs." However, FindFace is likely to expand its base to Odnoklassniki, another popular social network in Russia with an older audience than Vkontakte's.

Interest in FindFace skyrocketed after March 24, 2016, when a St.-Petersburg-based developed named Andrei Mima published a story (that looks suspiciously like a paid advertisement) about using FindFace to reconnect with two women from his past. Mima wrote that six years earlier, in the summer of 2010, he had photographed two young women on Nevsky Prospect. He wanted to send them the photo, but he didn't have their contact information. In 2016, he uploaded the photo to FindFace and identified the girls he had met. He nicknamed the service "Shazam for people."

A few weeks later, Yegor Tsvetkov, a photographer from St. Petersburg, tested the service in another way, taking pictures of random passengers on the subway. On social networks, people seemed more alive to him than in real life. He titled his project "Your Face Is Big Data." "Today we are deprived of the opportunity of doing anything without somebody finding out. CCTV cameras are everywhere now: in the streets, in shops, and on public transport. Now that we have services like FindFace, you don't stand a chance of concealing your identity and personal information. And people are welcoming this. The thought is quite scary, actually," the photographer argued.

Three days later, a link to the experiment was posted on 2ch ("Dvach," a Russian anonymous forum). The users created a thread titled "Looking for Sluts Who Acted in Porn or Worked the Streets," and coordinated a cyberbullying campaign. 2ch users encouraged each other to write to the husbands and friends of supposed porn actresses whose profiles were identified on Vkontakte using FindFace.

One of the people contacted by 2ch's cyberbullies endured the following interaction:

"Do you know she acts in porn movies?"

"Why tell it to everyone?"

"Do you think it's okay to do porn for a living?"

"It's nobody else's business. Who are you, by the way?"

"Are you saying, if your wife or daughter were doing it, you would still be okay with it? I'm just asking about opinions."

"It's nothing but a photo."

"A photo of a girl displaying her private parts for everyone to see. It's obscene."

"I don't give a crap."

Real-life photos of porn actresses—with their children and parents, and at university classes—were uploaded to dedicated forum threads on 2ch.

Then, a couple weeks later, FindFace helped to identify two arsonists in St. Petersburg, after their faces were captured on a CCTV camera in an elevator. The building's residents obtained footage from the managing company and recorded several screenshots. One of the tenants, Andrei Smirnov, uploaded the images to FindFace and found the wrongdoers' profiles, learning where they work, where they studied, and who their friends are. "In good old Soviet times, their employers and academic supervisors would be notified immediately," Smirnov says. The residents handed over the information to the police.

Perlin believes FindFace owes its popularity to stories like these. After the cyberbullying of porn actresses, he was even invited to a Russian Orthodox Internet channel called Tsargrad.

"Does this mean everybody is under control now?" the Internet show's host asked. "What if someone takes a picture of a man leaving an expensive boutique? His whereabouts could be traced, and the wrongdoers could ambush and mug him."

"We realized there might be violations of the law. But God's law is the priority. Our application creates lots of [more] acceptable opportunities."

"What can you do if you don't want to be in a global database?"

"You should ask yourself why you don't want to be there in the first place."

"What if I still don't want to?"

"You can pay for a premium account."

Perlin is an expert in viral campaigns. As early as in the fall of 2010, he and Vladimir Tabak, his fellow student at the Department of Journalism of the MSU, created an erotic calendar featuring female students to honor Vladimir Putin's 58th birthday. According to Perlin, the calendar made a nice profit, selling about 80,000 copies (the retail price was about 200 rubles, or $3). At that time, Perlin and Tabak were in charge of the Facultet Publishing House for Young Authors and also worked for the pro-Kremlin media manager Konstantin Rykov (the creator of the pro-Kremlin websites Vzglyad.ru and Dni.ru, and afterwards a deputy in the State Duma). They ran a number of programs on the Internet channel Russia.ru. Perlin recalls that Rykov encouraged them to post "any bullshit as long as there are a lot of views." This line of work shaped his subsequent professional approach.

Perlin often uses the Russian equivalent for "cool": "identifying porn actresses is fun and kind of cool"; "it's cool to do something good on a global level"; "it's cool to see pictures of some celeb who's been hacked." He confesses that he liked browsing through Super.ru, a recently closed online tabloid (now a part of Life.ru).

Both Perlin and Kukharenko are very enthusiastic about the prospect of FindFace providing assistance to the police. "They take unsolved cases, upload the suspects' photos to the application, and find their profiles, learning when they were last online. And then they file a request with Vkontakte to obtain the wrongdoers' last IP address," Kukharenko says. "And then the police are on their way." 

Russia's police like the application, too. "If such a mobile app is installed on police officers' phones—and not only on the police's phones—it will help to identify a detained person right on the spot," says Mikhail Pashkin, the chairman of the police officers' trade union. The Information and Public Relations administration of the Moscow City Department of Internal Affairs told Meduza that "the system will be in use as soon as the Information Technologies Department finishes its part of the work."

On its YouTube channel, FindFace promotes its facial recognition potential by showcasing searches of "a man with amnesia," "a con man," "the owner of a [lost] driver's license," "the one who beat me up in the army," "the girl who was reported missing on television," and "a pickpocket from a line."

Just like Kukharenko, Perlin is not really concerned about privacy-violation issues.

"I really don't get it when teens start panicking and saying, oh, now the Federal Security Service can read our messages," he says, commenting on the recent debates over Duma deputy Irina Yarovaya' set of "anti-terrorist" laws.

"This doesn't concern you in the least?"

"I don't give a hoot. I discuss all kinds of things on social networks. Let the feds read about my sex life and girlfriends. Business correspondence? Let them find out how much I make and who my customers are. I am genuinely convinced that no one is interested. Privacy is overrated. It's no big deal to gain access to someone's phone, so let the secret services read my conversations."

After receiving words of gratitude from the police, Perlin admits that he's become "somewhat of a megalomaniac." He is now keen on solving "global problems," such as locating missing persons. "I'll find great moral satisfaction in adding such functionality to every social network," he says. "I'll have the right to say I didn't just make some money—I solved a global problem. I could get away with murder after achieving something like that. Cool, ain't it?”

This text was translated from Russian by Ksenia Khudadyan.