My Not-Illegal Machine Learning Project

A long quest in legally obtaining images for building a machine learning model. Like FaceApp, but not so sneaky.

Since I arrived at Google just two months ago, I’ve been eager to develop a machine learning model which can detect fashion trends in images. While the current Google Vision AI tool can detect nearly twenty different kinds of snakes (including ‘corn snake’ — who knew this was a thing?), it cannot, unfortunately, detect whether or not an article of clothing or accessory is made of snakeskin. Additionally, although it can recognize a ‘bicycle saddle’ (again, what is this?), it will not be able to identify one of the hottest bag trends of the year; the saddle bag.

Since I was looking to detect these specific, niche elements for particular kinds of photos, the clear solution was to develop my own AutoML vision model, trained to identify current fashion trends. Fashion companies are notably behind with regards to technological advancements and machine learning integration, and building this model could aid improved fashion data analytics (or as I’ve entitled my project, dash).

However, in order to train a computer to be more fashionable, I was going to need lots of high-quality and diverse images of such recent fashion trends. As per AutoML’s specifications, for every trend I wanted to teach the computer (such as snakeskin or saddle bag), I would need approximately one hundred example images. Most importantly, every single image needed to be free of copyright and legally usable for product development.

Legal barriers to machine learning.

My first instinct was to turn to Instagram, scraping images from high-fashion accounts like Dior or Alexander Wang. Certainly, since these accounts are public and their photos are available for everyone to see, that means they are legally accessible? Wrong. Oh, so terribly wrong. It turns out that just because an image can be publicly accessed does not mean it is without copyright, or legal for anyone to take and use. Additionally, and after a brief back-and-forth with Google’s legal team, I came to learn that high-fashion images are actually some of the most heavily copyrighted. This made my job much, much harder.

If nothing I could publicly access online was legal for me to use, I had to turn elsewhere. Instinct number two was to run around NYC and take hundreds of pictures of myself wearing high-fashion, trendy outfits. After all, SoHo is only a short stroll away from Google’s NYC office. However, my boss and I concluded that running around New York taking selfies in Louis Vuitton stores was not, in fact, a good use of time.

My third and last option was simple yet daunting: I had to email absolutely everyone I knew could help me, or at least point me in a productive direction. The really wonderful thing about working at a company as large and as vast as Google is that, well, it’s a big company. There were so many different people I could turn to for help: Cloud customer engineers for fashion companies, strategic partner leads for magazine clients, and the very head of fashion & luxury at Google himself.

I began contacting everyone within the company who might have access to the images I needed. My first couple of emails were to product marketing managers in France, whom I found by exploring fashion projects at Google. In probably grammatically incorrect French (a preschool education in Belgium can only carry me so far), I inquired about Google’s role in fashion & technology and how I could go about building a fashion-catered machine learning model.

I was connected to Cloud customer engineers — some still in the Paris office and others here in NYC — who work developing products for fashion clients such as Alexander Wang, Dior, LVMH, and British Vogue. The engineers at Dior and Louis Vuitton connected me with representatives for their companies, and the British Vogue team connected me with the greater Conde Nast team here in the states. Later, my project’s pitch deck ended up in the inbox of Google’s Director of Global Partnerships of News & Publishing, who passed me along and introduced me to Google’s head of fashion & luxury. Not a bad network for an intern.

After countless conversations over Google Hangouts, I was still legal-image-less. Every conversation was incredibly helpful in the sense that I received much positive feedback and guidance for my project, but acquiring images legally permissible for machine learning training was starting to seem impossible. I was beginning to realize that images actually have a lot of rights.

Persistent to overcome this roadblock, I continued reaching out, this time to product managers at Google Shopping, where a manager in Mountain View connected me with engineers in Zurich which, to accommodate for different time zones, led to some early-bird meeting times. Good thing I’m a morning person.

The Zurich engineers shared with me a database they’d generated from Google Search data, demonstrating which fashion queries were most popular versus most “trendy”. Not only did this database help me with my personal fashion inquiries (should I really buy a snakeskin dress?), but it also proved to be a significant addition to my machine learning model. Once I get access to images, how am I supposed to teach a computer to recognize fashion trends if I don’t even know what’s trending?

Finally, one month later, I was connected to a Google Trends curator who forwarded me to a product marketing manager who forwarded me to a visual lead on the image search team. The image search team said yes, we do have images for you to use, due to Google’s tremendous connection with a vast number of brands.

These images were granted for Google to access and use, and though I am but a lowly intern, as an employee of Google my project falls well within the scopes of their usage.

I was granted five hundred (500) high quality images of fashion trends and outfits all from the past twelve months (trendy!). The images are perfect — ranging from runway shows from Prada to Balenciaga to Alexander Wang, to celebrity appearances, to high-end street fashion outfits. Thankfully, and as predicted, many of them include a saddle bag.

Data acquisition in the Big Data era.

For any of you hopeful machine learning engineers, know that acquiring data and databases is perhaps the most important yet lengthy step of the process, and although we are living in the ‘big data’ era, this data is well protected in the favor of its owners.

In fact, although the European Union has recently taken steps to allow further data access for machine learning purposes, these laws still maintain that data owners have the right to restrict for what purposes their data is used.

Perhaps you’ve heard of FaceApp, an app to which a user uploads a photo of their face and the app magically manipulates the image to showcase what you’d look like when you’re older. A virtual Botox calculator, if you will. Recently, FaceApp has made headlines for — as specified in its Terms & Conditions — reserving the rights to utilize any of the millions of user-uploaded photos however it may please. Yikes! Maybe it’s time we started paying more attention to the fine print.

FaceApp’s sleuth-y data acquisition is a brilliant example of the significant barriers in place when attempting to develop accurate and relevant machine learning models. Whatever engineers own FaceApp now have access to a fantastically large and diverse database of human images, which they are legally allowed to (and definitely will) use for machine learning purposes.

Acquiring data, let alone good data, is hard, as highlighted by my countless amounts of desperate email chains. Legal data acquisition, however, is essential to ensure that your product is not only trustworthy, but also usable by other engineers and clients.

So, as a data owner (or public-Instagram-account-poster), know your data is heavily protected and cannot be legally utilized for machine learning purposes without your consent. As a machine learning engineer, know that data accessibility is being improved, and chances are that, for whatever you wish to build, there exists a legal way to do it. Even if it means long chains of networking emails.

Thanks to everyone at Google who helped me eventually find these photos, & for providing so much feedback for my project. Thanks also to my intern host Anu, & to Dale, who taught me most of what I know about machine learning.