Online retailers have long lured customers with the ability to browse vast selections of merchandise from home, quickly compare prices and offers, and have goods conveniently delivered to their doorstep. But much of the in-person shopping experience has been lost, not the least of which is trying on clothes to see how they fit, how the colors work with your complexion, and so on.

Companies like Stitch Fix, Wantable, and Trunk Club have attempted to address this problem by hiring professionals to choose clothes based on your custom parameters and ship them out to you. You can try things on, keep what you like, and send back what you don’t. Stitch Fix’s version of this service is called Fixes. Customers get a personalized Style Card with an outfit inspiration. It’s algorithmically driven and helps human style experts match a garment with a particular shopper. Each Fix included a Style Card that showed clothing options to complete outfits based on the various items in a customer’s Fix. Due to popular demand, last year the company began testing a way for shoppers to buy those related items directly from Stitch Fix through a program called Shop Your Looks.

AI is a natural fit for such services, and Stitch Fix has embraced the technology to accelerate and improve Shop Your Looks. On the tech front, this puts the company in direct competition with behemoths Facebook, Amazon, and Google, all of which are aggressively building out AI-powered clothes shopping experiences.

Stitch Fix told VentureBeat that during the Shop Your Looks beta period, “more than one-third of clients who purchased through Shop Your Looks engaged with the feature multiple times, and approximately 60% of clients who purchased through the offering bought two items or more.” It’s been successful enough that the company recently expanded to include an entire shoppable collection using the same underlying technology to personalize outfit and item recommendations as you shop.

Stitch Fix data scientists Hilary Parker and Natalia Gardiol explained to VentureBeat in an email interview what drove the company to develop Shop Your Looks; how the team used AI to build it out; and the methods they used, like factorization machines.

In this case study:

• Problem: How to expand the scope of its service that matches outfits to online customers using a mix of algorithms and human expertise.
• The result is “Shop Your Looks.”
• It grew out of an experiment by a small team of Stitch Fix data scientists, then expanded across other units within the company.
• The biggest challenge was how to determine what is a “good” outfit, when taste is so subjective and context matters.
• Stitch Fix used a combination of human-crafted rules to store, sort, and manipulate data, along with AI models called factorization machines

This interview has been edited for clarity and brevity.

VentureBeat: Did Stitch Fix kind of fall in love with an AI tool or technique, using that as inspiration to make a product using that tool or technique? Or did the company start with a problem or challenge and eventually settle on an AI-powered solution?

Stitch Fix: To create Shop Your Looks, we had to evolve our algorithm capabilities from matching a client with an individual item in a Fix to now matching an entire outfit based on a client’s past purchases and preferences. This is an incredibly complex challenge because it means not only understanding which items go together but also which of these outfits an individual client will actually like. For example, one person may like bold patterns mixed together and another person may prefer a bold top with a more muted bottom.

To help us solve this problem, we took advantage of our existing framework that provides Stylists with item recommendations for a Fix and determined what new information we needed to feed into that framework, and how we could collect it.

First, it’s important to understand how clients currently share information with us:

• Style Profile: When a client signs up for Stitch Fix, we receive 90 different data points — from style to price point to size.
• Feedback at checkout: 85% of our clients tell us why they are keeping or returning an item. This is incredibly rich data, including details on fit and style — no other retailer gets this level of feedback.
• Style Shuffle: an interactive feature within our app and on our website where clients can “thumbs up” or “thumbs down” an image of an item or an outfit. They can do this at any time — so not just when they receive a Fix. So far, we’ve received an incredible 4 billion item ratings from clients.
• Personalized request notes to Stylists: Clients give their Stylists specific requests, such as if they are looking for an outfit for an event, or if they’ve seen an item that they really like.

For Shop Your Looks, we supplement this with information about what items go together. The outfits in Style Cards, outfits our Creative Styling Team builds, and outfits we serve to clients in Style Shuffle give us valuable additional insight into a client’s outfit style preferences

VB: How did you go about starting this project? Did you need to hire new talent?

SF: Data science is core to what we do. We have more than 125 data scientists who work across our business, including in recommendation systems, human computation, resource management, inventory management, and apparel design.

Data-driven experimentation is an important part of the team’s culture, so like many initiatives at Stitch Fix, Shop Your Looks was born out of an experiment from a small team of data scientists. As the project grew beyond the initial data collecting phase and into beta testing, the data science team worked with other groups across the business. For example, our Creative Styling Team is tuned in to customer needs and able to recommend looks that are approachable, aspirational, and inspirational.

VB: What was the biggest or most interesting challenge you had to overcome in the process of creating Shop Your Looks?

SF: Creating outfits for clients is a really complex problem because what makes a good outfit is so subjective to each individual. What one person believes is a great outfit, another might not. The toughest part of solving this problem is that an outfit is not a fixed entity — it’s fundamentally contextual. Tackling this problem required gathering new insights, not just about specific items that clients like, but also about how clients reacted to items grouped together.

And because style is so subjective, we had to rethink how we qualified a “good” outfit for our algorithms, since there’s not simply one perfect outfit that exists. Clients have different style preferences, so we believe a “good” outfit is one that a certain set of our clients like, but not necessarily all.

We learn a lot about how clients react to items grouped together when we share outfits with clients and ask them to rate them via Style Shuffle.

VB: What AI tools and techniques does Stitch Fix employ — generally, and for Shop Your Looks?

SF: Shop Your Looks combines AI models and human-crafted rules to store, sort, and manipulate data.

The system is roughly based on a class of AI models called factorization machines and has a few distinct steps. Because generating outfits is complicated, we can’t just create an outfit and call it good. In the first step, we create a pairing model, which is able to predict pairs of items that go well together, such as a pair of shoes and a skirt or a pair of pants and a T-shirt.

We then move on to the next stage — outfit assembly. Here we select a set of items that all come together to form a cohesive outfit (based on the predictions from the pairing model). In this system, we use “outfit templates,” which provide a guideline of what an outfit consists of. For example, one template is tops, pants, shoes, and a bag, and another is a dress, necklace, and shoes.

In the final phase of recommending outfits for Shop Your Looks, there are several factors that come into play. We set an anchor item, which is an item the client kept from a past Fix, which we’d like to build outfits around. The algorithm also has to factor in what inventory is available at any given time. Once that is done, the algorithm develops personalized recommendations tailored to each client’s preferences. Clients can then browse and shop these looks directly from the Shop tab on mobile or desktop. The outfit recommendations refresh throughout the day, so clients can regularly check back for new outfit inspiration.

VB: What did you learn that’s applicable to future AI projects?

SF: We introduced Shop Your Looks to a small number of our clients in the U.S. last year, and throughout this initial beta period we learned a lot about how they interact with the product and how our algorithms performed.

A key tenet of our personalization model is that the more information clients share, the better we are able to personalize their recommendations. We are usually able to adapt the model based on feedback from our clients; however, rules-based systems aren’t generally adaptive. We need the system to learn from client feedback on the outfits it recommends. We’re receiving immensely helpful feedback, from how clients engage with the outfit recommendations and also from a custom-built internal QA system. The model is in its early days, and we are continually adding more information to show clients more highly personalized outfits. For example, while seasonal trends are important overall, recommendations should be customized to a client’s local climate so that clients who experience summer weather earlier than others will start to receive summer items before those in cooler climates.

As we serve more clients, we are receiving an additional data set that strengthens the feedback loop and continues to make our personalization capabilities stronger.

VB: What’s the next AI-related project for Stitch Fix (that you can talk about)?

SF: One of the most interesting aspects of data science at Stitch Fix is the unusual degree to which the algorithms team is engaged with virtually every aspect of the business — from marketing to managing inventory and operations, and of course in helping our Stylists choose items our clients will love.

We believe that when we look to the future, the data science team will still be focused on improving personalization. This could include anything from sizing to predicting your styling needs before you even know you need something.