This article is Part 1 in a quarterly series that sets out
to demystify artificial intelligence and machine learning techniques that are
common in everyday life, and in the news. At Pandata, we firmly believe that
the high level why (or why not), when, and how can and should be accessible for
Seemingly customized recommendations are everywhere these
days – Netflix is making suggestions of what to watch next, Amazon recommends
products, LinkedIn highlights potential contacts, Pandora delivers you music
you will probably enjoy. While sometimes these recommendations can seem off
base, in general, they are fairly accurate at reflecting our interests and
sometimes present a welcome surprise. What is the science behind these
recommendations? Companies are using a state-of-the-art data science technique,
Recommender Systems, to leverage large datasets to efficiently guide the
While there are many variations on a recommender system, on
a general level, they work by using existing information about behavior to
predict preferences of the clients or end users. One approach, referred to as “Content-Based
Filtering” identifies characteristics of the end product that the customer
engaged with and identifies similar products. For example, if you recently
purchased warm winter boots, the shopping site may recommend a similar winter
item such as wool socks. Under the hood, each item in inventory is
characterized by text description, a series of features, or other such
descriptors. Then, one (or multiple) algorithms can be used to identify which
items are most similar. Then, when the customer purchases an item, the site can
recommend products that are most similar. However, it can be problematic to
recommend something too similar. If I just bought a pair of winter boots, I
likely don’t need another pair.
Recommending similar items based on item characteristics can
be useful in some circumstances. However, by just identifying similar items,
one is missing out on a voluminous and rich data source – human behavior. A
technique called “Collaborative Filtering” uses information about what items
people like or interact with to predict what any given person may prefer. As a
very simplified example, Person A bought winter boots, and cold medicine.
Person B bought winter boots, cold medicine, and diapers. Based on patterns in
behavior from Person A and B, when Person C adds winter boots to their cart,
the site may suggest “people who bought winter boots also bought cold medicine”.
One way collaborative filtering is done using explicit ratings,
such as movie ratings (such as 4-stars). A popular algorithm takes all the
ratings across all customers to make a big table, or matrix. Then, the
algorithm uses matrix factorization as a mathematical way to represent
information about the users and items.
These representations can then be used to compute
theoretical ratings for items that an individual may not have seen before. In
the end, this manifests as “People who purchased winter boots also purchased
cold medicine.” This approach differs from the previous one in that the
recommended item is not suggested because it is necessarily similar at all to
the original item, but rather patterns of human behavior suggest commonality.
However, many of us don’t bother rating movies or products,
yet still receive solid recommendations. Explicit ratings are great, but rare,
therefore, it is also possible to use the Collaborative Filtering technique
using what is called “implicit ratings”. This term refers to the idea that you
can infer what a person thinks about a product based on their behavior – did
they click on it and spend time on the product site, did they buy it, did they
only watch the first 3 minutes or did they binge the first 3 seasons.
Mathematically transforming information about how a person interacts with a
product can serve as a stand-in for ratings, although with some assumptions
baked in – such as that they bought it, therefore they liked it. Or they
watched three seasons, not just fell asleep with auto play on. While these
assumptions may not always be accurate, with the large volumes of data common
in streaming media or e-commerce, clear trends still emerge.
While recommender systems can be very powerful, they are not
without potential pitfalls. A big risk is that by using like to recommend like,
suggestions fall into a silo. At best, this causes recommendations to be
boring. At worst, recommendations can reflect social bias or discrimination
present in the underlying dataset. Siloing can be due to limitations in content
based filtering or due to the predictability and stereotypy of human behavior.
People who watch one horror movie probably watch multiple horror movies. Pretty
soon, the algorithms are only recommending horror movies. Human bias can also
be reflected. For example, if you are using a recommender system to suggest
college classes, a recommender system without additional modification may recommend
engineering classes to male students and early childhood education classes to
female students – based on gender biased enrollment patterns. That said, there
are statistical and mathematical steps one can take to avoid pigeonholing. A
truly effective recommender system involves a component to identify and address
bias and siloing.
Next time you are wondering how Amazon knows what shoes you like,
or Netflix plans the perfect Friday evening, you have a recommender system to
thank. If there is an AI concept that you would like to see explained, contact firstname.lastname@example.org.