Recommender Systems: The Power of Personalization Q&A

For context, please make sure you've attended the webinar, presented by Joseph A. Konstan.

Q: Is a recommender system being used by Facebook to show us news feeds of people we visit more often? Or [to] "like" more things of certain people?
A: While I don't know the details of their algorithm, yes, Facebook does use recommender systems (as do other social network sites like LinkedIn). The recommendations are used in multiple places—not just showing news, but also in recommending potential friends.

Q: How can we solve the cold start problem, when item ratings or user's profile are not available ? Which items can we recommend?
A: Cold start is a big problem, and it is addressed a number of different ways. In many systems that depend mostly on implicit ratings (e.g., purchase data), it is possible to simply ignore the problem.  Eventually, the new users sees enough products or the new product is seen by enough users. In other systems there are back-up content-based recommendations (e.g., recommend by genre), and even back-up popularity and demographic recommenders for new users. New items can be recommended by content attribute, or some systems simply recommend them to a small set of well-chosen users (chosen for their loyalty—not people who will disappear because of one bad recommendation) and perhaps for their depth of rating profile (so they connect to lots of other people).

Q: What open source tools exist? Are they easily integratable into any web site?
A: I mentioned a number of tools in the talk. Take a look at LensKit, MyMedia, or Mahout as a good starting point. Lots of people have integrated each into websites.

Q: Do you think that while recommending items to users, it is more interesting to show them items which they may like because of their profile, or is it more interesting to recommend items, which they may also like to explore?
A: This depends completely on the goals of your application. If you're running a recommender in a supermarket, it is easy to recommend likely "likes," but often of low value (people buy bananas and bread anyway). The goal is to find something new and interesting that might become a new like. Similarly, for music streaming, many people want to get introduced to some new music. On the other hand, at point of sale, it may be much more important to get a quick-yes add-on purchase.

Q: How to efficiently handle missing data?
A: Recommenders are entirely about missing data. The user-user and item-item methods are designed for sparse ratings matrices. For techniques like SVD, one common approach is to fill cells with a prediction of the cell value, or with the row-average/column-average combination.

Q: I'm curious what the phrase "deniability of preferences" meant on that previous slide near privacy.
A: Let me explain through an example. Suppose you're at a bookstore and the recommender suddenly suggests a pornographic magazine (while other people see it). You'd like the ability to have a credible denial that says—hey I've never told it I liked porn, and yet it is feasible that it recommended it anyway.  Or, if profiles are visible, you may want the ability to disclaim any particular rating (perhaps because the system added noise).

Q: Does time spent on a page count into calculating which pages I'm interested in?
A: It can. In early work, we found that time spent reading was a good surrogate for interest, up to about 5-minutes in Usenet news. After 5 minutes, we had to throw out the data point, because it was clear the user had gone onto something else. That's the same problem with the web—a minute on a page is probably good; 15 minutes is probably leaving to go read e-mail or check Facebook.

Q: Could you please tell me which is the best/latest survey paper related to RS?
A: There are several, depending on what your interest is. The Ekstrand et al. survey in Foundations and Trends in HCI is pretty good (disclaimer—I'm a co-author) on the technical side. I also like the two books I mentioned on the slides. All of these are pretty current.

Q: In the absence of explicit rating, what are the options for getting implicit rating, mainly in the case of e-commerce application?
A: Lots of them. Purchases, placing in market basket, re-visiting an item, time spent looking at an item.  Negative ratings associated with returns, very short time spent looking, perhaps skipping over an item to lower-ranked search results.

Q: How good is collaborative filtering in predicting ratings at the long tail region?
A: It varies, but a lot of work has been done on "coverage" metrics—metrics of how much of a catalog actually can be recommended—and on higher-serendipity/less-popular recommendations. In general, results have been pretty good.

Q: What is different between a recommender system and a rule-based system which groups together related things (genres, etc.) from a database and shows it to you?
A: There's certainly a continuum. A good catalog or search may seem like a recommender to some.  But I don't call it a recommender until it has some model of either user preferences, product associations, or some other similar data on which to build situational or collaborative suggestions.

Q: Are these matrices exportable and analyzable by the company owning the website?
A: In general, yes. That can be a big asset to the company. It is also the reason many researchers have worked on privacy-protecting recommenders (though few have been deployed beyond a research experiment)

Q: When considering item-item similarity and user-user similarity, which should we try to calculate first?
A: It depends. If the matrix is small, start with user-user. It tends to provide more interesting recommendations. If the matrix is large and the number of items is 10% of the number of users (or smaller), I'd probably go for item-item. Of course, with today's software tools, you can try both and compare them.

Q: Are these matrices customizable by the website owner [to] help serve their target audience?
A: Usually operators don't customize the matrix; instead they layer in business rules. Such rules can be simple (e.g., don't recommend anything out of stock, or anything on sale for less than 150% of cost). Or they can be less simple and involve tuning the recommender (e.g., tweak up or down an adjustment for how much to devalue popular items when recommending).

Q: Can you clarify item-item recommendation: If I rate a lamp with 5 and a fish 5 too, will the user buying a lamp get a recommendation for a fish?
A: If most people who rate the lamp high also rate the fish high (and if most people who rate the lamp low also rate the fish low), then when a new customer rates the lamp high, that customer is more likely to get a recommendation for the fish. Or, to put it more simply (and realistically), if people who buy low-fat cottage cheese also tend to buy skim milk, then a person who's bought one may be recommended the other. But if you're buying low-fat cottage cheese along with butter, bacon, and lard, the negative association with those three may knock skim milk off your recommendation list.

Q: Is there any way to reuse/readapt the recommendation information from one site to another? I mean, has somebody studied how to adapt it from one site to another (provided that there is any connection between the products from both sites)?
A: People have taken preference data from one site and used in in others (in fact, that's how we started MovieLens, with almost 3 million movie ratings from a different site). There are also examples of confederated sites that share a common profile.

Q: How can we introduce diversity in recommendations, to avoid providing the user with items that are too similar?
A: If you have a metric for assessing similarity, one of the simplest ways to diversify is to step further down the top-N list (throwing out items too close the the already-selected ones).  You can find an example of this at

Q: Why didn't Netflix implement the winner of the Netflix challenge?
A: (a) too complex an algorithm for the incremental value; (b) the main goal of Netflix isn't the same as the metric for the prize. They're much more concerned about good top-N recommendations and good recommendation coverage of their "long tail" of movies; the prize was based purely on accurate predictions. (c) ask them!

Q: Can you recommend some open sets of data to try and implement recommender systems on and practice using and implementing recommender systems?
A: Sure, there are a bunch of them out there, including MovieLens data, Jester data, data, Yahoo! Movie ratings, and more. Feel free to start at

Q: How do you validate that your recommender model works as intended? (i.e. there is a correlation between sales and the recommendations)
A: In the research world, we have a variety of metrics (there's a good summary of these at In the commercial world, they commonly do A/B testing and measure lift (the increase in sales), as well as longer-term measures of customer value (repeat business, etc.).

Q: Do you know of applications where you can get some tips about who your "peers" are? Or it will be an invasion of privacy?
A: Lots of social network sites have recommenders, including LinkedIn and ResearchGate.

Q: 10 years from now, where do you envision recommendation systems?
A: I think they'll be everywhere. But take a look at a "look back / look forward" article John Riedl and I recently wrote:

Q: What tools and languages are currently used for implementing Recommender Systems?
A: All over the map, from C#, C++, and Java to Mathematica to scripting languages.

Q: You indicated that LensKit was not commercial grade. Do you have a recommendation for an Open Source Commercial Grade tool?
A: LensKit is probably as close as it gets among open source tools.

Q: How do you support serendipitous browsing in a world filled with recommendations based on things you are already aware of through your interests?
A: I've seen a number of really nice ways to do this. You can deliberately include recommendations with very little support (e.g., one person's opinion). Some recommenders mix in a few random or just popular items you haven't seen. Some have "content" organizations that encourage serendipity (like a library's "browse by call number" feature).

Q: Do you know of successful applications of recommender systems for scientists "shopping" the literature for evidence related to their specific scientific questions?
A: Not specifically. There are experiments that have been successful in recommending research papers (but not for that purpose), and a number of systems to recommend experts or collaborators.

Q: Could you explain horting a bit more?
A: Horting is a graph theory-based approach described in detail in this article:

Q: Can we use recommender system for buying cars and vehicles?
A: You can, though given how infrequently most people buy cars, those recommenders tend to be content-based and more interview-based than built out of long-term product preferences.

Q: Is there any MSc [degree] or courses I could take in recommender sytems (especially distance learning)?
A: There are a few courses, but so far I don't know of one through distance learning. But wait awhile, and I expect we'll have one sometime about a year from now.

Q: Does personalization mean one needs to understand the culture, ethnicity of the people as many times this is very much dependent? Would that not mean intruding [on] privacy in some means?
A: Sometimes yes, sometimes no. If you have the population well-defined, you may be able to work simply from correlations in preferences.  But certainly there are plenty of marketers who include extensive profiling.

Q: You talked about metrics for computing the closeness of users/items. What are the most used metrics? (e.g., cosine similarity)
A: Pearson correlation and cosine similarity are the most commonly used.

Q: Have there been any effective uses of social network data in recommendation systems?
A: Lots of examples. Both specifically in recommending people (for example, see and in using the social graph to propagate product recommendations.

Q: Have there been any effective uses of social network data in recommendation systems?
A: Lots of examples. Both specifically in recommending people (for example, see and in using the social graph to propagate product recommendations.

Q: Is there any sample project available using LensKit? (Since LensKit is not well documented now.)
A: More is coming (including a lot more documentation).

Q: Should we consider the negative values as item similarities when adjusted cosine is used?
A: It isn't about whether adjusted cosine is used, but about whether you have reason to believe that negative values have real meaning. That's a domain and application-specific question.

Q: Recommending with social connection is very powerful. Telling the user what item his friends prefer is very effective. Do you think it would replace CF algorithms?
A: I think it has a role, but it won't replace CF algorithms. The problem with social connections is twofold. First, it is harder to aggregate. It isn't as meaningful to say "Hey, 20% of your friends like that song" when I don't know if those are my older relatives vs. work friends vs. hang-out friends. Second, there are always domains where your friends may just not be enough like you. Or where you don't have enough friends to cover the product space.

Q: Between precision/recall and RMSE, which could be a better measure of recommender system quality?
A: Depends on what you're trying to optimize. RMSE is a lousy way to tell whether the top-ten items you pick are any good. Top-N precision (or recall—the same for top-N) is better. But both have the problem of rewarding a "ground truth" that is usually biased towards things the user already knew. In an ideal world, you evaluate by deploying the recommender and measuring the success of the recommendations it makes.

Q: Can you please give one example for group recommendations?
A: Sure. Here are a couple:;

Q: What are the most useful strategies to combine profiles from multiple people for group recommendation?
A: A number of people have studied this problem. One clear result is that it is better to recommend for each and then combine the recommendations rather then combining the profiles into one "mega-profile." There are different combination functions that weigh how much you're concerned about average happiness, happiness relative to the user's mean, avoiding misery, etc.

Q: Any threats to the recommender systems from malicious users? Are there any security issues for the current recommender system? How to identify the trusted users and malicious users? How to protect the system from the malicious users and reviews?
A: Absolutely, there is lots of work on shilling and other forms of attack. Beyond the scope of this talk, but here are two pointers:;

Q: Is it possible to get the references to the recent conference that the presenter mentioned during the Q & A?
A: ACM Recommender Systems 2012.



Due to the volume of questions received during the September 20, 2012 ACM Learning Webinar, not all audience questions were asked during the live event. Presenter Joseph A. Konstan answered the questions listed here offline.


One's mind, once stretched by a new idea, never regains its original dimensions.


Safari Books Online                         Books 24x7