Morningstar Star Ratings: Do They or Don’t They Predict?

Ad Age ranks the “Does She or Doesn’t She” advertising campaign as the eighth best in history, right below “Tastes Great. Less Filling.” and “We Try Harder.” The “Does She or Doesn’t She” campaign involves a hair color product. Its effect looked so natural, when asked if a particular middle-aged woman used it, the announcer would state “Does she or doesn’t she? Only her hairdresser knows for sure.”

In a similar fashion, ever since we published “Have Mutual Fund Rating Agencies Lost Their Mojo?” (FiduciaryNews.com, January 14, 2013), we’ve been confronted with the same sort of question concerning Morningstar’s Star Ratings: Do they predict performance or don’t they? Most of the advisers we spoke with see Morningstar’s ratings as historical evaluations, not predictive devices. Morningstar, the proverbial hairdresser in this sense, appears to delight in leaving the answer rather ambiguous.

It wasn’t always this way. In 1996, an academic study showed Morningstar’s Star Ratings had no predictive value. At the time, Morningstar agreed with those results. This study seems to be what has prompted advisers to believe Morningstar’s ratings are not predictive. Unfortunately, Morningstar changed the way it calculates its star rating, making this 1996 obsolete.

Still, in the nearly two decades since that original study, Morningstar hasn’t deviated too far from its original response to the 1996 study. In the December 6, 1996 issue of Morningstar Mutual Funds, Amy C. Arnott said, “Many commentators insist on treating the star rating as a predictive measure or a short-term trading signal. The rating, which is clearly labeled as a historical profile, does neither.” Even though Morningstar changed their methodology in 2002, Christine Benz in her 2005 book Morningstar’s Guide to Mutual Funds, said, “It’s also worth noting that the star rating is based on how the fund did in the past. It won’t predict short-term winners.” Even Don Phillips, in his 2010 article “Star Wars, the Sequel,” said, “The Morningstar Rating for funds is a grade on past performance. Period. No one at Morningstar ever claimed that the stars have predictive power.”

Our “Lost Mojo” article was consistent with this view. We even quoted an adviser, referencing the Phillips piece, who said, “Morningstar was honest enough to state exactly that in a recently published article on their site where they pointed out that their proprietary star rating system is not a valid indicator of future investment performance or even relative performance.”

But now Morningstar’s tune might be changing. Apparently, the “Lost Mojo” quote didn’t sit too well with the folks in Chicago. Within hours of publication, Morningstar contacted FiduciaryNews.com. Being the fair and balanced media outlet we strive to be, we offered Morningstar a chance to have their say – as long as we got to do our standard reporting due diligence. They graciously agreed and, in fact, encouraged us to gather contrary opinions.

A spokesman for Morningstar told us, “We have always been very careful to make sure people understand that the rating is a purely quantitative measure of past, risk-adjusted performance and should not be used as the sole decision for investing in a mutual fund. That said, after we changed our methodology in 2002 and had enough history to run fund performance analysis, we did find some moderate predictive value—particularly on a relative basis. For example, from our April 2011 study we found that a domestic-stock fund rated 5 stars was 30% more likely to succeed than average and 2.2 times more likely to succeed than a 1-star fund.”

Edwin Choi, however, analyzed a similar Morningstar study from 2010 and concluded “Morningstar’s algorithm is horrible.” In his article, “Morningstar Ratings: Useful or Useless?” (Get Rich Slowly, October 20, 2010), Choi uses Morningstar’s own study to show the expense ratio was a better predictor of outcomes than the Morningstar Rating. The Phillips article may have been a response to articles like Choi’s. It clearly tells investors not to use Morningstar ratings to predict future returns. Practically for investors that means, don’t use star ratings to select funds for investment. But, in the end, the time period studied was probably too short to make it statistically significant. Mr. Choi did not agree to be quoted for this article, but his blog post is available at the above link.

Enter two Pace University professors – Matthew R. Morey and Aron A. Gottesman – who, in January 2006, published an independent study titled “Morningstar Mutual Fund Ratings Redux.” The researchers ran two separate studies on the Morningstar Rating – one before we changed the rating methodology and one after. The first study agreed with the 1996 study and concluded the Morningstar Rating did not have predictive ability. The second study reported, “…we find widespread support for the notion that the new Morningstar rating system can predict future performance…”

That being said, Morningstar own licensing agreement, consistent with the DALBAR SEC-No-Action Letter, after sketching out how the ratings are calculated, states “Past performance is no guarantee of future results.”

We’ll give Morningstar the last word on whether it believes the stars are predictive. Their spokesman said: “That depends on your interpretation of the word ‘predictive’ – [does it mean] guaranteed or associated with good outcomes? We believe the star ratings are associated with good outcomes, but we’ve always said we don’t want people buying funds simply because they have 5 stars. Morningstar’s star rating doesn’t ‘guarantee’ any results, but because high ratings are given to funds with low cost, low risk, and long manager tenure, they tend to highlight funds with a higher likelihood of better performance.”

So do they or don’t they?

Only their hairdresser knows for sure.

Interested in learning more about this and other important topics confronting 401k fiduciaries? Explore Mr. Carosa’s new book 401(k) Fiduciary Solutions and discover how to solve those hidden traps that often pop up in 401k plans. The book also contains a series of chapters on this subject, including how to create an investment policy statement that defines a set of menu options consistent with the “one portfolio” concept (as well as leaving room for those few remaining do-it-yourselfers).