The Star Rating Performs Well but Our Picks Are Better

This article originally appeared in Morningstar FundInvestor.

In our latest evaluation of the Morningstar Rating for funds, we find that it continues to be modestly successful in forecasting performance.

The star-rating methodology was reconstituted in the summer of 2002, and each year since then we have produced a study examining total returns, relative performance, and risk-adjusted performance at the end of June. Then we look at how it did at predicting those factors over the ensuing years so that we can understand how investors using the rating to aid their decisions would have fared.

How the Rating Works
It's important to note that the star rating is a purely quantitative measure, updated every month, that's based upon past returns and volatility. Specifically, it measures load-adjusted total returns from the past three-, five-, and 10-year periods. Then it adjusts for risk so that high-risk funds are taken down a peg or two, and low-risk funds are moved up a bit. The risk-adjusted measure is then graded on a curve within a category. We give 5 stars to 10% of the funds, 22.5% earn 4 stars, 35% take the 3-star rating, 22.5% take 2 stars, and 10% earn 1 star.

You'll notice many important things aren't included in the rating, such as our analysts' opinion, the manager's record, and expense ratios. The rating is a handy measure of past risk-adjusted performance, but it is not a recommendation and it certainly is not the whole story. Rather, it's a handy way to narrow your search for a fund and a way to keep tabs on the funds you own. The data in our study show why.

How the Rating Performed
In the aggregate, 5-star funds beat 4-star funds, and 4 beat 3, and so on. (See table.) This is true of returns, ensuing star ratings, and batting average. For example, using ratings of domestic-equity funds from 2003, 5-star funds returned an average of 9.90% annualized over the next five years compared with 9.20% for 4-star funds, 9.04% for 3-star funds, 9.22% for 2-star funds, and 8.70% for 1-star funds. The gap between each star rating group is hardly the Grand Canyon, but by the time you get out to 1-star funds, which lag the 5-star group by 120 basis points per year, that's a pretty big deal. The gap for the class of 2003 was smallest for international equity (63 basis points) and largest for balanced (313 basis points).

For 5-year star ratings over the next five years, 5-star funds earned an average of 3.07 stars compared with 2.73 for 1-star funds--again a modest differential.

The same is true for batting averages--a measure of what percent of funds beat their peer group averages, though in our example of domestic-equity ratings from 2003, there was a surprise. We saw 54% of 5-star funds beat their peers over the next five years. That figure declined to 42% for 3-star funds, but then picked up slightly for 2-star funds to 44%, and 1-star funds had a nifty 53% batting average. What happened? Extinction. Many 1- and 2-star funds were liquidated over the five years so they didn't show up to bring the batting average down. However, if you instead ask what percentage of funds survived and beat their peer group, the figures make more sense: 49% of 5-star funds met that hurdle, while 43% of 4-star funds did, down to 33%, 30%, and 36% for the bottom three rating groups.

How the Rating Performed Combined with a Cost Screen
I can think of a number of things that might improve the star rating's predictive power, but the one I'm certain of is factoring in expense ratios. Slicing the 5-star groups to include just those funds in the cheapest quartile of their category led to higher three- and five-year star ratings over the next three or five years 80% of the time. Factoring out the survivorship bias, the success ratio was improved 90% of the time when the expense screen was added in. In other words, if you buy funds that have the cheapest-quartile costs and 5 stars, you have a much greater chance of getting a mutual fund that will survive and outperform its peers.

Why would this be the case when expense ratios are indirectly factored into the star rating because it uses after-expense returns? There are a couple of reasons. First, the star rating requires only three years' worth of performance, yet an equity fund is something you should hold for 10 to 20 years. Thus, the negative compounding effect of expense ratios is understated. Over time, the difference between a 0.60% expense ratio and a 1.25% expense ratio can lead to a very big sum after 20 years, but not so much for three years. In addition, expense ratios change over time, and the fund's most current expense ratio, rather than past expense ratios, is the best indicator of where it will be in the future. For example,  Primecap Odyssey Aggressive Growth  (POAGX) started out with a 1.25% expense ratio, but now it's down to 0.78%.

Star Ratings Versus Fund Analyst Picks
Our  Fund Analyst Picks are forward-looking fundamental-driven selections whereas the star rating is a backward-looking performance-driven measure; yet it's natural to want to see how the two perform versus one another. To do that, I looked at our picks from June 2003 and then looked to see how they performed over the ensuing five years.

Score one for the humans! Our Fund Analyst Picks topped 5-star funds in all asset classes. In domestic equity, the picks enjoyed a success ratio of 64% versus 49% for 5-star funds. In international equity it was 66% for picks and 54% for 5-star funds. In balanced, the tally was 100% for picks versus 59% for 5 stars. In taxable-bond, picks won 81% to 70%, and in munis it was 100% to 78%.

How about if I limited our picks to just the cheapest quartile? Performance gets even better and is also better than that of low-cost 5-star funds. Although our picks have lower costs than 5-star funds and the typical choice of fund investors, it's clear that we, too, should be more disciplined in our selection. Except for the two asset classes where we already had 100% success rates, our cheaper picks fared better than our pricier ones. Among domestic-stock fund picks, our success ratio popped from 64% to 79% when limited to the cheapest quartile. (The picks from outside the cheapest quartile actually had success rates below 50%.) In international picks, the improvement was marginal while in taxable bonds it went from 82% to 91%, and once more our picks from outside the cheapest quartile had a success rate below 50%.

Conclusion
The star rating isn't a complete solution but rather an aid that helps you to narrow the field and improve your chances for success. Our Fund Analyst Picks do an even better job of that than the star rating, and because there are fewer picks than 5-star funds, it also narrows the list to fewer choices.

In addition, it's more apparent than ever that low costs are another vital piece. No matter how you evaluate mutual funds, you need to incorporate costs into the equation.

		FundInvestor Newsletter
	Morningstar FundInvestor is a 48-page newsletter dedicated to helping investors pick great mutual funds, build winning portfolios, and monitor their funds for greater		gains. This one-year subscription consists of 12 monthly issues, a Reader's Guide, and four free investing reports. Learn more.
	$115.00 for 12 Print Issues		$95.00 for 12 PDF Issues