How Morningstar's Fund Ratings Have Performed

In this piece, we review the performance of Morningstar's different fund rating systems in the United States--the Morningstar Rating for funds, or the "star rating," the Morningstar Analyst Rating, and the Morningstar Quantitative Rating for funds. For further background on the fund ratings, see the appendix at the end of this article.

On balance, the three ratings systems have performed pretty well. The star rating appears to point investors toward funds that are more likely to survive and outperform, and away from those likelier to fail. The Analyst Rating and Quantitative Rating look to sort funds effectively based on future risk-adjusted returns, though that's less evident by some other measures.

The findings also suggest that there are opportunities for us to improve the ratings in certain respects. We explore these areas and offer some concluding thoughts on enhancements that we might make to build on the ratings' strengths and drive even better performance in the future.

The Morningstar Rating for Funds 
The star rating is a quantitative, backward-looking rating system that ranks funds based on their trailing three-, five-, and 10-year risk-adjusted returns versus peers in their Morningstar Category.

We use two approaches to assess the star rating's performance. The first approach is event-study. Under event-study, we freeze the ratings cohorts as of a given time and then observe the performance of the funds in those cohorts over a subsequent period, like 12, 36, or 60 months. We repeat that exercise for other starting dates and then average the event-horizon observations over the full time span we are measuring, which in this case is the period between July 1, 2002, and March 31, 2019. (The study encompassed more than 15,500 unique rated funds, including dead funds, and nearly 3.6 million measurements. None of the data is simulated or hypothetical.)

To summarize Exhibit 1, 4- and 5-star funds outperformed their typical category peer by a larger margin than 1- and 2-star funds did, on average. In fact, 1-star funds lagged their average peer. This held over the 12-, 36-, and 60-month horizons we examined.

Exhibit 1: Average Annual Event-Horizon Excess Return (vs. Category Average)

Source: Morningstar. All star-rated U.S. open-end funds and ETFs for rolling 12-, 36-, and 60-month event horizons ended between July 1, 2002, and March 31, 2019; includes dead funds and all share classes.

The second approach we use to assess the star rating's performance is the future-star-rating test. Under this method, we freeze-rate funds at monthly intervals and compare the funds' five-year star ratings that month to their five-year star ratings 60 months hence. This has the advantage of testing the rating's risk-adjusted performance in the 60 months before and after the date selected. (The previous event-study test did not measure risk-adjusted performance.)

From Exhibit 2 we can see that 1- and 2-star funds survived and succeeded (a 4- or 5-star rating over the subsequent period) about a third as often as 4- and 5-star funds, on average. Conversely, 4- and 5-star funds failed (died or had 1- or 2-star rating over the subsequent period) about half as often as 1- and 2-star funds.

Exhibit 2: Average % Breakdown of Future 5-Year Rating by Starting Rating 

Source: Morningstar. Oldest share class of all star-rated U.S. open-end funds and ETFs for rolling 60-month event horizons ended between July 1, 2002, and March 31, 2019; includes dead funds.

While the star rating isn't meant to be predictive, and the findings above don't convincingly prove that it is, the rating's performance is consistent with expectations we'd have for a starting point for research. It appears to tilt the odds slightly in investors' favor, as higher-rated funds stand a better chance of surviving and succeeding than lower-rated ones.

The Morningstar Analyst Rating 
The Analyst Rating is a qualitative, forward-looking rating that Morningstar's manager research analysts assign based on their assessment of People, Process, Parent, Performance, and Price. Unlike the star rating, the Analyst Rating aims to predict a fund's future risk-adjusted performance over a market cycle.

We test the performance of the Analyst Rating using the event-study procedure. The time period for the study was December 2011 through March 2019. (The study spanned more than 2,300 unique funds, including dead funds, and nearly 169,000 measurements. None of the data is simulated or hypothetical.)

To summarize Exhibit 3, higher-rated funds beat their typical peer by a larger margin than lower-rated funds, on average. This held across all three event horizons, with Gold-rated funds posting the highest average excess returns and Negative-rated funds the lowest. These results are encouraging, as they suggest the Analyst Rating has succeeded directionally in sorting funds by future performance, as measured by excess returns versus the category average.

Exhibit 3: Average Annual Event Horizon Excess Return (vs. Category Average) 

Source: Morningstar. All rated U.S. open-end funds for rolling 12-, 36-, and 60-month event horizons ended between Dec. 1, 2011, and March 31, 2019; includes dead funds and all share classes.

But Exhibit 3 also shows there's room for improvement. For instance, Negative-rated funds had positive excess returns versus their typical peer, contrary to our expectations. (This notwithstanding the fact that there were very few funds rated Negative, and thus a small number of measurements.) In addition, Bronze, Neutral, and Negative funds performed more or less alike over the 60-month event horizon, on average, despite the fact we had greater conviction in Bronze-rated funds.

We see similar trends when we measure ratings performance by excess return versus the category index, as shown in Exhibit 4. Gold- and Silver-rated funds lagged their index by smaller margins than lower-rated funds. That said, Bronze-rated funds didn't perform much better than Neutral-rated funds over the three event horizons, on average. This doesn't meet our expectations.

Exhibit 4: Average Annual Event-Horizon Excess Return (vs. Category Index) 

Source: Morningstar. All rated U.S. open-end funds for rolling 12-, 36-, and 60-month event horizons ended between Dec. 1, 2011, and March 31, 2019; includes dead funds and all share classes.

That said, the Analyst Rating aims to predict funds' future risk-adjusted performance. Given that, it makes sense to perform the same event-study test but this time using capital asset pricing model, or CAPM, alpha to measure risk-adjusted performance. As shown in Exhibit 5, the Analyst Rating performed better on this basis, with Gold- and Silver-rated funds outperforming their average peers by a substantially larger margin than Neutral-rated funds did. (While Negative-rated funds outperformed by this measure, that cohort was very small.)

Exhibit 5: Average Annual CAPM Alpha (vs. Category Average)

Source: Morningstar. All rated U.S. open-end equity, fixed-income, alternative, and allocation funds for rolling 12-, 36-, and 60-month event horizons ended between Dec. 1, 2011, and March 31, 2019; oldest share class only, includes dead funds.

Lastly, we conduct the same test but this time regress the returns of rated funds against the category index, as shown in Exhibit 6. Here we find that Gold-rated funds added alpha over each event horizon, on average, whereas lower-rated funds did not. (Negative ratings were again an exception over the 60-month horizon, but there were very few observations owing to the small number of funds in this cohort.)

Exhibit 6: Average Annual CAPM Alpha (vs. Category Index) 

Source: Morningstar. All rated U.S. open-end equity, fixed-income, alternative, and allocation funds for rolling 12-, 36-, and 60-month event horizons ended between Dec. 1, 2011, and March 31, 2019; oldest share class only, includes dead funds.

All told, the Analyst Rating's performance has been encouraging. Gold- and Silver-rated funds tended to outperform their average peer and index by a larger amount than Neutral-rated funds, both before and after accounting for risk.

That said, there's room for improvement. The average Gold- and Silver-rated fund didn't outperform its index over longer event horizons before adjusting for risk (Gold-rated funds did so after risk). Moreover, Bronze-rated funds sometimes performed indistinctly from lower-rated funds, before and after adjusting for risk.

The Morningstar Quantitative Rating for Funds
The Morningstar Quantitative Rating for funds is a forward-looking rating that uses algorithmic techniques to emulate the way analysts evaluate funds and assigns ratings to funds that Morningstar analysts don't actually cover. Funds are given Quantitative Ratings of Gold, Silver, Bronze, Neutral, or Negative. Like the Analyst Rating, the Quantitative Rating aims to predict funds' future risk-adjusted performance over a market cycle. We launched the Quantitative Rating in 2018.

We test the Quantitative Rating's performance using the event-study procedure. The time period for the study was Feb. 28, 2018, through March 31, 2019. (The study spanned more than 9,200 measurements, including dead funds. None of the data is simulated or hypothetical.)

The Quantitative Rating succeeded in sorting funds based on their future 12-month excess returns (measured against the category average), as shown in Exhibit 7. For each step higher in rating, the average fund produced a larger payoff versus the average fund in its category.

Exhibit 7: Average 12-Month Event-Horizon Excess Return (vs. Category Average) 

Source: Morningstar. All rated U.S. open-end funds for rolling 12-month event horizons ended between March 1, 2018, and March 31, 2019; includes dead funds and all share classes.

The same more or less held true when we measured versus the category index--higher-rated funds outperformed lower-rated ones. The difference was magnitude--none of the cohorts outperformed its index, on average, as seen in Exhibit 8.

Exhibit 8: Average 12-Month Event-Horizon Excess Return (vs. Category Index)

Source: Morningstar. All rated U.S. open-end funds for rolling 12-month event horizons ended between March 1, 2018, and March 31, 2019; includes dead funds and all share classes.

In summary, the Quantitative Rating appears to be doing a good job of distinguishing between funds, with higher-rated funds tending to outperform their average peer by a larger amount than lower-rated funds. But it hasn't shown the ability to identify funds that will outperform an index, as Gold-, Silver-, and Bronze-rated funds lagged their category indexes.

Opportunities for Improvement
Encouraged as we might be by the performance of our ratings systems, we're always seeking ways to improve. Performance tests like these help us identify areas that might warrant further attention and, if necessary, methodology enhancement. In that vein, these are some of the areas where we're intensifying our focus:

• Performance versus the index: The Analyst Ratings and Quantitative Ratings appear to have performed pretty well versus the category average; the average Gold- and Silver-rated fund outperformed its typical peer by more than the average Neutral-rated fund. But no Morningstar Medalist cohort surpassed its index, on average. When we adjusted for risk, Gold-rated funds beat their index, but Silver- and Bronze-rated funds didn't.
• Performance of Bronze-rated funds: We expect Bronze-rated funds to outperform Neutral- and Negative-rated funds, on average. While we observed this with the Quantitative Rating, it wasn't evident with the Analyst Rating.
• Performance of Negative-rated funds: Though this was a very small cohort with relatively few measurements, it was nonetheless striking to see Negative-rated funds perform as strongly as they did, contrary to our expectations.

Next Steps
Given this, as we consider potential future modifications to the Analyst Ratings and Quantitative Ratings, we'd likely emphasize changes that would create greater separation between ratings cohorts and improve performance versus the index, before and after risk. (We are not weighing changes to the star rating methodology at present.)

What might that look like? We'll see, but our analysis suggests the following changes could build on the ratings' strengths and drive even better performance.

• Recenter our assessment around its more predictive elements--People, Process, and Parent. Our current framework already incorporates these three pillars. In the future, we could give them even greater weight, essentially having them drive our estimate of what a fund could achieve for investors before fees.
• Redouble our focus on fees and tailor ratings based on cost differences. We already include a Price Pillar in our assessment framework, with that pillar rating figuring into the overall score that drives a fund's Analyst Rating. In the future, we could re-express price in a way that would make it count as least as much as the analyst's assessment of a fund's potential before fees. We could also vary ratings by share class, taking fee differences into account. (Today, we assign the same Analyst Rating to each fund share class, irrespective of price differences between them.)
• Set an even higher bar for active funds. Today, our methodology doesn't insist that an active fund must beat both a relevant category benchmark index and a category average to earn a medalist rating. In the future, we could do so, in effect setting the bar higher. We could also more explicitly and formally consider how successful active investing has been in each area when setting expectations for how much value a given strategy could add before fees.

All told, we think enhancements like these could potentially make the Analyst Ratings and Quantitative Ratings even more useful to investors, explaining why we'll give them all due consideration.

Conclusion
The Morningstar ratings systems for funds are working pretty well. But our analysis finds there could be opportunities to build upon their strengths to drive even better ratings performance, yielding better outcome to investors. We're weighing a few potential enhancements at the moment and will introduce them if and when it makes sense. 

Appendix: Reviewing Our Fund Ratings
The Morningstar Rating for funds is a quantitative, backward-looking rating system that ranks funds based on their trailing three-, five-, and 10-year risk-adjusted returns versus peers in their Morningstar Category. The top 10% of funds receive 5 stars, the next 22.5% get 4 stars, the middle 35% receive 3 stars, the next 22.5% 2 stars, and the bottom 10% get 1 star. Given that the star rating is based entirely on past performance, it is not meant to be predictive, but rather to serve as a "report card" on funds' performance. We launched the current version of the star rating methodology in 2002.

The Morningstar Analyst Rating is a qualitative, forward-looking rating system. Morningstar's manager-research analysts assign ratings to funds based on their assessment of five pillars--People, Process, Parent, Performance, and Price. Unlike the star rating, the Analyst Rating aims to predict funds' future risk-adjusted performance over a market cycle. Launched in 2011, the Analyst Rating denotes the analysts' conviction on a scale of Gold, Silver, Bronze, Neutral, and Negative.

The Morningstar Quantitative Rating for funds is a forward-looking rating that uses algorithmic techniques to emulate the way analysts assign ratings to the strategies they cover. The algorithm applies those learnings in assigning Quantitative Ratings to strategies that our analysts don't cover. The Quantitative Rating uses a scale of Gold, Silver, Bronze, Neutral, and Negative. We launched this rating in 2018. 

We talk more about how to put the Morningstar Quantitative Rating into practice in this video.