Rating Morningstar’s Fund Ratings
Morningstar’s fund ratings have been effective in sorting funds based on future performance, though opportunities for improvement remain.
We recently conducted a study analyzing the performance of our fund rating systems. Those ratings systems include the Morningstar Rating for Funds (that is, the “star rating”), the Morningstar Analyst Rating, the Morningstar Quantitative Rating for funds, and the Morningstar Sustainability Rating for funds.
We found that, on balance, these ratings have done a good job of sorting funds based on future performance. Higher-rated funds were more likely to survive and outperform over subsequent horizons, while lower-rated funds were likelier to lag or go out of business. While this is encouraging, there are also opportunities for improvement.
An Overview of the Fund Ratings
Below is a table that summarizes the key attributes of the four fund rating systems that we evaluated.
- source: Morningstar Analysts
While the ratings share similarities, there are differences to note. The star rating is backward-looking, whereas the Analyst Rating and Quantitative Rating are forward-looking; the star rating and Quantitative Rating are quantitative in nature, while the other rating systems turn on analysts’ qualitative judgments; and the Sustainability Rating’s primary goal is to denote how much material unmanaged ESG risk funds court, not to predict their future performance.
Nevertheless, we know investors tend to use these rating systems to achieve better financial outcomes. Accordingly, we assessed whether the ratings tended to sort funds based on their future performance.
How We Analyzed Ratings Performance
We measured ratings performance in a few ways. We examined the trailing returns of U.S. funds and exchange-traded funds in each ratings cohort through Feb. 28, 2022. We also ran an “event study” in which we took monthly snapshots of the ratings since their inception, tracked performance of the rated funds over a subsequent event horizon, and then averaged the event horizon returns.
To control for stylistic differences, we compared rated funds’ returns to a relevant peer group average or to the benchmark index that corresponds to the category to which the funds had been assigned. In addition, in a few tests, we evaluated risk-adjusted returns by calculating rated funds’ capital asset pricing model alphas versus the relevant Morningstar Category average or the assigned category index.
The study utilized rated funds’ net-of-fee returns. We adjusted to control for multiple share classes (so that those funds didn’t have undue weight in the calculations) and also included dead funds whenever possible.
What We Found
In general, we found that the star rating, Morningstar Analyst Rating, and Morningstar Quantitative Rating were effective in sorting funds based on their future performance. Higher-rated funds were likelier to survive and outperform than lower-rated funds, though this was more pronounced when measured against the category average than against the costless benchmark index. The Sustainability Rating’s performance was mixed, but that was measured over a relatively short time frame.
We review each rating system’s performance in the sections below.
Morningstar Rating ("star rating") In our analysis, we found that the star rating effectively sorted funds based on their future performance, with higher-rated funds far likelier to survive and outperform average peers than lower-rated funds. That's evident in the chart below, which shows the trailing average annual excess returns of funds in each ratings cohort versus their average peer through Feb. 28, 2022.
In summary, though the star rating has struggled recently, it did a good job of sorting funds over the trailing three-, five-, and 10-year periods.
The same pattern held when we measured rated funds against their assigned benchmark indexes, as shown below.
True, not even the highest-rated cohorts were able to generate positive average annual excess returns versus their assigned indexes, reflecting the challenge of topping a costless benchmark. All the same, high-rated funds lagged their indexes by significantly smaller margins than low-rated funds.
These average annual excess returns do not, however, incorporate the results of funds that started but didn’t survive to the end of the trailing periods shown. To derive a sense of how well the star rating did in predicting the odds of success—which we defined as surviving to the end of a trailing period and outperforming a relevant peer-group average—we calculated success rates for each cohort.
In summary, higher-rated funds were 2 to 3 times more likely to succeed than lower-rated funds. And a big reason why they succeeded more often is because they survived while lower-rated funds died in droves, as shown in the death rate chart below. (We treat fund deaths as failures for purposes of calculating the success rates shown in the chart above.)
Finally, to assess the rating’s success at sorting funds based on their future risk-adjusted performance, we compared funds' starting and ending star ratings. Given that the star rating is based on funds’ risk-adjusted performance, it stood to reason that this comparison would yield a sense of whether the rating was just as successful at sorting funds when risk is taken into consideration.
In summary, funds that had 5-star ratings as of Feb. 28, 2012, were almost 4 times likelier to have a 4- or 5-star rating by Feb. 28, 2022, than funds that started with a 1-star rating. In fact, nearly 80% of funds that started with a 1-star rating finished the 10-year period with a 1- or 2-star rating or died before reaching the end.
Morningstar Analyst Rating We also found that Analyst Ratings have effectively sorted funds based on future performance. Higher-rated funds were likelier to survive and outperform, while lower-rated funds were more likely to lag or die, though this effect was less strong when measured against the category index.
In evaluating the Analyst Rating, it is worth keeping in mind that we assign Analyst Ratings to funds that Morningstar’s manager research analysts cover. Our research has found that covered funds tend to be larger by assets, possess longer track records, and boast lower expenses, on average, when compared with funds not under analyst coverage. They have also been higher performing, on average.
This is evident when we examine the trailing average annual excess returns of Analyst Rated funds through Feb. 28, 2022. We found the Analyst Ratings generally did a good job of sorting funds based on future performance, with higher-rated funds tending to outperform their average peer by a larger margin than lower-rated funds. But even lower-rated funds outperformed, partly reflecting the fact that covered funds are cheaper and higher performing, on average.
In reviewing the above, note that the Negative cohort contained relatively few funds on March 1, 2019, March 1, 2017, and March 1, 2012, for the trailing three-, five-, and 10-year periods, respectively. When taken together with funds assigned a Neutral rating, nonmedalists underperformed Morningstar Medalists over these trailing periods. Also note that as of March 1, 2012, manager research analysts had assigned Analyst Ratings to only a small portion of the coverage universe, as the Analyst Ratings were rolled out gradually from late 2011 through late 2012.
We can quiet some of the noise in the trailing average annual excess return data by examining the performance of the Analyst Ratings using the event-study procedure. This technique is less sensitive to starting and end point than trailing returns, as it takes monthly snapshots of rated funds, measures their performance over a subsequent event horizon (12, 36, or 60 months), and averages those measurements to give a fuller picture, as shown below.
Under the event-study approach, it is clearer that the Analyst Ratings have been effective at sorting funds. Higher-rated funds notched higher average excess returns than their typical peers compared with how lower-rated funds did, on average. Although the magnitude differed, this pattern held when we measured the performance of rated funds against their assigned category benchmark index, as shown below.
We also evaluated the risk-adjusted performance of funds assigned Analyst Ratings. In this analysis, we calculated CAPM alpha success ratios—that is, the percentage of rated funds that started a period, survived to the end, and generated positive alpha versus their assigned category index—across the ratings cohorts. We found that medalists succeeded about one-and-a-half times as often over the trailing periods ended Feb. 28, 2022.
(Note: The 10-year measurement included only those funds assigned ratings as of March 1, 2012, which was only a portion of the then-current coverage universe.)
Morningstar Quantitative Rating The Morningstar Quantitative Rating for funds uses algorithmic techniques to infer the rating a manager research analyst would assign to a fund if an analyst covered it. It does so by studying relationships between analysts' ratings-assignment tendencies and the characteristics of the funds they're assigning those ratings to. By applying those learnings to the universe of funds that analysts don't cover, the Morningstar Quantitative Rating can assign ratings in a way that emulates an analyst.
(Note that funds are eligible for an Analyst Rating or a Quantitative Rating, but not both. We assign Analyst Ratings to funds that manager research analysts cover and Quantitative Ratings to those they don’t cover.)
Although the Quantitative Rating is still relatively new, having launched only in 2017, the early results have been encouraging. It has done a good job of sorting funds based on their future performance, as shown in the chart below.
Although performance has been weaker recently, this pattern held even when we tallied funds’ average excess returns versus their assigned benchmark index, with higher-rated funds lagging by less than lower-rated funds.
Funds not covered by analysts tend to be smaller, less tenured, and costlier, on average. Given that, they die more often than covered funds, making it worthwhile to examine trailing success rates, which take dead funds into account (that is, funds that existed at the beginning of a trailing period but not at the end are treated as failures). As shown in the chart below, higher-rated funds succeeded more often than lower-rated funds, in part because they were much likelier to survive.
This also held true when we examined the risk-adjusted performance of funds assigned Quantitative Ratings. Medalists generated positive CAPM alpha (versus their assigned category index) more often than nonmedalists.
It is worth noting that one of the reasons the Analyst Rating and Quantitative Rating have sorted well based on future performance is because fees play an important role in each methodology. The costlier a fund, the lower its rating is likely to be, and vice versa. We consider this a feature, not a bug, of each rating system, as our research has found fees to be one of the most predictive variables of all.
Morningstar Sustainability Rating Lastly, we assessed the performance of the Morningstar Sustainability Rating. Unlike the other ratings systems, the Sustainability Rating's primary objective isn't to sort funds based on future performance. Rather, it aims to denote how much material unmanaged ESG risk a fund's holdings are courting when compared with other funds like it.
The Sustainability Rating is still relatively new—the current methodology was rolled out in 2019. For that reason, it is difficult to draw firm conclusions on how well the Sustainability Rating has sorted funds based on future performance. Nevertheless, it is worth assessing its success thus far.
Although the Sustainability Rating has sorted well recently, higher-rated (that is, lower ESG risk) funds have not outperformed their average peer by larger margins than lower-rated funds since inception. That said, there’s not evidence that investors who have sought to minimize ESG risks have incurred a large performance penalty, as performance was fairly similar across the ratings range.
- source: Morningstar Analysts
This picture changes when we limit the universe to “intentional” ESG funds. Intentional funds, unlike nonintentional, clearly prioritize environmental, social, and governance-related investing, typically by putting “ESG” in the fund’s name and making it a central part of the strategy’s mandate. When we evaluated the performance of intentional funds, we found that they’ve sorted future performance pretty well since inception, if less so recently.
We also examined how well U.S. stock funds’ ESG risk scores sorted future performance. To review, a fund’s Sustainability Rating is based on how its ESG risk score compares with that of others in its global category. The lower a fund’s ESG risk score versus its category peers, the higher the rating. But to those investors seeking to minimize ESG risk in absolute terms, the ESG risk score is more relevant.
Given that, we grouped U.S. stock funds by their ESG risk scores from least risky to riskiest and then averaged the trailing returns of the funds in each cohort. What we found is that less ESG-risky U.S. stock funds had slightly outperformed riskier U.S. stock funds since the Sustainability Rating’s inception.
Opportunities for Improvement
These findings are broadly encouraging, but we’d be remiss if we didn’t also address some opportunities for improvement:
- The star rating doesn't "corner" well
While the star rating has been effective in sorting funds based on future performance over the long haul, it has struggled lately, with lower-rated funds outperforming higher-rated funds. The star rating tends to sort the best when markets are trending higher but has been prone to falter amid transitions, when market leadership changes. This is a limitation of a rating system that is based entirely on past performance, explaining why we consider the star rating a starting point for research, not an all-in selection tool.
- Medalists haven't beaten their benchmarks
The forward-looking Analyst Rating and Quantitative Rating have generally been successful sorting funds based on future performance versus a category average, with higher-rated funds succeeding more often than lower-rated funds, on average. Nevertheless, Gold-, Silver-, and Bronze-rated funds have not outperformed their assigned benchmark indexes, on average. This leaves room for improvement, as these ratings systems aspire not just to sort well based on future performance but also to reserve the highest ratings for active funds that beat their market indexes over a full market cycle. So far, they haven’t met that mark.
- Confusion regarding ESG and performance
There is also an opportunity to further refine the way we present the Sustainability Rating to investors, especially to those who might view it as a predictor of future performance. While it is true that the Sustainability Rating doesn’t purport to predict how funds will perform in the future, there has been considerable debate about the relationship between material unmanaged ESG risk (which the rating denotes) and future returns. Some believe that minimizing this risk can lead to superior returns. But the data—albeit limited to the Sustainability Rating’s brief life—is mixed. This is encouraging insofar as it indicates that committed ESG investors aren’t incurring a big performance penalty. But it also raises questions about whether minimizing ESG risk will pay off in the form of above-average returns.
We found that the fund ratings have done a good job of sorting funds based on future performance. Higher-rated funds were more likely to survive and outperform over subsequent horizons, while lower-rated funds were likelier to lag or go out of business. This held especially true for the star rating and the forward-looking Analyst Rating and Quantitative Rating, if slightly less so for the Sustainability Rating (whose primary goal is to denote funds’ material unmanaged ESG risk).
Correction: Ex. 16 has been updated to label the globes correctly. The article was updated to remove extraneous text.
The author or authors do not own shares in any securities mentioned in this article. Find out about Morningstar’s editorial policies.