Efficacy of the Morningstar Quantitative Ratings

The Morningstar Quantitative Rating for managed investments has delivered solid performance since its launch in June 2017. Strategies with Quantitative Ratings of Gold, Silver, or Bronze have outperformed their Morningstar Category averages. Along with the outperformance, the Quantitative Rating model has proved to be stable over the past five years. As markets continue to remain tumultuous in 2022, we expect the ratings to remain resilient as the model goes through its first prolonged bear-market cycle.

Background

The Quantitative Rating uses a machine-learning model trained on Morningstar fund analysts' past rating inputs and the data supporting those decisions. The model extends the reach of Morningstar's research to strategies that our analysts don't cover. The Quantitative Rating has the same Morningstar Medalist iconography as the Morningstar Analyst Rating and can be interpreted in the same way—the higher a fund's rating, the higher the confidence that it will outperform over a full market cycle.

The Morningstar Quantitative Ratings model is expanding coverage to keep up with the growing managed investments market. Since the initial launch of Quantitative Ratings for open-end and exchange-traded funds in the United States, they have expanded across investment types, such as separate accounts and variable annuity accounts, and into new regions like Europe, Australia, South America, and Asia.

Nuts and Bolts of the Evaluation Studies

We looked at portfolio and event studies to evaluate the Quantitative Rating's performance and tracked month-to-month ratings changes to study its stability.

The portfolio study creates five portfolios that group the fund universe into buckets based on their ratings. Strategies from different asset classes and categories are in the same ratings buckets. For example, the Gold portfolio shows the performance versus a category investing in every Gold-rated fund each month on an equal-weighted basis. Returns are the rated strategies' returns minus the appropriate category average return.

An important test to evaluate ratings efficacy is an event study. It measures the typical experience an investor would have had using our ratings system on any given day across different holding periods. This study looked at the margin of outperformance among different ratings over various periods—one month, three months, six months, 12 months, 36 months, and 60 months—to see if managed investment products with the same Morningstar Analyst Rating or Morningstar Quantitative Rating, say Gold, outperformed over time. And if so, by how much.

Finally, ratings stability is measured by tracking the number of strategies that change ratings groups each month for the past year. The number of changes is used to create a transition matrix that determines the stability of each rating group. Morningstar Analyst Ratings are assigned on a yearly basis, but this study spreads the same rating over a 14-month period until an updated rating is found in future months or in past months.

Tale of the Tape

Portfolio Study Over different time horizons, the Analyst Rating and Quantitative Rating medalist products outperformed the Negative-rated products. The ratings sort the universe, and the trend is more prominent for the Quantitative Ratings. This is partly due to the larger volume of share classes that get Quantitative Ratings compared with Analyst Ratings. Exhibits 3 and 4 depict the performances of different ratings portfolios in discrete time periods. The trend persists across different time horizons; both ratings models show that medalist strategies outperformed the Negative-rated strategies.

In the three-year period and the since-inception period, which is nearly five years, medalist products outperformed the Negative-rated products compared with their category peers in both ratings models. The association of higher returns to higher rankings is more pronounced in Quantitative Ratings compared with Analyst Ratings. The strategies with MQRs of Gold or Silver outperformed their category peers at least by 0.5% in both time periods. Although, the shorter one-year period shows that even the medalist strategies across both models have not been immune to the market volatility we have seen recently. This has led to similar or worse performance compared the Neutral- and Negative-rated strategies.

Event Study The event study shows that investors in strategies with Quantitative Ratings and Analyst Ratings could experience this outperformance and that it is not the result of timing or coincidence.

Exhibit 5 below shows that buy-and-hold investors in time periods from one month to three years would have experienced gains from following the Quantitative Rating since its June 2017 launch. In all three-year holding periods, regardless of start date, Gold-rated strategies outperformed category peers by more than 2.5%. Negative-rated strategies underperformed category peers by nearly 1.0%. Exhibit 6 shows that buy-and-hold investors in all time periods of one month to five years also would have experienced gains investing in strategies with Analyst Ratings of Gold. In the five-year holding periods, regardless of start date, Gold-rated strategies outperformed category peers by approximately 1.00%. Negative-rated strategies underperformed category peers by nearly 6.0%. This shows that the Quantitative Rating and Analyst Rating both served investors well in recent years. We now await results from the turbulent market conditions.

Stability of the Morningstar Quantitative Ratings

Since the methodology update for the Quantitative Ratings in 2019, the ratings have been stable month over month. Exhibit 7 shows the monthly transition figures for the past year. When a fund receives an MQR of Negative, we would expect that it has only a 0.05% probability of receiving a Bronze rating a year later and no probability of receiving a Silver or higher rating. Similarly, when a fund receives an MQR of Gold, we would expect that the fund has a 91.04% chance of retaining its Gold rating one year later and only a 0.32% probability of receiving a Neutral rating and a 0.01% probability of receiving a Negative rating a year later. In other words, we tend to stick to our guns when rating strategies quantitatively. Similarly, in Exhibit 8, the Morningstar Analyst Ratings are extremely stable across the rating groups. The Quantitative Rating can closely follow Analyst Ratings when it comes to predicting fund ratings, signaling higher accuracy.

Conclusion

The Morningstar Quantitative Rating for managed investment products is off to a good start as a complement to the Morningstar Analyst Rating. The MQR is a reliable input for investors to use when Morningstar analysts do not cover a fund.