Improved measures give investors a clearer view of ETF performance.
This article first appeared in the August/September 2011 issue of Morningstar Advisor magazine. Get your free subscription today!
Passive funds offer investors a tempting trade-off: no risk of substantial underperformance but nearly zero chance of exceeding the benchmark. In the other words, passive investors should expect near-zero alpha; returns should trail the index only by the disclosed expenses of the fund.
In practice, things are not so clear. Passive funds certainly track their indexes much more closely than even the most diversified of active managers, but performance rarely matches that clean equation of index return minus disclosed expenses (Exhibit 1). Investors still need to be careful about choosing the right passive fund, and there are unique issues in measuring passive fund performance. Morningstar is introducing new data points--called Estimated Holding Cost, Tracking Volatility, and Market Impact--to help investors select the right ETF for their needs. But before we dive into these new measures, we first must explore how passive management should be assessed in theory.
Long-Term Tracking Differences
Investors who hold funds for many years may not care much about random short-term deviations, but any predictable long-term drag against the index should be minimized. These drags can occur for a variety of reasons, but most come from trading costs that aren't included in disclosed management or administrative fees. These trading costs are a function of the turnover inherent in the tracked index (small-cap and non-market-weighted indexes require far more trading each year), the liquidity of the market in the underlying securities (blue-chip global equities will incur minimal trading costs, while trading credit bonds or emerging-markets stocks is far more costly), and a portfolio manager's scale and efficiency. Other funds use swap contracts to provide synthetic index exposure or currency overlays, which pay costly spreads to the issuing banks. On the positive side for shareholders, passive fund managers also generate share-lending and repurchase-agreement revenue on their fairly stable holdings.
Trading costs, swap and derivative costs, and lending revenue persist from year to year, because they all depend on the policies and ability of the manager, as well as the nature of the index tracked. With this persistence in tracking ability, we should be able to look at index funds' return histories to identify the best performers, just as we do for traditional active funds.
Traders, hedgers, and portfolio managers using exchange-traded funds to provide liquidity all care far more about the quality of portfolio tracking on a day-to-day basis than they do about long-term differences. Short-term deviations from the index could result from cash inflows and outflows, leaving a portion of the portfolio uninvested for a day or two. Similar timing issues can occur with dividend reinvestment if payouts take time to reach the portfolio, and dividend withholdings can cause payout sizes to differ from those assumed in the index. Finally, many passive funds tracking unwieldy fixed-income or broad equity indexes choose to follow an "optimized" or "sampled" portfolio by investing in a subset of securities that closely resembles, but does not perfectly replicate, the index.
These short-term tracking errors also matter for long-term investors, though we cannot say beforehand whether these deviations from the index will be positive or negative. There's no way to know if the day that cash sits outside the market will be an up day or a down day, and daily return discrepancies have an average value that's near zero. However, small, accumulated short-term deviations could add up over time to bigger differences in trailing returns, so the average size of these daily discrepancies allows us to estimate the accuracy of our long-term tracking-difference measurements.
The Limits of "Tracking Error"
The oft-used but ill-defined term "tracking error" is applied to two measures that roughly capture these long-term return drags and short-term discrepancies. To measure the long-term drag on returns, analysts often measure the return difference between a fund and its index over periods of a year or more to estimate the long-term drag on returns. To approximate the short-term volatility of performance against the index, they take the standard deviation of differences between net asset value and index returns over shorter subperiods (such as daily or weekly). Unfortunately, neither of these common measures of tracking error shows much predictive power for future performance. Tracking differences can vacillate wildly depending on the dates chosen for the measurement period; estimates of tracking error using daily returns are often implausibly high. After long puzzling over these estimation errors, Morningstar's research team ultimately identified the indexes themselves as the source of the difficulties.
When to Trust the Index
Financial indexes try to approximate a platonic ideal of market value at any given moment. In practice, index providers use local market closing prices for the underlying securities as the basis for nearly all index prices. However, fund NAVs try to capture the precise value of the portfolio at the time that the fund's market, rather than the local market of the underlying securities, closes.
For most funds, this difference is negligible. But in the case of stocks in the MSCI Japan Index held by the U.S.-domiciled iShares MSCI Japan Index ETF EWJ, the Tokyo closing prices used in the index value are more than 15 hours old by the time the NAV for the fund is set. Fair value pricing for the NAV may include after-hours trades, trades in cross-listed shares, pricing of Nikkei futures, and any other sources of more up-to-date information on the portfolio value. On days when major market news comes out after the Japanese market closes, the difference between index values set at 8 a.m. and NAVs set at 4 p.m. can be more than 1 percentage point.
Measuring Performance Despite Stale Prices
A major problem with measuring passive fund performance relates to the issue of stale pricing in the index, which afflicts foreign securities and fixed-income funds most of all. Passive funds often incur costs and tracking errors of a few dozen basis points over a year, while day-to-day noise from stale prices can easily be as large as a percentage point. That level of noise means that the typical tracking-error or tracking-difference measurement is picking up far more noise than signal about management quality.
Morningstar's new measures of passive fund performance seek to address stale prices and noise versus the index while also separating long-term return drags from the short-term movements against the index. Estimated Holding Cost isolates the long-term return difference against the index over the past year and smooths the daily noise from stale prices in order to provide the most accurate possible value. We found our new estimates to be far more stable and, most important, far more predictive of future performance than traditional point-to-point estimates of tracking difference.
Tracking Volatility measures the day-to-day return discrepancies versus the index. It uses a sophisticated statistical technique that accounts for stale prices in the index that take a day or two to catch up with the pricing in the NAV. This factor for lagging prices shows incredible statistical significance and results in improved annualized tracking error estimates for credit bond index funds and foreign equity index funds compared with naive measures based on the standard deviations of daily return discrepancies. This new tracking-error measurement not only provides a useful method for identifying the most efficient funds for short-term traders and hedgers but also allows longer-term investors to see how confident they can be in our estimated holding cost for each fund. A larger tracking error means that the fund has experienced far more random deviations against its index, and there is a greater chance that the estimated holding cost has picked up random movement rather than persisting drags on return.
Although Estimated Holding Cost is statistically similar to alpha and Tracking Volatility is similar to tracking error, we have chosen to use distinct terminology to avoid confusion. The methodology is similar, and it will lead to identical results when stale pricing is not an issue and there is not predictive power in ETF price movements, such as when the ETF and the underlying asset trade in the same market during the same trading hours (as with S&P 500 funds). But for ETFs in which movements in the NAV may predict movement in the index, tracking volatility will be reduced to reflect the fact that the ETF should not be penalized for stale prices in the index.
In practice, there is some inconsistency in how the terminology is applied to index funds as opposed to active funds. For example, we want large and positive alphas for an active fund. For an index fund, however, we want a near-zero alpha but can accept negative alpha created by expense-ratio drag. Tracking error for an active fund refers to how closely the manager hugs the bench or swings for the fences. Tracking error on the order of 4% to 8% is not unexpected. But index funds are not supposed to swing for the fences.
The index fund is supposed to "track" its index, and the tracking error would ideally be zero. Because tracking error for index funds is expected to be zero, sometimes the term "tracking error" is applied to the fund's ability to match the index with a minimum holding cost.
The Estimated Holding Cost and Tracking Volatility measures are based on a fund's NAV. While large traders may be able to transact at or close to an ETF's NAV, most traders have to deal with market prices. ETFs have a number of advantages when compared with mutual funds, but trading complexity is a clear disadvantage. The market price can deviate from fund NAV, particularly when the ETF's underlying securities are illiquid. The less liquid an ETF's underlying holdings, the more the market price can move from NAV before it becomes profitable for an arbitrager to step in and bring the market price back in line with NAV. These past deviations go into Morningstar's new Market Impact measure. This measure is meant to capture the average impact (in basis points) of a $100,000 trade, as opposed to an instantaneous measure that will depend on current market conditions such as current limit orders or the bid/ask spreads on the underlying securities.
ETFs vs. Index Mutual Funds
While ETFs were the impetus for the development of these data points, there is no reason that they cannot be applied to index mutual funds. Obviously, mutual funds are created and redeemed directly with the fund company at NAV after market close, so there is no explicit Market Impact cost. ETFs hold up pretty well when comparing Estimated Holding Costs and Tracking Volatility. There is a clear linear relationship between expense ratio and holding costs (Exhibit 2).
IShares S&P 500 Index IVV and SPDR S&P 500 SPY both have a 0.09% expense ratio, but the more flexible legal structure of IVV allows it to engage in share lending and reinvest dividends, keeping the Estimated Holding Cost at 9 basis points and Tracking Volatility at just 5 basis points. SPY, which does not engage in share lending, ends up with a higher Estimated Holding Cost (16 basis points). The larger the Tracking Volatility, the wider the uncertainty and confidence interval for the Estimated Holding Cost.
Despite having a lower Estimated Holding Cost, IVV trails SPY in total volume. In fact, just about every security trails SPY--it trades an incredible $20 billion a day in average volume and maintains low Tracking Volatility. The result is a market impact cost of 0.1 basis points on a $100,000 trade. For IVV, which has ample liquidity (it trades about $370 million per day), the market impact is slightly higher, at 0.7 basis points. Rydex S&P Equal Weight RSP holds the exact same securities as SPY and IVV, but it equal-weights them, resulting in a higher weight in smaller and less-liquid names. Because a fund such as RSP appeals more to long-term investors than high-frequency traders or hedgers, daily trading volume is substantially lower, at about $50 million a day. Still, Market Impact averages around 2.2 basis points for a $100,000. When comparing fund expense ratios, it is important to take into account potential Market Impact costs.
We believe that these three new data points provide investors with far better measures of passive manager performance than are offered by any other fund data provider. By accounting for this tricky issue of stale prices in index values, and by disaggregating the short-term and long-term return differences against the index, these new measures allow investors of all time horizons to choose the funds that work best for their particular purposes. And as index funds continue to accumulate assets, we hope that these new data points help ensure that the best managers win out in the end.
Statistical Analysis of Fund Performance
The groundwork for the use of regression to analyze returns was laid by William Sharpe, in his 1964 paper that developed the Capital Asset Pricing Model. From this basic theory, sprung a number of risk-adjusted performance measures, such as the Sharpe ratio, which measures excess return relative to volatility, as well as Jensen's Alpha and the Treynor Ratio, which adjust return for the level of systematic risk. A practitioner's guide to the application of these theories can be found in the book Active Portfolio Management by Richard Grinold and Ronald Kahn.
The Alpha and the Beta
In finance, "beta" is widely used as a measure of an investment's sensitivity to market movements, but its name is borrowed from the common Greek symbols used to express a regression formula. The positive "alpha" for which active portfolio managers are incentivized to achieve derives its name from the intercept of the regression of portfolio returns regressed on benchmark returns. The residual volatility, fund movements that cannot be explained by the market, is the tracking error.
We value your feedback! Add your questions, comments, or criticism by clicking the link below.