The relationship between indexing and efficient-market theories is more nuanced than many believe.
Indexing is based on a simple proposition: Net of fees the markets are hard, if not impossible, to beat. The proposition has been tested many times, with supportive results. No surprise then that passive funds' market share has surged to 24% from 11% of all open-end and exchange-traded fund assets over the past decade. But indexing's well-deserved success has coincided with a disturbing abrogation of responsibilities by some investors and advisors. Many believe that they can't or shouldn't estimate expected returns of their investments. They've consigned valuation to the dustbin.
This is wrongheaded, motivated by a view of markets rejected decades ago. The early efficient-market theorists assumed that the market's expected returns, risks, and correlations were constant through time. Almost no financial economist believes this today. The market's expected returns change. And there's heaps of evidence that the market's returns are somewhat predictable over long horizons.
On an intuitive level, the market must be predictable to some extent. Otherwise, how could investors set prices for stocks versus bonds versus cash? We can also reasonably rule out certain scenarios, such as corporate earnings growing much faster than gross domestic product indefinitely, which would result in corporate earnings eventually taking over the entire economy. That returns are bounded by mean-reverting attributes of the economy points to predictability. Indeed, the evidence is compelling. In the August 2011 issue of The Journal of Finance, University of Chicago professor John Cochrane wrote: ". . . predictability is pervasive across markets. For stocks, bonds, credit spreads, foreign exchange, sovereign debt, and houses, a yield or valuation ratio translates one-for-one to expected excess returns, and does not forecast the cashflow or price change we may have expected." In other words, measures such as dividend/price predict future returns, especially over long horizons. Cochrane is a prominent efficient-markets theorist.
Adding return predictability to classical asset-pricing models, with changing risk, correlations, and expected returns, has surprising implications. In many cases, the hallowed market portfolio, containing all assets in the market weights, no longer guarantees the most return per unit of risk. There's no need to privilege total stock and bond market indexes, or static buy-and-hold strategies. The more realistic models suggest investors should time the market depending on how affected they are by recessions and their estimates of expected returns. An investor who can stomach a lot of volatility should increase his exposure to risky, high-expected-return assets during bad times. This sounds an awful lot like the dictum to "buy when there's blood in the streets." But everyone can't buy at the same time, nor should they. Investors with income or wealth sensitive to the business cycle should put less of their portfolios in value stocks, which are especially hurt by recessions, and possibly even hedge their exposures to their specific industries.
These new and improved models have their impracticalities. Until recently, sticking with a plain market-weighted index fund was perhaps the best course of action for the vast majority of investors. Trading was prohibitively expensive, and it was difficult to cheaply tailor one's exposures to various risk factors. No longer, as decimalization, financial innovation, and competition have slashed costs and expanded the menu of indexlike investments. Investors should take advantage of these circumstances to tailor more-efficient portfolios. However, demanding that advisors and individuals constantly update for every asset class estimate of expected returns, correlations, and standard deviations is impractical. A compromise is to adjust portfolio allocations based on expected returns, perhaps the most important of all three factors. As we'll see, estimating long-run (over a decade or more) expected returns isn't terribly hard.
Most expected returns can be decomposed into three parts: the current cash flow yield, the cash flow's expected growth rate, and the expected change in valuation (for example, a contraction or expansion of the dividend/price multiple). However, of the three, change in valuation multiples is often the least predictable, most volatile, and the least important in the long run, so investors should focus on current yields and expected cash flow growth. Current yields are easy to find. The trick, then, is to find the most appropriate and predictive cash flow growth figure. Fortunately, long-run historical growth rates provide a decent guide. For most major stock markets, dividend growth has averaged 1% to 2% annualized over the past century. For bond indexes, expected cash flow growth is negative owing to defaults. For U.S. Treasuries and investment-grade bonds, the default rate has historically been zero or close to it, so current yield (or better yet, real option-adjusted yield) provides a good guide to expected returns. According to Antti Ilmanen, U.S. high-yield bonds have since 1920 lost about 4.3% of value annually to defaults (2.6% after a 40% recovery rate is included).
Estimates of Real Long-Run Expected ReturnsNote: Data as of 10/10/11. Yields based on 12-month yields of various ETFs. 0.75% added to U.S. equity dividend yield to adjust for net share buybacks. Bond yields adjusted for inflation. Cash flow growth estimated by long-run growth figures for developed-country stock markets.
Adding a few bells and whistles seems to help forecasting power, but they're beyond the scope of this article. GMO, a respected asset manager, adds mean reversion in its models. An investor without the time, data, or inclination to estimate expected returns probably would do well to follow the regular valuation estimates GMO publishes for free at its website (registration required, unfortunately).