Double Bubble, Toil and Trouble

The NY Times today posed the question "Fed Missed This Bubble. Will It See a New One?" Both Fed chairmen Alan Greenspan and Ben Bernanke famously missed the development of the dot-com and the Great Recession bubbles (even conservative News.max accepts this analysis). So, how hard is it really to identify bubbles?

In an earlier post I showed how feather forecasting from a state-space model estimated up to 1990, showed that the market was over-priced during both the dot-com and the subprime mortgage bubbles. Implicitly, the feather forecast points to a sustainable level for the market but doesn't really display the level explicitly. To do that we need to remove the cyclical components and the month-to-month shocks from the model and run a counterfactual simulation for the stock market. The simulation is displayed above. Rather than reaching almost 1600 in the peak of the dot-com bubble, the simulation suggests that the market should have been at about 600. And, 1000 would have been a better level when subprime mortgage crisis broke.

Why is it so difficult for the US Federal Reserve to identify bubbles? Part of the problem is that one of the major cyclical state variable is the US economy is driven by Fed policy. Another cyclical state variable involves more fundamental economic factors such as housing, corporate profits, oil prices and unemployment. Whether the Fed is counter-cyclical or pro-cyclical will have to be a topic for another posting but clearly the Fed is in the middle of market cycles and has trouble looking outside the box.

And, compared to my pessimistic forecast for the S&P 500, the forecast eliminating the cyclical state variables for the US economy is optimistic. In the figure above, the S&P 500 is back to pre-crisis levels by 2015. The usual disclaimers apply even more strongly to this counterfactual forecast.

At best, financial reform might reduce but not totally eliminate bubbles in the US economy. Since late 20th Century bubbles seem to last about a half decade, the pattern suggests a bubble investing strategy that would differ from the standard diversified portfolio approach--a topic for a future posting.

Need To Know vs. Need To Share

The Cold-War culture that I was exposed to in the US military and still permeates the federal government is that information should be made available on a "need to know" basis. This approach might have made some sense during the cold war but, after 9/11, it seems to make much less sense today. The cultural change being pushed right now is the "need to share".

What hopefully will not get lost in the pendulum swing from one extreme to the other is that these approaches are not mutually exclusive. Some things need to remain in the federal government stove pipes and other things need to be shared. The Markle Task Force on National Security has developed a balanced set of initiatives for information sharing emphasizing: (1) Privacy and civil liberties protection, (2) Discoverability "... offering users the ability to 'discover' data that exists elsewhere without gaining access to the underling information until the user requesting access is authorized and authenticated", (3) An "authorized use" standard for information sharing and (4) Culture change from need-to-know to need-to-share.

Particularly to the second point on discoverability, I would (1) add the important motivational element of creating a publish/subscribe system--if agencies publish, they also get to subscribe ("share-to-play") and (2) emphasize the importance of starting with basic, standardized information about events of importance to national security. Here is the relevant paragraph from the Markle Foundation report on Discoverability:

Discoverability is the first step in an effective system for information sharing, offering users the ability to “discover” data that exists elsewhere. Data is tagged at the point of collection with standardized information (e.g., who, what, where, when) and submitted to a central index. Just as a card catalogue in a library serves as a central index, directing users to relevant books—but doesn’t provide the book itself—these “data indices” point users to data holders and documents, depending on the search criteria used.

The Dot-com and Subprime Mortgage Bubbles: An Example of State-Space Forecasting Techniques

In a prior post, I described state space models and why I think they are useful for time series forecasting. I'm using state-space models to generate forecasts for 2010 and plan to look at how well the models did in 2011. However, I can pretend that I was trying this same exercise in 1990 by estimating the model only on historical data up to 1990, using the 1990 model to forecast forward to the present and see how well the model would forecast two notorious crises that standard economic models seem unable to have predicted, the dot-com bubble from 1998-2000 and the subprime mortgage crisis from 2007-2010. For this exercise, I'll just look at the stock market, particularly the S&P 500 index using data from Yahoo! Finance.

First, as a point of comparison, I estimated a state-space model for the entire period. The model uses the state of the US economy as an input variable and the volumes and price data for the S&P 500 as the output variables. The fit of the model for S&P 500 volumes and prices in the figure above (dotted red line) is excellent.Using a model estimated only up to 1990, the price fit is still good but the volume forecast (upper panel above) starts to deteriorate after 2005. Notice for both series that the model tends to miss the turning points--a common feature of all forecasting models (the models don't know that the bubble has burst until after it has happened).

Although models appear unable to capture turning points, they are able to identify developing bubbles using feather forecasting. A feather forecast uses the model to predict a number of periods into the future (thirty-six in the figure above for monthly S&P 500 data), starting from each period in the sample. Up to 1995, the model forecasts basic linear growth for S&P 500 prices. In 1995, however, the feather forecast identifies the developing dot-com bubble as having started in 1995 rather than the conventional dating of 1998. From 1995 to well into 2001, the feather forecast is predicting a collapse in prices. Indeed, from 1995 to 2007 (the last point where a 36 month feather forecast can be made) the feather forecast is warning of a collapse in prices. A conservative investing strategy would have been not to ignore the developing bubble but realize that one would have to be vigilant and liquid with any new investments made after 1995.

This then brings us back to a future forecast made with the model estimated up to 1990. The model is essentially predicting a sell off for 2010 (increasing volumes and decreasing prices). It will be interesting to see what actually happens over the next year. The model cannot predict monthly rallies and sell-offs or the turning points for recovery, but studying the feather forecast suggests that the S&P 500 cannot realistically reach 1000 for a few more years into the future. In any event, the usual disclaimers apply to this prediction.

Forecasting Disclaimer

At the beginning of 2010, I'm publishing a collection of economic and environmental forecasts based on state-space time series models. Given that economists were unable to forecast the financial crisis of 2007-2010, I should probably explain why I think my approach is better.

First, I'm not sure my approach is better. That's part of the purpose for doing the forecasts in early 2010. At the start of 2011 (or any point in between), I can look back at the forecasts, see what went wrong (if anything) and try to figure out the reasons for any failures.

The state-space approaches are attractive because they address a number of known problems with existing forecasting models. State-space models forecast based on the "unobserved" state of the system being studied. The state variables are the minimum collection of independent, orthogonal variables that connect the model's input variables to the output variables. Typically, economic forecasting models have a rigid set of structural equations that are used to predict each output variable. Real human systems, however, are too flexible and too emergent to be modeled with a rigid set of equations.

A related attractive feature of state-space models is that there are two sources of error variation, one for the state-vectors and another for the output variables. Typically, economic forecasting models have one source of error for each output equation. And, typically the error distributions are modeled using a bell-shaped normal distribution. Nouriel Roubini, for one, has strongly criticized economic forecasting models both for their unrealistic structure, estimation techniques and assumptions about error terms. Rather than assuming that the functional form of the error term is known, state-space models can easily use a non-parametric bootstrapping technique to estimate confidence intervals around forecasts and the Kalman filter to compute the system state while forecasting (after an approximate system state is used to estimate the model).

Another interesting aspect of state-space models is the use of computer simulation to understand the forecasts. For example, one can estimate a state-space model for a based period (say 1950-1990 for the US economy) and then use simulation techniques to calculate the unobserved system state (using the Kalman filter) and the time path of the output variables to any point in the future. I'll present an example of the analysis in a later post.

For all these reasons, I find state-space models attractive for forecasting. For some of the answers to how well the models actually perform, we'll have to wait until 2011. However, we can also analyze two notorious historical episodes (the dot-com bubble and the financial crisis of 2007-2010) using a model estimated on US data up to 1990. Could these two crises have been predicted with state-space models? That will be the subject of my next post.

A Pessimistic S&P 500 Forecast for 2010-2012

It's probably appropriate that my first forecast for 2010 would involve the stock market. The US has gone through two stock market bubbles (the dot-com bubble from 1998--2000 and the subprime mortgage crisis from 2007--2010, hopefully) in addition to a direct terrorist attack on New York City (September 11, 1002) that paralyzed the financial system. Before presenting a forecast for 2010, let's look at how well my time series models dealt with this period of stock market instability.

The figure above displays a feather forecast of volume (the upper panel) and price for the S&P 500 (data from Yahoo! Finance). Although the models track the stock market data very well (displayed here), the feather forecasts (in this case, the feather forecast starts from every month in the sample and predicts forward three years) show that the S&P 500 was over-valued during the dot-com bubble, undervalued during 9-11 and over-valued again during the subprime mortgage crisis. In other words, using the models would have suggested a conservative investing strategy and warned of developing stock market bubbles.

The S&P 500 model is driven by the state of the US economy. Using financial terminology, the market fundamentals are determined by the state of the economy. Using systems terminology, the stock market is a hierarchical subsystem within the US economy. The three forecasts (dotted red, blue and green lines) in the graphic above suggest that the future will depend on the state of the US economy. Volumes will be flat while prices will either rally or remain flat based on the future of the US economy. Not really that surprising a range of forecasts.

When I use an actual forecast for the state of the US economy through 2012 as input for the model, the forecast is pessimistic: declining prices and increasing volumes, in other words, a sell off. You can compare my forecast to the one provided by the Financial Forecast Center here. Their forecast for prices also points downward (out to Jul 2010).

Of course, it's also possible to create an optimistic forecast for the S&P 500 based on a more optimistic forecast for the US economy and the usual disclaimers apply. We're at a point where uncertainty rules.

Forecasts for the New Year

In prior posts, I have made forecasts for the US economy and the World system. For the US economy, I have developed forecasts for nuclear power (optimistic) and for the airline industry (pessimistic). For the World system, I have forecasted relatively poor performance for the European Cap-and-Trade (emission trading) system. The forecasts were based on outputs from my time series models of the US economy and the World system.

My New Year's resolution is to make a series of forecasts, at least for the next few years and possibly for the entire decade. My intention will be to review these forecasts in January of 2011 to see how well the models performed.