Software Development Recap 2012

I have been working on a number of statistical packages written in the public domain R programming language. Last year I put two packages together. A package is a collection of R functions, usually built on other R packages, that implement a statistical procedure. The two packages were:

• hlmmc In the late summer of 2012, I taught a graduate-level class in Hierarchical Linear Models (HLMs, sometimes called Multilevel Models) at the University of Tennessee (UT-Knoxville). The hlmmc package was developed for the course to facilitate Monte Carlo studies (computer-intensive methods for the numerical analysis of messy problems) and teaching HLMs. More information about hlmmc (also a public domain package) is available here. I am also serializing my course lectures (here). The lectures were essentially demonstrations of the hlmmc package.
• ws This package allows state-space time series models I have developed to be simulated, re-estimated or extended. I have used the ws package to develop models of the world system (WL20), the US economy (USL20), other countries in the world system and various forecasting models (see examples of the forecasting models here and the US Stock Market models here). More information about the ws package is available here.

I use another set of routines to develop the models made available with the ws package. These routines  select the best model using the AIC criterion from a number of models starting with a Random Walk as the null hypothesis.  I believe that the routines I am currently using could be simplified for general use, but there is still some testing that needs to be done to make sure the simplified functions generate similar "best" models. For 2013, I should be able to complete the rw package, depending on how the testing goes.

As I developed this software, I realized that the I needed another outlet to develop the theory and rationale underlying the programs. In 2012, I created a number of other blogs for this purpose. As time permits in 2013, I will post results to these blogs:

• Economic Bubble Machine The world-system models can be used not only to produce forecasts but also to do counterfactual analysis. In a counterfactual analysis, we change the model parameters or something about how the model works and then run a new "fictional" simulation to generate counterfactual history. In the Economic Bubble Machine blog I will explore the use of counterfactual history to identify economic bubbles such as the Great Depression or the Subprime Mortgage Crisis.
• World System Conjectures Since I am developing world-system models, I need a place to test the models against world-system theory. The World System Conjectures blog will be devoted to that effort in 2013. The project involves historical analysis with quantitative models.
• Causal Macrosystems Finally, the Subprime Mortgage Crisis and the IPCC Emission Scenarios have pointed out the weaknesses of current quantitative macrosocietal models. During the current financial crisis, large-scale econometric models did not seem to provide enough (or any?) short-term early warning about potential problems. The IPCC emission scenarios that attempt to link climate change to human activity challenge the other extreme, that is, the ability of quantitative models to predict decades into the future. The Causal Macrosystems blog will expand some early attempts (here and here) to evaluate state-space models against their competitors.

NOTE: My major area of interest involves state space models and the application of systems theory to statistics, computer simulation and societal development. I have been working in this area since the late 1970's (here). For many years I had used my own software package to do state-space analysis. It was a collection of Fortran subroutines (see John Nash's book Compact Numerical Methods for Computers plus other routines from other sources) driven by an interpreted, stack-based RPN (Reverse Polish Notation) language similar to Postscript and also written in Fortran (based on A. Colin Day's Book Fortran Techniques: With Special Reference to Non-numerical Applications). When R became available in the late 1990's, I started writing the routines I am currently using in the R language. 

Intellectual Dishonesty at NCCAM?

NCCAM is the National Center for Complementary and Alternative Medicine (here) and it is the US Federal Government's lead agency for scientific research on complementary and alternative medicine (CAM). A recent investigation by the Chicago Tribune (here) concluded that "...precious research dollars could be better spent elsewhere." The types of studies funded by NCCAM have included inhalation of lavender and lemon to heal a wound (it didn't), coffee enemas for cancer (no effect), mind-body therapies (yoga, massage and acupuncture, the later found no better than sham acupuncture), energy healing, distant prayer, qigong (manipulation of a universal energy or life force), ginko biloba, saw palmetto (no benefit over placebo), etc.

In this post, I will look at one of the NCCAM studies (Sherman et. al 2011) funded to determine "whether yoga is more effective than conventional stretching exercises or a self-care book for primary care patients with chronic low back pain." In a press release, NCCAM concluded For Low-Back Pain, Yoga More Effective Than Self-Care But Not Stretching (the original study in the Archives of Internal Medicine is here). The study is of interest for a number of technical-statistical reasons which I will explore in my Random Variation blog (here). In this post, I'm just going to make a simple observation: the presented data do not seem to support the conclusions!

If you look carefully at the graphs from the article (presented above and discussed in a note below) you will notice that the error bars (confidence intervals) at the study's end point (week 26) overlap. Just eye-balling the graph, it would seem that self-care (the red line) will eventually be completely equal to either yoga or stretching which are clearly not different. What if they had run the study out to 32 weeks? To be fair to the authors, they based their conclusions on probability values (p-values), a commonly accepted approach to reporting significant results. Without getting into statistical details, p-values do not say anything about whether we are looking at an important difference in treatment and control effects. Differences of 1 point in the RDQ (Roland-Morris Disability Questionnaire) would seem to be of of questionable importance against 2 point confidence interval spreads (read more on expected values for the RDQ here).

From the standpoint of NCCAMs mission which is ambiguous (can they really conclude alternative medicine is a sham and expect continued funding?), the study certainly doesn't support the underlying theory (presented in the causal diagram above, click to enlarge). Yoga is supposed to be superior due to the "mental enhancement" of meditation. Since Yoga was no better than stretching, it seems for sure that the "mind-body" aspect of the trial was a failure.

However, the researchers go on to conclude that Yoga is safe and should be recommended by physicians for patients with low back pain. I don't follow these conclusions especially when there is anecdotal evidence of How Yoga Can Wreck Your Body. The researchers neither mention nor test these issues and I'm not sure why they would continue to recommend Yoga. And, in a related editorial in the same journal, Timothy S. Carey, MD, MPH, concludes that "The study by Sherman et. al. in this issue is an excellent example of a pragmatic comparative effectiveness trial (p. 2027)". The developing concensus about Comparative Effectiveness Research (CER) is important because it is a cornerstone of attempts to reform the US health care system. I will never have time to review all the CER studies, but I'm getting a little queasy feeling in my lower intestinal tract over CER and whether people will actually be treated based on CER studies let alone CAM.

NOTE: The four graphs above present two of the study outcome measures, the RDQ score and the Bothersome score (How bothersome was your back pain?). The left panel displays the raw scores and the right panel displays the adjusted scores to equalize initial conditions. Notice that no confidence intervals were provided for baseline. Statistically, I'm guessing that the initial conditions were not different. Any adjustment without expanding the end-point confidence intervals to reflect the uncertainty of the initial conditions seems questionable to me.