Last week, Herndon, Ash and Pollin (2013) released a paper challenging the findings of a widely cited study by Reinhart and Rogoff (2010) that was broadly used by political commentators and policymakers in support of European austerity measures and in budget debates in the U.S. The original findings argued that high debt to GDP ratios (namely, at a threshold of over 90%) lead to lower average GDP growth. Herndon, Ash and Pollin replicated this study, finding three types of errors which, if corrected, result in a much smaller difference between average growth rates on either side of the 90% debt to GDP ratio (Herndon et al., 2013). One error was an Excel coding mistake in which the authors excluded five countries that resulted in a -0.3 percentage point error in the average GDP growth in the highest public debt to GDP category. Another error involved selectively excluding time periods for certain countries (within the overall time period they examine) in which these countries had high debt and solid growth. Lastly, their measure of average real growth for each country is, according to Herndon, Ash and Pollin, unconventional as they take an average rather than weighting each year of data. This construct also strongly affects the overall results. This is a clear example of a research study that many in the mainstream media and public, as well as policy domain have accepted as conclusive without critically examining assumptions behind the study or how straightforward the implications actually are.
Schram and Soss (2002), in their analysis of research of the effects of the Temporary Assistance for Needy Families (TANF) program, argue that the “facts” of these impacts (e.g., 60% fewer people are on welfare than pre-TANF) are politically interpreted and their meanings far from obvious. Similarly, some have argued that although it has been widely accepted that higher debt causes lower economic growth, the causation is actually backwards; low growth motivates countries to issue debt in order to stimulate economic growth (Konczal, 2013). This possibility is acknowledged by Reinhart and Rogoff, but the lasting message that they clearly intend their study to get across is that high debt is unhealthy for the economy. Schram (2002) argues that a rampant problem in most empirically oriented research, is that “being an academic first often means couching one’s research in ostensibly neutral terms and allowing the ‘facts to speak for themselves.’” Reinhart and Rogoff suggest that higher debt acts as a signal of risk to investors. This suggestion is couched in neutral terms, but they provide no evidence to support this causal direction and go on to suggest other vulnerabilities associated with debt while not acknowledging their political orientation or what some might call a somewhat obvious alternative explanation. Beyond the particulars of the variable constructs and outright coding errors, how findings are interpreted was not clearly a question of debate for the original study's authors.
Open questions remain as to how this study may have been conducted or interpreted differently by researchers whose political orientations play a more overt their research. When research is informed by practice or policymaking, as in Schram’s “Weak Program,” do researchers and research consumers tend to be more critical of methods, constructs and interpretations?
References
Herndon, T., Ash, M., & Pollin, R. (2013). Does High Public Debt Consistently Stifle Economic Growth?: A Critique of Reinhart and Rogoff. Political Economy Research Institute. Amherst, MA.
Konczal, M. (2013). Shocking Paper Claims that Microsoft Excel Coding Error is Behind the Reinhart-Rogoff Study on Debt. Business Insider. Retrieved from http://www.businessinsider.com/thomas-herndon-michael-ash-and-robert-pollin-on-reinhart-and-rogoff-2013-4
Reinhart, C. M., & Rogoff, K. S. (2010). Growth in a Time of Debt.
Schram, S. F. (2002). Praxis for the Poor. New York, NY: New York University Press.
Schram, S. F., & Soss, J. (2002). Success Stories: Welfare Reform, Policy Discourse, and the Politics of Research. In S. F. Schram (Ed.), Praxis for the Poor (pp. 186–2000). New York: New York University Press.
Zoé
Schram and Soss (2002), in their analysis of research of the effects of the Temporary Assistance for Needy Families (TANF) program, argue that the “facts” of these impacts (e.g., 60% fewer people are on welfare than pre-TANF) are politically interpreted and their meanings far from obvious. Similarly, some have argued that although it has been widely accepted that higher debt causes lower economic growth, the causation is actually backwards; low growth motivates countries to issue debt in order to stimulate economic growth (Konczal, 2013). This possibility is acknowledged by Reinhart and Rogoff, but the lasting message that they clearly intend their study to get across is that high debt is unhealthy for the economy. Schram (2002) argues that a rampant problem in most empirically oriented research, is that “being an academic first often means couching one’s research in ostensibly neutral terms and allowing the ‘facts to speak for themselves.’” Reinhart and Rogoff suggest that higher debt acts as a signal of risk to investors. This suggestion is couched in neutral terms, but they provide no evidence to support this causal direction and go on to suggest other vulnerabilities associated with debt while not acknowledging their political orientation or what some might call a somewhat obvious alternative explanation. Beyond the particulars of the variable constructs and outright coding errors, how findings are interpreted was not clearly a question of debate for the original study's authors.
Open questions remain as to how this study may have been conducted or interpreted differently by researchers whose political orientations play a more overt their research. When research is informed by practice or policymaking, as in Schram’s “Weak Program,” do researchers and research consumers tend to be more critical of methods, constructs and interpretations?
References
Herndon, T., Ash, M., & Pollin, R. (2013). Does High Public Debt Consistently Stifle Economic Growth?: A Critique of Reinhart and Rogoff. Political Economy Research Institute. Amherst, MA.
Konczal, M. (2013). Shocking Paper Claims that Microsoft Excel Coding Error is Behind the Reinhart-Rogoff Study on Debt. Business Insider. Retrieved from http://www.businessinsider.com/thomas-herndon-michael-ash-and-robert-pollin-on-reinhart-and-rogoff-2013-4
Reinhart, C. M., & Rogoff, K. S. (2010). Growth in a Time of Debt.
Schram, S. F. (2002). Praxis for the Poor. New York, NY: New York University Press.
Schram, S. F., & Soss, J. (2002). Success Stories: Welfare Reform, Policy Discourse, and the Politics of Research. In S. F. Schram (Ed.), Praxis for the Poor (pp. 186–2000). New York: New York University Press.
Zoé
This is a very interesting perspective on the role of policy analyst and one that as you say raises a lot of questions. By pure coincidence, I happen to use Reinhart and Rogoff as an example and thinking over this I tried to make a distinction between the analysts and the researcher. Certainly the specific paper and researchers seem to have influenced the policy agenda. But are they necessary the policy analysts or was their work integrated in a policy context? In many ways, the above example raises also a lot of questions on the ethical role of the analyst or the academic; certainly the identity (researcher, analyst) influences the role. But does it influence the ethical responsibility?
A
RSS Feed