Working Paper: Negative Results in Social Science

The editors wrote a working paper that has first been made available by POLMETH, the online forum of the methodology section of the American Political Science Association. It has now been published by European Political Science. The working paper can be downloaded here.

Abstract: Do academic publication standards reflect or determine research results? The article proposes minimal criteria for distinguishing useful ‘unpublishable’ results from low- quality research, and argues that the virtues of negative results have been overlooked. We consider the fate these results have suffered thus far, review arguments for and against their publication, and introduce a new initiative—a journal to disseminate negative results and advance debate on their recognition and use. 

APSA 2006 Short Course: Rethinking Publication

Co-Sponsors:

The Journal of Spurious Correlations & The European Consortium for

Political Research Standing Group on Political Methodology

Presenters:

Prof. David Collier, University of California, Berkeley

Prof. Robert Boruch, Wharton School, University of Pennsylvania

Prof. Alan S. Gerber, Yale University

Dr. Christian Pfeffer, Harvard Medical School

Prof. Richard Scheines, Carnegie Mellon University

Description:

The short course revisited familiar strategies for evaluating and reporting research findings, for a critical examination of how implicit requirements of publication shape our knowledge of politics, above and beyond the explicit standards of the methods we use in research. Examples of specific research programs in the natural, clinical, applied and social sciences were intoduced to develop an interdisciplinary perspective on the relationship between evidence, theory and method. The debate contributes to understanding knowledge ccumulation and the process of social inquiry as a joint community enterprise.

An abstract of the debate will be available for download shortly.

Survey

JSpurC has polled the academic public seeking direct evidence of the recognition, use and publication of negative results in the social sciences. The survey of academics by the team of editors, was designed to solicit scholars experience with negative results. The answers describe the perceived function of research findings falling outside the current ‘publishable’ range, which are consequently only indirectly apparent as systematic shortcomings of the published form.

The survey data will receive full analysis in our journal. In the meantime, a brief summary is available for download. 

Download

Lehrer, David (2005) 'What is Spurious? An Inquiry Across Methods'
Working Paper Presented at the ECPR Budapest Conference, September 2005

What findings qualify as negative, unexpected, mistaken or unpublishable under major methodological approaches, and what does this tell us about the state of methods and of the discipline? The paper will consider theoretical, empirical and normative propositions on ‘spuriousness’ applied to political science, the social sciences and other fields of research. ‘Spurious’ will be employed as a general term to refer not to what is false, falsified or falsifiable but rather to the disciplinary effects of a broader category of results that are ‘unpublishable’ whether false or not. The paper will consider the potential utility of schemes for returning the ‘by-products’ of the research process to the shared public sphere of social science.

Download

literature

Bhaskar, R. (1998) The Possibility of Naturalism: A Philosophical Critique of the Contemporary Human Sciences (3rd edition) (London: Routledge).

Bonilla, J.P.Z. (2002) ‘Scientific Inference and the Pursuit of Fame: A Contractarian Approach’ Philosophy of Science, 69, pp. 300-323.

Browman, H.I. (Ed) (1999) ‘Theme Section: Negative Results’ Marine Ecology Progress Series, 191, pp. 301-309.

Charlton, B. (1987) ‘Think Negative’ New Scientist, 116, p. 72.

Charlton, B.G. ‘Why a Journal of Negative Results?’ (2004) Journal of Negative Results in Ecology and Evolutionary Biology, 1, pp. 6-7. 

Cleophas, R.C. and T.J. Cleophas (1999) ‘Is Selective Reporting of Clinical Research Unethical as Well as Unscientific?’ International Journal of Clinical Pharmacological Therapy, 37, pp. 1-7.

Collins, H.M. (2003) ‘Lead Into Gold: The Science of Finding Nothing’ Studies in History and Philosophy of Science, 34A, pp. 661-691.

Delamont, S. and P. Atkinson (2001) ‘Doctoring Uncertainty: Mastering Craft Knowledge’ Social Studies of Science, 31, pp. 87-107.

Empirical Economics (Heidelberg: Springer-Verlag).

Feyerabend, P., I. Lakatos and M. Motterlini (1999) For and Against Method: Including Lakatos’s Lectures on Scientific Method and the Lakatos-Feyerabend Correspondence (Chicago: University of Chicago Press).

Gerber, A.S., D.P. Green and D. Nickerson (2001) ‘Testing for Publication Bias in Political Science’ Political Analysis, 9, pp. 385-392.

Granger, C.W. (2003) ‘Time Series Analysis, Cointegration, and Applications’ Nobel Lecture, Stockholm. 

Granger, C.W.J. and P. Newbold (1974) ‘Spurious Regressions in Econometrics’ Journal of Econometrics, 2, pp. 111-120. 

Habermas, J. (1976). ‘A Positivistically Bisected Rationalism’ in T.W. Adorno, H. Albert, R. Dahrendorf, J. Habermas, H. Pilot, and K.R. Popper, Eds. The Positivist Dispute in German Sociology (Heinemann, London).

Habermas, J. (1973) ‘A Postscript to Knowledge and Human Interests’ Philosophy of the Social Sciences, 3, pp. 157-189.

Hall, P. ‘Aligning Ontology and Methodology in Comparative Politics’ (1993) in J. Mahoney and D. Rueschemeyer, Eds. Comparative Historical Analysis (Cambridge: Cambridge University Press).

Hedges, L.V. (1992) ‘Modeling Publication Selection Effects in Meta-Analysis’ Statistical Science, 7, pp. 246-255.

Hendry, D. (1980) ‘Econometrics—Alchemy or Science?’ Economica, 47, pp. 387-406.

Iyengar, S. and J.B. Greenhouse (1988) ‘Selection Models and the File-Drawer Problem (With Discussion)’ Statistical Science, 3, pp. 109-135.

Journal of Applied Econometrics (UK: John Wiley & Sons, Ltd.).

Journal of Articles in Support of the Null Hypothesis, <http://www.jasnh.com>.

Kennefick, D. (2000) ‘Star Crushing: Theoretical Practice and the Theoreticians’ Regress’ Social Studies of Science, 30, pp. 5-40.

King, G. (1995) ‘Replication, Replication’ PS: Political Science and Politics, 28, pp. 443-499.

King, G., R. Keohane and S. Verba (1994) Designing Social Inquiry (Princeton: Princeton University Press).

Knight, J. (2003) ‘Negative Results: Null and Void’ Nature, 422, pp. 554-555.

Kotze, D.J., C.A. Johnson, R.B. O’Hara, K. Vepsälaäinen and M.S. Fowler (2004) ‘Editorial: The Journal of Negative Results in Ecology and Evolutionary Biology’ Journal of Negative Results in Ecology and Evolutionary Biology, 1, pp. 1-5.

Mahoney, J. and G. Goertz (2004) ‘The Possibility Principle: Choosing Negative Cases in Comparative Research’ American Political Science Review, 98, pp. 653-669.

Mahoney, J.M. (2004) ‘Revisiting General Theory in Historical Sociology’ Social Forces, 83, pp. 459-489.

Mahoney, M.J. (1977) ‘Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System’ Cognitive Therapy and Research, 1, pp. 161-175.

Malhotra, S., N. Shafiq and P. Pandhi (2004) ‘Advantages of Reporting, Publishing, and Reading “Negative” Studies’ Medscape Pharmacists, 5. 

Massie, B.M. (2003) ‘Editorial: Negative, Neutral, and Discordant Results: What Should an Editor Do?’ Journal of Cardiac Failure, 2, pp. 77-79.

Maxwell, C. (1981) ‘Clinical Trials, Reviews, and the Journal of Negative Results’ British Journal of Clinical Pharmacology, 1, pp. 15-18.

Mestre, M.P. (2002) ‘Editorial: The Relevance of Research Papers With “Negative Results”’ Revista Clínica Española, 202, pp. 575-576.

Mittelstaedt, R. and T.S. Zorn ‘Econometric Replication: Lessons from the Empirical Sciences’ Quarterly Journal of Business and Economics, 23.

Moe, T.M. (1979) ‘On the Scientific Status of Rational Models’, American Journal of Political Science, 23, pp. 215-243.

Moses, J., B. Rihoux and B. Kittel (2005) ‘Mapping Political Methodology: Reflections on a European Perspective’, European Political Science, 4, pp. 55-68.

Philips, P.C.B. (1988) ‘Reflections on Econometric Methodology’ Economic Record, 64, pp. 344-359.

Phillips, P.C.B (1986) ‘Understanding Spurious Regressions in Econometrics’ Journal of Econometrics, 33, pp. 311-340.

Popper, K. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge (UK: Routledge). 

Prechelt, L. (1997) ‘Why We Need an Explicit Forum for Negative Results’ Journal of Universal Computer Science, 3.

Ragin, C.C. and J. Sonnett (2004) ‘Between Complexity and Parsimony: Limited Diversity, Counterfactual Cases, and Comparative Analysis’ in S. Kopp and M. Minkenberg (Eds) Vergleichen in der Politikwissenschaft (Wiesbaden: VS Verlag für Sozialwissenschaften). 

Sartori, G. (1993) ‘Totalitarianism, Model Mania and Learning from Error’ Journal of Theoretical Politics, 5, pp. 5-22.

Sartori, G. (2004) ‘Where Is Political Science Going?’ PS: Political Science and Politics, 4, pp. 785-787.

Scarbrough, E. and E. Tanenbaum (1998) Research Strategies in the Social Sciences: A Guide to New Approaches. (Oxford: Oxford University Press).

Schmitter, P.C. (2001) ‘Seven (Disputable) Theses Concerning the Future of ‘Transatlanticized’ or ‘Globalized’ Political Science’, < http://www.iue.it/SPS/People/Faculty/CurrentProfessors/bioSchmitter.shtml>.

Shepperd, M. (2003) ‘Empirically-Based Software Engineering’ Upgrade: The European Journal for the Informatics Professional, 4, pp. 37-41.

Sigelman, L. (1999) ‘Publication Bias Reconsidered’ Political Analysis, 8, pp. 201-210.

Stigler, G. (1982) ‘The Process and Progress of Economics’ Nobel Lecture, Stockholm.

Ward, A. (2004) ‘How One Mistake Leads To Another: On the Importance of Verification/Replication’ Political Analysis, 12, pp. 199-200.