A group of social scientists in Europe and the US has established a new journal of negative and unpublishable results in the social sciences. The mission of The Journal of Spurious Correlations (JSpurC) is to provide a legitimate venue for exploring pure and applied methodological questions in the social sciences in the company of colleagues without fear of professional embarrassment or reprisal. While a number of the present organizers are political scientists, such an initiative may be relevant to other social science disciplines as well, and to a range of methodological approaches beyond the ‘quantitative.’

Why establish The Journal of Spurious Correlations?  

• A tremendous amount of potentially useful information is currently lost to the social sciences through selective reporting of results.  The Journal of Spurious Correlations will represent the first systematic effort within the social sciences to address this very significant problem.
   

• Other fields are now developing initiatives like this, particularly in the ‘hard’ sciences.  Some of these are quite recent, but it may be only a matter of time before similar initiatives, in some format or other, appear in the social sciences.
   

• Airing and discussing researchers’ ‘mistakes’ could enhance quality control and community building in the disciplines of social science.
   

• An initiative such as this could also catalyze a (much-needed) wider discussion about the use of methods, about ‘What is spurious?’ in the social sciences, and about how scholarly disciplines should handle their ‘mistakes.’
   

• Such discussion could lead to greater dialogue across social science disciplines and across national borders on questions central to the vocation of social scientific research.

collaboration

The Journal is an affiliated project of the Research Committee on Logic and Methodology of the International Sociological Association, and draws participating editors, advisory board members and contributors from around the world.

contribution

Despite widely accepted notions of the practices that ‘good’ social scientific research ‘should’ entail—i.e. the imperative to report one’s research findings—most negative findings—those that do not fit with the researcher’s own theoretical presuppositions or with accepted tenets of methodological appropriateness—generally are not reported.  Researchers do not submit these findings to journals, and journals do not seek to publish them.  If such findings were published, they might aid social scientists at large in refining theory or method, but at the same time they would likely undermine the individual careers of the few researchers who dared to publish them.  Instead, ‘negative’ findings are buried in desk drawers, deleted from hard drives, or filed in circular files, and anlayses are re-run and readjusted until more salutary results are produced.

No longer.  Through www.jspurc.org, The Journal of Spurious Correlations will provide a forum for disseminating the rigorous but otherwise ‘unpublishable’ findings that are, strictly speaking, necessary products of quantitative (and qualitative) social science research.  The Journal will publish such findings and commentary upon them by their own authors and by other scholars, as well as articles on methods of a more general nature.  The Journal and its related discussion groups and activities will thus provide a forum for collaborative exploration of the ‘dark side’ of social scientific research (or at least of its uncharted gray areas).

In providing a legitimate venue in which to explore methods’ usefulness (as well as efficacy) without fear of professional embarrassment or reprisal, The Journal of Spurious Correlations may aid graduate education and information exchange; foster scholarly debate on established methods and their application; and develop new perspectives on the ‘stylized facts’ of various social science subfields by bringing previously submerged findings to light.  In making ‘negative results’ available, The Journal may also make possible more comprehensive meta-analysis by political scientists of various research questions and subfields (Hedges 1992).  By subjecting its ‘negative’ or anomalous findings, and the methods by which they have been reached, to critique by a community of scholars, The Journal may enhance the sophistication and validity of social scientists’ understanding of what they themselves do.  This might particularly apply to its postgraduate contributors, editors and readers, who may be engaging with particular research methods for the first time.

This ambitious project is made possible by the collaborative spirit of inquiry of the rising generation of social science researchers.  This (growing) group of researchers recognizes that the rewards of publication in JSpurC  are likely to include the enhanced opportunity for each of us to learn from our own ‘mistakes’; access to a shared venue in which to subject individual ‘mistakes’ to the critical scrutiny and scholarly discussion of our peers; and the possibility of contributing to a wider discussion of what constitutes an analytic or research ‘mistake’ and of how those ‘mistakes’ should be handled by the discipline.

At present, there is no single place where the best of the discarded data and analysis, the most interesting and relevant of the flotsam and detritus of social scientific inquiry, is gathered and subjected to critical review.

Who knows?  In aggregate, it might well have something useful to tell us.

content

Dream papers might fall into several categories:

A number of researchers independently submit similar ‘negative results’ that, collectively, force reassessment of established orthodoxies in a particular social science subfield.  These orthodoxies had gone unchallenged for years because researchers had hesitated to publish conflicting results.

• A negative result published in The Journal helps social scientists to reinterpret or contextualize understanding of a phenomenon by providing an upper or lower bound of observation that researchers had previously been unaware of.
   

•  Brief write-ups of ‘unpublishable’ results, published anonymously, are accompanied by cogent, and perhaps conflicting, invited signed commentary by prominent scholars in one or more subfields, discussing issues of the approach and implications of the result.
   

• Future meta-analyses of research in various subfields will cite The Journal extensively because it will be a repository for data not reported elsewhere.
   

• Articles in The Journal will introduce data sets that were not previously available because the researchers who developed them have not yet published an article analyzing those data.
   

•  Researchers in several different institutions might decide to abort a particular research program, because someone else has already attempted something very much like it, with results of demonstrably little utility…yet travel down this blind alley has continued to be replicated because prior researchers’ results were never reported.
   

• Both in commentary and in specially commissioned or submitted articles, important technical questions of methods are addressed in a workshop format, making The Journal a crucial tool for graduate students and established scholars seeking to understand the advantages and pitfalls of particular methods.
   

• Editorials and articles engage in ongoing debate, important to social scientists but now scattered sporadically across various books and journals, on the ‘deeper’ questions of methodology and on the reconcilability of heterogeneous methods in the social sciences.  While early iterations of this debate might center on the concept of the ‘negative,’ ‘spurious’ or ‘unpublishable’ in the context of various methods, and on whether or not there are social science methods to which this concept does not apply, a vigorous debate might address still broader questions of the application of ‘methodologies and methods.’
   

 We are more interested in The Journal becoming a ‘meeting-point’ for new, at times unconventional discussion of methods, than for it simply to cater to a specialized group or school (however well-organized) such as only those using sophisticated methods.

tasks

The Journal does not require articles to be written with a narrow substantive focus.  Rather, The Journal seeks to collect material that has already been produced and that is constantly in production as a by-product of the authorship of articles for other journals.  Thus the potential pool of source material is large.  The Journal’s challenges will be:

• To attract a sufficient amount of interesting material that already exists, by convincing social scientists to reclassify ‘junk’ as ‘submissions’.
   

• To develop an adequate peer-review system for filtering various kinds of ‘unpublishable’ submissions to determine what is worthy of inclusion in The Journal.
   

• To filter what is potentially a huge volume of submissions, larger than that which other social science journals typically face.

The Journal is less interested in results that have been rejected by other journals, and more interested in those that have not even been submitted.

precursors

Why hasn’t anyone thought of this before?  Some have.

In the natural and physical sciences, the problem of rescuing and airing negative results is now coming into its own.  New journals are answering the calls for such fora in the ‘hard’ and ‘life sciences that have been sounded in such prominent scientific journals as Nature and New Scientist (Kotze et al 2004).

In the clinical and applied sciences, such as medicine, pharmacology, clinical psychology, cognitive and computer science and software engineering (Prechelt 1997), the value to scientific advancement of publishing negative results has spurred recent activity in this direction, e.g. in biomedicine and other fields.  While social science methods might differ from those of the more experimental and applied disciplines, problems of explicit and implicit censoring of published results, and the disciplinary impacts of this censoring, are morphologically similar.

In the social and policy sciences, Political Analysis, the journal of the Society for Political Methodology of the American Political Science Association, includes a section on ‘Replications and Extensions’ of previously published work.  Empirical Economics, the journal of the Institute for Advanced Studies in Vienna, and the Journal of Applied Econometrics have also recognized the value of including a section containing brief replications of prior studies.  Such activities serve an important purpose in advancing social science theory and methods, but one that is distinct from what JSpurC will do.

JSpurC will, instead, focus on the publication of original, rather than replicated, negative, spurious, questionable and ‘unpublishable’ results.  The remit of The Journal of Spurious Correlations will not be limited only to publication of spurious regression results.  The Journal will embrace all manner of results and submissions on the frontier of social scientific research that advance its mission and that might not find a place within existing social science journals.  These results will be peer-reviewed and vetted for research quality and scientific and heuristic value, and will generally be accompanied by commentary and analysis, both by their authors and by invited scholars.

activities

The Journal will explore uncharted territory.  In doing so, in a collegial spirit of deconstructive play, it will provide a platform for unlocking insights that may place methodological orthodoxies—quite seriously—in a new light. 

In the future, The Journal  could lead to further print publications such as an edited book compiling its key submissions and debates, or to a conference or panels in existing disciplinary conferences to discuss issues it will have raised.  Email lists or online discussion groups for readers and contributors might also be natural outgrowths of The Journal.

Other potential contributions of The Journal might include methodology clinics, held online or as sessions of ECPR or other conferences, in which actual and potential mistakes and issues encountered by users of various methods might be discussed.  These could include what to do when one’s results appear to be spurious or to reflect multicollinearity or measurement error (both how to resolve such problems using particular quantitative techniques and how to decide among methods that might shift analytic weight away from the source of the ‘problem’).

Another practical contribution of this initiative could be to generate and disseminate insights on data sources.  If new data sets are not published because analyses performed by the researchers who compiled them fail to yield publishable results, then all or part of those data, along with the analysis of the data, might never see the light of day.  In such cases one researcher’s rubbish may indeed be another’s dinner, if the data sets discussed in JSpurC articles were subsequently to be employed by researchers in novel ways.  The Journal might thus address the problem of ‘data selection bias,’ i.e. the state of affairs in which only data sets accompanied by ‘successful’ analyses are likely to be published and disseminated, even if the data that has been compiled and used ‘unsuccessfully’ by one researcher might have been analyzed ‘successfully’ by another.  The Journal might include among its publication activities an ‘anonymous’ archive on its website of data sets or ideas for data sets that have inspired JSpurC submissions and/or that have not been published elsewhere.

approach

Must we be positive to be spurious?  Not necessarily!

The Journal will, as a matter of both principle and interest, welcome submissions from practitioners of all approaches to political and social science, be they positivist, postpositivist, formal, informal, quantitative, discursive, analytic, hermeneutic, critical, applied, inductive, deductive, ideographic, nomothetic, fuzzy, emancipatory, historical or otherwise.  While The Journal’s mission resonates with recent trends in the natural sciences, it does not aim to remake political or social science in the image of the other sciences.

In addition to publishing ‘spurious’ results of all stripes, The Journal will include substantive articles and editorials considering questions of how and in what way the concepts of ‘negative results’ or ‘unpublishable findings’ might apply to work generated within non-quantitative or non-positivist paradigms.  A core aim of The Journal will not be to wall itself off from non-positivist or non-quantitative approaches, but rather to open and sustain dialogue among the various ‘methodologies and methods,’ approaches and paradigms employed by those who investigate political and social life.

Production of The Journal will be made facilitated by the possibility of triple-blind (anonymous) publication, as well as by brief summaries of results, without the implicit requirement of extensive theory-building or literature review in articles reporting ‘positive’ results, and by reliance on the contributions of both ‘regular’ and ‘guest’ editors, reviewers and commentators.  Submissions will be generated both via  word of mouth and via formal Calls for Papers.

JSpurC  will also include theoretical and applied articles on method, and commentary on the analytic results that it publishes, both of which will most likely be signed by their authors.  The Journal will generally publish analytic results accompanied by brief author summaries of perhaps 500 to 2500 words, since many researchers have some ‘spurious’ findings to share, but few may be inclined to make these the basis of an overly long article.

theory

Is this a normative project?  Yes and no.

The Journal will engage in and, it is hoped, stimulate debate not only on what is false, falsified or falsifiable, but also on the broader category of what is unpublishable whether ‘false’ or not.  The Journal will not seek to promote or enforce a single orthodoxy with respect to which some results may be deemed ‘negative;’ nor will it seek to pass judgment on what is ‘positive.’ 

All methods contain an implicit normative dimension: application of a given method includes some results or statements as acceptable and excludes others.  The implicit craft rules and conventions for applying methods (including quantitative, qualitative and other ‘methods’) and for interpreting results are no less important to research practice than are the explicit, formal rules by which methods are defined.  These normative dimensions of a given method (conventions such as the five percent test of significance, and craft rules governing which statistical test to apply to findings or how to assign relative analytic weight when multiple statistical tests are performed) may change over time.

Social norms within disciplines prescribing what ought to be done with negative results may also change over time.  The Journal’s normative dimension consists primarily in its exploration of what social scientists ought to do with their negative results, and of what practice conventions, craft rules, tests or ‘methods’ might be developed for analyzing, aggregating and disseminating those negative results.  Such shared disciplinary norms, institutions and practices remain notably undeveloped.

A set of results may be deemed publishable because it has been anticipated by theory and/or because it falls within the range of previously accepted ‘positive’ results.  Results might also be validated procedurally, e.g. if they have been arrived at via scrupulously observed explicit and implicit rules defining the method employed, and if the data or objects analyzed are deemed to have been reliably observed or appropriately considered.  Yet many ‘negative,’ spurious or unpublishable results are arrived at via impeccable procedures that follow the implicit and explicit rules that define the particular methods employed.  Validation of such findings largely by reference to prior results biases reporting across the discipline, privileging results arrived at earlier in time and diminishing researchers’ capacity to reconsider those earlier results in light of more recent ones.

Differing evidentiary cultures across institutional settings might also determine which results are considered ‘negative:’  some findings may not be submitted to journals by researchers from one group that might have been submitted had they been reached by a researcher from another group.  While distinctions among craft cultures in the social sciences might be blurring, cross-country studies indicate ongoing segregation of publishing communities along national lines (Schmitter 2001)—which leaves open the possibility that results are inconsistently filtered.  Competition for acceptance by prestigious journals increases pressure on researchers to submit articles introducing both new data and new theory.  This means that the less novel result might be neglected by researchers and not developed into an article, even when it might be formally sound and substantively interesting: not highly marketable, but rather confirmatory or otherwise useful to some disciplinary subgroup.

Social norms and normative rules might both determine the evaluation of results in practice.  Methods themselves might include ‘pure’ normative claims about what constitute ‘good’ and ‘bad results (based on procedural rules, accepted thresholds, or algorithms for determining results’ internal consistency).  Theories underlying particular research designs may also introduce ‘applied’ normative claims that qualify or disqualify various sets of results.  Unpublished or unpublishable results may be methodologically sound, yet still might be discarded because they do not accord with the expectations of theory; or they may confirm hypotheses yet fail for one reason or another to pass the explicit or implicit tests of the methods by which they were arrived at.

Against these norms and normative rules for the testing, validation and dialogue of theory, methods, data and findings, the ‘spurious,’ if widely disseminated, might serve as a critical benchmark: ‘Positive’ results can be interpreted, or reinterpreted, in terms of ‘negative’ ones.  A ‘negative’ result might be rehabilitated to serve as the upper or lower limit of observations of a phenomenon (Collins 2003).  One set of results (including ‘theoretical’ results or statements (Kennefick 2000)) is typically validated in terms of its conformity with another; enhanced availability of negative results could therefore increase the scope for validating new findings through processes of differentiation from as well as of conformity to, thereby refining the joint exercise of social science researchers’ judgment.