Home Introduction Concepts Domains Positions Science studies Citation index Search

Empirical studies of the quality of science

A kind of science studies is the concrete, empirical study of errors, misinterpretations, methodological problems etc. in the published scholarly and scientific literature.

The examples provided in the references below seem to indicate that the condition of (parts of) the academic literature is bad, and this calls for a kind of "source criticism" when using that literature.

Katzer, Cook & Crouch (1998) write "The literature is probably worse that you think" and refer to six empirical studies as evidence:
1. In a study of the quality of educational research (Wandt, 1965, pp 1-7) researcher acting as editors found that only 7% of the articles were worthy of publication.
2. A critic (Fischer, 1970, p. 305) identified one hundred different fallacies common or prevalent in historical writings, which seriously weakened the conclusions drawn..
3. An analysis of rew data in psychology (Wolins, 1962, 657-658) found major mistakes in 3 out of seven examined cases.
4. Morgenstern (1963) found large and apparently unacknowledged errors in foreign trade statistics, prices, production figúres, employment statistics, national income statistics, and economic growth rate indexes.
5. Several investigators (Tversky & Kahneman, 1971; Brewer, 1972; Duggan & Dean, 1968 and Katzer & Sodt, 1973) found that practicing researchers in psychology, education, sociology, and communication are not fully knowledgeable about their craft and many of them believe things to be true about the process of research that are clearly not true.
6. Rensberger (1977, p.44) was quoted: "Dr, Richard W. Roberts, director of the National Bureau of Standards, estimates that half or more of the numerical data published by scientists in their journal articles are unusable because there is no evidence that the researcher accurately measured what he thought he was measuring or no evidence that possible sources of error were eliminated or accounted for".

Katzer, Cook & Crouch (1998) also refer to Bernam (1981, 1986) for further information.

A related issue are errors in official statistics. In August 2008 Finn Bo Frandsen brought to the press information about the unreliability of the official Danish statistics concerning building published by "Danmarks Statistik", which reported a decline in new buildings on 13%. In reality, however the rate of building increased 6%! http://www.danskbyggeri.dk/C1256C8D005F5C1B/0/77FF11AD2492234CC1256FAF00336E91


Brewer, J. K. (1972). A note on the power of statistical tests in the American Educational Research Journal. American Educational Research Journal, 9, 391-401.

Brock, A. (1993). Something old, something new - the reappraisal of of Wilhelm Wundt in textboks. Theory & Psychology, 3(2) , 235-242 "Subsequent generations of textbook writers had merely copied from other textbooks without consulting original sources. Boring's flawed account had become increasingly distorted as it was passed on from one generation to the next."

Burnam, Tom (1981). More misinformation. New York: McKay.

Burnam, Tom (1986). Dictionary of misinformation. Reissued. New York : Perennial Library. (First published 1975).

Dewsbury, D.A. (1995). Robert Yerkes' Rabbit. Theory & Psychology, 5(3), 449-450. (Note). Abstract: A case of distortion in the history of psychology literature is described. Three significant details of an incident in the boyhood of Robert Yerkes were distorted in a recent article. It is important that such errors be corrected because they can greatly alter interpretations of the significance of historical events and they are difficult to eliminate from the secondary literature once perpetrated.

Duggan, T. J. & Dean, C. W. (1968). Common misinterpretations of significiance levels in sociology journals. The American Sociologist, 3, 45-46.

Fischer, David Hackett (1970). Historians' fallacies. Towards a logic of historical thought. New York: Harper & Row.

Harris, Ben (1980). Ceremonial versus critical history of psychology. American Psychologist, 35(2) , 218-219. (Note).

Harris, Ben (1979). Whatever Happened to Little Albert? American Psychologist, 34, 2, pp. 151-160. http://www.sussex.ac.uk/psychology/documents/harris_-1979.pdf

Katzer, J. & Sodt, J. (1973). An analysis of the use of statistical testing in communication research. The Journal of Communication, 23, 251-265.

Katzer, Jeffrey; Cook, Kenneth H. & Crouch, Wayne W. (1998). Evaluating Information: A Guide for Users of Social Science Research. 4 ed. Boston, MA: McGraw-Hill.

Lindsey, Duncan & Lindsey, T. (1978). The Outlook of Journal Editors and Referees on the Normative Criteria of Scientific Craftmanship. Quality and Quantity: European and American Journal of Methodology, 12, 45-62.

Lindsey, Duncan (1978). The scientific publication system in social science: A study of the operation of leading professional journals in psychology, sociology, and social work. San Francisco: Jossey-Bass.

Mahl, G. F. (1997). Distortion in secondary sources. Contemporary Psychology, 42(3), p. 272. (Letter).

Morgenstern, O. (1963). On the accuracy of economic statistics. 2nd. ed. Princeton, N.J.: Princeton University Press.

Rensberger, B. (1977). Fraud in research is a rising problem in science. New York Times, January 23, 1977, p. 44.

Samelson, F. (1980). J.B. Watson's Little Albert, Cyril Burt's twins, and the need for a critical science. American Psychologist, 35, 619–625.

Schoeneman, T. J.; Brooks, S.; Gibson, C; Routbort, J; Jacobs, D. (1994). Seeing the insane in textbooks of abnormal psychology - The uses of art in histories of mental illness. Journal for the Theory of Social Behavior, 24(2), 111-141. Overall, the pictures in historical chapters of textbooks examined in this study support a ''Whiggish'' account of history that celebrates the present and gives a distorted, incomplete rendering of the past.

Seligman, M.E.P. (1980). Harris on selective misrepresentation - A selective misrepresentation of Seligman. American Psychologist, 35(2), 214-215. (Note).

Stars, Susan Leigh & Gerson, Elihu M. (1987). The management and dynamics of anomalies in scientific research. Sociological Quarterly, 28(2), 147-169. This article examines the dynamics of anomalies as part of scientific work. Several types of anomaly are identified: mistakes, artifacts, fraud, and discovery; and typical trajectories for artifacts are described: the establishment of suspected artifacts, changes from unacceptable to acceptable, changes in significance, visibility, and means of control. The conditions under which an anomaly changes status are examined. A detailed example of an anomaly trajectory is presented, analyzing an anomaly in neuroscience research and tracing its career from 1870 to the present.

Teigen, K. H. (1994). Dodson - A law for all seasons. Theory & Psychology, 4(4) , 525-547. Abstract: The paper traces the vicissitudes of the Yerkes-Dodson law from 1908 to the present. In its original form, the law was intended to describe the relation between stimulus strength and habit-formation for tasks varying in discrimination difficultness. But later generations of investigations and textbook authors have rendered it variously as the effects of punishment, reward, motivation, drive, arousal, anxiety, tension or stress upon learning, performance, problem-solving, coping or memory; while the task variable has been commonly referred to as difficulty, complexity or novelty, when it is not omitted altogether. These changes are seldom explicitly discussed, and are often misattributed to Yerkes and Dodson themselves. The various reformulations are seen as reflecting conceptual changes and current developments in the areas of learning, motivation and emotion, and it is argued that the plasticity of the law also reflects the vagueness of basic psychological concepts in these areas.

Thomas, R. K. (2007). Recurring errors among recent history of psychology textbooks. American Journal of Psychology, 120(3), 477-495.
Abstract: Five recurring errors in history of psychology textbooks are discussed. One involves an identical misquotation. The remaining examples involve factual and interpretational errors that more than one and usually several textbook authors made. In at least 2 cases some facts were fabricated, namely, so-called facts associated with Pavlov's mugging and Descartes's reasons for choosing the pineal gland as the locus for mind-body interaction. A fourth example involves Broca's so-called discovery of the speech center, and the fifth example involves misinterpretations of Lloyd Morgan's intentions regarding his famous canon. When an error involves misinterpretation and thus misrepresentation, Thomas shows why the misinterpretation is untenable.

Tversky, A. & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110.

Vicente, K. J. & Brewer, W.F. (1993). Reconstructive remembering of the scientific literature. Cognition , 46(2) , 101-128. Abstract: In this paper we investigate the role of reconstructive memory in citation errors that occur in the scientific literature. We focus on the case of de Groot's (1946) studies of the memory for chess positions by chess experts. Previous work has shown that this research is very often cited incorrectly. In Experiment 1 we show that free recall of this work by research psychologists replicates most of the errors found in the published literature. Experiment 2 shows that undergraduates reading a correct account of the de Groot study also make the same set of errors in recall. We interpret these findings as showing that consistent errors in secondary accounts of experimental findings are frequently reconstructive memory errors due to source confusion and schema-based processes. Analysis of a number of other examples of scientific literature that have been frequently cited incorrectly add additional support to the reconstructive account. We conclude that scientists should be aware of the tendency of reconstructive memory errors to cause violations of the scientific norm of accurate reporting of the scientific literature.

Wandt, E. (Ed.). (1965). A cross-s educational research. New York: McKay. (pp. 1-7).

Wolins, L. (1962). Responsibility for raw data. American Psychologist, 17, 657-658.


Back to Quality - - Fallacist's fallacy - - - Source criticism

Entry Added: July 27, 2008
Last Update: October 23, 2008