Evidence · Research methods

Citation analysis: what citation counts do and don't tell you

Citation counts measure influence, not validity. In a large UK research assessment, field-normalised citations correlated positively with expert quality scores in every field, but only weakly to moderately, and in the behavioural and brain sciences citations and impact factors barely predicted replicability or accurate statistics. Who cites whom also reflects social ties, so citations work better as an aggregate signal than as a verdict on a single paper.

Definition

Citation analysis, part of scientometrics, uses the references papers make to one another to map the structure of a field and to measure the influence of articles, authors and journals. It began as an information retrieval tool that let readers follow ideas through the literature, and later became a tool for evaluating journals, researchers and institutions. Most debate about its validity concerns that second use.

What the research finds

From retrieval tool to evaluation tool

The citation index was proposed in 1955 as a way to link documents through the ideas they share, launched as the Science Citation Index in 1964 and extended to ranking journals in 1972. Its originator later described how its main use drifted from retrieval to research evaluation. Citation data can also locate a field's historical roots: counting cited references by their publication year reveals peaks that point to seminal works.^[1]^[2]^[3]^[4]^[5]^[6]

Citations correlate with quality, but loosely

Across 87,739 journal articles in 34 UK assessment units, field-normalised citations correlated positively with expert quality scores in all fields, from very weak (0.1) to strong (0.5), and no citation threshold guaranteed excellent quality. Authors' own quality ratings of their papers also rise with citations, most for scientific importance, but the link is too weak to judge single articles reliably.^[7]^[8]

Citations barely track replicability

Using 45,144 articles, 667,208 statistical tests and 190 replication attempts in the behavioural and brain sciences, citation counts and journal impact factors were weak and inconsistent predictors of accurate statistical reporting, evidential value and replicability, and were sometimes negatively related to them.^[9]

Citing is partly a social act

Among UK economists, an author was more likely to cite another if they had co-authored, worked in the same department, studied at the same university or shared political views, beyond working on similar topics. Individual authors also build stable citation identities over time; social ties shape them, but the perceived relevance of the cited work remains the main motive.^[10]^[11]

What predicts more citations

A broad review found many factors associated with more citations in some fields, years or journals, but the strength and even direction of each varies with the papers examined and causal evidence is rare. The most consistent predictors are the citation rate of the journal, the number of authors, properties of the reference list and international collaboration. Citation data predict article quality better in medicine, health and the physical sciences than in the social sciences.^[12]

Indicators need normalisation and care with skew

Citation indicators depend on the database used, which items are counted, normalisation for field differences and how co-authored papers are credited. Because citation distributions are highly skewed, top-percentile indicators can mislead when the least-cited papers deviate from the expected pattern. Counting distinct citing authors, rather than citations, has been proposed to capture the breadth of a work's influence.^[13]^[14]^[15]

In marketing academia, prestige signals shape careers

Five years of data on marketing doctoral candidates in the US showed that the research productivity of the degree-granting department was the strongest cue for job-market interviews, campus visits and starting salary. A top publication, a defended proposal with data and attendance at a leading doctoral consortium carried a clear salary premium.^[16]

How strong is the evidence?

Most of this literature comes from the natural and health sciences, economics and information science. No study reviewed here measures what drives citations to marketing articles specifically, so those patterns are carried over rather than tested in marketing. Two of the indicator studies are available as preprints (one a widely cited review), and all the citation-driver findings are correlational: nothing here shows that any factor causes more citations. The consistent message is that citation counts are skewed, field-dependent and partly social, so they are more trustworthy at aggregate levels than for judging a single paper.

References

Eugene Garfield (1955). Citation Indexes for Science: A New Dimension in Documentation Through Association of Ideas. Science, 122(3159).
Eugene Garfield (1964). 'Science Citation Index' — A New Dimension in Indexing. Science 144(3619); reprinted in Essays of an Information Scientist, Vol. 7.
Eugene Garfield (1972). Citation Analysis as a Tool in Journal Evaluation. Science.
Eugene Garfield (2006). Commentary: Fifty years of citation indexing. International Journal of Epidemiology.
Werner Marx, Lutz Bornmann, Andreas Barth, Loet Leydesdorff (2013). Detecting the historical roots of research fields by reference publication year spectroscopy (<scp>RPYS</scp>). Journal of the Association for Information Science and Technology.
Lutz Bornmann, Andreas Thor, Werner Marx, Loet Leydesdorff (2016). Identifying seminal works most important for research fields: Software for the Reference Publication Year Spectroscopy (RPYS). Collnet Journal of Scientometrics and Information Management.
Mike Thelwall, Kayvan Kousha, Emma Stuart, Meiko Makita, Mahshid Abdoli, Paul Wilson, Jonathan Levitt (2023). In which fields are citations indicators of research quality?. Journal of the Association for Information Science and Technology.
Dag W. Aksnes, Fredrik Niclas Piro, Lone Wanderås Fossum (2023). Citation metrics covary with researchers’ assessments of the quality of their works. Quantitative Science Studies.
Michael R. Dougherty, Zachary Horne (2022). Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences. Royal Society Open Science.
Carlo D’Ippoliti, Lucio Gobbi, Christian Alexander Mongeau Ospina, Giulia Zacchia (2023). Social determinants of citations: An empirical analysis of UK economists. Kyklos.
Howard D. White (2001). Authors as citers over time. Journal of the American Society for Information Science and Technology.
Kayvan Kousha, Mike Thelwall (2023). Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper. Journal of the Association for Information Science and Technology.
Ludo Waltman (2016). A review of the literature on citation impact indicators. Journal of Informetrics.
Alonso Rodríguez-Navarro (2024). Distinct citation distributions complicate research evaluations. A single indicator that universally reveals research efficiency cannot be formulated. arXiv preprint 2407.09138.
Beniamino Cappelletti–Montano, Silvia Columbu, Stefano Montaldo, Monica Musio (2021). New perspectives in bibliometric indicators: Moving from citations to citing authors. Journal of Informetrics.
Angeline G. Close, Julie Guidry Moulard, Kent B. Monroe (2010). Establishing human brands: determinants of placement success for first faculty positions in marketing. Journal of the Academy of Marketing Science.

Bibliographic details only. Follow the DOI to read each study at its publisher.

Building on this evidence?

ScholarSignal checks a new research idea against the closest existing work and an editor rubric, and tells you whether the contribution is new enough for the journals you target.

Evaluate my idea for free