Conference Paper · 2023
Best Short Paper Nominee · 2023
Wang, G., & Burton-Jones, A. (2023). Miscellaneous citation groups are buried in journal impact factors: A case of the information systems field. In Proceedings of the 44th International Conference on Information Systems (pp. 1–9). Association for Information Systems. https://aisel.aisnet.org/icis2023/generalis/generalis/11/
Explore the original contribution
Interactive paper explorationStart
All citations enter the same numerator, obscuring differences among the communities that generated them.
Reconsider
The paper's five recalibrations produce substantially different rankings for the same set of journals.
Implication
The next question is what community the metric represents—and whether that is the influence the journal intends to have.
Disagreement between JIF rankings and expert judgments is often approached as a metric-design problem or attributed to particular journal and article characteristics.
The paper's five recalibrations of 19 IS journals reveal different rankings across citing communities. Citation sources differ both by disciplinary affiliation and by ranking category, and those groups are buried inside an aggregate JIF.
Concept explorer
Select a concept to see its role in this paper’s contribution.
The paper foregrounds distinct, nested citing communities that aggregate JIFs conceal.
Its exploratory calculation retains the five-year denominator but restricts the numerator to citations from a chosen journal group.
Group definitions can shift, overlap or be subdivided, so a group-specific metric is not automatically an objective solution.
Five recalibrations compare AIS-recommended journals and higher- or moderately ranked IS and non-IS journals. The resulting differences reveal a source-composition explanation of ranking disagreement.
PDF pp. 3–8; printed pp. 2–7.
The paper argues that refining an algorithm alone may miss the nested and changing social groups that give citation counts their meaning.
PDF p. 9; printed p. 8.
Editors and evaluators can examine the audiences generating citations before drawing conclusions about a journal's influence.
PDF pp. 8–9; printed pp. 7–8.
Treat group-specific calculations as a prompt for investigation, rather than a ready-made replacement ranking. The paper explicitly says their effectiveness and general applicability remain untested.
PDF pp. 9–10; printed pp. 8–9.
Why this matters
A multidisciplinary field can appear different depending on which audience its metrics privilege. Evaluation systems need to examine these audience boundaries instead of treating a field as a single homogeneous citation community.
Discuss this researchAn observational, exploratory research-in-progress paper using November 2022 JCR data for 19 selected IS journals. Findings are provisional and require broader replication and systematic statistical validation. It continues the authors' 2023 MISQ editorial analysis; it is not an independent definitive replication.
Data selection, five citing-group definitions, calculation and comparative results.
Fluid boundaries, limited inference, untested applicability and potential metric gaming.
This exploration is based on the author-supplied full text. It distinguishes the paper’s own contribution from established concepts it draws on. For formal citation, consult the publication.