ScholarGPS™ Ranking Terminology and Methodology

The ranking terminology and methodology used by ScholarGPS™ was developed by a panel of distinguished scholars from academia and other non-academic institutions over a period spanning several years. After its initial development, the methodology was submitted to various academic leaders for their comments and critical review for further refinement. Previous ranking methodologies have been plagued with problems such as use of subjective qualitative components, a lack of Field /Discipline /Specialty sensitivity, confusion over the inclusion or exclusion of self-citations, whether to account for the number of authors on individual publications, and a perceived bias in favor of senior scholars. ScholarGPS™ Rankings were developed on several foundational principles: that a ranking terminology and methodology should exhibit simplicity, transparency and fairness, and that the population of scholars considered within the methodology should encompass every individual (active or retired) across the entire globe who meets our definition of a scholar. In consideration of these foundational principles, ScholarGPS™ Rankings are determined purely quantitatively, and are based on metrics which capture the productivity, the impact, and the quality of individual scholars as well as institutions. The ranking methodology is presented in graphical form in the Ranking Flowchart.

Ranking Terminology

Rankings for both scholars and institutions are evaluated in four ranking categories:

  1. Overall (All Fields)
  2. by Field
  3. by Discipline
  4. by all Specialties with which they are associated

Four ranking metrics are calculated across each of the preceding four categories for scholar rankings:

  1. Productivity (archival publication count)
  2. Impact (citation count)
  3. Quality (h-index)
  4. ScholarGPS™ Ranks(geometric mean[4] of productivity, impact, and quality scores)

Additionally, three ranking options are available for each ranking metric for scholar rankings:

  1. Duration:
    • Lifetime publications, or
    • Publications from the prior five years
  2. Author contribution credit:
    • No weighting based on the number of authors, or
    • Weight authors' publication and citation counts based on the number of authors on a publication. For example, if two authors are listed on a publication, each scholar will be credited with 0.5 publications and half of the publication's citations. The scholar's fractional h-index[3] is also calculated based on these weighted citation counts.
  3. Citation inclusion:
    • Include all citations, or
    • Exclude self-citations (citations from publications by the same author)

Ranking metrics are presented both in terms of standard competition rank[1] (e.g., "1,2,3,3,5,6...") and top percentage ( based on the complement of the percentile rank[2], e.g., "Top 3%").

Scholar rankings form the basis of the institutional rankings -- See Institutional Rankings below.

Scholar Ranking

Scholars are ranked in each category and by each ranking metric for various criteria. The ranking proceeds as follows.

First, the top percentage rank[2] of a scholar within a category for any ranking metric is calculated as follows:

`N` = Total scholars within a category

`R` = Standard competition rank of the scholar within a category

`F` = Frequency of scholars with rank `R`

Top Percentage Rank (`TPR`) = `100 - [((N - R + 1) - (0.5 xx F))/N xx 100]`

where

Productivity `TPR` = `TPR_p` = Top percentage rank by publication

Impact `TPR` = `TPR_c` = Top percentage rank by citation count

Quality `TPR` = `TPR_h` = Top percentage rank by h-index

Note that `TPR` is based on the standard Percentile Rank[2], but is not rounded to the nearest integer i.e., is given as a decimal number.

Next, the ScholarGPS™ Ranksof each individual scholar, which is reported in terms of a pair of rankings `[SR, STPR]` is determined by calculating the composite score (defined as the geometric mean[4])

`S = \root{3}{TPR_p xx TPR_c xx TPR_h}`

From the distribution of these composite scores, we determine the pair

`text(ScholarGPS™ Ranks) = [text(ScholarGPS Rank),text(ScholarGPS Top Percentage Rank)]`

where

`text(ScholarGPS Rank (SR)) = text(Standard Competition Rank(S))`

and

`text(ScholarGPS Top Percentage Rank (STPR)) = text(Top Percentage Rank(SR))`

As evident, a scholar's ScholarGPS™ rank `SR` and ScholarGPS™ top percentage rank `STPR` are determined by that scholar's `S` score relative to the distribution of `S` scores of scholars in the population to which the scholar is being compared. A scholar's ScholarGPS™ Rankspair can be determined relative to the universe of scholars (Overall), relative to scholars in the scholar's Field or Discipline, or relative to scholars in any Specialty associated with the scholar.

Recall that each ranking metric can be determined relative to a Field, a Discipline, or a Specialty; can be computed with or without author and citation weighting; and can be over the scholar's lifetime or the prior five-year period. In fact, each ranking metric `M` is a multi-indexed entity `M_{F,T}^{\alpha \beta \gamma}` where `\alpha` is the duration over which the metric is determined (lifetime, prior five years), `\beta` indicates the weighting type (weighted/unweighted authors/citations), `\gamma` indicates whether self-citations are or are not included, `F` indicates the category over which the metric has been determined (Overall, Field, Discipline, or Specialty) and `T` indicates the metric type (productivity, impact, quality, or composite score). To avoid this cumbersome notation, and because it will be clear from the context what the complete specification of the metric is, we drop all but the most necessary indices. For example, `M_p` would be some metric related to productivity, and the context will identify the other circumstances under which it was determined.

We define the metric pSTPR to be the STPR metric computed with respect to publications and citations weighted by author, with no self-citations. This metric personalizes the STPR metric to individual scholars and is routinely used by ScholarGPS to determine such rank distinctions as Highly Ranked Scholar or Top Scholar (see below).

Scholar Activity and Scholar Recognition

For certain purposes (such as scholar ranking or institutional ranking), ScholarGPS classifies all scholars as being active or inactive. ScholarGPS defines an active scholar as one who has published an archival publication within the preceding five year period of the current year. Where the distinction applies, users will be given the option to see a result based on active scholars or on all scholars. All scholars thus includes retired and even deceased scholars (but that might be included in the active category due to the five-year capture window). The five year window was chosen to capture scholars whose publication activity level might be conducted over longer periods of time (e.g., scholars in certain humanities).

ScholarGPS identifies scholars who attain levels of excellence that merit special recognition. ScholarGPS introduces two such classes: Highly Ranked Scholars and Top Scholars. In both cases, the figure of merit used to determine such scholars is the personalized ScholarGPS Top Percentage Rank (pSTPR, defined above). Scholars may be placed into one of these categories based on their lifetime achievements, or based on their ranking as determined over the preceding five year period. Scholars are compared within the peer group appropriate to the recognition, which may be: (i) the entire universe of scholars ("Overall"); (ii) within the scholar's specific Field; (iii) within the scholar's specific Discipline or (iii) within each Specialty that the scholar is associated with. Thus, a Highly Ranked Scholar or Top Scholar may be given the award for their overall performance, and/or their performance in their Field, Discipline and Specialties. Some scholars receive the award in multiple categories.

Highly Ranked Scholars represent the most elite group of scholars whose scholarly excellence places them at the very top of their peer group. Such scholars lie in the top 0.05% of all scholars in that peer group as determined by the relevant pSTPR (Overall, Field, Discipline or Specialty). Highly Ranked Scholars are always identified on their profile pages, as well as being easily found using the ScholarGPS Highly Ranked Scholar page.

Top Scholars are those individuals whose pSTPR places them in the top 0.5% of all scholars. The Top Scholar category is thus more encompassing than the elite Highly Ranked Scholar category, and allows for the recognition of more individuals particularly in domains of expertise where there might be few Highly Ranked Scholars. Top Scholars still represent a class of excellent scholars, and should be considered subject experts. ScholarGPS allows for the identification of Top Scholars in three ways: Top Scholars by Expertise, which identifies Top Scholars by Field, Discipline or Specialty, Top Scholars by Institution, which identifies Top Scholars who are associated with a specific institution, and Top Scholars by Country, which identifies Top Scholars by the country they are associated with.

Institution Ranking

Institutional Rankings are based on the quality and quantity of active scholars in each institution who are highly productive (number of publications) and have generated outstanding work of meaningful impact (citations) and excellent quality (h-index). Data used in the calculations of institutional rankings are based on the lifetime contributions of the included scholars, weighting each publication and citation by the number of authors, and excluding self-citations. Uniquely, ScholarGPS™ rankings are available separately for academic institutions (universities and colleges), and non-academic institutions (private or public industries including health care systems, and research institutions).

In general, the ranking categories for an institution are representative of the following areas within an institution:

  1. Overall (All Fields): The institution overall
  2. Fields: Schools or Colleges (e.g., College of Engineering)
  3. Disciplines: Departments or divisions
  4. Specialties: Institutes or research centers

The critical mass of quality scholars necessary for institutional rankings was determined based on an extensive optimization analysis to (1) make institutional rankings nearly independent of institution size and (2) avoid penalizing institutions for scholars who are inactive, or active in only a small number of focused research areas.

Specialties

Since Specialties are characterized by focused scholarly activities that may span across multiple Disciplines, and because publication and citation traditions can vary substantially from Discipline-to-Discipline and Specialty-to-Specialty, care must be taken in the ranking of institutions relative to this ranking category. However, regardless of the Specialty, the Institution's ranking in any Specialty must reflect both the quality and quantity of outstanding scholars associated with the Specialty. Based on extensive internal research and the recommendations of our panel of distinguished advisors, we calculate the institutional rankings in Specialties based on the active scholars in each Specialty.

The institutional Specialty Rank Score is defined as:

`l_s` = `sum_{a \in A}(100-STPR_a)`

where

`l_s` is the Institutional Rank Score in Specialty `s`

`A` is the set of admissible scholars in the institution relative to Specialty `s`

`STPR_a` is the ScholarGPS™ Top Percentage Rank (see Scholar Ranking above) of Scholar `a`


Disciplines

As for Specialties, Disciplines are characterized by focused scholarly activities, and because publication and citation traditions can vary substantially from Discipline-to-Discipline, care must be taken in the ranking of institutions relative to Disciplines. Based on extensive internal research and the recommendations of our distinguished advisors, we calculate the institutional rankings in Disciplines based on the active scholars in each Discipline.

The institutional Discipline Rank Score is defined as:

`l_d` = `sum_{a \in A}(100-STPR_a)`

where

`l_d` is the Institutional Rank Score in Discipline `d`

`A` is the set of admissible scholars in the institution relative to Discipline `d`

`STPR_a` is the ScholarGPS™ Top Percentage Rank (see Scholar Ranking above) of Scholar `a`


Fields

ScholarGPS associates each Discipline with one and only one Field. Therefore, institutional excellence in the Disciplines that comprise a Field will yield a high institutional ranking in the Field. Based on our research and the recommendations of our advisors, we calculate the institutional rankings in Fields based on the active scholars in each Discipline for those Disciplines belonging to the Field of interest.

The institutional Field Rank Score is defined as:

`l_f` = `sum_{a \in A}(100-STPR_a)`

where

`l_f` is the Institutional Rank Score in Field `f`

`A` is the set of admissible scholars associated with the institution relative to the Disciplines belonging to Field `f`

`STPR_a` is the ScholarGPS™ Top Percentage Rank (see Scholar Ranking above) of Scholar `a`


Overall (All Fields)

An institution's overall ranking correlates to the institution rankings in the Disciplines that comprise the institution. Based on our research and the recommendations of our advisors, we calculate the institutional rankings based on the active scholars in each Discipline.

The institutional Overall (All Fields) Rank Score is defined as:

`l_o` = `sum_{a \in A}(100-STPR_a)`

where

`l_o` is the Overall (All Fields) Institutional Rank Score

`A` is the set of admissible scholars associated with the institution across all Disciplines

`STPR_a` is the ScholarGPS™ Top Percentage Rank (see Scholar Ranking above) of Scholar `a`


Other Considerations

Publications with over 20 authors are not included in scholarly profiles or in ScholarGPS™ Ranking schemes because in such cases it is often too difficult to ascertain individual contributions to the publication.

A scholar's ranking in a category is presented only if it is in the top 50%.

In order for a Specialty to be included in Institutional Rankings, it must have a critical mass of at least 6,000 scholar profiles associated with it.

A maximum of up to 100 institutions may be ranked for a Specialty.

Notice

It is likely impossible to find any encompassing set of metrics that would create the perfect scholarly ranking model — one that would be embraced by all scholars and which would rank scholars with absolute and complete fairness and accuracy. ScholarGPS™ recognizes that great care should be taken in using any scores (whether those from ScholarGPS™ or any other ranking system) as the final statement of any scholar’s true productivity or value. Users should therefore not construe a lower score or ranking as necessarily representative of lesser influence or prestige. While ScholarGPS™ metrics are derived largely from the traditional metrics used in the sciences and social sciences, other parameters including quality of teaching, outreach activities, as well as other modes of scholarly or artistic dissemination such as exhibitions, performances, and musical compositions should be considered by others when warranted.

References

[1] Ranking. (2022, March 24). In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Ranking&oldid=1078999715#Standard_competition_ranking_(%221224%22_ranking)

[2] Percentage Rank. (2022, January 10). In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Percentile_rank&oldid=1064785741

[3] Koltun, V., & Hafner, D. (2021). The h-index is no longer an effective correlate of scientific reputation.
PLOS ONE, 16(6). doi: 10.1371/journal.pone.0253397

[4] Geometric mean. (2023, February 16). In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Geometric_mean&oldid=1139765330

[5] Standard score. (2023, March 29). In Wikipedia.
https://en.wikipedia.org/w/index.php?title=Standard_score&oldid=1147224272