At many non-profit and for-profit companies, it is important to measure the reach, influence, and success of the organization’s efforts. This is not always easy and sometimes not even feasible, but when quantification is possible, it should be carried out to gauge the impact of a task on an audience. This is often done through media monitoring, but what about measuring your organizations other outputs?
An organization’s publications (e.g. reports, conference proceedings, authored works, etc.) are a good starting place where impact can be measured, and in organizations where an information professional is employed, the work is cut out for them. Tracking and measuring the impact of your organization’s output is a clear-cut way to demonstrate your value, contribute to department goals, and generate useful statistics that will benefit your employer.
Many information professionals will be familiar with the concept of citation analysis, and its application in the activities of scholarly research. Citation analysis is a method of bibliometrics that attempts to gauge the impact of an author’s work through the frequency of being cited in other works. The process of citing acknowledges previous works, and debatably, implies that a work is significant. The practice of measuring the number of cites is predominantly of interest to authors and stakeholders in the sciences, and some in economics, but is helpful in other fields as well. Citation analysis is also prevalent (and of greater interest) in academic environments where an author’s published research enhances their clout, and professional profile.
Because of its wide acceptance in academic settings, many of the tools available to analyze citations do not incorporate “non-scholarly” literature (often termed “grey literature” e.g. working papers), since they do not undergo the same scholarly publishing process. The majority of grey literature is not indexed in proprietary or free citation databases, because it is not controlled by commercial publishing. Consequently, the trickiest aspect of measuring the impact of your organization’s output is that citation indexes will often not include the publications you seek to evaluate.
With these obstacles is mind, I compiled a list of tools for conducting citation analysis, some are fee-based and others are free:
- Thomson Reuter’s Web of Science/Knowledge: (http://thomsonreuters.com/products_services/science/science_products/a-z/web_of_science/) (subscription required)
Web of Science (WoS) is the premier citation index and includes an extensive range of very impressive analysis tools. With a lofty price tag, not every organization will be able to afford this product (and unfortunately, they do not offer a non-profit rate). The indexed content is drawn almost exclusively from scholarly journals and conference proceedings, and its historical coverage is unmatched. Free 30-day trials are available, and you can select from seven subject database packages based on subject area. Thomson Reuters also provides access to “Highly Cited Research” (http://www.highlycited.com), a free resource to identify highly cited authors and works. WoS is tremendously useful for tracking cites in academic journals, but quite disappointing for measuring citations from grey literature.
- Elsevier’s Scopus: http://www.scopus.com/home.url (subscription required)
Scopus is a slightly less expensive alternative to Web of Science, and boasts having “the largest abstract and citation database of research literature and quality web sources.” Scopus has a free author/organization lookup tool, which I highly recommend testing. This will give you an idea of your organization’s cited works in the Scopus indexed content. Similar to WoS, grey literature is relatively absent from this database.
- Google Scholar: (http://scholar.google.ca/) (free)
Using the advanced Google Scholar search option, you can perform searches by author name, or by affiliated organization to retrieve articles where either are mentioned or cited. This can be tricky however, as it is not possible to narrow search results to the bibliography or footnotes. Your search results might include many unwanted items. That said, Google does a much better job than WoS or Scopus of retrieving obscure and grey literature citation mentions.
- Publish or Perish: (http://www.harzing.com/pop.htm) (free)
Public or Perish (PoP) uses Google Scholar citations, and runs as standalone software that links to the web. You must first download PoP, then perform your citation queries to retrieve the analysis of an author or publisher’s works. It is less accurate than the fee-based products like Scopus or WoS, but PoP is also quite straightforward about its shortcomings. PoP performs much better than Google on the web, and as a free software, is worthwhile and effective.
- CiteSeerX: (http://citeseerx.ist.psu.edu/) (free)
CiteSeerX focuses on (but is not limited to) literature in the areas of computer and information-science. As an index database, CiteSeerX is unique for adding complex metadata to its contents, which enable a greater capacity for linking documents, and locating related materials. Documents are automatically harvested from the web, so indexed content is continuously up-to-date. Try this tool in combination with others, as depending on the field of your organization and its outputs, CiteSeerX’s subject specificity might not track your publications.
- Scirus: (http://www.scirus.com/) (free)
Scirus is a science-specific index database encompassing a very wide range of web pages, government resources, academic articles, and special information sources (e.g. patent data from LexisNexis, technical reports from NASA, institutional digital repositories, etc.). Scirus’ capabilities rival some fee-based indexes, and it also includes an impressive range of non-scholarly grey literature content.
- RePEc: (http://repec.org/) (free)
Research Papers in Economics (RePEc) is a database of over one million items, largely in the social sciences, economics, finance, and computer sciences. The items in RePEc are unpublished papers, non-commercially published materials, and also pre-published versions of academic articles. RePEc is useful for many types of organizations that produce literature because it encourages free and open dissemination of these materials throughout a variety of outlets. It works by users (e.g. authors of works, or information professionals on behalf on their organization) uploading publications and providing the metadata and bibliographic information. To perform citation analyses, RePEc has an embedded system for tracking these uploaded items. With this, you can quantify how often your publications are downloaded, cited, and shared. RePEc is also a collaborative effort with many other repositories, such as EconLit, EDIRC, and IDEAS.
Although these options may not provide the complete picture of your organization’s impact, I found that using a combination of these tools can produce a rough, but useful solution. Optimistically, there are ongoing developments in this field, such as Google Scholar Citations (http://scholar.google.ca/intl/en/scholar/citations.html), and Open Grey (www.opengrey.eu), and the maturation of these products will benefit the efforts of citation analysis for grey literature.
If your organization produces publications for stakeholders or a general audience, consider tracking cites through the abovementioned means. It will strengthen your value to the organization, and provide them with a quantifiable source of evidence-based measurement for organizational outputs.
Listed below are additional sources that review and compare citation analysis tools:
- Badia, Giovanna. “Google Scholar out-performs many subscription databases when keyword searching.” Evidence Based Library and Information Practice, Vol. 5, No. 3, 2010: 39-41. Available from https://ejournals.library.ualberta.ca/index.php/EBLIP/article/view/8543
- Bosman, J., et al. “Scopus reviewed and compared.” Utrecht University Library, 2006. Available from http://igitur-archive.library.uu.nl/DARLIN/2006-1220-200432/Scopus%20doorgelicht%20&%20vergeleken%20-%20translated.pdf
- Connor, E. “Searching for science: a descriptive comparison of CiteSeer, FirstGov for Science, and Scirus.” Journal of Electronic Resources in Medical Libraries, Vol. 2, No. 2, 2005: 35-47.
- Pauly, D. And Stergiou K. I. “Equivalence of results from two citation analyses: Thomson ISI’s Citation Index and Google’s Scholar service.” Ethics in Science and Environmental Politics, December 22, 2005: 33-35. Available from http://www.int-res.com/articles/esep/2005/E65.pdf
- Science Intelligence and InfoPros. “Google Scholar Vs Web of Science: McDonalds Vs a Gourmet restaurant?!” December 7, 2010. Available from http://scienceintelligence.wordpress.com/2010/12/07/google-scholar-vs-web-of-science-macdonalds-vs-a-gourmet-restaurant/
Zack Osborne 