Citation Analysis: Measuring impact and delivering value to your organization

At many non-profit and for-profit companies, it is important to measure the reach, influence, and success of the organization’s efforts. This is not always easy and sometimes not even feasible, but when quantification is possible, it should be carried out to gauge the impact of a task on an audience. This is often done through media monitoring, but what about measuring your organizations other outputs?

An organization’s publications (e.g. reports, conference proceedings, authored works, etc.) are a good starting place where impact can be measured, and in organizations where an information professional is employed, the work is cut out for them. Tracking and measuring the impact of your organization’s output is a clear-cut way to demonstrate your value, contribute to department goals, and generate useful statistics that will benefit your employer.

Many information professionals will be familiar with the concept of citation analysis, and its application in the activities of scholarly research. Citation analysis is a method of bibliometrics that attempts to gauge the impact of an author’s work through the frequency of being cited in other works. The process of citing acknowledges previous works, and debatably, implies that a work is significant. The practice of measuring the number of cites is predominantly of interest to authors and stakeholders in the sciences, and some in economics, but is helpful in other fields as well. Citation analysis is also prevalent (and of greater interest) in academic environments where an author’s published research enhances their clout, and professional profile.

Because of its wide acceptance in academic settings, many of the tools available to analyze citations do not incorporate “non-scholarly” literature (often termed “grey literature” e.g. working papers), since they do not undergo the same scholarly publishing process. The majority of grey literature is not indexed in proprietary or free citation databases, because it is not controlled by commercial publishing. Consequently, the trickiest aspect of measuring the impact of your organization’s output is that citation indexes will often not include the publications you seek to evaluate.

With these obstacles is mind, I compiled a list of tools for conducting citation analysis, some are fee-based and others are free:

  • Thomson Reuter’s Web of Science/Knowledge: (http://thomsonreuters.com/products_services/science/science_products/a-z/web_of_science/) (subscription required)
    Web of Science (WoS) is the premier citation index and includes an extensive range of very impressive analysis tools. With a lofty price tag, not every organization will be able to afford this product (and unfortunately, they do not offer a non-profit rate). The indexed content is drawn almost exclusively from scholarly journals and conference proceedings, and its historical coverage is unmatched. Free 30-day trials are available, and you can select from seven subject database packages based on subject area. Thomson Reuters also provides access to “Highly Cited Research” (http://www.highlycited.com), a free resource to identify highly cited authors and works. WoS is tremendously useful for tracking cites in academic journals, but quite disappointing for measuring citations from grey literature.
  • Elsevier’s Scopus: http://www.scopus.com/home.url (subscription required)
    Scopus is a slightly less expensive alternative to Web of Science, and boasts having “the largest abstract and citation database of research literature and quality web sources.” Scopus has a free author/organization lookup tool, which I highly recommend testing. This will give you an idea of your organization’s cited works in the Scopus indexed content. Similar to WoS, grey literature is relatively absent from this database.
  • Google Scholar: (http://scholar.google.ca/) (free)
    Using the advanced Google Scholar search option, you can perform searches by author name, or by affiliated organization to retrieve articles where either are mentioned or cited. This can be tricky however, as it is not possible to narrow search results to the bibliography or footnotes. Your search results might include many unwanted items. That said, Google does a much better job than WoS or Scopus of retrieving obscure and grey literature citation mentions.
  • Publish or Perish: (http://www.harzing.com/pop.htm) (free)
    Public or Perish (PoP) uses Google Scholar citations, and runs as standalone software that links to the web. You must first download PoP, then perform your citation queries to retrieve the analysis of an author or publisher’s works. It is less accurate than the fee-based products like Scopus or WoS, but PoP is also quite straightforward about its shortcomings. PoP performs much better than Google on the web, and as a free software, is worthwhile and effective.
  • CiteSeerX: (http://citeseerx.ist.psu.edu/) (free)
    CiteSeerX focuses on (but is not limited to) literature in the areas of computer and information-science. As an index database, CiteSeerX is unique for adding complex metadata to its contents, which enable a greater capacity for linking documents, and locating related materials. Documents are automatically harvested from the web, so indexed content is continuously up-to-date. Try this tool in combination with others, as depending on the field of your organization and its outputs, CiteSeerX’s subject specificity might not track your publications.
  • Scirus: (http://www.scirus.com/) (free)
    Scirus is a science-specific index database encompassing a very wide range of web pages, government resources, academic articles, and special information sources (e.g. patent data from LexisNexis, technical reports from NASA, institutional digital repositories, etc.).  Scirus’ capabilities rival some fee-based indexes, and it also includes an impressive range of non-scholarly grey literature content.
  • RePEc: (http://repec.org/) (free)
    Research Papers in Economics (RePEc) is a database of over one million items, largely in the social sciences, economics, finance, and computer sciences. The items in RePEc are unpublished papers, non-commercially published materials, and also pre-published versions of academic articles. RePEc is useful for many types of organizations that produce literature because it encourages free and open dissemination of these materials throughout a variety of outlets. It works by users (e.g. authors of works, or information professionals on behalf on their organization) uploading publications and providing the metadata and bibliographic information. To perform citation analyses, RePEc has an embedded system for tracking these uploaded items. With this, you can quantify how often your publications are downloaded, cited, and shared. RePEc is also a collaborative effort with many other repositories, such as EconLit, EDIRC, and IDEAS.

Although these options may not provide the complete picture of your organization’s impact, I found that using a combination of these tools can produce a rough, but useful solution. Optimistically, there are ongoing developments in this field, such as Google Scholar Citations (http://scholar.google.ca/intl/en/scholar/citations.html), and Open Grey (www.opengrey.eu), and the maturation of these products will benefit the efforts of citation analysis for grey literature.

If your organization produces publications for stakeholders or a general audience, consider tracking cites through the abovementioned means. It will strengthen your value to the organization, and provide them with a quantifiable source of evidence-based measurement for organizational outputs.

Listed below are additional sources that review and compare citation analysis tools:

  1. Badia, Giovanna. “Google Scholar out-performs many subscription databases when keyword searching.” Evidence Based Library and Information Practice, Vol. 5, No. 3, 2010: 39-41. Available from https://ejournals.library.ualberta.ca/index.php/EBLIP/article/view/8543
  2. Bosman, J., et al. “Scopus reviewed and compared.” Utrecht University Library, 2006. Available from http://igitur-archive.library.uu.nl/DARLIN/2006-1220-200432/Scopus%20doorgelicht%20&%20vergeleken%20-%20translated.pdf
  3. Connor, E. “Searching for science: a descriptive comparison of CiteSeer, FirstGov for Science, and Scirus.” Journal of Electronic Resources in Medical Libraries, Vol. 2, No. 2, 2005: 35-47.
  4. Pauly, D. And Stergiou K. I. “Equivalence of results from two citation analyses: Thomson ISI’s Citation Index and Google’s Scholar service.” Ethics in Science and Environmental Politics, December 22, 2005: 33-35.  Available from http://www.int-res.com/articles/esep/2005/E65.pdf
  5. Science Intelligence and InfoPros. “Google Scholar Vs Web of Science: McDonalds Vs a Gourmet restaurant?!” December 7, 2010. Available from http://scienceintelligence.wordpress.com/2010/12/07/google-scholar-vs-web-of-science-macdonalds-vs-a-gourmet-restaurant/
Advertisements

Professional Profiles Series

Recently I was interviewed by the President of the SLA-Toronto Student Group, Zoe Cliff.  Zoe asked if we could meet to chat about my career in special libraries for for their blog’s professional profiles series. I thought it would be neat to share some insight into job hunting, my experiences in libraries, and landing a job.

Below is a transcript of the interview. To read other professional profiles, visit the SLA-Toronto Student Group Blog here: http://slatsg.blogspot.com/

1. Describe your current position
I’m a Research Librarian in a special library at CIGI, a non-profit research organization (“think tank”) that focuses on international governance concerns: energy and environment, economics, development and security. I provide a wide range of library and research services to the organization’s researchers and experts. These users have strong academic credentials, extensive international research experience, and a wide range of policy expertise. As a solo-librarian, I’m responsible for nearly all aspects of maintaining the library: acquisitions, cataloguing, collection development, subscriptions and serials management, research requests and support, special projects, outreach and communicating with the wider organization. It’s a lot of fun!

2. Describe your educational background
I have an honours Bachelor of Arts from Acadia University in history and sociology, and a Master of Information Studies (MISt) from the University of Toronto, with a focus on Library & Information Science.

3. Describe your first job as a librarian or information professional etc. and subsequent career path
This is my very first job as a librarian. After graduating from my MISt, I worked briefly at the Information Centre of De Beers Canada (the mining company), and then landed my current position. I also held a handful of library positions while at the iSchool: I worked part-time at the Robarts Reference & Research Services desk, and did some cataloguing and reference for the Data, Map & Government Information department. In addition to this, I worked at the Royal Ontario Museum Library & Archives.Regarding my current job, I was initially hired on a contract basis as the Collections Librarian, but after some restructuring, I was promoted and hired as a permanent full-time Research Librarian.

4. How did your information training and background prepare you for the job you now have?
I think that a combination of my experiences prepared me for the job I have now. Specifically, I owe a great deal to my work experiences while studying at U of T. I worked hard to get as much library experience as I could, because I felt that I would get more out of hands-on experience in libraries than I would in the classroom. I was very lucky, and landed some really terrific student jobs. In those roles I had the opportunity to assist with in-depth reference for a range of academic disciplines (humanities, social sciences, business, government documents, data and statistics), I performed original and copy-cataloguing, got involved with knowledge management and Web 2.0, and worked with archives. I was exposed to a diversity of information settings, user groups, and subject areas, and I think that a mixture of these experienced helped me to thrive in my current position where I handle most aspects of the Library. When I saw the job posting at CIGI, I thought to myself, “…this job description was written for your background and experience: apply.”

5. What advice would you give someone who is currently doing his or her Master in Information?
As a relatively new grad myself, I don’t think that I have a ton of wisdom to share, but here are some bits of advice that you might find helpful:
  • I know it’s easy to get into doing this, but don’t just take soft skill “bird” courses. Especially if you’re looking for a job in special libraries, it’s crucial to have those practical, “hard skills”. For example: aspects of cataloguing; abilities with open source; knowing something about HTML, MySQL, or web design, data, etc. I think that employers are looking for people who can walk into a job with as little training as possible and who can make a real impact. Impress them with your knowledge, skills and innovative ideas!
  • Speak to a few people in the field who have jobs that you want: inquire about how they got to where they are, ask them to critique your resume and/or CV, and in general, get your name and face out there. A similar suggestion is to request to speak to someone in Human Resources at a large library where you’d like to work. While I was job hunting, I met with the Head of Human Resources at UTL to have a mock interview and resume critique, and it was an invaluable experience. They see a lot of resumes, after all.
  • Create an online presence! You should want to be found in a Google search on sites like LinkedIn, through your personal website, and on Twitter. It will show that you’re web-savvy, relevant, and have something to say. (N.B. At the same time, be vigilant about Facebook privacy settings… you don’t need potential employers to know too much about you.)
  • Get experience. Realistically, everyone at the iSchool will graduate with the same degree. So what really counts (and what will make you stand out), is your experience. I think any experience is good for your resume and getting a job. If you can volunteer, great. If you can get part-time jobs, even better.
  • In the end, I think that getting a job relies on the following: who you know, your experience, luck, and timing.

6. Any general advice for new information professionals?
Network with colleagues, collaborate and be nice. Sometimes I feel like there’s an aura of competition that exists within special libraries (but maybe it’s other places too), and I think it can make people uptight and insular. Participate in socials, activities, listservs, and collaborate. It would make our community even more pleasant than it already is.

7. What helpful lessons did you learn early in your career? Do any of them still apply today?
Don’t be overwhelmed, stay organized, and take it one day at a time. When I was at De Beers, I worked with the Manager of Information Services (it was just the two of us), and I was always impressed with her ability to stay calm under duress. The library was never in shambles, but there were some problem areas that I would have lost sleep over. I asked how she dealt with that kind of stress or pressure, and she said, “No library is perfect. You just have to deal with things in stride, and prioritize your time.” I sometimes feel overwhelmed when I think of all of the things that need to be done at my library, e.g. backlog of cataloguing, responding to requests, maintaining print collections, broken links on the digital library, etc.; but I try to stay organized and prioritize competing long-term tasks, one day at a time.

I <3 Veronica Mars

If you have never watched Veronica Mars… then you need to start! I’d heard of the show a few years ago, but it never really interested me because I thought it was just about a subversive girl with a catchy name, trudging through her high school years. Well, it kind of is like that.. but what I didn’t know, is that she is a detective! Stay with me here…

For any librarian who loves conducting research, helping others find what they’re seeking, and who really enjoys the problem-solving aspects of the profession, Veronica Mars will inspire and re-introduce how challenging and rewarding doing research can be. Librarians aren’t detectives, I know, but a lot of what research and reference librarians do is, in a way, detective work: Someone approaches you with a problem (e.g. reference question), you use the necessary reasoning to determine how to help them. You draw upon resources available to you, think of creative ways to find answers, and determine ways around obstacles to obtain valuable information. Veronica Mars uses databases, newspapers on microform, indexes, public search engines, personal contacts, libraries, police reports (classified information, but she has friends in important places), etc. She is a natural (and impeccable) librarian. Granted, Veronica Mars often uses unethical means to acquire information (trespassing, lying, disguising herself to mislead others, to name a few), her research tactics are brilliant, inventive, and almost always successful!

Plus, I love that she goes to the library. Give this show a chance. There are three seasons on DVD. You will be hooked.

ch-ch-ch-ch-changes

So far, 2010 has been a great year. The weather in Toronto has been fantastic, I had a terrific time attending the Super Conference, my first paper was published, and to top all of that, I was offered a dream job. Four weeks ago I started as the Collections Librarian with the Centre for International Governance Innovation (CIGI) in Waterloo, Ontario. CIGI is an independent research centre/think-tank that focuses on issues related to international governance and foreign relations, and has partnerships with academic institutions, policy-makers and businesses all over the world. Their library houses a specialized collection of books (formerly the print collection from the Canadian Institute of International Affairs), a strong digital collection of electronic resources and publications, and also the Canadian Foreign Relations Index (CFRI). In my role as a collections librarian, I work with the Manager of Content Services (“Head Librarian”) to manage the print resources, perform acquisitions and cataloguing, conduct outreach and promote CIGI’s unique and specialized collections, and assist with research requests. This position is truly a dream job, because it combines my longstanding interests in special libraries, advocacy and access to rare special collections and research resources, establishing or improving access points (cataloguing/indexing), and assisting with research. Overall I’m extremely pleased and feel very lucky to experience this challenging and satisfying position. While it is only a 14 month contract, it’s the perfect kind of commitment for right now.

Just wanted to post an update on what’s new! More to come in 2010!

“Home” for the Holidays

Ho ho ho, it’s Christmas Eve day!

I’m Presently in Fredericton, New Brunswick, spending time with my Mother over the Holidays. She moved here in January 2009 from my childhood home in Oshawa (A.K.A., The ‘Shwa’).

It was a bit of a pleasant shock to land at the airport and suddenly be immersed into a Winter Wonderland. There has been little to no snow in Toronto, contributing to my Scrooge-like attitude, but upon entering this snowy paradise, I am pumped for Yuletide traditions!

Spending the holidays in New Brunswick for the first time also makes me feel reflective and observant about my new “home-base” in the Maritimes. So on that note, Here Are A Few of My Favourite Things… about Fredericton:

  • City-wide Wireless Internet:
    Aptly called the “Fred-eZone“, the Municipality of Fredericton provides free Wi-Fi to its residents and visitors! What a marvelous Gift!
  • Fredericton reminds me of a smaller Ottawa:
    Being the capital city of the Province, Fredericton is dabbled with Provincial headquarters, historic buildings, and a certain feel that I find is similar to Ottawa… lots of lights downtown, and a lot of funding for parks, infrastructure, etc. It’s just a really nice place to be!
  • University of New Brunswick (UNB) & St. Thomas University:
    Atop the hill of Fredericton are the campuses  of UNB and St. Thomas University. I definitely have a thing for small primarily-undergraduate universities, so I have an appreciation and love for visiting campuses of these types of institutions. The buildings and campuses tend to be quaint, old, and historic, and they remind me of my own academic experiences at Acadia. They also have gorgeous academic libraries and research centres!
  • Friendliness:
    It’s not just a stereotype about the Maritimes… it is actually true! I studied at Acadia University in Wolfville, Nova Scotia for four years, and my combined experiences of being in the Atlantic Provinces (including Newfoundland) has demonstrated to me that people here are genuinely, proportionately nicer and more considerate than city-dwellers of Ontario. No offense intended, but it’s been my objective observation for years, and I’m continuously reminded of this each time I visit the Maritimes. It’s nice to say “Hi” to a stranger in the grocery store, or converse with a friendly passer-by on the street. It is refreshing and puts a smile on your face!

I leave you now with the trailer for my favourite Christmas movie of all time: A Christmas Story (1983). Happy Holidays!!!

Rookie blogger

Howdy visitors! This homepage and blog is dedicated to my newfound free-time. I have very recently completed my Master of Information Studies degree from U of T, and have the ability (i.e. the time) to do things like read for pleasure, cook, create and contribute to a blog, etc. So this is one of the outlets that I’m using to fill my time, network, get experience with web 2.0 tools, be creative and all that fun stuff.

It feels like such a heavy weight has been lifted being done a term of school… but it still hasn’t really sunk in that I won’t be returning to a classroom in September. On Sunday night I was writing the last assignment for my final Masters course. When I was finished proofreading, I emailed the document to my prof., took a deep breath, and started to cry! I was overwhelmed with joy, pride, fear, and strangely, I knew that I was really going to miss all of those stressful and crazy times inherent with student life.

So. I’ve whiped my tears away, and am looking forward to new challenges and opportunities! Now that I’m done school, I’m still working very part-time with Robarts Reference, and cataloguing with Data, Map, & Government information services, while actively looking for full-time employment. Wish me luck!