Summary

The Linguistic Society of America values the open sharing of scholarship, and encourages the fair review of Open Scholarship in hiring, tenure, promotion, and awards. The LSA encourages scholars, departments, and evaluation committees to actively place value on Open Scholarship in their evaluation process with the aim of encouraging greater accessibility, distribution, and use of linguistic research.

Preamble

Open Scholarship is broadly defined as a set of practices that enable the free sharing, reuse, and repurposing of scholarly work at all stages of the research lifecycle. The practice of Open Scholarship promotes free access to and reusability of all scholarly products not just for colleagues in academia, but for the general public and scholars from a variety of institutions and contexts. Open Scholarship encompasses the sharing of data, methods, publications, preprints, software, code, infrastructure, media files, public scholarship, pedagogical materials, and other products and processes of research. We specifically include the practices of Open Education and Open Pedagogy into the Open Scholarship umbrella; these principles involve the use and creation of openly licensed educational materials for the classroom. Open Scholarship intersects with ethics and community-based scholarship through careful review of human subjects research guidelines and community-driven plans for sharing the products of research within and beyond the community. Open Scholarship also relates to practices governing the ethical sharing of data. The LSA recognizes the need to practice Open Scholarship within the context of adherence to human subjects protections and protocols, as well as granting agencies’ requirements for Responsible and Ethical Conduct of Research (RECR). Open Scholarship improves our field by enabling and encouraging greater reproducibility in linguistic research through the ethical and timely availability of data, code, and source material (see Berez-Kroeker et al. 2018).

In order for our field and the public to reap the benefits of Open Scholarship, linguists must be incentivized to adopt Open Scholarship practices and rewarded for doing so. Key points where those incentives and rewards come in are in the processes of hiring, promotion, and tenure, i.e. moments of formalized research assessment, and the recommendations in this document are focused on those areas. However, many of the points made below could apply to other aspects of scholarship as well, such as graduate training and review of manuscripts and grant applications. The LSA is a signer of the San Francisco Declaration on Research Assessment (DORA; http://sfdora.org), which includes the following recommendations:

“consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.”

The Linguistic Society of America has, in previous statements, recognized the scholarly merit of language documentation, found here, and released a statement on the evaluation of language documentation for hiring, tenure, and promotion, found here. The LSA has a number of publications and resources that are Open Access, found here. In continuation of this work, this statement expands these previous statements to encompass Open Scholarship in all disciplines of linguistics. 

Open Scholarship has positive benefits for research, including increased impact, better reliability and reproducibility of research, and improved opportunities for collaboration. Allen and Mehler (2019) describe these opportunities, and also address challenges for researchers practicing Open Scholarship. These challenges include increased time and cost for producing and sharing open work, but crucially, a primary challenge is a lack of incentive in the hiring, tenure, and promotion process for scholars who practice Open Scholarship. This challenge was also outlined in the 2009 White Paper from the Cyberling Workshop regarding the sharing of linguistic data, specifically that “Receiving academic credit for publication of data would provide a needed incentive for doing the extra work needed” (Palmer et al., 2009).

As part of the growing attention both within linguistics and across many other fields to assessment of varied research outputs, the goal of the present document is to make recommendations concerning appropriate means of evaluation of the practice of Open Scholarship, for the use of committees such as those considering the hiring, review, tenure, promotion, and awards for linguists.

Incentivizing Open Scholarship

A necessary first step in the incentivization of Open Scholarship is to highlight its importance in the language of job advertisements, hiring criteria, internal review materials, and external letter requests. Alperin and colleagues (2019) published a wide-ranging study of language about Open Scholarship in the review, promotion, and tenure (RPT) documents of 129 universities in North America. They found almost no mentions of Open Access publishing or Open Scholarship, and when they were mentioned, it was in a negative context. Despite a number of universities having an explicitly stated public mission, terms and concepts related to public and community were mostly mentioned in RPT documents in relation to service, whereas evaluation of publications and research output were limited to academic impact and not the public accessibility or usability of the research output. Alperin and colleagues also recommend that scholars who practice Open Scholarship need to be “allowed and likely encouraged to produce other types of outputs beyond the six traditional outputs [books, conference proceedings, grants, journal articles, monographs and presentations]” and “for the public availability of these and other outputs to be valued, that too may need to be explicitly rewarded.” In sum, scholars who practice Open Scholarship are often doing so outside of the explicitly stated values and evaluation mechanisms associated with academic positions, and a path to fostering more Open Scholarship necessarily involves revisiting and revising the documents that specify how a scholar’s academic work will be evaluated, whether that is in the review and promotion process that Alperin and colleagues studied or in job advertisements and other academic policy and evaluation documents.

As one example of explicitly rewarding Open Scholarship in academic work, Odell and colleagues (2016) wrote a case study of Promotion and Tenure guidelines and Open Scholarship at Indiana University-Purdue University Indianapolis (IUPUI). This paper provides a model for working with partners across campus to effectively incorporate Open Scholarship into RPT documents with cooperation from stakeholders at all levels. 

The National Academies of Sciences, Engineering, and Medicine (NASEM) launched the Roundtable on Aligning Incentives for Open Science in 2019 to fundamentally improve the correlation between open practices, credit/reward systems, and research missions and values. It brings together senior leaders from universities, funding agencies, societies, foundations, and industry to generate an actionable open science toolkit that can be adapted and adopted by specific research communities to suit disciplinary norms. One component of the Roundtable’s workstream is the drafting, road testing, and refinement of sample language that can be deployed at key leverage points within the research lifecycle.  The LSA is collaborating with the Roundtable through the Committee on Scholarly Communication in Linguistics (CoSCiL) to develop language that can be used by linguistics departments in job postings, annual reporting, tenure & promotion instructions, and external reviewer guidelines to signal support for open practices. This statement is an outcome of that collaboration.

Based on these and related developments, we recommend that review, promotion, and tenure committees revisit the language of their guidelines and look for opportunities to explicitly mention the value of Open Scholarship, and encourage their scholars to submit Open Scholarship work in connection to the public mission of the institution (if applicable) or the field as a whole.  Department chairs, provosts, deans, and other leaders of research and teaching institutions are encouraged to consult the language samples prepared in conjunction with the NASEM Roundtable.

Evaluating Open Scholarship

Open Scholarship can be a key component for a scholar’s portfolio in a number of situations, including but not limited to hiring, review, promotion, and awards. Because Open Scholarship can take many forms, evaluation of this work may need different tools and approaches from publications like journal articles and books.  In particular, citation counts, a common tool for evaluating publications, are not available for some kinds of Open Scholarship in the same form or from the same providers as they are from publications. Here we share recommendations on how to assess the use of Open Scholarship materials including and beyond citations, including materials that both have formal peer review and those that do not not.

For tenure & promotion committees, program managers, department chairs, hiring committees, and others tasked with evaluating Open Scholarship, NASEM has prepared a discipline-agnostic rubric that can be used as part of hiring, review, or promotion processes. Outside letters of evaluation can also provide insight into the significance and impact of Open Scholarship work. Psychologist Brian Nosek (2017) provides some insight into how a letter writer can evaluate Open Scholarship, and includes several ways that evaluation committees can ask for input specifically about contributions to Open Scholarship. Nosek suggests that letter writers and evaluators comment on ways that individuals have contributed to Open Scholarship through “infrastructure, service, metascience, social media leadership, and their own research practices.” We add that using Open Scholarship in the classroom, whether through open educational materials, open pedagogy, or teaching of Open Scholarship principles, should be included in this list. Evaluators can explicitly ask for these insights in requests to letter writers, for example by including the request to “Please describe the impact that [scholar name]’s openly available research outputs have had from the research, public policy, pedagogic, and/or societal perspectives.” These evaluations can be particularly important when research outputs are not formally peer reviewed.

For scholars preparing hiring, review, promotion, or other portfolios that include Open Scholarship, we recommend not only discussing the Open Scholarship itself, but also its documented and potential impacts on both the academic community as well as broader society. Many repositories housing Open Scholarship materials provide additional metrics such as views, downloads, comments, and forks (or reuse cases) alongside citations in published literature. The use and mention of material with a Digital Object Identifier (DOI) can be tracked using tools such as ImpactStory, Altmetric.com, and other alternative metrics. To aid with evaluation of this work, the creator should share these metrics where available, along with any other qualitative indicators (such as personal thank-yous, reuse stories, or online write-ups) that can give evaluators a sense of the impact of their work. The Metrics Toolkit provides examples and use cases for these kinds of metrics. This is of potential value when peer review of these materials may not take the same form as with published journals or books; thoughtful use and interpretation of metrics can help evaluators understand the impact and importance of the work.

The Linguistic Society of America reaffirms its commitment to fair review of Open Scholarship in hiring, tenure, and promotion, endorses all of these approaches to peer review and evaluation of Open Scholarship, and encourages scholars, departments, and personnel committees to take them into careful consideration and implement language about Open Scholarship in their evaluation processes.

References

Allen C., Mehler D.M.A. (2019). Open science challenges, benefits and tips in early career and beyond. PLoS Biology, 17(5): e3000246. https://doi.org/10.1371/journal.pbio.3000246

Alperin, J. P., Nieves, C. M., Schimanski, L. A., Fischman, G. E., Niles, M. T., & McKiernan, E. C. (2019). Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents? eLife, 8, e42254. http://doi.org/10.7554/eLife.42254

Berez-Kroeker, A. L., Gawne, L., Kung, S. S., Kelly, B. F., Heston, T., Holton, G., Pulsifer, P., Beaver, D. I., Chelliah, S., Dubinsky, S., Meier, R. P., Thieberger, N., Rice, K., & Woodbury, A. C. (2018). Reproducible research in linguistics: A position statement on data citation and attribution in our field. Linguistics, 56(1), 1–18. https://doi.org/10.1515/ling-2017-0032

Nosek, B. (2017). Are reproducibility and open science starting to matter in tenure and promotion review. Center for Open Science. Retrieved February 1, 2021, from https://cos.io/blog/are-reproducibility-and-open-science-starting-matter-tenure-and-promotion-review/

Odell, J., Coates, H., & Palmer, K. (2016). Rewarding open access scholarship in promotion and tenure: Driving institutional change. College & Research Libraries News, 77(7), 322-325. https://doi.org/10.5860/crln.77.7.9518

Palmer, A., Anderson, D., Kansa, E., Mihaylov, P., Nichols, J., & Wassink, A. (2009). White Paper: Working Group 2. In “Cyberling 2009 Workshop: Towards a Cyberinfrastructure for Linguistics Workshop Report”, Emily M. Bender, PI, 32-62. https://www.linguisticsociety.org/sites/default/files/Cyberling2009Workshop_FinalReport.pdf

Document Evolution

  • First draft presented for Open Comments (Autumn 2020) 
  • Revision 1 presented at the CoSCiL Open Meeting, January 14, 2021
  • Revision 2 presented to the Executive Committee of the LSA, March 15, 2021
  • This version was approved by the Executive Committee on April 29, 2021.

Contributors to this document

Members of the Committee on Scholarly Communication in Linguistics (2020-21)

  • Lauren B. Collister, University of Pittsburgh; Chair
  • Emily M. Bender, University of Washington
  • Andrea Berez-Kroeker, University of Hawai'i
  • Paul De Decker, Memorial University
  • Arienne M. Dwyer, University of Kansas
  • Jeff Good, University of Buffalo
  • Emily Moeng,  University of North Carolina at Chapel Hill
  • Carla D. Morris,  Gallaudet University
  • Geoffrey Nathan, Wayne State University
  • Alexis Palmer, Saarland University
  • Visvajit Sriramrajan, Hofstra University
  • Laurel Smith Stvan, University of Texas at Arlington

Guests who assisted with drafting this document at the Open Meeting in January 2021:

  • Greg Tananbaum, Open Research Funders Group
  • Eric Baković, University of California San Diego
  • Dan Villareal, University of Pittsburgh
  • Carolyn Anderson, Wellesley College

And the many commenters who shared their responses to the first draft of the Statement.

Executive Summary and Resources

The Linguistic Society of America acknowledges the need to incentivize Open Scholarship particularly in times of formal academic assessment (e.g. hiring, review, promotion, tenure, and awards).

In order to support the growth of Open Scholarship in our field, the LSA recommends several practices. Additionally, the LSA recognizes that institutions likely have their own local scholarly communication expertise and recommends that linguists consult their libraries and other research support offices for institution-specific guidance.

For tenure & promotion committees, program managers, department chairs, hiring committees, and others tasked with evaluation:

  • Insert language about Open Scholarship where possible and appropriate in job advertisements, hiring criteria, review criteria, award information, and other policy documents.
  • Evaluate Open Scholarship in ways appropriate to its form and impact as part of the whole portfolio of a scholar’s work.
  • Ask for commentary on Open Scholarship when soliciting outside letters of evaluation.
    • Resource: Nosek’s (2017) guidance for incorporating Open Scholarship in evaluation letters.

For scholars pursuing Open Scholarship and preparing portfolios for evaluation:

  • Track indicators of impact for your Open Scholarship to share as evidence of the importance and use of work beyond citation counts.
    • Resource: The Metrics Toolkit documentation of different evaluation and impact metrics for different kinds of work.
  • Provide metrics and other indicators of impact of Open Scholarship work that help committees do evaluation.