Comments on this draft are being accepted from LSA members through 1 December 2020. Please use the comment box at the very bottom of this page to post your comments. Thank you.

Summary

The Linguistic Society of America values the open sharing of scholarship, and encourages the fair review of open scholarship in hiring, tenure, and promotion. The LSA encourages scholars, departments, and personnel committees to actively place value on open scholarship in their evaluation process with the aim of encouraging greater accessibility, distribution, and use of linguistic research.

Preamble

The Linguistic Society of America has, in previous statements, recognized the scholarly merit of Language Documentation, found here, and released a statement on the evaluation of language documentation for hiring, tenure, and promotion, found here. Both of these statements highlight language documentation as a practice that includes text collections as well as “archives of primary data, electronic databases, corpora, critical editions of legacy materials, pedagogical works designed for the use of speech communities, software, websites, or other digital media.”

Much of this work, especially when shared openly, relates to the principles of Open Scholarship, which are relevant to the entire discipline. Open Scholarship is broadly defined as a set of practices that enable the free sharing, reuse, and repurposing of scholarly work at all stages of the research lifecycle. Open Scholarship encompasses not only Open Access publication, but also open sharing of data, methods, software, code, infrastructure, pedagogical materials, and other products of research. In addition, we include the practice of Open Education and Open Pedagogy into the Open Scholarship umbrella; these principles involve the use and creation of openly licensed educational materials for the classroom. The practice of Open Scholarship promotes greater and freer access to all scholarly products not just for colleagues in academia, but for the general public and scholars from a variety of institutions and contexts. In addition, Open Scholarship enables and encourages greater reproducibility in linguistic research through the availability of data, code, and source material (see Berez-Kroeker et al. (2018)).

The LSA is a signer of the San Francisco Declaration on Research Assessment (DORA; http://sfdora.org), which includes the recommendations, “consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice”.

Open Scholarship has been found to have positive benefits for research, including increased citation counts, better reliability and reproducibility of research, and improved opportunities for collaboration. Allen and Mehler (2019) describe these opportunities, and also address challenges for researchers practicing Open Scholarship. These challenges include increased time and cost for producing and sharing open work, but crucially, a primary challenge is a lack of incentive in the hiring, tenure, and promotion process for scholars who practice Open Scholarship.

As part of the growing cross-disciplinary attention to assessment of varied research outputs, the goal of the present document is to make recommendations concerning appropriate means of evaluation of the practice of Open Scholarship.

Incentivizing Open Scholarship

A necessary first step in the incentivization of Open Scholarship is to highlight its importance in the language of job advertisements, internal review materials, and external letter requests. Juan Pablo Alperin and colleagues (2019) published a wide-ranging study of language about Open Access in the RPT documents of 129 universities in North America. They found that, despite a number of universities having an explicitly stated public mission, terms and concepts related to public and community were mostly mentioned in review, promotion, and tenure (RPT) documents in relation to service, whereas evaluation of publications and research output were limited to academics and not the public. This study also found almost no mentions of Open Access publishing or Open Scholarship, and when they were mentioned, it was in a negative context. Alperin and colleagues recommend that “publicly orientated faculty work may first need to be considered on par with activities for which there are ‘quantifiable research metrics,’ since these are the ones that appear to be the most valued”, and that scholars who do Open Scholarship need to be “allowed and likely encouraged to produce other types of outputs beyond the six traditional outputs [books, conference proceedings, grants, journal articles, monographs and presentations] we searched for. Relatedly, for the public availability of these and other outputs to be valued, that too may need to be explicitly rewarded.”

As one example of explicitly rewarding Open Scholarship in academic work, Jere Odell and colleagues (2016) wrote a case study of Promotion and Tenure guidelines and Open Scholarship at Indiana University-Purdue University Indianapolis (IUPUI). This paper provides a model for working with partners across campus to effectively incorporate Open Scholarship into RPT documents with cooperation from stakeholders at all levels. 

The National Academies of Sciences, Engineering, and Medicine launched the Roundtable on Aligning Incentives for Open Science in 2019 to fundamentally improve the correlation between open practices, credit/reward systems, and research missions & values. It brings together senior leaders from universities, funding agencies, societies, foundations, and industry to generate an actionable open science toolkit that can be adapted and adopted by specific research communities to suit disciplinary norms. One component of the Roundtable’s workstream is the drafting, road testing, and refinement of sample language that can be deployed at key leverage points within the research lifecycle.  The LSA is collaborating with the Roundtable through the Committee on Scholarly Communication in Linguistics (CoSCiL) to develop language that can be used by linguistics departments in job postings, annual reporting, tenure & promotion instructions, and external reviewer guidelines to signal support for open practices. This statement is an outcome of that collaboration.

Based on these and related developments, we recommend that review, promotion, and tenure committees revisit the language of their guidelines and look for opportunities to explicitly mention the value of Open Scholarship, and encourage their scholars to submit Open Scholarship work in connection to the public mission of the institution (if applicable) or the field as a whole.  Department chairs, provosts, deans, and other leaders of research and teaching institutions are encouraged to consult the language samples prepared by NASEM.

Evaluating Open Scholarship

One issue with evaluating Open Scholarship beyond Open Access Publications is that citation counts, a common tool for evaluating publications, are not available in the same form or from the same providers as they are from publications.

Here we share recommendations on how to assess the use of Open Scholarship materials including and beyond citations. The Transparency and Openness Promotion guidelines (https://cos.io/top/) contain guidance for journals on how to address citations to open data sets and code with the aim of increasing citation to these materials in published articles. Many repositories housing Open Scholarship materials provide metrics such as views, downloads, comments, and ‘forks’ (or reuse cases) in addition to citations in published literature. The use and mention of material with a Digital Object Identifier (DOI) can be tracked using tools such as ImpactStory, Altmetric.com, and other alternative metrics. To evaluate this work, the creator should share these metrics where available, along with any other qualitative indicators (such as personal thank-yous, reuse stories, or online write-ups) that can give evaluators a sense of the impact of their work.

Outside letters of evaluation can also provide insight into the significance and impact of Open Scholarship work. Psychologist Brian Nosek (2017) provides some insight into how a letter writer can evaluate Open Scholarship, and includes several ways that evaluation committees can ask for input specifically about contributions to Open Scholarship. Nosek suggests that letter writers and evaluators comment on ways that individuals have contributed to open scholarship through “infrastructure, service, metascience, social media leadership, and their own research practices”. We would add that using Open Scholarship in the classroom, whether through open educational materials, open pedagogy, or teaching of open scholarship principles, should be included in this list. Evaluators can explicitly ask for these insights in requests to letter writers, for example by including the request to “Please describe the impact that [scholar name]’s openly available research outputs have had from the research, public policy, pedagogic, and/or societal perspectives.”

In all cases, scholars who practice Open Scholarship should be ready to discuss both the acts of Open Scholarship that they engage in as well as their documented and potential impacts on both the academic community as well as broader society.

The Linguistic Society of America reaffirms its commitment to fair review of open scholarship in hiring, tenure, and promotion, endorses all of these approaches to peer review and evaluation of open scholarship, and encourages scholars, departments, and personnel committees to take them into careful consideration and implement language about open scholarship in their evaluation process.

References

Allen C, Mehler DMA (2019) Open science challenges, benefits and tips in early career and beyond. PLoS Biol 17(5): e3000246. https://doi.org/10.1371/journal.pbio.3000246

Alperin, J. P., Nieves, C. M., Schimanski, L. A., Fischman, G. E., Niles, M. T., & McKiernan, E. C. (2019). Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents?. eLife, 8, e42254. http://doi.org/10.7554/eLife.42254 

Berez-Kroeker, A. L., Gawne, L., Kung, S. S., Kelly, B. F., Heston, T., Holton, G., Pulsifer, P., Beaver, D. I., Chelliah, S., Dubinsky, S., Meier, R. P., Thieberger, N., Rice, K., & Woodbury, A. C. (2018). Reproducible research in linguistics: A position statement on data citation and attribution in our field. Linguistics, 56(1), 1–18. https://doi.org/10.1515/ling-2017-0032

Nosek, B. (2017). Are reproducibility and open science starting to matter in tenure and promotion review. Center for Open Science. Retrieved September 11, 2019, from https://cos.io/blog/are-reproducibility-and-open-science-starting-matter-tenure-and-promotion-review/

Odell, J., Coates, H., & Palmer, K. (2016). Rewarding open access scholarship in promotion and tenure: Driving institutional change. College & Research Libraries News, 77(7), 322-325. https://doi.org/10.5860/crln.77.7.9518

Contributors to this document

Lauren B. Collister, University of Pittsburgh

Greg Tananbaum, Open Research Funders Group

Members of the Committee on Scholarly Communication in Linguistics for the Linguistic Society of America:

Emily Bender, University of Washington

Andrea Berez-Kroeker, University of Hawai'i

Paul De Decker, Memorial University

Arienne M. Dwyer, University of Kansas

Emily Moeng,  University of North Carolina at Chapel Hill

Carla D. Morris,  Gallaudet University

Geoffrey Nathan, Wayne State University

Alexis Palmer, Saarland University

Visvajit Sriramrajan, Hofstra University

Laurel Smith Stvan, University of Texas at Arlington

 

Comments

Please, in the sentence: "Open Scholarship has been found to have positive benefits for research, including increased citation counts, better reliability and reproducibility of research, and improved opportunities for collaboration" -- please add that Open Scholarship makes for higher quality scholarship, improving our overall field.

Thank you for your comment. We will take this into account in our edits.

I recommend revising the preamble by eliminating the first paragraph and the first sentence of the second paragraph. The important initiatives of open access and encouraging open scholarship should not come second to a passing mention of language documentation, which concerns only a small subset of the field, and which is orthogonal to the aims of this document. There are many administrators who won’t read past the first paragraph, and who may well disagree fundamentally with that paragraph’s assessment—which is irrelevant to the goal of this document. Why potentially distract any readers at all by beginning with an irrelevant aside?

Thank you for this input. The idea of this ordering was to situate this statement in the context of other statements and advocacy previously done by the LSA. The committee will consider how to keep that context while avoiding the distraction that you mention.

The statement is not explicit about how "open scholarship" is interpreted.
The statement does not address issues of collegiality. Departments should be environments in which all research areas are considered equal and none is claimed to be superior or to require more linguist-skills than another.

Thank you for your comment. We do include this definition of Open Scholarship: "Open Scholarship is broadly defined as a set of practices that enable the free sharing, reuse, and repurposing of scholarly work at all stages of the research lifecycle. Open Scholarship encompasses not only Open Access publication, but also open sharing of data, methods, software, code, infrastructure, pedagogical materials, and other products of research."
Could you let us know your thoughts on how this could be improved to address your first concern?

Collegiality is out of the scope of this particular statement, although it underpins our assertion that Open Scholarship -- no matter which subdiscipline of linguistics -- should be given merit and evaluated fairly.

There are many good points in here, but I worry that the effort could be ineffective because it groups a confusing mixture of issues under the single heading of "open", and because it presumes influence where it has little. Things grouped here include: (i) 'open access", a business model in journal and book publishing; (ii) "open resources" (for want of a better term), including software, datasets, or other resources that are free or available for modification; (iii) other products that don't pass through traditional peer review processes. This could even include publications in relatively traditional journal formats that apply different approaches to peer-review, e.g., publishing based on soundness only and relying on post-publication metrics as impact measures.

I think there are two distractions to avoid here, and one piece where an LSA resource could make a difference. The first possible distraction is the use of "open" to refer to a free-to-read business model. It's interesting, and it has many benefits. But this document won't make a difference, and there's a risk that this document will be mainly taken as valuing that business model over others (with risk of hypocrisy). The second possible distraction is an argument about what hiring and promotion authorities should value highly. They are not going to change their values because the LSA says that they should. That's just not going to happen. The piece where the LSA can be very helpful is in providing very concrete ways for the value and impact of these works to be assessed.

Also, concretely, what can a scholarly society do to assign value to these kinds of products, independent of the confines of the traditional review process. For example, if the society published a list of "Top 10 pieces of open scholarship in 202x", that would give an immediate boost to that kind of work, as it would assign value to it.

As my comment is similar to this one, I'm putting it here as a "reply", though it's really just a follow-up and largely an expression of agreement. I see two issues here: 1) what should be evaluated, and 2) how to evaluate it when it is published in an open format. These are very different questions. For example, open access publications vary greatly in the nature and rigor of their peer review processes, in many cases, not differing from traditional scholarly publications. The LSA could provide guidelines for cases of open scholarship where there is peer review vs. where there is not, for example. One could also sharpen and extend the text already in the document about how evaluation can be made of specific sorts of non-peer reviewed output. This latter question is obviously challenging -- to give just one example, the number of downloads from an academic or other social media site may or may not be correlated with research quality or ultimate impact.

As for what should be evaluated, this is also a complex and no doubt controversial issue. Perhaps the most helpful thing the LSA could do on this latter point is to prepare a separate document or documents on additional types of scholarly output that it considers specifically relevant for evaluation in linguistics, along the lines of what it has already done for language documentation materials.

Thank you for your thoughtful comment. Defining "open scholarship" is indeed challenging, as there seem to be new ways to share scholarship created all the time, and the landscape is so varied throughout linguistics that listing all of the possibilities would be a lifetime's work. Our committee was also concerned that making a "definitive list" would be detrimental to those linguists who are exploring new kinds of analysis or data collection that might not fit neatly into that list.

We do attempt to list types of scholarly output in this sentence: "Open Scholarship encompasses not only Open Access publication, but also open sharing of data, methods, software, code, infrastructure, pedagogical materials, and other products of research." Is there something that perhaps we could add to this list (which is going to be edited based on other comments) that would move towards what you desire to see?

Regarding the peer review question, this was our thinking behind the language for use in soliciting evaluation letters in particular. We will revise this section as well as the metrics section to include a mention of how this kind of evaluation may be particularly helpful when there is not peer review in the same sense as a journal article or book. Thank you for this helpful pointer.

I want to second Colin’s point about the importance of separating the different kinds of “open” activities. Given the fact that Language itself is not a fully open-access publication, it might be odd for the LSA to be preaching the merit of “open access” as model in journal and book publishing.

In terms of the idea of publishing a list of “Top 10 pieces of open scholarship”, I wonder if it might be more useful for the society to maintain a list of non-predatory journals that publish linguistic research on a regular basis. Just like not all printed journals are created equal, not all open access journals are created with the same egalitarian views in mind. Often times it is the budding scholars who fall prey to the predatory practices of “open access” journals.

One last thing I’d recommend the LSA to do is to not use the label “Open Linguistics” since that happens to be the name of an open access linguistic journal. I don’t think the LSA would want to create the impression that it is endorsing that journal somehow.

Thank you for your thoughtful comment. We had the same feeling about "Open Linguistics" and avoided that phrasing intentionally. We also intentionally avoided mention of any specific business models, although it seems that this came through anyhow. Was there any particular phrase or paragraph that brought a business model to your attention that we should revise in our second draft?

Hello, thank you for your thoughtful comment.

With respect, this statement does not intend to refer to any business models, but rather an umbrella term of "open scholarship" which we define in the statement as follows:
"Open Scholarship encompasses not only Open Access publication, but also open sharing of data, methods, software, code, infrastructure, pedagogical materials, and other products of research. In addition, we include the practice of Open Education and Open Pedagogy into the Open Scholarship umbrella; these principles involve the use and creation of openly licensed educational materials for the classroom."
Do you think the confusing/distracting part is in separating out "Open Access publication" at the beginning of that definition, or is there something else in the statement itself that is causing this? In our revisions, we can rephrase this sentence to include publication in the long list.
Regarding the "free-to-read" issue - the very first part of the definition of Open Scholarship calls out "free sharing, reuse, and repurposing of scholarly work at all stages of the research lifecycle". Do you believe that more is necessary to indicate that the discussion is beyond "free-to-read" here?

Regarding "very concrete ways for the value and impact of these works to be assessed" -- we have included 1.) suggestions for individuals on including meaningful metrics and discussion of those metrics in their portfolios, and 2.) language samples for deans/chairs/administrators to use in job ads, evaluation documents, and instructions for external evaluations on these materials. Is there something more specific that you would like to see in terms of concrete advice -- but, crucially, which would not be too restricting and/or impossible to use in a variety of assessment contexts?

Finally, I personally like the idea of the "Top 10" list you mention, and we will suggest it to the Society! We had also briefly considered suggesting an award for Open Scholarship in Linguistics, but that was outside of the scope of this statement.

Thank you again for your comment.

"Open access" in books and journals primarily refers to a business model where readers don't pay for access. Either authors (or their institutions) pay, or some other institution pays. This is entirely independent of the peer review process used. Unfortunately, there is widespread confusion around this, leading many researchers to express opinions on whether publishing in an open access outlet is advisable or meritorious, particularly for younger scholars. This makes no sense, because it confuses business models with peer review models ... but I encounter this quite frequently. Hence the suggestion to explicitly put this aside. As I understand it, the aim of this statement is to support linguists navigate the challenges of receiving recognition for open scholarship.

The guidance for candidates and department-level administrators is useful, as they are the most motivated in the process. The more specific, the better. The kind of piece that might be more useful for higher committees that do not know linguistics would be a factual statement about different publication and impact conventions in linguistics. It may be surprising to some to learn just how varied the practices are within one field. So, for example, citation metrics may be useful for computational linguists or psycholinguists, but they are much less relevant for somebody engaged in documentation / grammar writing. Citation half lives vary hugely across different sub-fields, also.

I disagree with Colin Phillips' suggestion that this statement should side-step the issue of open access business models. The confusion about these models is intimately bound up in confusion about how to 'count' these and other forms of open scholarship contributions, so I think it's important for there to be a statement that encompasses them all. The fact that some may remain confused is not (to me) a reason not to have a statement that addresses the issue.

Perhaps to address Colin's underlying concern, it would be useful for CoSCiL to create & maintain a separate resource (or set of resources) about more specific types of scholarly communication contributions. In that vein, I would suggest as a place to start a broader version of Louise McNally's suggestion that the the LSA "provide guidelines for cases of open scholarship where there is peer review vs. where there is not". Restricting any such guidelines to open scholarship suggests -- wrongly, as Colin points out -- that there the presence vs. absence of peer review is only an issue for open scholarship, when in fact there are plenty of non-open publications that involve little to no peer review. (And this is not to mention the different approaches to peer review that exist, as Colin also points out.)

For much the same reason I would broaden Alan Yu's suggestion about maintaining "a list of non-predatory journals" to both open and non-open publications. (Alan may have in fact been suggesting this broader interpretation; if so, it was unclear to me.) Perhaps the most useful thing would be to create & maintain a list of all linguistics journals with pertinent, regularly updated information about their review practices, acceptance rates, average time from initial submission to publication, etc. This would be much more useful, it seems to me, than a list of "acceptable" (or "unacceptable") journals. (So as not to appear that I'm suggesting dumping more on CoSCiL's lap, this seems to me to be a better project for the Committee of Editors of Linguistics Journals.)

Finally, I'm personally opposed to the "top 10 pieces of open scholarship" idea, with all that such lists entail -- who decides? what are the criteria? what does it say about instances of excellent work that doesn't happen to make the list? -- not to mention the fact that it will mainly serve to amplify the signal of just those instances selected for the list rather than amplifying the signal of "open scholarship" generally. Perhaps instead the LSA could profile individuals, teams, and institutions who make significant, sustained, model contributions to open scholarship in its various forms.

Thank you, Eric, for these thoughtful comments. Spotlighting Open practices is a wonderful idea.

I think a list of all linguistics journals, not filtered for any "acceptable" criteria, would be helpful. A listing of many Open Access journals in linguistics are available via the Directory of Open Access Journals with much of the information you suggest, and so part of the work is already done. We will take this feedback and send it to the EC in conjunction with our statement.

CoSCiL will take your suggestion about resources to create on broader scholarly communication topics. We will need a new project after this one is done, so the committee will discuss the idea. We have in the past done blog posts and webinars on various topics, so this effort could fit in there as well.