Adopted by the LSA Executive Committee on September 25, 2018. A pdf of this statement may be viewed or downloaded here.

The Linguistic Society of America (LSA) has expressed its support for the recognition of language documentation “as scholarly contributions to be given weight in the awarding of advanced degrees and in decisions on hiring, tenure, and promotion of faculty. It supports the development of appropriate means of review of such works so that their functionality, import, and scope can be assessed relative to other language resources and to more traditional publications.” The document quoted, the ”Resolution Recognizing the Scholarly Merit of Language Documentation”, can be found here. The resolution makes it clear that “documentation” in this sense is scholarship that includes “not only grammars, dictionaries, and text collections, but also archives of primary data, electronic databases, corpora, critical editions of legacy materials, pedagogical works designed for the use of speech communities, software, websites, or other digital media”. In addition, the Linguistic Society of America is a signer of the San Francisco Declaration on Research Assessment (DORA), which includes the recommendation:

5. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

As part of the growing cross-disciplinary attention to assessment of varied research outputs, the goal of the present document is to make recommendations concerning appropriate means of evaluation of such scholarship.

Documentary scholarship is generally regarded as a very heavy time commitment, and the issue of just how to assess it has been discussed in the community of documentary linguistics, including in published articles, among which are Austin 2016, Berez-Kroeker et al. 2018, Haspelmath and Michaelis 2014, Thieberger et al. 2016, and Woodbury 2014. The discussion below is closely based on this literature, and the terminology is borrowed from Thieberger et al. 2016, but in the present document the “corpus” is the body of language documentation scholarship that is deposited in a language archive (with no implication that annotated texts are necessarily included).

The aspect of documentary linguistics that makes the present statement necessary is that, unlike traditional publications, documentation is not ordinarily peer-reviewed. Thus, the approaches to evaluation described here involve ways of having documentary scholarship peer-reviewed, for candidates for whom this seems necessary. Each is described only briefly here, and those interested in more detail, including advantages and disadvantages of each approach, are referred to the sources cited.

The first type, the corpus review, is parallel to the review of a book. It would consist of a brief summary and written evaluation of the scholarly corpus by a person not at all involved in the project, and it would be published in a recognized journal, just as a book review is. In a dossier for hiring, tenure, or promotion, it would be treated like a book review. In particular, a personnel committee does not itself undertake to assess a scholarly book; it relies instead on the testimony of a book review. Similarly, a personnel committee could rely on the corpus review for documentary linguistics.

In the second approach, the corpus overview article, the scholar herself would write a description of her own documentary corpus, submitting it to a peer-reviewed journal. Woodbury 2014 provides guidelines for how a scholar can take authorial control of her work, insuring that the scholarship is accompanied by appropriate metadata, guides, and search aids. In this case, the peer review of the descriptive article provides to personnel committees the same assurance that peer review of traditional articles provides.

Third, the corpus journal or corpus-as-article is the publication of the actual corpus of documentation in a peer-reviewed journal. Recommended by Haspelmath and Michaelis 2014, this approach shares with the corpus overview article the parallel to a traditional article and the assurance provided by peer review of the published material.

Finally, outside letters of evaluation should be solicited from senior scholars who have expertise in documentation, just as letters evaluating a candidate in phonetics would be sought from senior phoneticians or letters evaluating one in syntax would be sought from senior syntacticians. In cases of collections that are restricted in order to respect communities’ or speakers' wishes, an assessment of the collection could be made based on the available catalogue metadata and other publicly available information. Alternatively, the community might be asked to make an assessment of restricted materials.

In addition to using one or more of these four methods of peer review, a candidate may want to include a data quality checklist in their dossier to indicate that their data do meet minimum standards accepted in the field of language documentation. The Digital Endangered Languages and Musics Archives Network (DELAMAN) has posted a "Minimal Checklist for the Digital Preservation of Language Documentation Materials" here.

A candidate may also wish to take advantage of “other” or “non-traditional output” categories available in some instances.

Another way to recognize the value of documentation is through awards given to corpora of especially high quality. DELAMAN, for instance, gives out its Franz Boas Award to junior scholars who have done outstanding documentary work in creating a rich multimedia documentary collection of a particular language that is endangered or no longer spoken.

Two examples of the corpus overview article are Caballero 2017 and Salffner 2015. The first Franz Boas Award is reported here. We are not aware of examples of the other two approaches that have been published at this time.

The Linguistic Society of America endorses all of these approaches to peer review and evaluation and encourages scholars, departments, and personnel committees to take advantage of them. The LSA reaffirms its commitment to fair review of language documentation in hiring, tenure, and promotion.

References

Austin, Peter K. 2016. Language documentation 20 years on. Endangered languages and languages in danger: Issues of documentation, policy, and language rights, ed. by Luna Filipović and Martin Pütz, 163-164. Amsterdam: John Benjamins.

Berez-Kroeker, Andrea L., Lauren Gawne, Susan Smythe Kung, Barbara F. Kelly, Tyler Heston, Gary Holton, Peter Pulsifer, David I. Beaver, Shobhana Chelliah, Stanley Dubinsky, Richard P. Meier, Nick Thieberger, Keren Rice, and Anthony C. Woodbury. 2018. Reproducible research in linguistics: A position statement on data citation and attribution in our field. Linguistics 56(1): 1-18. Online: https://doi.org/10.1515/ling-2017-0032.

Caballero, Gabriela. 2017. Choguita Rarámuri (Tarahumara) language description and documentation: A guide to the deposited collection and associated materials. Language Documentation and Conservation, vol. 11. http://nflrc.hawaii.edu/ldc/

Haspelmath, Martin, and Susanne Maria Michaelis. 2014. Annotated corpora of small languages as refereed publications: A vision. http://dlc.hypotheses.org/691

Salffner, Sophie. 2015. A guide to the Ikan language and culture documentation. Language Documentation & Conservation 9: 237-267.

Thieberger, Nick, Anna Margetts, Stephen Morey, and Simon Musgrave. 2016. Assessing annotated corpora as research output. Australian Journal of Linguistics 36: 1-21. https://doi.org/10.1080/07268602.2016.1109428

Woodbury Anthony C. 2014. Archives and audiences: Toward making endangered language documentations people can read, use, understand, and admire. Language Documentation and Description 12. http://www.elpublishing.org/publicationpage/12