textreuse: Detect Text Reuse and Document Similarity
Date
2015-11-05
Authors
Mullen, Lincoln
Journal Title
Journal ISSN
Volume Title
Publisher
rOpenSci
Abstract
This R package provides a set of functions for measuring similarity among documents and detecting passages which have been reused. It implements shingled n-gram, skip n-gram, and other tokenizers; similarity/dissimilarity functions; pairwise comparisons; minhash and locality sensitive hashing algorithms; and a version of the Smith-Waterman local alignment algorithm suitable for natural language. It is broadly useful for, for example, detecting duplicate documents in a corpus prior to text analysis, or for identifying borrowed passages between texts. The classes provides by this package follow the model of other natural language processing packages for R, especially the NLP and tm packages. (However, this package has no dependency on Java, which should make it easier to install.)
Description
Keywords
Textreuse, Text reuse, Document similarity, R, Jaccard similarity, Minhash, Locality sensitive hashing, Smith-Waterman local sequence alignment, Natural language processing
Citation
Lincoln Mullen (2015). textreuse: Detect Text Reuse and Document Similarity. R package version 0.1.2. https://github.com/ropensci/textreuse