Suppose I have a large corpus of text documents (say 100-million - in a common human language - e.g. English). I suspect that various subsets of these documents are 'similar'... by this, I mean they appear to have been created using common templates (no-longer available to me - except by inference from the corpus) containing substantial boilerplate. After applying standard NLP tokenisation to 'words' - I define the common-runs of a pair of documents as the longest sequences of words found in both documents. I define valid-common-runs as the longest sequence of common-runs that occur in the same order in both documents. Hence if "Abracadabra Systems" was (only) present at the end of one document and the beginning of another, it would be a common-run... but it would only be a valid-common-run if there were no other (distinct) common-runs in the documents. My ultimate objective is to cluster the documents using valid-common-runs (perhaps weighted by a simple function ranging over the valid-common-run text). My question is about efficient ways to compute the 'valid-common-runs' and find groups of similar documents. A naive approach to finding common-runs might be to consider every pair of documents in turn... but 100-million documents would require 10-quadrillion document comparisons... which seems prohibitively resource intensive. I would like to know (before I try to re-invent wheels) are there any well-established efficient algorithms to determine anything similar to 'valid-c…

Full article content could not be extracted automatically. Read the original below.