A Dynamic Graph Model for Representing Streaming Text Documents

Abstract

This work presents new techniques for representing an evolving stream of text documents. Text processing is traditionally performed on a fixed corpus of documents by representing the documents as vectors in a high-dimensional space with each dimension corresponding to a different word in the lexicon. The lexicon is formed by the set of unique words in the corpus. The vector entries are equal to the counts of the word in the document and often weighted by the inverse of the probability of the corresponding word occurring in a document. The probability of word occurrence, also called the document frequency, is needed in order to create document vectors which emphasize the informative words in each document. In order to apply statistical text processing techniques to a changing corpus of documents, a generalization of the vector space model is introduced. The generalization relies on managing a changing lexicon of words and approximating the probability of word occurrence over documents in the document stream. The methods presented here can be used to represent any new document as a vector, including documents that contain words that have not been seen previously in the document stream. Additionally, this work presents a graph model for representing a dynamic corpus of text documents. The graph model differs from other methods for text clustering which act on a fixed corpus of documents. The vertices in the graph represent topics and evolve as the document stream changes. The vertices contain statistics on documents of a similar topic. Each vertex has an associated lexicon and document frequency which can be used to provide information about the document stream. The graph model is demonstrated on a dataset of news articles collected over several years.