Combining Statistical Language Models via the Latent Maximum Entropy Principle

TitleCombining Statistical Language Models via the Latent Maximum Entropy Principle
Publication TypeJournal Article
Year of Publication2005
AuthorsF. Peng, Y. Zhao, Shaojun Wang, D. Schuurmans
Abstract

We present a unified probabilistic framework for statistical language modeling which can simultaneously incorporate various aspects of natural language, such as local word interaction, syntactic structure and semantic document information. Our approach is based on a recent statistical inference principle we have proposed-the latent maximum entropy principle-which allows relationships over hidden features to be effectively captured in a unifiedmodel. Our work extends previous research on maximum entropy methods for language modeling, which only allow observed features to be modeled. The ability to conveniently incorporate hidden variables allows us to extend the expressiveness of language models while alleviating the necessity of pre-processing the data to obtain explicitly observed features.We describe efficient algorithms for marginalization, inference and normalization in our extended models. We then use these techniques to combine two standard forms of language models: local lexical models (Markov N-gram models) and global document-level semantic models (probabilistic latent semantic analysis). Our experimental results on the Wall Street Journal corpus show that we obtain a 18.5% reduction in perplexity compared to the baseline tri-gram model with Good-Turing smoothing.

Full Text

S. Wang, D. Schuurmans, F. Peng and Y. Zhao, 'Combining Statistical Language Models via the Latent Maximum Entropy Principle,' Machine Learning Journal, Special Issue on Learning in Speech and Language Technologies, Vol. 60, pp. 229-250, 2005
pages: 229 - 250
publisher: Kluwer Academic Publishers
year: 2005
hasEditor: Pascale Fung
hasURL: http://knoesis.wright.edu/library/publications/StatModelMaxEntropy.pdf
hasBookTitle: Machine Learning Journal, Special Issue on Learning in Speech and Language Technologies