Journal Article

Association pattern discovery via theme dictionary models

Liu, Jun S., Deng, Ke, Geng, Zhi

2014

Overview

Discovering patterns from a set of text or, more generally, categorical data is an important problem in many disciplines such as biomedical research, linguistics, artificial intelligence and sociology. We consider here the well‐known ‘market basket’ problem that is often discussed in the data mining community, and is also quite ubiquitous in biomedical research. The data under consideration are a set of ‘baskets’, where each basket contains a list of ‘items’. Our goal is to discover ‘themes’, which are defined as subsets of items that tend to co‐occur in a basket. We describe a generative model, i.e. the theme dictionary model, for such data structures and describe two likelihood‐based methods to infer themes that are hidden in a collection of baskets. We also propose a novel sequential Monte Carlo method to overcome computational challenges. Using both simulation studies and real applications, we demonstrate that the new approach proposed is significantly more powerful than existing methods, such as association rule mining and topic modelling, in detecting weak and subtle interactions in the data.

• Publisher: Blackwell Publishers,Blackwell Publishing Ltd,John Wiley & Sons Ltd,Oxford University Press
• Subject: Artificial intelligence / Association rule mining / Biomedical research / Biomedicine / Co-occurrence pattern recognition / Computational methods / Data / Data mining / Dictionaries / Information retrieval / Linguistics / Market basket analysis / markets / Medical research / Miners / Monte Carlo method / Monte Carlo simulation / Nominal measurement / Probability / Rejection control sampling / Research methods / Sequential Monte Carlo methods / Simulation / Sociology / Statistical methods / Statistics / Studies / Subsets / Topic modelling