Journal Article

Multi-armed bandits with dependent arms

Sun, Yin, Singh, Rahul, Liu, Fang, Shroff, Ness

2024

Overview

We study a variant of the multi-armed bandit problem (MABP) which we call as MABs with dependent arms. Multiple arms are grouped together to form a cluster, and the reward distributions of arms in the same cluster are known functions of an unknown parameter that is a characteristic of the cluster. Thus, pulling an arm i not only reveals information about its own reward distribution, but also about all arms belonging to the same cluster. This “correlation” among the arms complicates the exploration–exploitation trade-off that is encountered in the MABP because the observation dependencies allow us to test simultaneously multiple hypotheses regarding the optimality of an arm. We develop learning algorithms based on the principle of optimism in the face of uncertainty (Lattimore and Szepesvári in Bandit algorithms, Cambridge University Press, 2020), which know the clusters, and hence utilize these additional side observations appropriately while performing exploration–exploitation trade-off. We show that the regret of our algorithms grows as O ( K log T ) , where K is the number of clusters. In contrast, for an algorithm such as the vanilla UCB that does not utilize these dependencies, the regret scales as O ( M log T ) , where M is the number of arms. When K ≪ M , i.e. there is a lot of dependencies among arms, our proposed algorithm drastically reduces the dependence of regret on the number of arms.

• Publisher: Springer US,Springer Nature B.V
• Subject: Algorithms / Artificial Intelligence / Clusters / Computer Science / Control / Decision theory / Exploitation / Machine Learning / Mechatronics / Multi-armed bandit problems / Natural Language Processing (NLP) / Robotics / Simulation and Modeling / Tradeoffs