Random rotational embedding Bayesian optimization for human-in-the-loop personalized music generation

Abstract

Generative deep learning models, such as those used for music generation, can produce a wide variety of results based on perturbations of random points in their latent space. User preferences can be incorporated in the generative process by replacing this random sampling with a personalized query. Bayesian optimization, a sample-efficient nonlinear optimization method, is the gold standard for human-in-the-loop optimization problems, such as finding this query. In this paper, we present random rotational embedding Bayesian optimization (ROMBO). This novel method can efficiently sample and optimize high-dimensional spaces with rotational symmetries, like the Gaussian latent spaces found in generative models. ROMBO works by embedding a low-dimensional Gaussian search space into a high-dimensional one through random rotations. Our method outperforms several baselines, including other high-dimensional Bayesian optimization variants. We evaluate our algorithm through a music generation task. Our evaluation includes both simulated experiments and real user feedback. Our results show that ROMBO can perform efficient personalization of a generative deep learning model. The main contributions of our paper are: we introduce a novel embedding strategy for Bayesian optimization in high-dimensional Gaussian sample spaces; achieve a consistently better performance throughout optimization with respect to baselines, with a final loss reduction of 16%-31% in simulation; and complement our simulated evaluations with a study with human volunteers (n = 16). Users working with our music generation pipeline find new favorite pieces 40% more often, 16% faster, and spend 18% less time on pieces they dislike than when randomly querying the model. These results, along with a final survey, demonstrate great performance and satisfaction, even among users with particular tastes.