Bidirectional reinforcement learning neural network for constrained molecular design

Abstract

Abstract We present BiRLNN, a bidirectional molecular design framework that combines recurrent neural networks with reinforcement learning to optimize drug-like properties of generated compounds. We examined the use of Self-Referencing Embedded Strings representations, which ensures 100% syntactic validity of generated molecules. By generating molecular sequences in both forward and backward directions, we enabled more balanced exploration of chemical space while maintaining constraint requirements during molecular design. To guide generation towards desirable pharmacological targets, we implement a multi-objective reward function based on quantitative estimate of drug-likeness and synthetic accessibility, and apply policy gradient-based reinforcement learning for fine-tuning. We demonstrate that our bidirectional model covers the full constrained chemical space compared to unidirectional ones using pharmaceutically relevant fragments, allowing it to explore regions containing molecules unreachable by the latter. Moreover, the reinforcement learning process successfully steers the constrained generation process toward desirable compound classes with improved reward metrics. Our results demonstrate that BiRLNN offers a robust and flexible strategy for navigating chemical space in multi-objective drug design tasks.