Efficient and accurate search in petabase-scale sequence repositories

Abstract

Abstract The amount of biological sequencing data available in public repositories is growing rapidly, forming a critical resource for biomedicine. However, making these data efficiently and accurately full-text searchable remains challenging. Here we build on efficient data structures and algorithms for representing large sequence sets 1–6 . We present MetaGraph, a methodological framework that enables us to scalably index large sets of DNA, RNA or protein sequences using annotated de Bruijn graphs. Integrating data from seven public sources 7–13 , we make 18.8 million unique DNA and RNA sequence sets and 210 billion amino acid residues across all clades of life—including viruses, bacteria, fungi, plants, animals and humans—full-text searchable. We demonstrate the feasibility of a cost-effective full-text search in large sequence repositories (67 petabase pairs (Pbp) of raw sequence) at an on-demand cost of around US$100 for small queries up to 1 megabase pairs (Mbp) and down to US$0.74 per queried Mbp for large queries. We show that the highly compressed representation of all public biological sequences could fit on a few consumer hard drives (total cost of around US$2,500), making it cost-effective to use and readily transportable for further analysis. We explore several practical use cases to mine existing archives for interesting associations, demonstrating the use of our indexes for integrative analyses, and illustrating that such capabilities are poised to catalyse advancements in biomedical research.