What is GENERator
GENERator is a generative genome foundation model launched by the AI for Science team at Alibaba Cloud's Feitian Lab, focusing on the design and generation of DNA sequences. The model is based on the Transformer decoder architecture, has a context length of 98k base pairs and 1.2 billion parameters, and its training data covers 386 billion bp of eukaryotic DNA. It performs excellently in multiple benchmarks, can generate DNA sequences with structures similar to natural protein families, and demonstrates optimization potential in tasks such as promoter design.
Main features of GENERator
- DNA sequence generation: GENERator can generate biologically meaningful DNA sequences that can encode proteins with structures similar to known families. It can successfully generate entirely new variants of the histone and cytochrome P450 families.
- Promoter design: Through fine-tuning, GENERator can design promoter sequences with specific activity for regulating gene expression. Experiments show that the generated promoter sequences differ significantly from natural samples in activity, demonstrating strong potential for gene expression regulation.
- Genome analysis and annotation: In gene classification and taxonomic group classification tasks, GENERator performs excellently, can efficiently identify gene locations, predict gene functions, and annotate gene structures.
- Sequence optimization: GENERator shows significant potential in sequence optimization, generating DNA sequences with specific activity through instructions, providing a new tool for synthetic biology and genetic engineering.
Technical principles of GENERator
- Transformer decoder architecture: GENERator adopts the Transformer decoder architecture, achieving efficient sequence modeling through multi-head self-attention mechanisms and feedforward neural networks. The decoder can handle long sequences, avoid seeing future information during generation, and ensure that the generated sequences conform to biological logic.
- Ultra-long context modeling: The model has a context length of 98k base pairs and can handle complex gene structures. It maintains coherence and biological meaning when generating long sequences.
- 6-mer tokenizer: GENERator uses a 6-mer tokenizer to split DNA sequences into nucleotide fragments of length 6. It performs better than single-nucleotide tokenizers and BPE tokenizers in generation tasks, balancing sequence resolution and context coverage.
- Pretraining strategy: The model is pretrained on large-scale data, and the dataset contains 386 billion bp of eukaryotic DNA. The pretraining task adopts Next Token Prediction (NTP), learning the semantics of DNA sequences by predicting the next nucleotide.
- Downstream task adaptation: GENERator performs excellently in multiple downstream tasks, including gene classification, taxonomic group classification, and promoter design. Through fine-tuning, the model can generate promoter sequences with specific activity, demonstrating strong gene expression regulation capability.
- Biological validation: The DNA sequences generated by the model can encode proteins with structures similar to natural protein families. By using Progen2 to calculate the perplexity (PPL) of the generated sequences and using AlphaFold3 to predict their three-dimensional structures, the biological significance of the generated sequences was verified.
Project address of GENERator
- Project official website：https://generteam.github.io/
- Github repository：https://github.com/GenerTeam/GENERator
- HuggingFace model library：https://huggingface.co/GenerTeam
- arXiv technical paper：https://arxiv.org/pdf/2502.07272
Application scenarios of GENERator
- DNA sequence design and optimization: GENERator can generate biologically meaningful DNA sequences, for example for customization of protein families. It can generate DNA sequences structurally similar to natural protein families, such as variants of the histone and cytochrome P450 families.
- Genomic Analysis and Annotation: In genomics research, GENERator can efficiently identify gene locations, predict gene functions, and annotate gene structures.
- Synthetic Biology and Genetic Engineering: GENERator provides a new tool for designing and optimizing gene expression regulatory elements, such as promoters and enhancers. It has important application value in synthetic biology and genetic engineering.
- Precision Medicine and Drug Design: By generating gene sequences associated with specific diseases, GENERator can provide support for precision medicine and drug design. It can be used to design targeted sequences for gene therapy.
- Sequence Optimization in Biotechnology: GENERator can generate DNA sequences with specific functions through instructions, providing new possibilities for sequence optimization in biotechnology.