Version Compatibility
Reference examples tested with: REINVENT 4.0+, RDKit 2024.09+, PyTorch 2.1+, MolMIM (NVIDIA BioNeMo), chemprop 2.0+.
Before using code patterns, verify installed versions match. If versions differ:
- Python:
pip show <package>thenhelp(module.function)to check signatures
If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.
Generative Molecular DesignGenerate novel molecules biased toward desired properties using deep generative models. REINVENT 4 (Loeffler et al. 2024, AstraZeneca) provides four generator families: Reinvent (de novo), Libinvent (scaffold decoration and library design), Linkinvent (linker design), and Mol2Mol (similarity-constrained molecular optimization). These support design tasks including R-group replacement and scaffold hopping and can be used with transfer learning, reinforcement learning, and curriculum learning. For specific niches: MolMIM (NVIDIA BioNeMo) for latent-space property optimization, DiffSMol / DiGress for diffusion-based generation, and JT-VAE for latent-space optimization. The art of generative design is in the scoring function: poorly-designed scoring rewards uninteresting molecules, while well-designed scoring captures both activity and developability.
For QSAR/scoring models that feed generative design, see chemoinformatics/qsar-modeling. For synthetic feasibility, see chemoinformatics/retrosynthesis. For library enumeration as alternative, see chemoinformatics/reaction-enumeration.
Generator Mode Taxonomy
| Mode | Input | Output | Use case | Fails when |
|---|---|---|---|---|
| De novo | Empty seed or training set | Novel molecules | Wide chemical space exploration | Synthetic feasibility weak |
| Scaffold decoration | Scaffold + attachment points | Decorated molecules | Series expansion | Generation diversity limited by scaffold |
| Linker design | 2 fragments | Linker molecules | PROTAC, ternary complex | Few linker geometric options |
| R-group replacement | Scaffold + existing R-groups | New R-group set | Optimize one position | Single-position only |
| Molecular optimization | Lead molecule | Improved analogs | Lead optimization | Improvement window narrow |
| Constrained generation | Hard constraints (MW, fragments) | Compliant molecules | Patent / IP design | Constraints overly restrictive |
Learning Algorithm Taxonomy
| Algorithm | Use | Pro | Con |
|---|---|---|---|
| Transfer learning (TL) | Adapt prior model to focused training set | Stable, simple | Limited optimization power |
| Reinforcement learning (RL) | Reward-driven generation | Powerful for MPO | Reward hacking risk |
| Curriculum learning (CL) | Gradual constraint introduction | Better convergence | Slower; tuning sensitive |
Decision Tree by Scenario
| Scenario | Generator | Algorithm | Scoring |
|---|---|---|---|
| New target, no SAR | De novo | Benchmark RL against simpler search | Validated target evidence + developability objectives |
| Series expansion | Scaffold decoration | TL on series + RL | QSAR ensemble + QED |
| PROTAC linker | Linker design | Project-specific constrained workflow | Validated geometry/ternary-complex evidence; no generic DC50 surrogate |
| Lead optimization MPO | Molecular optimization | CL with staged constraints | Multi-task: activity + ADMET |
| Diverse hit set | De novo with diversity bonus | RL + Tanimoto distance to known | Activity + diversity |
| Patent space carve-out | Constrained de novo | RL + structural constraints | Activity + novelty |
| Hit-to-lead | R-group replacement | TL on lead + RL | Activity + Lipinski |
| ADMET-aware design | De novo or optimization | RL | hERG + CYP + AMES + QED |
REINVENT 4 Setup
REINVENT 4 uses a TOML configuration file specifying generator, algorithm, prior model, and scoring functions.
Goal: Configure a reinforcement-learning REINVENT 4 run with a prior, agent, sampling parameters, and a QED scoring component.
Approach: Build a release-matched REINVENT 4 staged-learning TOML config with [parameters] for the prior/agent checkpoints, [[stage]] blocks, and one or more [[stage.scoring.component]] blocks. Validate the config with the installed release because component parameters evolve between versions.
run_type = "staged_learning"
device = "cuda:0"
[parameters]
prior_file = "priors/reinvent.prior"
agent_file = "priors/reinvent.prior"
batch_size = 64
unique_sequences = true
[[stage]]
termination = "simple"
min_steps = 25
max_steps = 500
[stage.scoring]
type = "geometric_mean"
[[stage.scoring.component]]
[stage.scoring.component.QED]
[[stage.scoring.component.QED.endpoint]]
name = "QED"
weight = 1
# The REINVENT 4 CLI binary is `reinvent` (not `reinvent4`).
reinvent -l logfile.log config.toml
Output: a live stage CSV using summary_csv_prefix, plus the configured chkpt_file at stage termination or graceful interruption. Post-process the CSV to select molecules; REINVENT does not emit a checkpoint and SMILES file at every iteration by default.
Scoring Function Design (Most Important Part)
A good scoring function:
- Returns 0-1 (normalized)
- Combines multiple endpoints
- Penalizes pathological generations (PAINS, unstable, unsynthesizable)
Goal: Build a multi-component generative reward that balances predicted activity, drug-likeness, synthesizability, and novelty.
Approach: Combine a QSAR sigmoid on pIC50, QED, SA-score reverse-sigmoid, and Tanimoto-similarity reverse-sigmoid via geometric mean so any zero component zeroes the total.
In REINVENT 4, define these under the active stage's [stage.scoring] section, with each component using the exact component and endpoint tables from the installed release's configuration examples. Do not reuse REINVENT 3 [scoring_function] or [[scoring_function.components]] syntax in a REINVENT 4 config. The accompanying example is deliberately limited to built-in, documented component structure; add predictive-property endpoints only after validating their release-specific model-container parameters.
geometric_mean ensures all components must be reasonably high (one zero -> zero total). arithmetic_mean allows compensation.
Multi-Parameter Optimization (MPO)
Lead optimization commonly involves multiple objectives. The following component types illustrate a project-specific scoring design; weights and transforms must be fit to the actual assays and decision context.
| Component | Weight | Transformation |
|---|---|---|
| Target activity (predicted pIC50) | 0.3 | sigmoid 5-8 |
| Selectivity (off-target ratio) | 0.2 | sigmoid 1-100 |
| QED | 0.1 | identity |
| Synthetic accessibility (SA score) | 0.1 | reverse sigmoid 1-4 |
| hERG predicted prob | 0.1 | reverse sigmoid 0.3-0.7 |
| AMES predicted prob | 0.1 | reverse sigmoid 0.3-0.7 |
| Tanimoto novelty vs known | 0.1 | reverse sigmoid 0.4-0.6 |
The weights and transformation bounds above are repository starting examples only. Normalize weights as required by the selected aggregation and tune every bound against project assay distributions and prospective behavior.
Reward Hacking (Production Pitfall)
RL agents will find ways to maximize reward without learning the intended behavior:
- Trivial scaffolds that score high on QED
- Repeat structural motifs that game similarity scoring
- Out-of-distribution molecules that exploit QSAR overconfidence
- Trivial SMILES (e.g., "CCC...C") that match generic scoring
Mitigations:
- Include one or more synthesis-aware signals when they have been validated for the project; SA score alone does not establish route feasibility
- Use ensemble QSAR with uncertainty (penalize high-uncertainty predictions)
- Include diversity bonus (Tanimoto to reference)
- Add fingerprint similarity penalty within batch (prevent mode collapse)
- Validate generated samples on held-out QSAR test set
Synthetic Accessibility Scoring
sa_score (Ertl 2009) measures synthetic accessibility: 1 (easy) to 10 (very hard).
from rdkit.Contrib.SA_Score import sascorer
from rdkit import Chem
def sa_score(smi):
mol = Chem.MolFromSmiles(smi)
if mol is None:
return None
return sascorer.calculateScore(mol)
sascorer is shipped in RDKit Contrib in current RDKit distributions. Use the namespaced import above; do not install an unrelated top-level package.
The score is a 1-to-10 heuristic derived from fragment contributions and molecular complexity, with lower values intended to indicate easier synthesis. It is not a route planner, cost estimate, or calibrated feasibility probability. Use it as one audited reward component or annotation, never as an absolute filter.
Diffusion-Based Generation (Modern Alternatives)
| Tool | Approach | Strength | Status |
|---|---|---|---|
| DiGress (Vignac 2023) | Discrete diffusion on graphs | Conditional generation | Public |
| DiffSMol (Chen 2025) | Equivariant diffusion | 3D molecule generation | Public |
| MolDiff (Peng 2023) | Full-atom diffusion | Joint atom/bond generation | Public |
| TargetDiff (Guan 2023) | Pocket-conditioned equivariant diffusion | Structure-based design | Public |
Diffusion models iteratively denoise molecular representations, whereas REINVENT generators autoregressively construct SMILES. Diversity, validity, and drug-likeness depend on the model, training data, conditioning, and evaluation protocol; compare them on a matched benchmark for the intended task.
Constrained / Goal-Directed Generation
Goal: Enforce hard structural requirements (e.g., must contain hydroxyl) and exclude PAINS without letting constraint satisfaction game the reward.
Approach: Stage transfer learning then RL. In REINVENT 4, CustomAlerts is a global structural-alert filter: a match produces zero and it is applied before score aggregation. MatchingSubstructure is a scoring component (1 for a match and 0.5 otherwise), so it is a soft penalty rather than a hard inclusion constraint. Apply a separate post-generation SMARTS validation step when presence of a feature is mandatory.
[[stage.scoring.component]]
[stage.scoring.component.CustomAlerts]
[[stage.scoring.component.CustomAlerts.endpoint]]
name = "Unwanted SMARTS"
params.smarts = ["PAINS_SMARTS_1", "BRENK_SMARTS_1"]
[[stage.scoring.component]]
[stage.scoring.component.MatchingSubstructure]
[[stage.scoring.component.MatchingSubstructure.endpoint]]
name = "Hydroxyl preference"
weight = 0.1
params.smarts = ["[OX2H]"]
There is no REINVENT 4 filter_only option for these components. Treat structural alerts as triage flags where appropriate, and separately verify any true hard inclusion or exclusion rule on the generated structures.
MolMIM (NVIDIA BioNeMo)
MolMIM encodes SMILES into a learned latent space, uses gradient-free CMA-ES to optimize a user-defined property objective, and decodes candidate molecules.
# Pseudo-code; requires NVIDIA NIM access
# from bionemo.molmim import MolMIMOptimizer
# optimizer = MolMIMOptimizer(model="molmim-property-optimizer")
# optimized = optimizer.optimize(seed_smiles, target_property="logp", target_value=2.0)
Tradeoffs against REINVENT depend on the oracle budget, objective, and implementation; benchmark both under matched constraints when selecting a generator.
Per-Tool Failure Modes
REINVENT RL -- mode collapse
Trigger: The reward, learning strategy, or diversity control favors a narrow chemotype.
Mechanism: Agent finds a high-scoring local maximum and stops exploring.
Symptom: Diversity and scaffold coverage collapse relative to a project-defined baseline while reward continues to rise.
Fix: Add diversity bonus to scoring; reduce sigma; reset agent if collapsed.
REINVENT TL -- overfitting
Trigger: Transfer learning data are too small or homogeneous for the intended generalization task.
Mechanism: Generator memorizes training set; no generalization.
Symptom: Generated molecules near-identical to training set actives.
Fix: Use larger training set; mix with diverse external sample; apply RL after TL.
Generated molecule unsynthesizable
Trigger: SA score missing from reward.
Mechanism: The reward omits synthesis evidence, allowing candidates with no plausible validated route to score well.
Symptom: AiZynthFinder cannot solve route; medchem rejects.
Fix: Combine audited synthesis-aware annotations with reaction- or route-based validation for selected candidates.
PAINS in generation
Trigger: No structural alerts in scoring.
Mechanism: Curcumin / rhodanine / quinone scaffolds optimize for activity (false positives in training data).
Symptom: Generated molecules match PAINS_A.
Fix: Flag relevant structural alerts for orthogonal assay review or apply a project-justified penalty; never reward a PAINS match.
Diffusion model OOD
Trigger: Pocket-conditioned diffusion on novel target family.
Mechanism: Training distribution covered specific protein families; novel targets extrapolate.
Symptom: Generated molecules look like training distribution, not optimized for target.
Fix: Validate on target-family-held-out evaluation; supplement with classical methods.
Validation set leakage
Trigger: Same molecules in training generators and downstream QSAR.
Mechanism: Scoring model has seen the molecule; predictions optimistic.
Symptom: Held-out QSAR validation fails on top generated.
Fix: Use scaffold-split QSAR; ensure scoring model trained on a held-out set vs generation samples.
Reconciliation: REINVENT vs Diffusion
| Aspect | REINVENT 4 | Diffusion |
|---|---|---|
| Speed | Implementation-, hardware-, and oracle-dependent | Implementation-, hardware-, and sampling-step-dependent |
| Output diversity | Model/training/reward-dependent | Model/training/conditioning-dependent |
| Drug-likeness of output | Training- and reward-dependent | Training- and conditioning-dependent |
| Scoring flexibility | Excellent (TOML config) | Method-specific |
| Production maturity | High | Emerging |
| When to use | When its priors and scoring system validate well for the task | When a matched benchmark supports the selected diffusion model |
Common Errors
| Symptom | Cause | Fix |
|---|---|---|
| REINVENT generates invalid SMILES | Prior/tokenization/model mismatch or sampling issue | Inspect invalid-token logs, prior compatibility, and release-matched sampling settings; sigma is not a token sampling rate |
| QSAR score all 0.0 | Out-of-domain molecules | Ensemble + uncertainty; reject high-uncertainty |
| All generations duplicates | unique_sequences=False |
Set unique_sequences=true |
| Generated SMILES too long | Prior/model sequence behavior | Use a release-documented sampling constraint or post-generation project rule; do not invent a staged-learning max_length field |
| Reward stuck at 0.5 | Constraints conflict | Inspect scoring components; reduce constraint count |
| Diffusion model crashes | Input violates model-specific pocket/size contract | Follow that model release's documented preprocessing and limits |
| MolMIM cold-start slow | Latent search exhaustiveness | Reduce search budget |
| Optimization converges trivially | Reward gradient dominated by one term | Use geometric_mean; rebalance weights |
References
- Loeffler et al., J. Cheminformatics 16:20 (2024) -- REINVENT 4 framework and four generator families (DOI 10.1186/s13321-024-00812-5).
- Olivecrona M et al., J. Cheminformatics 9:48 (2017) -- REINVENT original (DOI 10.1186/s13321-017-0235-x).
- Vignac et al., ICLR (2023) -- DiGress discrete diffusion.
- Chen H et al., Nat. Mach. Intell. 7:758-770 (2025) -- DiffSMol structure-based 3D molecular generation (DOI 10.1038/s42256-025-01030-w).
- Peng X, Guan J, Liu Q, Ma J. Proc. ICML, PMLR 202:27611-27629 (2023) -- MolDiff full-atom molecular diffusion.
- Guan J et al., ICLR (2023) -- TargetDiff pocket-conditioned 3D equivariant diffusion (OpenReview: kJqXEPXMsE0).
- Reidenbach D, Livne M, Ilango RK, Gill M, Israeli J. MLDD Workshop at ICLR (2023) -- MolMIM and CMA-ES latent-space optimization (OpenReview: iOJlwUTUyrN).
- Jin W, Barzilay R, Jaakkola T. Proc. ICML, PMLR 80:2323-2332 (2018) -- JT-VAE junction-tree.
- Ertl P, Schuffenhauer A. J. Cheminformatics 1:8 (2009) -- SA score (DOI 10.1186/1758-2946-1-8).
- REINVENT 4 official repository and release-matched configs: https://github.com/MolecularAI/REINVENT4
- RDKit SA-score implementation: https://github.com/rdkit/rdkit/tree/master/Contrib/SA_Score
Related Skills
- chemoinformatics/qsar-modeling - Build scoring models for generative
- chemoinformatics/retrosynthesis - Validate synthetic feasibility post-generation
- chemoinformatics/molecular-standardization - Standardize generated SMILES
- chemoinformatics/admet-prediction - ADMET in scoring components
- chemoinformatics/substructure-search - PAINS / BRENK filter for generation
- chemoinformatics/scaffold-analysis - Scaffold-aware generation control
- chemoinformatics/reaction-enumeration - Alternative to generative for combinatorial
- chemoinformatics/virtual-screening - Validate generated against target