SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsAlayaWorld

AlayaWorld

AlayaWorld is a full-stack interactive long-horizon world model open-sourced by AlayaLab, fine-tuned on LTX-2, focusing on "playable video world generation." It adopts an autoregressive world model architecture, supports real-time camera control, prompt switching during generation, and maintains memory consistency over longer time spans. The project is built around four core features: interactivity, consistency, stability, and runtime: trajectory-aware navigation is achieved through a rendered 3D cache combined with AdaLN camera modulation, and new events are introduced mid-generation through block-level prompt switching; spatial recall is performed using explicit 3D cache reprojection, and temporal continuity is maintained with compressed frame history embeddings; long-horizon error accumulation is suppressed through training on drifted history and an error bank reinjection mechanism; real-time interaction is achieved through few-step DMD distillation and short time blocks. The repository provides a unified launcher for training, validation, and four inference paths, and has open-sourced the inference code, full-stack training code, v1.1 weights (AR + DMD), Stage1 bidirectional weights, and part of the training data, and provides a browser-based interactive demo. The project is limited to academic research and non-commercial use, and adopts the LTX-2 community license agreement.

Open website

About this tool

Purpose

AlayaWorld is a full-stack open-source interactive long-horizon world model, positioned for "Long-Horizon and Playable Video World Generation." It is an interactive autoregressive world model with real-time camera control, prompt switching, and long-horizon memory consistency, mainly used to generate interactive continuous video worlds rather than short video clips generated in a single pass.

Main Capabilities

The project is built around four core properties:

  • Interaction: Provides two control channels—a rendered 3D cache combined with lightweight AdaLN camera modulation to achieve grounded, trajectory-aware navigation; and block-level prompt switching, which can introduce new events during generation.
  • Consistency: Uses two complementary memories—explicit 3D cache reprojection to the query viewpoint to achieve spatial recall, plus compressed frame history embeddings to maintain temporal continuity, so that revisited locations remain recognizable.
  • Stability: Trains on drifted history, and introduces an error bank to reinject accumulated artifacts into memory and targets, preventing errors from continuously accumulating over minute-level rollouts.
  • Runtime: Achieves real-time interaction through few-step DMD distillation and short time blocks, with prompt switching occurring at block boundaries to minimize visual and semantic latency.

Usage

The repository uses a unified launcher to drive training, validation, and all four inference paths, with the specific paths selected entirely by configuration.

Environment: A single Python environment covers training and all inference paths (verified on torch 2.7.1 + CUDA 12.8). Weight paths are located under paths: in each configuration and must point to the actual download location; the full directory structure is shown in docs/vigeo/README.md.

Weights: The Stage1 bidirectional weights are shared by the DA3 version and the v1.1 ViGeo version; both start the subsequent autoregressive and spatial memory stages from the same Stage1 checkpoint. Stage1 I2V itself uses neither DA3 nor ViGeo, and does not require camera trajectories; the subsequent AR/DMD checkpoints still correspond to their respective versions.

Bidirectional inference requires downloading the two checkpoint files to the directory expected by configs/infer_i2v_bidir.yaml ; at the same time, place the LTX-2.3 base model in weights/ltx-2.3/ltx-2.3-22b-dev.safetensors, Gemma in weights/ltx-2.3/google/gemma-3-12b-it-qat-q4_0-unquantized, or adjust the configuration's paths.base_transformer、paths.vae、paths.gemma and paths.resume_checkpoint. The Stage1 release contains only the transformer and its configuration; the VAE and text encoder dependencies must be obtained separately.

Inference: All four paths use the same launch form. Paths a–c render a fixed camera trajectory to a file; path d generates a bidirectional I2V clip, by default based on playground/case1, at 481 frames, 960×544, 24 fps (about 20 seconds). To use your own images and prompts, under validation.modes.i2v_bidir.dataset set image_dir and prompt_file；captions_json can provide per-image prompts. Path d does not require a pose file or a spatial memory checkpoint.

To drive the camera and switch prompts during generation runtime,reactor/ provides path a as a live stream in the form of a browser demo. Note: b/c/d run through the trainer's validation loop, so they require VALIDATE_ONLY=1; path a is dispatched directly to the inference pipeline (in the configuration da3_infer.enabled), ignoring that variable. The vigeo spatial path is incompatible with FA3; if FA3 has been built, launch with ALAYA_USE_FA3=0 . For the full CLI options of path a, see inference/README.md.

To run b/c on your own images and trajectories, use scripts/infer/generate_video.sh to prepare the inputs in one step (--image/--prompt together with --extrinsics or --synth-frames) and launch c. The cases for path a consist of three files with a shared prefix.

Training: The four stages use the same launcher, and each stage's paths: points to the previous stage's checkpoint. For dataset download, data format, details of each stage, and launcher parameters, see docs/vigeo/README.md.

Release Notes

  • [2026-07-08] Project page and technical report released.
  • [2026-07-16] Inference code released, pretrained weights available on Hugging Face.
  • [2026-07-21] Full technical report released.
  • [2026-08-17] Full-stack training + inference code, v1.1 weights (AR + DMD), and part of the training data open-sourced, with the v1.1 technical report.
  • [2026-08-20] Browser interactive demo: you can drive the camera with the keyboard and change prompts mid-rollout, with streaming output as it generates, see reactor/.
  • [2026-09-26] Stage1 bidirectional weights released, shared by the DA3 version and the v1.1 ViGeo version, supporting image-to-video inference via a unified launcher.

Weights and Data

  • Pretrained weights: Hugging Face AlayaLab/AlayaWorld
  • Shared Stage1 bidirectional weights:AlayaWorld-stage1
  • v1.1 pretrained weights: AR is AlayaWorld-v1.1-stage2b, DMD is AlayaWorld-v1.1-stage3
  • Training data (partial):AlayaWorld-v1.1-data

Applicable Scenarios and Limitations

This project is based on LTX-2 from Lightricks Ltd. Portions of the original LTX-2 codebase (ltx2/) were modified by Alaya Lab for academic and research purposes, and the released weights (merged_infer.safetensors) are fine-tuned from LTX-2.3. Therefore the project (code and weights) is released under the LTX-2 Community License Agreement, retaining all of LTX-2's original copyright, license, patent, trademark, and attribution notices.

For academic research and non-commercial use only. Commercial use of LTX-2 and its derivatives requires contacting Lightricks Ltd. (entities with annual revenue ≥ 10 million USD require a commercial license). Third-party weights (Gemma-3 text encoder, Depth-Anything-3) are not redistributed in this repository and must be obtained from their original sources under their respective licenses. For full attribution, see NOTICE and THIRD_PARTY_LICENSES.md.

Team and Contact

Core lead: Kaipeng Zhang; Lead: Chuanhao Li; Core contributors: Chuanhao Li, Kaipeng Zhang, Yifan Zhan, Yongtao Ge, Yuanyang Yin; Other contributors include Jiaming Tan, Kang He, Liaoyuan Fan, Mingliang Zhai, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, Zihui Gao. Collaboration or business inquiries: kaipeng.zhang@shanda.com.

Related tools

Browse category