Textual Inversion is an open-source project for personalized text-to-image generation. Its core idea is to learn new "words" in the embedding space of a frozen text-to-image model from a small number of example images to represent a unique concept. The project was proposed by researchers from Tel Aviv University and NVIDIA, and the paper is titled "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion."
In terms of use, this method aims to solve the problem of how to use language to guide the model to generate a specific concept, for example, turning the user's cat into a painting, or imagining new products based on a favorite toy. With only 3-5 concept images provided by the user (such as a certain object or a certain style), the model can learn to represent that concept with a new "word" in the embedding space. These "words" can be combined into natural language sentences to achieve personalized creation in an intuitive way. The paper found that a single word embedding is enough to capture unique and diverse concepts, and presents concepts more faithfully than multiple baseline methods across a variety of applications and tasks.
In terms of main capabilities, this repository provides official code, data, and example inversion results. Features include: performing inversion training on an image set to learn the embedding corresponding to a concept; using the learned placeholder to generate new images; merging multiple checkpoints (LDM and SD embeddings) to use multiple concepts simultaneously in one prompt; releasing datasets and pretrained embeddings (some datasets are temporarily unavailable due to image copyright); and adding Stable Diffusion support (as of August 29, 2022, SD embedding merging is available, and SD key tuning code is in WIP status). The update log also shows that gradient storage and checkpoint mechanisms were optimized, reducing memory requirements and training time by about 55%.
In terms of usage, the project is based on and shares the environment dependencies of Latent Diffusion Models (LDM). You first need to set up the LDM environment and download the official LDM text-to-image checkpoint. When inverting an image set, you need to specify the initialization word (init_word). This word is a rough single-token description (such as 'toy', 'painting', 'sculpture') and serves only as the starting point for optimization, not as the placeholder that later represents the concept; if the input is more than one token, a replacement prompt will be shown. The paper used 5k training iterations, but some concepts (especially styles) may converge faster. For multiple GPUs, you can passin a comma-separated list of GPU indices via the --gpus parameter. Embeddings and output images are saved in the log directory. More options (such as changing the placeholder string, default "*", maximum number of training iterations, checkpoint saving frequency, etc.) can be configured in configs/latent-diffusion/txt2img-1p4B-finetune.yaml. When generating new images of a learned concept, use the placeholder string; merging checkpoints can be performed separately for LDM or SD embeddings.
Applicable scenarios include: personalized image generation for a specific object or style, combining concepts into new scenes or new characters, and studying the interpretability and personalization of the embedding space of text-to-image models.
In terms of limitations: the repository does not provide specific fee or pricing information (not yet available). Training set images should remain upright; images taken with mobile phones may be oriented incorrectly due to rotation metadata, and need to be manually corrected or await support for metadata parsing. Results may be sensitive to random seeds, so you can try changing the seed and re-running the inversion. Stable Diffusion support is still in progress. Some datasets are unavailable due to image ownership, and the pretrained model is "coming soon." Adding "a photo of" to the prompt usually improves target consistency.
