aphantasia

Aphantasia is a collection of text-to-image/video generation tools, derived from the artwork of the same name. It is based on OpenAI's CLIP model and the Lucent library, uses FFT/DWT/RGB parameterizers, and can generate images without a GAN. The project has been updated to replace the old depth estimation method with Depth Anything 2, and has been tested on Python 3.7-3.11 and PyTorch 1.7.1 to 2.3.1.

Main capabilities

• Generates large amounts of detailed texture, similar to a deepdream effect.
• Supports Full HD/4K and higher resolutions.
• Supports multiple CLIP models.
• Continuous mode processes a list of phrases (for example, illustrating lyrics).
• Supports pan/zoom motion, with smooth interpolation.
• Direct RGB pixel optimization (very stable).
• 3D appearance based on Depth Anything 2.
• Complex queries: text and/or images as the main prompt, with separately configurable style prompts and subject prompts to subtract (avoid).
• Can start/resume the generation process from saved parameters or an image.

Usage After installing dependencies such as CLIP, images can be generated via the command line. For example, generate an image from a text prompt (size can be set):

python aphantasia.py --in_txt "prompt" --size 512

Reproduce an image:

python aphantasia.py --in_img image.jpg

If the --sync X parameter is greater than 0, an LPIPS loss is added to keep the composition similar to the original image. Text and image prompts can be combined. For non-English languages, use --translate(Google Translate).

Set a more specific query:

python aphantasia.py --in_txt "good prompt :1 | also good prompt :1 | bad prompt :-0.5"

Other options:

• --model M Select the CLIP vision model: ViT-B/32 (default), ViT-B/16, RN50, RN50x4, RN50x16, RN101. You can also set --dualmod to use both ViT-B/32 and ViT-B/16 (recommended).
• --dwt Switch to the DWT (wavelet) generator instead of FFT. There are several methods, selected by --wave X , such as db2, db3, coif1, coif2, etc.
• --align XX Regarding composition (or sampling distribution): uniform may be the most suitable; overscan can create semi-seamless tileable textures.
• --steps N Set the number of iterations. 100-200 is enough to get started; 500-1000 will be more refined.
• --samples N Set the number of image cuts (samples) processed per step. The more samples there are, the fewer iterations can be used to achieve similar results (and vice versa). 200/200 is a good guess. Note: GPU memory is mainly consumed by this number (not resolution)!
• --aest X Enhance overall cuteness through aesthetic loss. Try various values (possibly negative).
• --decay X(composition softness),--colors X(saturation), and --contrast X may be useful, especially for ResNet models (they tend to burn colors).
• --sharp X may help increase sharpness if the image becomes "myopic" after increasing decay. It affects other color parameters, so it is best to adjust them together! Current default values:--decay 1.5 --colors 1.8 --contrast 1.1 --sharp 0。
• --transform X Apply some enhancements, which usually improves the results (but is slower). There are several choices; fast seems optimal.
• --optimizer Can be adam, adamw, adam_custom, or adamw_custom. The custom options are noisier but stable; pure adam is softer, but may tend toward colorful blur.
• --invert Invert the entire criterion if you want to check the "exact opposite".
• --save_pt myfile.pt Save the FFT/DWT parameters so that the next query can be resumed with --resume myfile.pt . You can also start/resume directly from an image file.
• --opt_step N Tell it to save every Nth frame (useful for high iteration counts, default 1).
• --verbose(Enabled by default) Enables some print output and real-time image preview.

Some experimental tricks (effectiveness is not very certain):

• --enforce X Add more detail by enhancing the similarity between two parallel samples. A good starting value is about 0.1.
• --expand X Increase diversity by forcing a difference between the front/back samples. A good starting value is about 0.3.
• --noise X Add some noise to the parameters, which may make the composition less clogged (to a certain extent).
• --macro X(0 to 1) Will shift the generation toward larger forms and less scattered compositions. It should not be too close to 1, because quality depends on the diversity of the samples.
• --prog Set a progressive learning rate (from 0.1 times to 2 times the lrate setting). In some cases it may promote the creation of macro forms.
• --lrate Control the learning rate. The range is fairly wide (at least tested between 0.001 and 10).

Text to Video [Continuous Mode] There are two ways to make a video from a text file, processing line by line, all at once.

The new method: interpolate the subject into a constant stream, with permanent pan/zoom motion and an optional 3D appearance. Make a video from two text files, processing line by line, rendering 100 frames per line:

python aphantasia.py --in_txt file1.txt --in_txt2 file2.txt --steps 100

Make a video from two phrases, with a total length of 500 frames:

python aphantasia.py --in_txt "phrase1" --in_txt2 "phrase2" --steps 500

Prefix (-pre), suffix (-post), and "stop words" (--in_txt0) can also be loaded as phrases or text files. All text inputs understand weighted syntax, such as good prompt :1 | also good prompt :1 | bad prompt :-0.5(within a line). Images can also be used as references, via the --in_img parameter. Explore other parameters for more explicit control. This method works best for direct RGB pixel optimization, but it can also be used with FFT parameterization:

python aphantasia.py --in_txt file.txt --dwt

To add a 3D appearance, add --depth 0.01to the command.

Generate a separate image for each line of text (with sequence and training video, as in the single-image mode above), then render the final video from these images (blending the images in FFT space), with a length of duration seconds. Make a video from a text file, processing line by line:

python aphantasia.py --in_txt file.txt --steps 100 --duration 10

There is a --keep X parameter that controls the degree to which the next line/image generation follows the previous one. 0 means random initialization, and higher values keep the original composition more strictly. Safe values are 1~2 (higher numbers may cause the image to get stuck). Make a video from a directory of saved *.pt snapshots (interpolation only):

python aphantasia.py --in_pt dir

Other Generators

• VQGAN from Taming Transformers: one of the best methods for color/tone/detail (especially the new Gumbel-F8 model); but the resolution is fairly limited (about 800x600 max on Colab).
• CPPN + export to HLSL shaders: one of the earliest methods, can be exported to TouchDesigner, vvvv, Shadertoy, etc.
• SIREN + Fourier feature modulation: another early method, not very interesting by itself.

Use Cases

• Generate large amounts of detailed textures, similar to deepdream effects.
• Create images at full HD/4K and above.
• Illustrate lists of phrases such as lyrics to generate continuous videos.
• Image generation requiring complex queries (text and/or image prompts).
• Resume the generation process from saved parameters or images.

Limitations

• Dependencies such as CLIP need to be installed; the specific installation steps are not provided in the materials.
• The test environment is Python 3.7-3.11 and PyTorch 1.7.1 to 2.3.1; other versions may be incompatible.
• GPU memory consumption is mainly affected by --samples Quantity matters, not resolution.
• Some experimental techniques have uncertain effects.
• The VQGAN generator has limited resolution.

Acknowledgements Based on OpenAI's CLIP model (paper). The FFT encoding is taken from the Lucent library, and the 3D depth processing was done by deKxi. Thanks to Ryan Murdock, Jonathan Fly, Hannu Toyryla, @eduwatch2, and torridgristle for their ideas.