Word-As-Image is a semantic typography technique whose goal is to automatically transform a word into an illustration that visualizes the meaning of that word while maintaining the readability of the text. For example, the letter shapes of the word "bird" may be adjusted into an outline with bird-like features, but the word can still be recognized overall as "bird".
Main capabilities:
- Automatically adjust the geometric shapes of letters according to the semantics of the word, so that the text illustration can express the meaning of the word.
- Maintain the legibility of the text and preserve the style of the original font.
- Focus on concise, black-and-white geometric design, without changing color or texture and without using decoration.
- Can handle a variety of semantic concepts and supports multiple fonts.
- Can also be applied to Chinese characters (one Chinese character can represent one word).
How to use it (based on the technical pipeline described officially):
- Process each letter of the input word individually, representing each letter as a closed vector shape.
- Given the set of control points P of a letter and a semantic concept, iteratively optimize the new positions P̂ of the deformed letter.
- At each iteration, use a differentiable rasterizer (DiffVG) to backpropagate the raster-based loss gradients to the shape parameters.
- Feed the augmented rasterized deformed letters into a pretrained frozen Stable Diffusion model, and use Score Distillation Sampling (L_sds loss) to drive the letter shapes to embody the semantic concept.
- At the same time, use two additional loss functions: L_tone preserves local tone and structure by comparing the result after a low-pass filter (LPF) with the original letters; L_acap regularizes shape modification by constraining the deformation on the letter shape triangulation (D) to remain as shape-preserving as possible.
- Stable Diffusion 2's Depth-to-image can also be used as a post-processing step to add color and texture to the result.
Applicable scenarios:
- Occasions in creative design such as text illustrations, logos, and posters that need to combine text with the meaning of an image.
- Font design and semantic typography research.
- Text visualization that requires a black-and-white tone, a concise style, and an emphasis on readability.
Limitations:
- The official description emphasizes that it deliberately does not change color or texture and does not use decoration, so the default output is a black-and-white geometric style; if color and texture are needed, an additional post-processing step is required.
- Specific availability, pricing method, platform support, installation and deployment details, etc. are not yet provided.
