CLIPasso is a method for converting object images into sketches, capable of achieving different levels of abstraction while preserving the key visual features of the object. Even with minimal representations (for example, drawing a flamingo and a horse with only a few strokes), the semantics and structure of the depicted subject can still be recognized.
Abstraction is central to sketching, because line drawings are simple and minimal. Abstraction requires identifying the essential visual attributes of an object or scene, which demands semantic understanding and prior knowledge of high-level concepts. Therefore, abstract depiction is challenging for artists, and even more so for machines. CLIPasso proposes an object sketching method that can achieve different levels of abstraction under the guidance of geometric and semantic simplification.
Unlike sketch generation methods that typically rely on explicit sketch datasets for training, CLIPasso leverages the ability of CLIP (Contrastive Language-Image Pretraining) to extract semantic concepts from sketches and images. This method defines a sketch as a set of Bezier curves and uses a differentiable rasterizer to directly optimize the curve parameters against a CLIP-based perceptual loss. The degree of abstraction is controlled by varying the number of strokes. The generated sketches exhibit multiple levels of abstraction while maintaining recognizability, the underlying structure, and the essential visual components of the depicted object.
How it works: CLIPasso is optimization-based, so it does not require any explicit sketch dataset. It uses the CLIP image encoder to guide the process of converting a photo into an abstract sketch. CLIP encoding provides semantic understanding of the concepts shown, while the photo itself provides the geometric basis of the sketch for the specific object. A sketch is defined as N black strokes placed on a white background. Different levels of abstraction are created by varying the number of strokes N.
Given a target image I, the goal is to synthesize the corresponding sketch S while preserving the semantic and geometric attributes of the object. First, the salient regions of the input image are extracted to define the initial positions of the strokes. Then, at each optimization step, the stroke parameters are fed into a differentiable rasterizer to generate a rasterized sketch. The generated sketch and the original image are input into CLIP together to define a CLIP-based perceptual loss. The key to the method's success is using the intermediate layers of the pretrained CLIP model to constrain the geometry of the output sketch. Without this term, the output sketch will not resemble the input image. The loss is backpropagated through the differentiable rasterizer, and the stroke control points are directly updated at each step until the loss function converges.
Unlike traditional sketching methods, CLIPasso does not use a sketch dataset for training, but instead optimizes under the guidance of CLIP. Therefore, the method is not limited to the specific categories observed during training, because no category definitions are introduced at any stage. This makes the method robust to various inputs.In addition, CLIPasso supports editing brush styles on SVG and demonstrates results on "in the wild" sketches, such as processing 100 random cat images in SketchCOCO.
