Purpose
This repository is the PyTorch implementation of the paper "No Token Left Behind: Explainability-Aided Image Classification and Generation". The project focuses on using explainability methods to assist image classification and generation tasks.
Main Capabilities
The repository contains the following Notebooks and running instructions:
- Spatially conditioned image generation Notebook
- Image editing Notebook
- Image generation Notebook
- Prompt engineering running instructions
Usage
First, install the datasets according to DATASETS.md. Then create the required environment (the official materials do not provide the complete specific commands). Next, clone and install dassl in the external directory. Finally, run the experiments (the official materials do not provide the complete specific running commands).
Applicable Scenarios
Suitable for researchers and developers in directions such as explainability-aided image classification and generation, spatially conditioned generation, image editing, and prompt engineering.
Limitations and Dependencies
The project depends on multiple external projects: the image manipulation code is based on StyleCLIP, the image generation code is based on FuseDream, the spatially conditioned image generation code is based on VQGAN+CLIP and VQGAN, the prompt engineering code is based on CoOp and Dassl, and the explainability method code is based on Transformer-MM-Explainability. The example code is released under the terms of LICENSE. Some specific running commands and environment configuration details are not yet provided in the official materials.
