Skip to content

README.md

Latest commit

 

History

History
164 lines (110 loc) · 5.03 KB

README.md

File metadata and controls

164 lines (110 loc) · 5.03 KB

Data Engine of MINIMA (CVPR 2025) for Multi-modal Image Generation

MINIMA Data Engine is an open-source project that integrates cutting-edge methods from the community to enable multi-modal transformations.
Currently, the engine supports the following modalities:

framework

Online Demo

The online demo is under development. Stay tuned!

Environment Setup

The engine requires the following dependencies:

pip install -r engine_extra_requirements.txt

Checkpoints Preparation

We recommend placing all checkpoints in the data_engine/weights directory.
The download_weights.sh script can be used to download all the required weights and place them in the correct directory following the instructions below:

cd data_engine
bash download_weights.sh

You can also download the weights manually and place them in the data_engine/weights directory.

Note: If you encounter infrared generation errors, please refer to #25. Deleting the .cache_tuner folder might help.
Installing xformers can help reduce GPU memory usage when generating infrared images.

The weight files should maintain the exact folder structure shown below for the program to locate them correctly:

Weight Files Structure

The directory structure should be like this:

weights/
├── stylebooth/
│ ├── step-210000/
│ └── stylebooth-tb-5000-0.bin
├── clip-vit-large-patch14/
│ ├── tokenizer.json
│ └── ...
├── depth_anything_v2/
│ └── depth_anything_v2_vitl.pth
├── dsine/
│ └── dsine.pt
├── paint_transformer/
│ └── model.pth
└── anime_to_sketch/
  └── improved.bin

Per-Modality Weight Download Instructions

Infrared Generation

The infrared generation code is based on scepter

Please download the weights from styleBooth weights, clip-vit-large-patch14.
And our style tuner is available for download from
The weight files structureGoogle Drive or Hugging Face.

NOTE: Generation a 1024x1024 image requires a GPU with about 12GB of memory.

Depth Generation

The depth generation code is based on Depth-Anything-V2

Please download the weights from Depth-Anything-V2-Large

Event Generation

The event generation module is a simple simulation implemented with basic code.

NOTE: Since this is a simulated process, no checkpoint is required.

Normal Generation

The normal generation code is based on DSINE

Please download weights

Sketch Generation

The sketch generation code is based on Anime2Sketch

Please download the weights

Paint Generation

The paint generation code is based on PaintTransformer

Please download weights

MINIMA Data Engine

To run the engine, you can use the following command:

cd data_engine
python modality_engine.py --modality <modality> --input_path <input_path> --output_dir <output_dir>
# --modality: Choose from [infrared, depth, event, normal, sketch, paint]
# --input_path: Supports both a single image or a directory that contains images

# Example
python modality_engine.py --modality infrared --input_path ./figs/origin_image.jpg --output_dir './result'

Acknowledgements

We sincerely appreciate the contributions from the open-source community.

Special thanks to:

Your support and feedback help improve MINIMA Data Engine. We welcome contributions and collaborations from the community!