Skip to content

Commit b0381aa

Browse files
LeoXing1996LeoXing1996
committed
upload code 🎉
1 parent 845fa91 commit b0381aa

File tree

100 files changed

+15327
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

100 files changed

+15327
-1
lines changed

‎.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "live2diff/MiDaS"]
2+
path = live2diff/MiDaS
3+
url = [email protected]:lewiji/MiDaS.git

‎.pre-commit-config.yaml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
repos:
2+
- repo: https://github.com/astral-sh/ruff-pre-commit
3+
# Ruff version.
4+
rev: v0.3.5
5+
hooks:
6+
# Run the linter.
7+
- id: ruff
8+
args: [ --fix ]
9+
# Run the formatter.
10+
- id: ruff-format
11+
- repo: https://github.com/codespell-project/codespell
12+
rev: v2.2.1
13+
hooks:
14+
- id: codespell
15+
args: ["-L", "warmup,mose,parms", "--skip", "*.json"]
16+
- repo: https://github.com/pre-commit/pre-commit-hooks
17+
rev: v4.3.0
18+
hooks:
19+
- id: trailing-whitespace
20+
- id: check-yaml
21+
- id: end-of-file-fixer
22+
- id: requirements-txt-fixer
23+
- id: fix-encoding-pragma
24+
args: ["--remove"]
25+
- id: mixed-line-ending
26+
args: ["--fix=lf"]

‎README.md

Lines changed: 139 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,13 @@
1010
[![arXiv](https://img.shields.io/badge/arXiv-2407.08701-b31b1b.svg)](https://arxiv.org/abs/2407.08701)
1111
[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://live2diff.github.io/)
1212

13-
## Code will be release in **one week**, stay tuned!
13+
## Release
14+
15+
* [2024/07/18] We release [HuggingFace space](https://huggingface.co/spaces/Leoxing/Live2Diff), code, and [checkpoints](https://huggingface.co/Leoxing/Live2Diff).
16+
17+
## TODO List
18+
19+
- [ ] Support Colab
1420

1521
## Key Features
1622

@@ -33,8 +39,140 @@ The speed evaluation is conducted on **Ubuntu 20.04.6 LTS** and **Pytorch 2.2.2*
3339
| 768 x 512 | **On** | **12.15** |
3440
| 768 x 512 | Off | 6.29 |
3541

42+
## Installation
43+
44+
### Step0: clone this repository and submodule
45+
46+
```bash
47+
git clone https://github.com/open-mmlab/Live2Diff.git
48+
# or vis ssh
49+
git clone [email protected]:open-mmlab/Live2Diff.git
50+
51+
cd Live2Diff
52+
git submodule update --init --recursive
53+
```
54+
55+
### Step1: Make Environment
56+
57+
Create virtual envrionment via conda:
58+
59+
```bash
60+
conda create -n live2diff python=3.10
61+
conda activate live2diff
62+
```
63+
64+
### Step2: Install PyTorch and xformers
65+
66+
Select the appropriate version for your system.
67+
68+
```bash
69+
# CUDA 11.8
70+
pip install torch torchvision xformers --index-url https://download.pytorch.org/whl/cu118
71+
# CUDA 12.1
72+
pip install torch torchvision xformers --index-url https://download.pytorch.org/whl/cu121
73+
```
74+
75+
Please may refers to https://pytorch.org/ for more detail.
76+
77+
### Step3: Install Project
78+
79+
If you want to use TensorRT acceleration (we recommend it), you can install it by the following command.
80+
81+
```bash
82+
# for cuda 11.x
83+
pip install ."[tensorrt_cu11]"
84+
# for cuda 12.x
85+
pip install ."[tensorrt_cu12]"
86+
```
87+
88+
Otherwise, you can install it via
89+
90+
```bash
91+
pip install .
92+
```
93+
94+
If you want to install it with development mode (a.k.a. "Editable Installs"), you can add `-e` option.
95+
96+
```bash
97+
# for cuda 11.x
98+
pip install -e ."[tensorrt_cu11]"
99+
# for cuda 12.x
100+
pip install -e ."[tensorrt_cu12]"
101+
# or
102+
pip install -e .
103+
```
104+
105+
### Step4: Download Checkpoints and Demo Data
106+
107+
1. Download StableDiffusion-v1-5
108+
109+
```bash
110+
huggingface-cli download runwayml/stable-diffusion-v1-5 --local-dir ./models/Model/stable-diffusion-v1-5
111+
```
112+
113+
2. Download Checkpoint from [HuggingFace](https://huggingface.co/Leoxing/Live2Diff) and put it under `models` folder.
114+
115+
3. Download Depth Detector from MiDaS's official [release](https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt) and put it under `models` folder.
116+
117+
4. Apply the download token from [civitAI](https://education.civitai.com/civitais-guide-to-downloading-via-api/) and then download Dreambooths and LoRAs via the script:
118+
119+
```bash
120+
# download all DreamBooth/Lora
121+
bash scripts/download.sh all YOUR_TOKEN
122+
# or download the one you want to use
123+
bash scripts/download.sh disney YOUR_TOKEN
124+
```
125+
126+
5. Download demo data from [OneDrive](https://pjlab-my.sharepoint.cn/:f:/g/personal/xingzhening_pjlab_org_cn/EpefezlxFXNBk93RDttYLMUBP2bofb6AZDfyRIkGapmIrQ?e=A6h2Eb).
127+
128+
Then then data structure of `models` folder should be like this:
129+
130+
```bash
131+
./
132+
|-- models
133+
| |-- LoRA
134+
| | |-- MoXinV1.safetensors
135+
| | `-- ...
136+
| |-- Model
137+
| | |-- 3Guofeng3_v34.safetensors
138+
| | |-- ...
139+
| | `-- stable-diffusion-v1-5
140+
| |-- live2diff.ckpt
141+
| `-- dpt_hybrid_384.pt
142+
`--data
143+
|-- 1.mp4
144+
|-- 2.mp4
145+
|-- 3.mp4
146+
`-- 4.mp4
147+
```
148+
149+
### Notification
150+
151+
The above installation steps (e.g. [download script](#step4-download-checkpoints-and-demo-data)) are for Linux users and not well tested on Windows. If you face any difficulties, please feel free to open an issue 🤗.
152+
153+
## Quick Start
154+
155+
You can try examples under [`data`](./data) directory. For example,
156+
```bash
157+
# with TensorRT acceleration, please pay patience for the first time, may take more than 20 minutes
158+
python test.py ./data/1.mp4 ./configs/disneyPixar.yaml --max-frames -1 --prompt "1man is talking" --output work_dirs/1-disneyPixar.mp4 --height 512 --width 512 --acceleration tensorrt
159+
160+
# without TensorRT acceleration
161+
python test.py ./data/2.mp4 ./configs/disneyPixar.yaml --max-frames -1 --prompt "1man is talking" --output work_dirs/1-disneyPixar.mp4 --height 512 --width 512 --acceleration none
162+
```
163+
164+
You can adjust denoising strength via `--num-inference-steps`, `--strength`, and `--t-index-list`. Please refers to `test.py` for more detail.
165+
166+
## Troubleshooting
167+
168+
1. If you face Cuda Out-of-memory error with TensorRT, please try to reduce `t-index-list` or `strength`. When inference with TensorRT, we maintian a group of buffer for kv-cache, which consumes more memory. Reduce `t-index-list` or `strength` can reduce the size of kv-cache and save more GPU memory.
169+
36170
## Real-Time Video2Video Demo
37171

172+
There is an interactive txt2img demo in [`demo`](./demo) directory!
173+
174+
Please refers to [`demo/README.md`](./demo/README.md) for more details.
175+
38176
<div align="center">
39177
<table align="center">
40178
<tbody>

‎configs/base_config.yaml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
pretrained_model_path: "./models/Model/stable-diffusion-v1-5"
2+
3+
motion_module_path: './models/live2diff.ckpt'
4+
depth_model_path: './models/dpt_hybrid_384.pt'
5+
6+
unet_additional_kwargs:
7+
cond_mapping: true
8+
use_inflated_groupnorm: true
9+
use_motion_module : true
10+
motion_module_resolutions : [ 1,2,4,8 ]
11+
unet_use_cross_frame_attention : false
12+
unet_use_temporal_attention : false
13+
14+
motion_module_type: Streaming
15+
motion_module_kwargs:
16+
num_attention_heads : 8
17+
num_transformer_block : 1
18+
attention_block_types : [ "Temporal_Self", "Temporal_Self" ]
19+
temporal_position_encoding : true
20+
temporal_position_encoding_max_len : 24
21+
temporal_attention_dim_div : 1
22+
zero_initialize : true
23+
24+
attention_class_name : 'stream'
25+
26+
attention_kwargs:
27+
window_size: 16
28+
sink_size: 8
29+
30+
noise_scheduler_kwargs:
31+
num_train_timesteps: 1000
32+
beta_start: 0.00085
33+
beta_end: 0.012
34+
beta_schedule: "linear"
35+
steps_offset: 1
36+
clip_sample: False

‎configs/disneyPixar.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# good s0.4
2+
base: "./configs/base_config.yaml"
3+
4+
prompt_template: "masterpiece, best quality, intricate, print, pattern, {}"
5+
6+
third_party_dict:
7+
dreambooth: "./models/Model/disneyPixarCartoon_v10.safetensors"
8+
clip_skip: 2
9+
10+
num_inference_steps: 50
11+
t_index_list: [30, 36, 42]

‎configs/kFelted.yaml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# good, s06-4 and s05-4
2+
base: "./configs/base_config.yaml"
3+
4+
prompt_template: "masterpiece, best quality, felted, {},"
5+
6+
third_party_dict:
7+
dreambooth: "./models/Model/revAnimated_v2RebirthVAE.safetensors"
8+
lora_list:
9+
- lora: './models/LoRA/kFeltedReV.safetensors'
10+
lora_alpha: 1
11+
clip_skip: 2
12+
13+
num_inference_steps: 50
14+
t_index_list: [25, 34, 43]
15+
# or
16+
# num_inference_steps: 50
17+
# t_index_list: [20, 27, 34, 41]

‎configs/moxin.yaml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
base: "./configs/base_config.yaml"
2+
3+
prompt_template: 'shukezouma, negative space, shuimobysim, official art,extremely detailed CG,unity 8k wallpaper,chinese ink painting, {}'
4+
5+
third_party_dict:
6+
lora_list:
7+
- lora: "./models/LoRA/MoXinV1.safetensors"
8+
lora_alpha: 0.7
9+
dreambooth: "./models/Model/3Guofeng3_v34.safetensors"
10+
clip_skip: 2
11+
12+
num_inference_steps: 50
13+
t_index_list: [30, 36, 42]

‎configs/origami.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# good, s04!!!!
2+
3+
base: "./configs/base_config.yaml"
4+
5+
prompt_template: "(masterpiece),best quality, a origami paper of {}"
6+
7+
third_party_dict:
8+
dreambooth: "./models/Model/helloartdoor_V122p.safetensors"
9+
lora_list:
10+
- lora: "./models/LoRA/ral-origami-sd15.safetensors"
11+
lora_alpha: 1
12+
clip_skip: 2
13+
14+
num_inference_steps: 50
15+
t_index_list: [30, 36, 42]

‎configs/pixart.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# good, 0.4 & 0.3
2+
3+
base: "./configs/base_config.yaml"
4+
5+
prompt_template: "(masterpiece), best quality, {}"
6+
7+
third_party_dict:
8+
dreambooth: "./models/Model/aziibpixelmix_v10.safetensors"
9+
clip_skip: 2
10+
11+
num_inference_steps: 4
12+
strength: 0.6

‎configs/toonyou.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
base: "./configs/base_config.yaml"
2+
3+
prompt: "masterpiece, best quality, intricate, print, pattern, {}"
4+
5+
third_party_dict:
6+
dreambooth: "./models/Model/toonyou_beta6.safetensors"
7+
clip_skip: 2
8+
9+
num_inference_steps: 50
10+
t_index_list: [25, 31, 37, 43]

‎demo/.gitattributes

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
*.7z filter=lfs diff=lfs merge=lfs -text
2+
*.arrow filter=lfs diff=lfs merge=lfs -text
3+
*.bin filter=lfs diff=lfs merge=lfs -text
4+
*.bz2 filter=lfs diff=lfs merge=lfs -text
5+
*.ckpt filter=lfs diff=lfs merge=lfs -text
6+
*.ftz filter=lfs diff=lfs merge=lfs -text
7+
*.gz filter=lfs diff=lfs merge=lfs -text
8+
*.h5 filter=lfs diff=lfs merge=lfs -text
9+
*.joblib filter=lfs diff=lfs merge=lfs -text
10+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
11+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
12+
*.model filter=lfs diff=lfs merge=lfs -text
13+
*.msgpack filter=lfs diff=lfs merge=lfs -text
14+
*.npy filter=lfs diff=lfs merge=lfs -text
15+
*.npz filter=lfs diff=lfs merge=lfs -text
16+
*.onnx filter=lfs diff=lfs merge=lfs -text
17+
*.ot filter=lfs diff=lfs merge=lfs -text
18+
*.parquet filter=lfs diff=lfs merge=lfs -text
19+
*.pb filter=lfs diff=lfs merge=lfs -text
20+
*.pickle filter=lfs diff=lfs merge=lfs -text
21+
*.pkl filter=lfs diff=lfs merge=lfs -text
22+
*.pt filter=lfs diff=lfs merge=lfs -text
23+
*.pth filter=lfs diff=lfs merge=lfs -text
24+
*.rar filter=lfs diff=lfs merge=lfs -text
25+
*.safetensors filter=lfs diff=lfs merge=lfs -text
26+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27+
*.tar.* filter=lfs diff=lfs merge=lfs -text
28+
*.tar filter=lfs diff=lfs merge=lfs -text
29+
*.tflite filter=lfs diff=lfs merge=lfs -text
30+
*.tgz filter=lfs diff=lfs merge=lfs -text
31+
*.wasm filter=lfs diff=lfs merge=lfs -text
32+
*.xz filter=lfs diff=lfs merge=lfs -text
33+
*.zip filter=lfs diff=lfs merge=lfs -text
34+
*.zst filter=lfs diff=lfs merge=lfs -text
35+
*tfevents* filter=lfs diff=lfs merge=lfs -text

‎demo/.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
__pycache__/
2+
venv/
3+
public/
4+
*.pem
5+
!lib/
6+
!static/

0 commit comments

Comments
 (0)