Video2E

Video2E is a ModoForge subproject.

ModoForge is a creative space that brings together more than one hundred models for exploring and building AI-powered media workflows.

Video2E is a command-line frontend for PyTorch and ONNX Runtime video and still-image enhancement. It supports CPU, NVIDIA CUDA, and Apple Silicon through CoreML, while intentionally excluding NCNN and TensorRT.

Acknowledgements

Video2X provided early inspiration for Video2E.

Video2E maintains a reviewed and reduced source snapshot of selected PyTorch components from REAL-Video-Enhancer. We thank both projects, their maintainers, and contributors for their work.

Usage

Built-in models

The built-in model catalogue is grouped by capability below. Quality tiers are 🥉 baseline, 🥈 balanced, and 🥇 high. Run video2e models list --capability <capability> --params to see the same catalogue together with the download state for the configured model directory.

Upscaling

Model	Quality	Speed	Use case	Scaling factor	Profile	Real-CUGAN noise level
animejanai-3	🥈	very_fast	anime	2	—	—
animejanai-3.1	🥈	fast	anime	2	—	—
animesr-2	🥇	slow	anime	4	—	—
anisd	🥈	fast	anime	2	source_quality_high, source_quality_medium, source_quality_low	—
dat2-bhi	🥇	very_slow	live action	4	general, noisy, clean	—
nomos8k	🥈	fast	live action	4	weak, medium, strong	—
openproteus	🥈	very_fast	live action	2	—	—
realcugan-pro	🥇	medium	anime	2, 3	—	-1, 0, 3
realcugan-se	🥈	medium	anime	2, 3, 4	—	-1, 0, 1, 2, 3
realesrgan	🥉	very_fast	live action	4	—	—
realesrgan-anime	🥈	very_fast	anime	4	—	—
realesrgan-xplus	🥇	slow	live action	2, 4	—	—
realesrgan-xplus-anime	🥇	medium	anime	4	—	—
realplksr-anisd	🥇	slow	anime	2	—	—
realplksr-anitoon	🥈 🥇	medium, very_slow	anime	2	small_size, large_size	—
realplksr-bhi	🥇	slow	live action	4	—	—
realplksr-public	🥇	slow	live action	2	—	—

Restoration

Model	Quality	Speed	Use case
deh264-rtmosr	🥈	fast	anime, live action
deh264-span	🥈	fast	anime, live action
deh264-super-ultra-compact	🥉	very_fast	anime, live action
drunet-color	🥈	slow	anime, live action
realplksr-deh264	🥇	very_slow	live action

Frame interpolation

Model	Quality	Speed	Use case	Profile	Frame-rate multiplier	Scene threshold
gmfss	🥈	medium	anime, live action	—	2..8	0..100
gmfss-pro	🥇	slow	anime, live action	—	2..8	0..100
ifrnet-vimeo90k	🥉	fast	anime, live action	—	2..8	0..100
rife-4.25	🥈	medium	anime, live action	—	2..8	0..100
rife-4.26	🥇	medium, slow	anime, live action	standard, heavy	2..8	0..100

Local installation

Python 3.10 through 3.13 is supported. Install the profile for the current machine; make install selects Apple Silicon on an arm64 Mac and CPU elsewhere, while NVIDIA CUDA 12.8 is selected explicitly. The installer creates and reuses the project-local .venv instead of modifying the system Python environment:

export PATH="$HOME/.local/bin:$PATH"
# Choose one profile:
make install                 # Apple Silicon on arm64 macOS, CPU elsewhere
# make install PROFILE=cuda  # NVIDIA CUDA 12.8
video2e --help

Add ~/.local/bin to the shell startup configuration if it is not already on PATH. make install links the virtual-environment command into that directory and fails rather than reporting success when video2e would resolve elsewhere. Set BIN_DIR to select another writable directory already on PATH. Set PYTHON to choose the interpreter used to create the environment, or VENV_DIR to use another virtual-environment directory. Virtual-environment activation remains optional; .venv/bin/video2e can also be invoked directly.

Shell completion

video2e completion generates completion code for Bash, Zsh, or Fish. Load it for the current shell session with one of the following commands:

source <(video2e completion bash)

source <(video2e completion zsh)

video2e completion fish | source

To enable completion on startup, save the generated code in the shell's completion directory. For example:

video2e completion bash > ~/.local/share/bash-completion/completions/video2e

video2e completion zsh > "${fpath[1]}/_video2e"

video2e completion fish > ~/.config/fish/completions/video2e.fish

Video2E discovers model weights in this order:

1. VIDEO2E_MODEL_DIR, used directly.
2. MODOFORGE_MODELS_DIR, with /video2e appended.
3. ~/.video2e/models when that directory exists.
4. ~/.modoforge/models/video2e when that directory exists.

If neither conventional directory exists, ~/.video2e/models is used as the default location. An explicit --model-dir overrides the complete search. After placing the selected weight there, choose the command for the processing scene and a built-in model name or stable model id:

video2e upscale \
  -i input.mp4 \
  -o output.mp4 \
  --model openproteus \
  -s 2

--device defaults to auto: CUDA is selected when available, followed by CoreML for ONNX models on Apple Silicon, then CPU. Pass --device only to force a specific backend; use --device-index to select a non-default CUDA GPU.

Progress integration

Media-processing commands and video2e models pull accept --progress-format human|jsonl|none. Human-readable progress is the default. Applications should request jsonl and read standard output one line at a time. Each event starts with the literal VIDEO2E_PROGRESS followed by a tab and one compact JSON object. Events include a schema version, monotonic sequence number, operation, phase and status, with optional completed/total units, phase progress, FPS and ETA. Other output remains ordinary diagnostic text and should not be parsed as progress. Consumers should ignore unknown fields and stale sequence numbers so future schema additions remain safe.

Container image

Prebuilt CUDA and CPU images are published on GHCR:

docker pull ghcr.io/modoforge/video2e:cuda
docker pull ghcr.io/modoforge/video2e:cpu

Release builds pushed from a Git tag also publish immutable profile-specific tags such as v0.1.0-cuda and v0.1.0-cpu. The floating cuda and cpu tags track the latest tagged release for each runtime profile. Non-release builds use the nearest Git tag, or v0.0.0 when none exists, plus the commit short SHA and profile.

Model weights are not bundled. Mount the model and work directories when running the CUDA image:

docker run --rm --gpus all \
  -v "$PWD/models:/models:ro" \
  -v "$PWD/work:/work" \
  ghcr.io/modoforge/video2e:cuda \
  upscale \
  -i /work/input.mp4 -o /work/output.mp4 \
  --model openproteus -s 2

Use video2e models list to inspect available models, video2e devices to inspect inference devices, and video2e <command> --help for all options.

Supported capabilities and runtimes

Capability	Runtime	Input	Devices
Upscaling	PyTorch and ONNX Runtime	Video and still images	CPU, NVIDIA CUDA, Apple CoreML for ONNX
Restoration	PyTorch	Video and still images	CPU and NVIDIA CUDA
Frame interpolation	PyTorch	Video	CPU and NVIDIA CUDA

Model architecture selection is handled internally after resolving the model name or stable id. The model catalogue is the authoritative source for each model's runtime, supported parameters, scale, intended use, and license data.

Run video2e models list to scan the model directory and report model names, capabilities and download state in a readable table. Add --params to include supported model parameters and their accepted values; the table does not mark default values. Partially downloaded parameter variants are shown as partial (N/M). Use --capability upscale, --capability restore, or --capability interpolate to filter the catalogue. A non-empty expected model file is considered downloaded. Add --json to emit the complete machine-readable catalogue, including stable variants, paths, relative quality and speed tiers, intended video types, input recommendations, model risks, weight source URLs, weight licenses, license evidence URLs, license risks, and observed weight update times. A weight update time is an observed release asset or distribution-mirror timestamp; it is not a training date. License metadata is evidence-oriented and conservative: a code repository license is not treated as a weight license unless the upstream wording covers the project, released models, or linked model contents. unknown means no weight-scoped grant was confirmed, not that a weight is free to use. This metadata is not legal advice and does not replace review of the upstream terms. Run video2e devices to inspect available inference devices; that command emits structured JSON on standard output. See the model support plan for the current 48-model inventory, upstream status, licensing policy, and phased roadmap for new restoration, denoising, upscaling, interpolation, and temporal models.

Development installation

Python 3.10 through 3.13 is supported. make install selects the Apple Silicon profile on an arm64 Mac and the CPU profile elsewhere. Select CUDA or another profile explicitly when needed:

export PATH="$HOME/.local/bin:$PATH"
# Choose one profile:
make install                 # Apple Silicon on arm64 macOS, CPU elsewhere
# make install PROFILE=cuda  # NVIDIA CUDA 12.8

The installation is editable. The profile-specific scripts remain available for direct use. Each script creates and reuses .venv by default:

cd video2e
./scripts/install-cpu.sh

cd video2e
./scripts/install-cuda.sh

cd video2e
./scripts/install-macos.sh

The CPU and Apple Silicon profiles install onnxruntime. The NVIDIA profile pins onnxruntime-gpu below 1.27 because this project's CUDA 12.8 base is not compatible with the CUDA 13 default introduced by ONNX Runtime 1.27. Do not install both ONNX Runtime packages in the same environment.

The required REAL-Video-Enhancer PyTorch components are maintained as a curated source snapshot under src/video2e/vendor/rve. No external RVE checkout is required at build time or runtime.

This snapshot is intentional. REAL-Video-Enhancer does not currently expose a versioned backend library API, so depending on its internal src.* modules would make Video2E releases sensitive to upstream directory and argument changes. The vendored code is pinned in NOTICE.md, reduced to the supported architectures, and can be updated as an explicit reviewed change.

Model usage guide

--scaling-factor selects a native model weight. It may be omitted when the model name has exactly one scale; names with multiple scales require an explicit value. Enum selectors such as --profile follow the same rule: a single value is selected automatically, while multiple values require an explicit choice. --output-scale independently controls the final dimensions relative to the input. When it differs from --scaling-factor, the selected model first runs at its native scale, then the result is resized to the requested output scale; video is encoded by FFmpeg after that resize. For example, -s 4 --output-scale 2 runs the native 4× model and produces a final 2× output. Stable model ids remain available for reproducible low-level selection, but model selectors apply only to names.

Restoration uses its own command and accepts only restoration models:

video2e restore \
  -i input.mp4 \
  -o restored.mp4 \
  --model deh264-span

Still images

Still-image mode is selected automatically from the input extension or image content. It bypasses video probing, frame-rate, audio, subtitle, and video encoding logic while reusing the same PyTorch/ONNX model, device, scale, and tile implementation:

video2e upscale \
  -i input.png \
  -o output.png \
  --model realcugan-pro \
  -s 2 \
  --realcugan-noise-level 0

Supported still-image input and output formats are PNG, JPEG, WebP, BMP, and TIFF. PNG, WebP, and TIFF outputs preserve an input alpha channel; JPEG and BMP flatten transparency onto white. EXIF orientation is applied before inference, and ICC profiles are retained when the output encoder supports them. Animated images are rejected rather than silently processing only their first frame.

Image mode accepts upscale and restore models. Interpolation models and the video-only --frame-rate-mul, --start-time, and --end-time options are rejected. The output filename must use a supported image extension.

Interpolation uses the Video2X-compatible frame-rate multiplier:

video2e interpolate \
  -i input.mp4 \
  -o output.mp4 \
  --model gmfss-pro \
  --device cuda \
  -m 2

RIFE 4.26 is the general default candidate, RIFE 4.25 is retained as the prior standard baseline, and Heavy trades speed and memory for a larger feature encoder aimed at difficult animation motion. Download rife4.25.pkl, rife4.26.pkl, or rife4.26.heavy.pkl from the REAL-Video-Enhancer model release into the model directory, then select the matching stable id:

video2e interpolate -i input.mp4 -o output.mp4 \
  --model rife-4.25 --device cuda -m 2

video2e interpolate -i input.mp4 -o output.mp4 \
  --model rife-4.26 --profile standard --device cuda -m 2

video2e interpolate -i input.mp4 -o output.mp4 \
  --model rife-4.26 --profile heavy --device cuda -m 2

The five built-in Real-ESRGAN ids use official PyTorch weights. Download the files from the matching official Real-ESRGAN releases into the model directory:

Model id	Architecture	Official weight	Native scale
realesr-general-x4v3	Compact	realesr-general-x4v3.pth	4×
realesr-animevideov3-4x	Compact	realesr-animevideov3.pth	4×
realesrgan-x2plus	ESRGAN	RealESRGAN_x2plus.pth	2×
realesrgan-x4plus	ESRGAN	RealESRGAN_x4plus.pth	4×
realesrgan-x4plus-anime-6b	ESRGAN	RealESRGAN_x4plus_anime_6B.pth	4×

The Real-ESRGAN project publishes these official weights within its BSD-3-Clause-licensed project. Video2E records the five weights as BSD-3-Clause with project-scope evidence; redistribution must preserve the required copyright notice, license conditions, and disclaimer.

For example:

video2e upscale -i input.mp4 -o output.mp4 \
  --model realesrgan-general \
  --model-dir models --device cuda --tile-size 256 -s 4

video2e upscale -i input.mp4 -o output.mp4 \
  --model realesrgan-xplus \
  --model-dir models --device cuda --tile-size 256 -s 2

The General and AnimeVideo v3 models reuse the lightweight Compact loader. The x2plus, x4plus, and Anime 6B models use the vendored ESRGAN/RRDB loader. PyTorch CPU runs in float32; NVIDIA CUDA defaults to float16. Apple Silicon currently runs these PyTorch weights on CPU because the CLI does not expose MPS.

The five built-in RealPLKSR ids use the following upstream weights. The source, file size, SHA-256, and license lock is maintained in realplksr-model-sources.json.

Model id	Official weight	Native scale	Use	License
public-realplksr-2x	2xPublic_realplksr_dysample_layernorm_real.safetensors	2×	Real-world mixed degradation	Apache-2.0
bhi-realplksr-real-4x	4xBHI_realplksr_dysample_real.safetensors	4×	Real photography degradation	CC BY 4.0
anisd-realplksr-medium-2x	2x_AniSD_AC_RealPLKSR_127500.pth	2×	Medium/heavy WEB and DVD anime degradation	CC BY-NC 4.0
anitoon-small-2x	2x_AniToon_RPLKSRS_242500.pth	2×	Faster old cartoon/anime restoration	CC BY-NC 4.0
anitoon-large-2x	2x_AniToon_RPLKSRL_280K.pth	2×	High-capacity old cartoon/anime restoration	CC BY-NC 4.0

Download the selected file into the model directory, then run for example:

video2e upscale -i input.mp4 -o output.mp4 \
  --model realplksr-public \
  --model-dir models --device cuda --tile-size 256 -s 2

The Public and BHI files use the upstream safetensors assets. AniSD and AniToon are only published as PyTorch .pth files. AniSD and AniToon are covered here by the publisher repository's CC BY-NC 4.0 project license, rather than a separate per-asset license statement, and must not be selected for commercial output without obtaining additional rights. The Public model's LayerNorm variant supports CUDA float16; the other four models are automatically run in float32 because their GroupNorm variants do not declare safe FP16 support.

The dat2-bhi family exposes three 4× DAT-2 weights from Philip Hofmann. Exact sources, byte sizes, SHA-256 values, and CC BY 4.0 evidence are maintained in models/dat2-model-sources.json.

Profile	Official weight	Recommended input
general	4xBHI_dat2_real.safetensors	General real-world input with realistic noise, blur, JPEG, or WebP recompression
noisy	4xBHI_dat2_otf.safetensors	Input with noticeable noise and compression damage
clean	4xBHI_dat2_otf_nn.safetensors	Clean or already-denoised input with resize and JPEG damage

Download all three profiles with video2e models pull dat2-bhi, or run a selected profile directly after downloading it:

video2e upscale -i input.mp4 -o output.mp4 \
  --model dat2-bhi --profile general --scaling-factor 4 \
  --model-dir models --device cuda --tile-size 128

The selectively vendored Spandrel DAT loader detects these weights as the 11,212,131-parameter DAT-2 variant. DAT currently declares no safe float16 path and therefore runs in float32. All three profiles support inference batch sizes 1 and 2; start with batch size 1 and a conservative tile size when VRAM is limited. All three weights are single-frame models and can produce temporal texture flicker on video.

The 17 built-in Real-CUGAN ids cover every official SE and Pro PyTorch weight in the official updated_weights and weights_pro folders. Exact Google Drive file ids, byte sizes, and SHA-256 values are locked in realcugan-model-sources.json.

Family	Native scale	Model id suffixes	Official files
SE	2×	conservative, no-denoise, denoise1, denoise2, denoise3	up2x-latest-*.pth
SE	3×	conservative, no-denoise, denoise3	up3x-latest-*.pth
SE	4×	conservative, no-denoise, denoise3	up4x-latest-*.pth
Pro	2×	conservative, no-denoise, denoise3	pro-*-up2x.pth
Pro	3×	conservative, no-denoise, denoise3	pro-*-up3x.pth

Each denoise strength is a separate weight and stable model id; it is not a runtime noise parameter. conservative is the restrained restoration choice, no-denoise is for clean sources, and denoise1 through denoise3 increase denoising strength. Pro has a distinct input/output range and is detected from the official weight marker. There is no official Pro 4× weight in this set. The separate NCNN models-nose architecture is not supported.

Download the selected official file into the model directory, then run for example:

video2e upscale -i input.mp4 -o output.mp4 \
  --model realcugan-pro --scaling-factor 2 --realcugan-noise-level 0 \
  --model-dir models --device cuda --tile-size 256

PyTorch CPU uses FP32 and NVIDIA CUDA supports FP16/FP32. Apple Silicon runs these weights on CPU because the CLI does not expose MPS. Real-CUGAN generic tile mode enforces a 40-pixel overlap to cover its wider model boundary, but SE statistics are still calculated independently per tile. It is therefore not the upstream cache-based seamless algorithm; use --tile-size 0 when full-frame equivalence is required. The upstream alpha default remains 1. The official weight distribution does not state a weight license, so users must verify intended use and redistribution rights independently.

PyTorch CPU always uses float32, while PyTorch CUDA defaults to float16. ONNX precision is defined by the model itself: auto accepts the declared input type, while an explicit value must match it. --device auto selects CUDA first, then Apple CoreML on Apple Silicon, and otherwise CPU.

AnimeJaNai V3.1 Balanced SPANF3 is an ONNX FP16 model. Download the official 2x_AnimeJaNai_HD_V3.1_Balanced_SPANF3_b8f64_unshuffle_fp16.onnx file into the model directory, then run:

video2e upscale -i input.mp4 -o output.mp4 \
  --model animejanai-3.1 \
  --device auto --tile-size 256

On an Apple Silicon Mac, use --device coreml to require CoreML. The upstream release is distributed from a repository licensed CC BY-NC-SA 4.0; no separate per-weight license statement was found. Treat it as non-commercial unless the rightsholder grants additional rights.

Select a Video2E encoder preset; each preset resolves its FFmpeg codec and quality-control option:

video2e upscale -i input.mp4 -o output.mp4 \
  --model nomos8k --profile medium \
  -c x264_nvenc --crf 18 -e preset=p7

Parameters

Parameters are exposed only by the commands that support them. Use video2e <command> --help for the authoritative command-specific view.

Video inputs can group independent frames into one model invocation with --inference-batch-size. Supported values are 1, 2, 4, and 8; the default is 1, and the final batch may contain fewer frames. The same option applies to full-frame inference and --tile-size inference: in tile mode each spatial tile is evaluated for the whole frame batch. Each built-in model declares its accepted batch sizes in the model catalog, so unsupported values are rejected before loading the inference runtime. Stateful AnimeSR, interpolation models, still-image inputs, and the built-in AnimeJaNai ONNX model accept only batch size 1. Runtime shape validation remains a consistency check for ONNX files. Video2E does not retry CUDA or provider OOM failures with a smaller batch; choose a lower value explicitly.

For example:

video2e upscale -i input.mp4 -o output.mp4 \
  --model realesrgan-xplus --scaling-factor 2 \
  --device cuda --inference-batch-size 4 --tile-size 256

Choose --tile-size according to the available memory:

Environment	Recommended value
CPU	128 or 256
4-6 GB VRAM	128 or 256
8-12 GB VRAM	256 or 512
16 GB or more VRAM	512 or 0
CUDA out of memory	Halve the current value.

Start with 0 when memory is sufficient. Tiling is intended for single-frame upscaling and restoration models; it does not reduce the temporal model memory used by gmfss, ifrnet, or rife.

Build

The binary distribution is a PyInstaller onedir application because PyTorch and CUDA consist of large dynamic libraries:

make package

The result is written to dist/video2e/. CPU and CUDA distributions should be built in separate virtual environments using the corresponding PyTorch wheel. Set BINARY_NAME to override the distribution name.

Docker

The default image uses the shared CUDA and FFmpeg base:

make image

Run it with the NVIDIA Container Toolkit and mount model and work directories:

docker run --rm --gpus all \
  -v "$PWD/models:/models:ro" \
  -v "$PWD/work:/work" \
  ghcr.io/modoforge/video2e:cuda \
  upscale \
  -i /work/input.mp4 -o /work/output.mp4 \
  --model openproteus -s 2

The same Dockerfile can build the CPU profile:

make image-cpu

License

Video2E is free and open-source software licensed under the GNU Affero General Public License v3.0 or later.

Third-party components included in the source tree remain subject to their respective licenses and attribution requirements. See NOTICE.md for details.