Skip to content

feat: keep 4:4:4 video sharp when importing - #4333

Open
richiemcilroy wants to merge 23 commits into
mainfrom
feat/import-preserve-source
Open

richiemcilroy wants to merge 23 commits into
mainfrom
feat/import-preserve-source

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Importing a 4:4:4 screen capture re-encoded it to 4:2:0, which smeared coloured text in the editor. The import now also keeps the source's H.264 4:4:4 video next to the re-encode, and the paused editor view reads it, so coloured text is sharp when you stop on a frame. Scrubbing, playback, seeking and export read the same re-encode as before, so export output and speed are exactly main's.

Source frame, re-encoded import (before), kept import (after)
flowchart LR
    S[source] --> T[transcode_video<br/>unchanged] --> D[display.mp4]
    S --> P{plan_keep<br/>packet scan, no decode}
    P -- "H.264 4:4:4, avc1, limited range,<br/>no rotation or SAR, even size,<br/>keyframes at most 4.5 s apart, at most 1080p60" --> K[copy video packets<br/>and decode each one to check] --> DS[display-source.mp4]
    P -- anything else --> N[nothing kept]
    D --> E1[scrub, playback, seek, export]
    DS --> E2[paused editor frame]
Loading

Import

  • import_video (apps/desktop-gpui/src/import.rs) runs the existing transcode_video unchanged, then keep_source_video writes display-source.mp4 next to display.mp4 when the source qualifies. A source that does not qualify, or a copy that fails its check, leaves the project exactly as main writes it.
  • Eligibility comes from the stream and a packet scan, not the extension: H.264 in MP4/MOV with an avc1 sample entry, yuv444p limited range, BT.601/709 tags only, progressive, even dimensions, square pixels, no display matrix, a keyframe first, strictly ordered timestamps, no discarded packets.
  • The kept file is video only. Timestamps are shifted so the first frame is at zero, as the re-encode does. An untagged source gets the tag the re-encode writes (BT.709 for HD, BT.601 for SD), in the container (colr) too, so both files show the same colours.
  • Editor clip staging keeps display-source.mp4 along with display.mp4 and audio.ogg.

Paused editor frame

  • RecordingSegmentDecoders finds display-source.mp4 next to a display.mp4 and opens a decoder for it the first time a paused frame needs it. Only FrameFetch::Still reads it; playback, scrubbing and export keep reading display.mp4.
  • The editor preview renders every request from display.mp4 as before. 150 ms after the last paused frame, and when playback stops, it renders that frame again from the kept file, both sides of a transition included. Any new request or playback start cancels it. A new ForgetLastFrame renderer message makes the display layer upload that frame even though its recording time matches the one on screen.
  • cap_video_decode::ffmpeg::chroma_is_420 reads the pixel format, or chroma_format_idc from avcC and chromaFormat from hvcC. Hardware decoders, AVAssetReader and Media Foundation output 4:2:0, so a full-chroma file goes to FFmpeg software decode.
  • FrameConverter converts limited-range yuv444p with a row-parallel fixed-point converter (cpu_yuv::yuv444p_to_rgba_with_matrix). Other full-chroma formats use swscale BILINEAR, because FAST_BILINEAR halves chroma (35.8 dB vs 53.9 dB on the fixture). Every other format keeps its existing path.

Measurements

Main at 93622e4 against this branch, on the same runners.

Paused editor frame on macOS, coloured text region against the source frame, at the window's 1248×702 preview size:

Before After
1920×1080 4:4:4 text fixture 31.4 dB 37.3 dB
8 s 1080p60 4:4:4 test clip 38.4 dB 47.1 dB

Editor latency bench on macOS, three rounds per side on two 4:4:4 imports:

  • Scrub input to paint p50 is 17.5 to 18.5 ms on both sides, and fine scrub p95 is 20.1 to 20.5 ms on both.
  • Playback presents at the same rate within run-to-run spread.
  • The one extra paint is the sharpened frame. When playback reaches the end or a scrub stops, the same frame number is drawn again from the kept file about 150 ms later. The bench counts it as one repeated frame and one long interval.

Unchanged:

  • Export output: for 4:4:4 imports, every export of a kept import decodes to the same frames as main's export of the same source (27 exports per fixture, three fixtures). On Linux the files are byte-identical. On macOS main's own repeated exports already differ in bytes, and the decoded frames match.
  • Export time, 9 interleaved rounds, branch over main median: macOS 1.000, 1.000, 1.000; Linux 1.015, 1.039, 1.051, against 1.015, 1.020, 1.040 for main's own export path run with this branch's tools.
  • Scrubbing, playback and cold seeks read display.mp4, which decodes to the same frames as main's on macOS, Linux and Windows. macOS cold seek p50 / max on the text fixture: 22.7 / 27.4 ms before, 21.7 / 26.2 ms after.
  • 4:2:0 and HEVC imports write the same files as main and decode identically. Projects imported on main render and export byte-identically with this branch.
  • An untagged 1080p 4:4:4 file of eight flat colour swatches renders the same colours from the kept file as from the re-encode.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

Keeps eligible 4:4:4 source video for sharper paused previews while retaining the re-encoded video for playback and export.

  • Qualifying 4:4:4 imports keep a copy of the source video.
  • The editor redraws settled frames from the kept source video.
  • Full-chroma frames keep their color detail through decoding and conversion.

The earlier color, transition, cancellation, and comment concerns are addressed. No new actionable issues were found.

Reviews (14) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

Hardware and platform decoders hand frames back as 4:2:0. Video with fuller
chroma now decodes in software on every platform, AVAssetReader and Media
Foundation decline it at init so the FFmpeg fallback takes it, and the RGBA
conversion keeps every chroma sample.
A re-encode subsamples it to 4:2:0, which blurs coloured text. The import now
copies the source packets into display.mp4 when every editor decoder reads
them as they are, and re-encodes exactly as before otherwise.
@sandchest-com

sandchest-com Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 789b360: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 2 credits · Review details

@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 19, 2026 3:34am UTC

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 2 other findings in Capy.

Open in Capy

Comment thread apps/desktop-gpui/src/import.rs Resolved
Comment thread apps/desktop-gpui/src/import.rs Resolved
Comment thread apps/desktop-gpui/src/import.rs Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 18, 2026 00:28 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 18, 2026 02:18 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

…source

# Conflicts:
#	crates/rendering/src/project_recordings.rs

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 high).

View 1 other finding in Capy.

Open in Capy

Comment thread crates/editor/src/editor_instance.rs Resolved

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

The display layer reuses its texture for a frame at the same recording time, so the settled frame decoded from the kept source never replaced the one already shown. The renderer now forgets its last frame before a settled render. At the end of playback the settled frame is scheduled directly instead of first redrawing the same frame from display.mp4.
…source

# Conflicts:
#	crates/editor/src/editor_instance.rs

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 3 other findings in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

Comment thread crates/editor/src/editor_instance.rs Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

Comment thread crates/editor/src/editor_instance.rs Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 1 other finding in Capy.

Open in Capy

Comment thread crates/rendering/src/lib.rs Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 19, 2026 01:29 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai, please re-review the PR

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 19, 2026 03:20 Inactive

This branch was successfully deployed

1 active and 1 inactive (outdated) deployments
Preview – cap-web — 789b3602 Deployed Nov 19, 2026 by vercel[bot]
Preview – cap-status — 6d8ab69b Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant