feat: add auto-follow zoom mode with cursor tracking#257
feat: add auto-follow zoom mode with cursor tracking#257xKeCo wants to merge 5 commits intosiddharthvaddem:mainfrom
Conversation
📝 WalkthroughWalkthroughAdds a per-zoom-region focus mode ("manual" | "auto") that can use recorded cursor telemetry to drive zoom focus with adaptive smoothing; UI exposes a focus-mode selector and telemetry is threaded through playback and export pipelines to produce consistent auto-follow behavior. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant SettingsPanel
participant VideoEditor
participant VideoPlayback
participant ZoomRegionUtils
participant CursorFollowUtils
participant FrameRenderer
User->>SettingsPanel: choose "auto" focus mode
SettingsPanel->>VideoEditor: onZoomFocusModeChange("auto")
VideoEditor->>VideoEditor: pushState update selected zoom.focusMode="auto"
Note over VideoPlayback,FrameRenderer: During playback/export frame loop
VideoPlayback->>ZoomRegionUtils: findDominantRegion(timeMs, cursorTelemetry, viewportRatio)
ZoomRegionUtils->>CursorFollowUtils: interpolateCursorAt(cursorTelemetry, timeMs)
CursorFollowUtils-->>ZoomRegionUtils: interpolated cursor focus
ZoomRegionUtils-->>VideoPlayback: region with resolved focus (auto)
VideoPlayback->>CursorFollowUtils: adaptiveSmoothFactor / smoothCursorFocus(prev, raw)
CursorFollowUtils-->>VideoPlayback: smoothed focus
VideoPlayback-->>FrameRenderer: render frame with smoothed focus
FrameRenderer-->>VideoPlayback: frame rendered / exported
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contains the zoom region configuration used in the PR demo video: two auto-follow zoom regions and one manual zoom region. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 163b12d6fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| targetProgress = strength; | ||
|
|
||
| // Apply deadzone + time-based smoothing for auto-follow mode | ||
| if (region.focusMode === "auto" && !transition) { |
There was a problem hiding this comment.
When transition is present, this condition bypasses the auto-follow update path entirely, so smoothedAutoFocus stays at its pre-transition value. In exports with connected auto zoom regions, the first non-transition frame then smooths from stale focus and produces a visible camera lurch right after the pan. Updating/resetting the smoothed state during transition frames (or before the first post-transition frame) avoids this discontinuity.
Useful? React with 👍 / 👎.
| targetProgress = strength; | ||
|
|
||
| // Apply deadzone + smoothing for auto-follow mode | ||
| if (region.focusMode === "auto" && !transition) { |
There was a problem hiding this comment.
The preview path has the same transition gating, so during connected pans in auto mode the smoothed focus ref is not refreshed while the transition runs. When transition ends, the next tick reuses stale focus and briefly pulls toward an outdated cursor position, causing a visible jump in playback that disagrees with the intended continuous auto-follow behavior.
Useful? React with 👍 / 👎.
This reverts commit 5c66212.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
src/components/video-editor/videoPlayback/cursorFollowUtils.ts (1)
49-54: Clamp smoothing factor to guard against future misuse.
factoris currently assumed valid; clamping to[0,1]avoids accidental overshoot if a caller ever passes an out-of-range value.Suggested defensive tweak
export function smoothCursorFocus(raw: ZoomFocus, prev: ZoomFocus, factor: number): ZoomFocus { + const clampedFactor = Math.min(1, Math.max(0, factor)); return { - cx: prev.cx + (raw.cx - prev.cx) * factor, - cy: prev.cy + (raw.cy - prev.cy) * factor, + cx: prev.cx + (raw.cx - prev.cx) * clampedFactor, + cy: prev.cy + (raw.cy - prev.cy) * clampedFactor, }; }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/components/video-editor/videoPlayback/cursorFollowUtils.ts` around lines 49 - 54, smoothCursorFocus currently uses the caller-provided factor directly; clamp factor to the [0,1] range before computing the interpolated cx/cy to prevent overshoot if callers pass out-of-range values. In the smoothCursorFocus( raw: ZoomFocus, prev: ZoomFocus, factor: number ) function, compute a clampedFactor = Math.max(0, Math.min(1, factor)) (or equivalent) and use clampedFactor in the interpolation expressions for cx and cy so ZoomFocus results are always a proper interpolation.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/components/video-editor/VideoEditor.tsx`:
- Around line 1604-1610: The focus-mode selector is currently hidden whenever
cursorTelemetry is empty due to hasCursorTelemetry={cursorTelemetry.length > 0},
which blocks switching an existing region out of "auto"; update the visibility
logic so the selector remains visible if the selected region's focusMode is
"auto" (or any persisted value) even when cursorTelemetry is empty. Concretely,
change the condition that sets hasCursorTelemetry (used where
selectedZoomFocusMode, onZoomFocusModeChange, hasCursorTelemetry are passed) to
true when either cursorTelemetry.length > 0 OR the selectedZoomId corresponds to
a zoomRegions entry whose focusMode === "auto" (or simply when a selectedZoomId
exists and zoomRegions.find(z => z.id === selectedZoomId)?.focusMode is truthy),
so handleZoomFocusModeChange remains accessible for existing auto regions.
In `@src/components/video-editor/VideoPlayback.tsx`:
- Around line 390-391: The early return in VideoPlayback (the block checking
region.focusMode === "auto" before calling onSelectZoom(region.id)) prevents
focus dragging but leaves the overlay interactive; update the overlay logic so
when region.focusMode === "auto" the overlay element used for zoom/focus is
rendered or styled with pointer-events: none (e.g., toggle a CSS class or inline
style on the overlay element) so it doesn't intercept pointer events, or ensure
the early-return path also disables the overlay (via the same prop/state used to
render the overlay) rather than only returning before onSelectZoom.
- Around line 870-903: The smoothed auto-follow state
(smoothedAutoFocusRef.current) can persist across seeks/pauses/gaps because it’s
only updated inside the active auto-region branch; clear it whenever auto mode
is not actively driving the camera by setting smoothedAutoFocusRef.current =
null in the other cases: specifically when region.focusMode !== "auto" (already
present) and also when transition is truthy or when targetProgress < 0.999 and
not isZoomingIn (i.e., not actively zooming-in), so that smoothed state is reset
before the next auto zoom; locate and update the logic around region.focusMode,
transition, targetProgress, isZoomingIn and prevTargetProgressRef to implement
this.
In `@src/lib/exporter/frameRenderer.ts`:
- Around line 540-542: The smoothing delta is currently computed from the source
timestamp (timeMs) inside renderFrame, which makes smoothing depend on playback
speed; change the dtMs calculation to use the exporter output timestamp (the
output-frame timestamp provided to the exporter callback) instead of timeMs
while leaving timeMs usage for cursor/region lookup. Specifically, in
renderFrame replace the dtMs := this.prevAnimationTimeMs != null ? timeMs -
this.prevAnimationTimeMs : 0 assignment to compute dtMs :=
this.prevAnimationTimeMs != null ? outputTimestampMs - this.prevAnimationTimeMs
: 0 (keeping the same fallback), so framesElapsed and factor (using
AUTO_FOLLOW_SMOOTHING_FACTOR) are derived from the output-frame delta; update
any references to prevAnimationTimeMs/timeMs usage accordingly so only
cursor/region logic still uses the source time.
- Around line 537-572: The code fails to clear the auto-follow accumulator when
an auto region ends, causing future auto zooms to resume smoothing from stale
state; update the branch that handles non-auto regions (check region.focusMode
!== "auto") to reset both this.smoothedAutoFocus and this.prevTargetProgress
(e.g., set smoothedAutoFocus = null and prevTargetProgress =
null/undefined/initial value) so the next auto follow starts fresh; locate the
logic around region.focusMode, smoothedAutoFocus and prevTargetProgress in
frameRenderer.ts and add the reset there.
---
Nitpick comments:
In `@src/components/video-editor/videoPlayback/cursorFollowUtils.ts`:
- Around line 49-54: smoothCursorFocus currently uses the caller-provided factor
directly; clamp factor to the [0,1] range before computing the interpolated
cx/cy to prevent overshoot if callers pass out-of-range values. In the
smoothCursorFocus( raw: ZoomFocus, prev: ZoomFocus, factor: number ) function,
compute a clampedFactor = Math.max(0, Math.min(1, factor)) (or equivalent) and
use clampedFactor in the interpolation expressions for cx and cy so ZoomFocus
results are always a proper interpolation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 2d2c2bab-0e99-4028-83fa-7c233b3c8c86
📒 Files selected for processing (15)
src/components/video-editor/SettingsPanel.tsxsrc/components/video-editor/VideoEditor.tsxsrc/components/video-editor/VideoPlayback.tsxsrc/components/video-editor/projectPersistence.tssrc/components/video-editor/types.tssrc/components/video-editor/videoPlayback/constants.tssrc/components/video-editor/videoPlayback/cursorFollowUtils.tssrc/components/video-editor/videoPlayback/overlayUtils.tssrc/components/video-editor/videoPlayback/zoomRegionUtils.tssrc/i18n/locales/en/settings.jsonsrc/i18n/locales/es/settings.jsonsrc/i18n/locales/zh-CN/settings.jsonsrc/lib/exporter/frameRenderer.tssrc/lib/exporter/gifExporter.tssrc/lib/exporter/videoExporter.ts
| selectedZoomFocusMode={ | ||
| selectedZoomId | ||
| ? (zoomRegions.find((z) => z.id === selectedZoomId)?.focusMode ?? "manual") | ||
| : null | ||
| } | ||
| onZoomFocusModeChange={(mode) => selectedZoomId && handleZoomFocusModeChange(mode)} | ||
| hasCursorTelemetry={cursorTelemetry.length > 0} |
There was a problem hiding this comment.
Keep the focus-mode control visible for already-"auto" regions.
focusMode is persisted, but hasCursorTelemetry={cursorTelemetry.length > 0} hides the entire selector as soon as telemetry is unavailable. If a project is opened without its cursor samples, the region can stay in "auto" while manual focus dragging remains disabled, leaving no UI path back to "manual".
Suggested change
- hasCursorTelemetry={cursorTelemetry.length > 0}
+ hasCursorTelemetry={
+ cursorTelemetry.length > 0 ||
+ (selectedZoomId
+ ? zoomRegions.find((z) => z.id === selectedZoomId)?.focusMode === "auto"
+ : false)
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| selectedZoomFocusMode={ | |
| selectedZoomId | |
| ? (zoomRegions.find((z) => z.id === selectedZoomId)?.focusMode ?? "manual") | |
| : null | |
| } | |
| onZoomFocusModeChange={(mode) => selectedZoomId && handleZoomFocusModeChange(mode)} | |
| hasCursorTelemetry={cursorTelemetry.length > 0} | |
| selectedZoomFocusMode={ | |
| selectedZoomId | |
| ? (zoomRegions.find((z) => z.id === selectedZoomId)?.focusMode ?? "manual") | |
| : null | |
| } | |
| onZoomFocusModeChange={(mode) => selectedZoomId && handleZoomFocusModeChange(mode)} | |
| hasCursorTelemetry={ | |
| cursorTelemetry.length > 0 || | |
| (selectedZoomId | |
| ? zoomRegions.find((z) => z.id === selectedZoomId)?.focusMode === "auto" | |
| : false) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/components/video-editor/VideoEditor.tsx` around lines 1604 - 1610, The
focus-mode selector is currently hidden whenever cursorTelemetry is empty due to
hasCursorTelemetry={cursorTelemetry.length > 0}, which blocks switching an
existing region out of "auto"; update the visibility logic so the selector
remains visible if the selected region's focusMode is "auto" (or any persisted
value) even when cursorTelemetry is empty. Concretely, change the condition that
sets hasCursorTelemetry (used where selectedZoomFocusMode,
onZoomFocusModeChange, hasCursorTelemetry are passed) to true when either
cursorTelemetry.length > 0 OR the selectedZoomId corresponds to a zoomRegions
entry whose focusMode === "auto" (or simply when a selectedZoomId exists and
zoomRegions.find(z => z.id === selectedZoomId)?.focusMode is truthy), so
handleZoomFocusModeChange remains accessible for existing auto regions.
| if (region.focusMode === "auto") return; | ||
| onSelectZoom(region.id); |
There was a problem hiding this comment.
Don't leave the hidden auto overlay interactive.
This early return stops focus dragging, but the overlay still sits above the stage with pointer events enabled whenever a zoom is selected. In auto mode that invisible layer will eat webcam drags and other underlying pointer interactions unless the overlay-effect logic also switches it to pointer-events: none.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/components/video-editor/VideoPlayback.tsx` around lines 390 - 391, The
early return in VideoPlayback (the block checking region.focusMode === "auto"
before calling onSelectZoom(region.id)) prevents focus dragging but leaves the
overlay interactive; update the overlay logic so when region.focusMode ===
"auto" the overlay element used for zoom/focus is rendered or styled with
pointer-events: none (e.g., toggle a CSS class or inline style on the overlay
element) so it doesn't intercept pointer events, or ensure the early-return path
also disables the overlay (via the same prop/state used to render the overlay)
rather than only returning before onSelectZoom.
| // Apply deadzone + smoothing for auto-follow mode | ||
| if (region.focusMode === "auto" && !transition) { | ||
| const raw = targetFocus; | ||
| const isZoomingIn = | ||
| targetProgress < 0.999 && targetProgress >= prevTargetProgressRef.current; | ||
| if (targetProgress >= 0.999) { | ||
| // Full zoom: apply deadzone + smoothing for stable follow | ||
| const prev = smoothedAutoFocusRef.current ?? raw; | ||
| const dx = Math.abs(raw.cx - prev.cx); | ||
| const dy = Math.abs(raw.cy - prev.cy); | ||
| if (dx > AUTO_FOLLOW_DEADZONE || dy > AUTO_FOLLOW_DEADZONE) { | ||
| const smoothed = smoothCursorFocus(raw, prev, AUTO_FOLLOW_SMOOTHING_FACTOR); | ||
| smoothedAutoFocusRef.current = smoothed; | ||
| targetFocus = smoothed; | ||
| } else { | ||
| smoothedAutoFocusRef.current = prev; | ||
| targetFocus = prev; | ||
| } | ||
| } else if (isZoomingIn) { | ||
| // Zoom-in: track cursor directly so zoom always aims at current cursor | ||
| // position; keep ref in sync to avoid snap when full-zoom begins | ||
| smoothedAutoFocusRef.current = raw; | ||
| } else { | ||
| // Zoom-out: keep smoothing for continuity — avoids snap at zoom-out start | ||
| const prev = smoothedAutoFocusRef.current ?? raw; | ||
| const smoothed = smoothCursorFocus(raw, prev, AUTO_FOLLOW_SMOOTHING_FACTOR); | ||
| smoothedAutoFocusRef.current = smoothed; | ||
| targetFocus = smoothed; | ||
| } | ||
| } else if (region.focusMode !== "auto") { | ||
| smoothedAutoFocusRef.current = null; | ||
| } | ||
| prevTargetProgressRef.current = targetProgress; | ||
|
|
There was a problem hiding this comment.
Clear follow smoothing whenever auto mode is not actively driving the camera.
These refs are only updated inside the active-region branch. After a seek, a pause in the unzoomed view, or a gap between auto zooms, the next zoom-in can reuse the previous region's smoothed focus and drift from the wrong spot.
Suggested change
- if (region && strength > 0 && !shouldShowUnzoomedView) {
+ if (!region || strength <= 0 || shouldShowUnzoomedView) {
+ smoothedAutoFocusRef.current = null;
+ prevTargetProgressRef.current = 0;
+ }
+
+ if (region && strength > 0 && !shouldShowUnzoomedView) {
const zoomScale = blendedScale ?? ZOOM_DEPTH_SCALES[region.depth];
const regionFocus = region.focus;
@@
- } else if (region.focusMode !== "auto") {
+ } else if (region.focusMode !== "auto") {
smoothedAutoFocusRef.current = null;
+ prevTargetProgressRef.current = 0;
}
prevTargetProgressRef.current = targetProgress;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/components/video-editor/VideoPlayback.tsx` around lines 870 - 903, The
smoothed auto-follow state (smoothedAutoFocusRef.current) can persist across
seeks/pauses/gaps because it’s only updated inside the active auto-region
branch; clear it whenever auto mode is not actively driving the camera by
setting smoothedAutoFocusRef.current = null in the other cases: specifically
when region.focusMode !== "auto" (already present) and also when transition is
truthy or when targetProgress < 0.999 and not isZoomingIn (i.e., not actively
zooming-in), so that smoothed state is reset before the next auto zoom; locate
and update the logic around region.focusMode, transition, targetProgress,
isZoomingIn and prevTargetProgressRef to implement this.
src/lib/exporter/frameRenderer.ts
Outdated
| // Apply deadzone + time-based smoothing for auto-follow mode | ||
| if (region.focusMode === "auto" && !transition) { | ||
| const raw = targetFocus; | ||
| const dtMs = this.prevAnimationTimeMs != null ? timeMs - this.prevAnimationTimeMs : 0; | ||
| const framesElapsed = dtMs > 0 ? dtMs / (1000 / 60) : 1; | ||
| const factor = 1 - Math.pow(1 - AUTO_FOLLOW_SMOOTHING_FACTOR, Math.max(1, framesElapsed)); | ||
| const isZoomingIn = targetProgress < 0.999 && targetProgress >= this.prevTargetProgress; | ||
| if (targetProgress >= 0.999) { | ||
| // Full zoom: apply deadzone + smoothing for stable follow | ||
| const prev = this.smoothedAutoFocus ?? raw; | ||
| const dx = Math.abs(raw.cx - prev.cx); | ||
| const dy = Math.abs(raw.cy - prev.cy); | ||
| if (dx > AUTO_FOLLOW_DEADZONE || dy > AUTO_FOLLOW_DEADZONE) { | ||
| const smoothed = smoothCursorFocus(raw, prev, factor); | ||
| this.smoothedAutoFocus = smoothed; | ||
| targetFocus = smoothed; | ||
| } else { | ||
| this.smoothedAutoFocus = prev; | ||
| targetFocus = prev; | ||
| } | ||
| } else if (isZoomingIn) { | ||
| // Zoom-in: track cursor directly so zoom always aims at current cursor | ||
| // position; keep ref in sync to avoid snap when full-zoom begins | ||
| this.smoothedAutoFocus = raw; | ||
| } else { | ||
| // Zoom-out: keep smoothing for continuity — avoids snap at zoom-out start | ||
| const prev = this.smoothedAutoFocus ?? raw; | ||
| const smoothed = smoothCursorFocus(raw, prev, factor); | ||
| this.smoothedAutoFocus = smoothed; | ||
| targetFocus = smoothed; | ||
| } | ||
| } else if (region.focusMode !== "auto") { | ||
| this.smoothedAutoFocus = null; | ||
| } | ||
| this.prevTargetProgress = targetProgress; | ||
|
|
There was a problem hiding this comment.
Reset the auto-follow accumulator when no zoom region is active.
smoothedAutoFocus and prevTargetProgress are only cleared for non-auto regions. After an auto region ends, the next auto zoom after a gap can be treated as a continuation of the previous one and start smoothing from stale coordinates.
Suggested change
- if (region && strength > 0) {
+ if (!region || strength <= 0) {
+ this.smoothedAutoFocus = null;
+ this.prevTargetProgress = 0;
+ }
+
+ if (region && strength > 0) {
const zoomScale = blendedScale ?? ZOOM_DEPTH_SCALES[region.depth];
const regionFocus = this.clampFocusToStage(region.focus, region.depth);
@@
- } else if (region.focusMode !== "auto") {
+ } else if (region.focusMode !== "auto") {
this.smoothedAutoFocus = null;
+ this.prevTargetProgress = 0;
}
this.prevTargetProgress = targetProgress;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lib/exporter/frameRenderer.ts` around lines 537 - 572, The code fails to
clear the auto-follow accumulator when an auto region ends, causing future auto
zooms to resume smoothing from stale state; update the branch that handles
non-auto regions (check region.focusMode !== "auto") to reset both
this.smoothedAutoFocus and this.prevTargetProgress (e.g., set smoothedAutoFocus
= null and prevTargetProgress = null/undefined/initial value) so the next auto
follow starts fresh; locate the logic around region.focusMode, smoothedAutoFocus
and prevTargetProgress in frameRenderer.ts and add the reset there.
src/lib/exporter/frameRenderer.ts
Outdated
| const dtMs = this.prevAnimationTimeMs != null ? timeMs - this.prevAnimationTimeMs : 0; | ||
| const framesElapsed = dtMs > 0 ? dtMs / (1000 / 60) : 1; | ||
| const factor = 1 - Math.pow(1 - AUTO_FOLLOW_SMOOTHING_FACTOR, Math.max(1, framesElapsed)); |
There was a problem hiding this comment.
Use output-frame delta for smoothing, not source-media delta.
dtMs here is derived from the source timestamp passed into renderFrame(). Inside speed regions that makes export smoothing depend on playback speed, so auto-follow becomes snappier/slower than the live preview. The exporter callbacks already have an output timestamp available; use that delta for the smoothing factor and keep source time only for cursor/region lookup.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lib/exporter/frameRenderer.ts` around lines 540 - 542, The smoothing
delta is currently computed from the source timestamp (timeMs) inside
renderFrame, which makes smoothing depend on playback speed; change the dtMs
calculation to use the exporter output timestamp (the output-frame timestamp
provided to the exporter callback) instead of timeMs while leaving timeMs usage
for cursor/region lookup. Specifically, in renderFrame replace the dtMs :=
this.prevAnimationTimeMs != null ? timeMs - this.prevAnimationTimeMs : 0
assignment to compute dtMs := this.prevAnimationTimeMs != null ?
outputTimestampMs - this.prevAnimationTimeMs : 0 (keeping the same fallback), so
framesElapsed and factor (using AUTO_FOLLOW_SMOOTHING_FACTOR) are derived from
the output-frame delta; update any references to prevAnimationTimeMs/timeMs
usage accordingly so only cursor/region logic still uses the source time.
|
@xKeCo can you explain how this is any different from the existing auto zoom capabilities that you can apply based on the cursor telemetry and magic want button. |
@siddharthvaddem Hey! Good question — they actually solve different problems and complement each other. The existing cursor telemetry feature (Magic Wand) looks at your recording after the fact, finds moments where the cursor was relatively still, and creates zoom regions with a fixed focus point based on the average cursor position during those dwell periods. It's essentially a smart shortcut for generating zoom regions quickly. What this PR adds is a Focus Mode toggle (Manual / Auto) within an existing zoom region. Instead of locking the camera to a static point, Auto mode tracks the cursor's position frame by frame during playback — with smoothing and a small deadzone to avoid jitter — and the same logic is applied identically when exporting. Here's a side-by-side comparison to make it more concrete: Without Auto-Follow (Magic Wand only): the zoom regions are created based on where the cursor was dwelling, but the focus point stays static. If the cursor moves significantly during a zoom, the active area drifts out of frame. Screen.Recording.2026-04-02.at.11.13.52.AM.mp4With Auto-Follow: the camera continuously pans with the cursor throughout the zoom region, keeping the active area centered the whole time. Screen.Recording.2026-04-02.at.11.07.57.AM.mp4So to summarize: Magic Wand helps you create zoom regions based on cursor activity. Auto-Follow makes the camera dynamic inside those regions rather than staying locked to a single point. You can even use both together — let the Magic Wand generate the regions, then toggle Auto on the ones where the cursor moves a lot. |
|
okay this is cool and makes sense on a high level. This needs some minor tweaks as when we have sudden movements - the focus is so rigid - it feels super snappy. Happy to merge once you have this addressed. |
|
This is a nice improvement ! Well done |
|
That’s an interesting feature! I’d love to see it implemented. Thanks for making this, but I’ve noticed an issue in vertical mode. The space between a window and the container at the top and bottom is excessively large. This happens when the cursor is positioned at the top or bottom of the window. I expect the window to remain at the top of the container without much spacing, and the same should apply to the bottom. Here’s the result from your current branch: |
There was a problem hiding this comment.
🧹 Nitpick comments (2)
src/components/video-editor/videoPlayback/zoomRegionUtils.ts (1)
70-73:ViewportRatiois duplicated fromfocusUtils.ts.As noted in
focusUtils.ts, consider consolidating this interface definition.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/components/video-editor/videoPlayback/zoomRegionUtils.ts` around lines 70 - 73, The ViewportRatio interface declared in zoomRegionUtils.ts duplicates the same type in focusUtils.ts; remove the local interface and import the single shared definition from focusUtils.ts (or move it to a common types module if more appropriate) so both zoomRegionUtils.ts and focusUtils.ts reference the same ViewportRatio symbol; update any references in functions within zoomRegionUtils.ts that use ViewportRatio to use the imported type and ensure exports/imports are adjusted accordingly.src/components/video-editor/videoPlayback/focusUtils.ts (1)
42-45: Consider extractingViewportRatioto a shared types file.This interface is duplicated in
zoomRegionUtils.ts(lines 70-73). Moving it to a shared location (e.g.,types.tsor a dedicatedvideoPlayback/types.ts) would reduce duplication and ensure consistency.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/components/video-editor/videoPlayback/focusUtils.ts` around lines 42 - 45, Extract the duplicated interface ViewportRatio into a shared types module and import it where needed: create a new types file (e.g., video playback types) that exports interface ViewportRatio { widthRatio: number; heightRatio: number; }, replace the local declarations in focusUtils.ts (reference: ViewportRatio) and zoomRegionUtils.ts (where ViewportRatio is duplicated) with an import from the new module, and update any imports/usages to ensure both files use the single shared type.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@src/components/video-editor/videoPlayback/focusUtils.ts`:
- Around line 42-45: Extract the duplicated interface ViewportRatio into a
shared types module and import it where needed: create a new types file (e.g.,
video playback types) that exports interface ViewportRatio { widthRatio: number;
heightRatio: number; }, replace the local declarations in focusUtils.ts
(reference: ViewportRatio) and zoomRegionUtils.ts (where ViewportRatio is
duplicated) with an import from the new module, and update any imports/usages to
ensure both files use the single shared type.
In `@src/components/video-editor/videoPlayback/zoomRegionUtils.ts`:
- Around line 70-73: The ViewportRatio interface declared in zoomRegionUtils.ts
duplicates the same type in focusUtils.ts; remove the local interface and import
the single shared definition from focusUtils.ts (or move it to a common types
module if more appropriate) so both zoomRegionUtils.ts and focusUtils.ts
reference the same ViewportRatio symbol; update any references in functions
within zoomRegionUtils.ts that use ViewportRatio to use the imported type and
ensure exports/imports are adjusted accordingly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fe131dfd-1372-4677-aa76-1dd8911bf661
📒 Files selected for processing (6)
src/components/video-editor/VideoPlayback.tsxsrc/components/video-editor/videoPlayback/constants.tssrc/components/video-editor/videoPlayback/cursorFollowUtils.tssrc/components/video-editor/videoPlayback/focusUtils.tssrc/components/video-editor/videoPlayback/zoomRegionUtils.tssrc/lib/exporter/frameRenderer.ts
✅ Files skipped from review due to trivial changes (1)
- src/components/video-editor/videoPlayback/constants.ts
|
@siddharthvaddem @GarryLaly @FabLrc Thanks for the feedback! Here's what was addressed: 1. Smoother auto-follow transitionsProblem: Sudden cursor movements caused rigid/snappy camera behavior due to a binary deadzone — the camera would either freeze completely or lurch forward, creating an unpleasant jerk. Fix: Replaced the hard deadzone with velocity-adaptive smoothing. The camera now moves faster when the cursor is far from the current focus and naturally decelerates as it gets closer, eliminating the stop/start feel. The base responsiveness was also increased from 0.05 to 0.1. 2. Excessive vertical spacing in portrait mode (9:16)Problem: In vertical/stacked layouts the focus bounds calculation assumed the baseMask (recording area) filled the full stage. In portrait mode the baseMask is shorter than the stage, so the camera could pan past the recording edges, showing blank space at top and bottom. Fix: The focus bounds now account for the stageSize / baseMask ratio per axis, so the camera is correctly clamped to the recording content. A safety cap at 0.5 prevents an edge case in Picture-in-Picture portrait layouts where the ratio would exceed the zoom scale and invert the bounds. ExamplesWide (16:9)Making of making-v2-wide.mp4Result result-v2-wide.1.mp4Portrait (16:9)Making of making-v2-mobile.mp4Result result-v2-mobile.1.mp4Tip If the auto-follow transitions still feel too abrupt for a given recording, adjusting the Motion Blur slider in the Video Effects panel can significantly smooth out the motion between frames and make the camera follow feel more cinematic. |
|
@xKeCo that's awesome! It's working well now and looks perfect. Thanks for the update |
4 participants
Summary
FrameRendererDemo
export-1775028836571.mp4
Example project file: auto-follow-zoom-example.openscreen — load this in Openscreen to reproduce the demo (update
screenVideoPathto point to your own recording)How it works
Playback and export share the same 3-phase camera logic per zoom region:
progress < 1and increasingprogress ≥ 0.999progress < 1and decreasingExport uses a time-based smoothing factor (
1 - (1 - BASE)^(dt / 16.67ms)) so the smoothing is frame-rate-independent regardless of export resolution or speed regions.Changes
New file
src/components/video-editor/videoPlayback/cursorFollowUtils.tsinterpolateCursorAt— binary search + linear interpolation on cursor telemetry arraysmoothCursorFocus— exponential smoothing helperModified
types.ts—ZoomFocusMode = "manual" | "auto", added optionalfocusModetoZoomRegionconstants.ts—AUTO_FOLLOW_SMOOTHING_FACTOR = 0.05,AUTO_FOLLOW_DEADZONE = 0.06zoomRegionUtils.ts— threadscursorTelemetrythroughfindDominantRegion→getResolvedFocus; pre-computes cursor position once per connected-zoom transition pair to avoid a duplicate binary searchoverlayUtils.ts— hides the indicator overlay in Auto modeVideoPlayback.tsx— 3-phase smoothing in the PixiJS ticker;cursorTelemetryprop wired inVideoEditor.tsx—handleZoomFocusModeChange, forwardscursorTelemetryto both exportersSettingsPanel.tsx— Focus Mode button group (rendered only when cursor telemetry is available)projectPersistence.ts— normalizesfocusModeon project loadframeRenderer.ts— mirrors the playback smoothing with time-based factor scalingvideoExporter.ts/gifExporter.ts— forwardcursorTelemetrytoFrameRendereri18n/locales/{en,es,zh-CN}/settings.json— Focus Mode translations (nested object to fix key traversal)Summary by CodeRabbit
Release Notes
New Features
Localization