AI-Powered Virtual Background and Blur for Video Calls

[miroslavpejic85]

Feature request

Implement AI-powered virtual background and blur functionality, allowing users to either replace their background with a custom image or blur it for enhanced privacy. This feature should work in real-time without requiring a green screen, ensuring smooth performance while maintaining video quality.

Pros

Pros of Virtual Background and Blur in Video Calls

✅ Privacy Protection

• Hides sensitive or personal surroundings, preventing unwanted background visibility.
• Useful for professionals working from home or public spaces.

✅ Professional Appearance

• Helps maintain a clean and distraction-free look, especially in business meetings.
• Can create a branded or customized background for companies.

✅ Focus on the Speaker

• Blurring reduces background distractions, keeping attention on the person speaking.
• Virtual backgrounds can be used to create a consistent visual experience.

✅ Enhanced User Experience

• Makes remote communication more engaging and fun.
• Users can personalize their background to suit different settings (casual, professional, themed).

✅ Flexible Environment Usage

• No need for a dedicated workspace—users can take calls from anywhere while maintaining a professional look.
• Helps in low-budget setups where users might not have an ideal background.

✅ Reduces the Need for Green Screens

• AI-based segmentation removes the need for physical green screens, making it accessible to everyone.

Additional context

Using MediaPipe can be a good solution.

✅ Pros of Using MediaPipe

1. Real-time Performance – Optimized for fast execution on both CPU and GPU.
2. No Green Screen Needed – Uses AI-based segmentation to separate the user from the background.
3. Cross-Platform Support – Works on web (JavaScript), mobile (Android/iOS), and desktop.
4. Lightweight & Efficient – Compared to heavier models like TensorFlow, MediaPipe is optimized for speed.
5. Google-backed & Actively Maintained – Reliable with frequent updates.
6. Works In-Browser – Can run client-side using WebAssembly (WASM) or WebGL, reducing server load.

❌ Potential Challenges

• Quality Variations – Might struggle with fine details like hair in low-light conditions.
• Performance on Low-End Devices – Can be CPU-intensive, so optimization is needed for weaker devices.
• Limited Customization – Pre-trained models may not be as flexible as custom ML solutions.

Alternative Solutions

• TensorFlow.js – More customizable but heavier.
• WebGPU/WebGL-based solutions – For better performance on supported devices.

[miroslavpejic85]

[miroslavpejic85]

[miroslavpejic85]

mirotalk-sfu-image-file-url.mp4

[miroslavpejic85]

[miroslavpejic85]

[miroslavpejic85]

[miroslavpejic85]

Browser	Chromium-Based	Virtual Background Support
Google Chrome	✅ Yes	✅ Yes
Microsoft Edge	✅ Yes	✅ Yes
Opera	✅ Yes	✅ Yes
Brave	✅ Yes	✅ Yes
Vivaldi	✅ Yes	✅ Yes
Safari	❌ No	❌ No
Mozilla Firefox	❌ No	❌ No

[cs35-owncloud]

Hello,
sorry it took me a while to test the features, it's really cool, thanks a lot !
Is it possible to support it also on Firefox ?

[miroslavpejic85]

[miroslavpejic85]

[avatar1024]

Looks and work great! Is a working version for firefox on the roadmap at all? I've told my users to switch to chrome-based browsers to use this but we're all using firefox as a default browser for most other things so it'd be interesting to know if this is coming or not. Thanks again for all your work developing this!

[miroslavpejic85]

Looks and work great! Is a working version for firefox on the roadmap at all?

Thank you for your feedback! I'm glad to hear it's working well for you.

Firefox does not support the following experimental Web APIs, which are crucial for advanced real-time media processing like virtual backgrounds and video enhancements:

1. MediaStreamTrackProcessor – Allows breaking down a MediaStreamTrack into individual frames for processing.
2. MediaStreamTrackGenerator – Enables generating synthetic media streams from processed frames.
3. TransformStream in the context of media streams – Useful for applying transformations to streams efficiently.

These APIs are part of the WebCodecs and Streams API standards and are currently only supported in Chromium-based browsers (Chrome, Edge, Brave, etc.).

Workarounds for Firefox:

• For virtual backgrounds, you might need to use less efficient alternatives like Canvas-based processing or WebGL shaders.
• Some libraries (like TensorFlow.js or WASM-based solutions) can help but may have higher latency.

Best Performance Recommendation:

For the smoothest experience with real-time video effects, Chromium-based browsers are indeed recommended due to their support for these modern APIs. If Firefox adds support in the future, compatibility will be automatically resolved.

[avatar1024]

Thanks! Very useful info. I'll keep track of support for those APIs on firefox (actually seems like TransportStream is already supported and MediaStreamTrackProcessor is coming 🙂).

Oh while testing one thing I have noticed is that the Privacy mode (circle around face) seems gone, both on Firefox and Chrome-based browser, is this the intended behaviour or am I missing something (as it move somewhere else maybe)?

[miroslavpejic85]

Thanks! Very useful info. I'll keep track of support for those APIs on firefox (actually seems like TransportStream is already supported and MediaStreamTrackProcessor is coming 🙂).

Very Good!

Oh while testing one thing I have noticed is that the Privacy mode (circle around face) seems gone, both on Firefox and Chrome-based browser, is this the intended behaviour or am I missing something (as it move somewhere else maybe)?

Yes, it works on any device! Privacy Mode was actually the first fun feature I implemented as an alternative to virtual backgrounds in the beginning. Just position your face inside the circle, and you're good to go! 😊 It can also now be combined with a virtual background as well! 🎉

[avatar1024]

Yes, it works on any device!

Sorry if I'm stupid but I just can't see the ui to activate Privacy Mode any longer...

I used to use this a while ago, and indeed it was a very clever approach you came up with to enable privacy before implementing virtual background :).

[miroslavpejic85]

Sorry if I'm stupid but I just can't see the ui to activate Privacy Mode any longer...

Before, the menu bar was always visible. Now, it's hidden by default for a cleaner UI. Click on the desired video element to toggle the top menu bar and access features like Circle Privacy Mode and more.