Klippy

AI background removal

Klippy ships a real-time background removal workflow powered by MediaPipe Selfie Segmentation. This guide explains how the pipeline works, how to tune quality and edge controls, and what happens during preview, export, and batching so you can confidently integrate or extend the feature.

Where it lives

  • Open the AI panel from the editor sidebar. The panel focuses on background removal for selected video clips.
  • Settings are persisted per media file inside mediaFile.aiProcessing, enabling clip-specific tuning without affecting the global project.
  • The modal AIVideoProcessingModal provides progress feedback when you commit a high-quality offline render.

Processing architecture

Background removal combines live preview on the Remotion canvas with offline processing for exports:

  • AIVideoProcessorMediaPipe.tsx drives live playback. Each rendered frame is segmented using useMediaPipeSegmentation, then refined with smoothing, edge contraction, and spill suppression before compositing.
  • BackgroundRemovalProcessor.ts encapsulates mask construction, temporal smoothing, edge operations, and TensorFlow fallback. It is reused by the export pipeline so previews and renders stay consistent.
  • TensorFlowManager (lazy loaded) provides an alternate model for environments where MediaPipe is unavailable; the processor falls back automatically.
  • useBackgroundMedia and related hooks inject virtual backgrounds (blur, color, image, or video) when configured.

Controls & quality modes

The Background removal panel exposes the following knobs (stored per clip):

Core options

  • Quality: fast, balanced, quality alter segmentation resolution and cadence.
  • Threshold: controls matte aggressiveness; lower values preserve more detail.
  • Edge smoothing: gaussian blur applied to the mask for softer transitions.
  • Temporal smoothing: blends masks frame-to-frame to reduce flicker.

Advanced adjustments

  • Edge contract (erosion) tightens the matte to reduce halos.
  • Mask contrast applies gamma correction to emphasise foreground confidence.
  • Spill suppression desaturates edge pixels to remove green-screen tint.
  • Overlay mode visualises the active mask for debugging.

Settings modify both the live preview and the exported render. Use Balanced for laptop demos andQuality when running the offline processor before publishing.

Preview vs committed render

  • While editing, frames are processed on the fly inside Remotion. Resolution is capped at 1080p to protect memory.
  • Click Process video to queue a higher fidelity render. The modal uses workers to iterate the source file, writes the processed clip back into the media library, and swaps timeline references.
  • Exports respect the processed result: Standard (FFmpeg) composites the transparent PNG/WebM frames; Transparent (WebCodecs/MediaBunny) keeps the alpha channel for sticker workflows.

Performance notes

  • Segmentation runs in a dedicated worker; useMediaPipeSegmentation throttles requests to avoid blocking the UI.
  • Temporal smoothing keeps a previous mask buffer (prevMaskDataRef) so toggling the preview resets stabilization—expect a short catch-up period.
  • When capturing full-HD webcam footage, prefer Balanced mode and disable overlay to save GPU time.
  • GPU acceleration is browser dependent. Safari may fall back to CPU; for production, test on target devices and adjust quality accordingly.

Virtual backgrounds & effects

Background removal feeds other AI features:

  • Virtual background: configure blur, color, image, or looping video sources via useBackgroundMedia.
  • Object detection overlay: optional bounding boxes (driven by useObjectDetectionFromMask) help validate segmentation.
  • Export transparency: processed clips can be layered over footage using the Transparent export path described in Rendering.

Troubleshooting

  • Model fails to load: ensure cross-origin isolation headers are active; MediaPipe assets require COOP/COEP (already configured in public/_headers and middleware.ts).
  • Low-light footage: raise mask contrast and lower threshold so the matte favours darker foreground regions.
  • Laggy preview: disable overlay, reduce quality to Fast, or temporarily mute AI preview via the panel toggle.
  • Green spill: increase edge contract (1–2 px) and apply spill suppression ≈0.4; consider adding a subtle color-correct adjustment downstream.