Dunfey · Hotel WWDC as data, est. 1983
Front desk everything
Years
Topics

WWDC25 · 17 min · Photos & Camera / Audio & Video

Enhance your app with machine-learning-based video effects

Discover how to add effects like frame rate conversion, super resolution, and noise filtering to improve video editing and live streaming experiences. We’ll explore the ML-based video processing algorithms optimized for Apple Silicon available in the Video Toolbox framework. Learn how to integrate these effects to enhance the capabilities of your app for real-world use cases.

Watch at developer.apple.com ↗

Transcript all transcripts

Chapters

  • 0:00 — Introduction
  • 1:04 — Video effects overview
  • 4:55 — Add effects to your app
  • 6:46 — Example use: frame rate conversion
  • 11:16 — Example use: motion blur effect
  • 13:46 — Example use: low latency super resolution effect
  • 14:55 — Example use: low latency frame interpolation

Code shown on screen · 4 snippets

Frame rate conversion configuration swift · at 8:06 ↗ 
// Frame rate conversion configuration


let processor = VTFrameProcessor()

guard let configuration = VTFrameRateConversionConfiguration(frameWidth: width,
                                                            frameHeight: height,
                                                     usePrecomputedFlow: false,
                                                  qualityPrioritization: .normal,
                                                               revision: .revision1)
else {
     throw Fault.failedToCreateFRCConfiguration
}

try processor.startSession(configuration: configuration)
Frame rate conversion buffer allocation swift · at 8:56 ↗ 
// Frame rate conversion buffer allocation

//use sourcePixelBufferAttributes and destinationPixelBufferAttributes property of VTFrameRateConversionConfiguration to create source and destination CVPixelBuffer pools

sourceFrame = VTFrameProcessorFrame(buffer: curPixelBuffer, presentationTimeStamp: sourcePTS)
nextFrame = VTFrameProcessorFrame(buffer: nextPixelBuffer, presentationTimeStamp: nextPTS)

// Interpolate 3 frames between reference frames for 4x slow-mo
var interpolationPhase: [Float] = [0.25, 0.5, 0.75]

//create destinationFrames
let destinationFrames = try framesBetween(firstPTS: sourcePTS,
                                           lastPTS: nextPTS,
                            interpolationIntervals: intervals)
Frame rate conversion parameters swift · at 9:48 ↗ 
// Frame rate conversion parameters

guard let parameters = VTFrameRateConversionParameters(sourceFrame: sourceFrame,
                                                         nextFrame: nextFrame,
                                                       opticalFlow: nil,
                                                interpolationPhase: interpolationPhase,
                                                    submissionMode: .sequential,
                                                 destinationFrames: destinationFrames)
else {
     throw Fault.failedToCreateFRCParameters
}

try await processor.process(parameters: parameters)
Motion blur process parameters swift · at 12:35 ↗ 
// Motion blur process parameters

sourceFrame = VTFrameProcessorFrame(buffer: curPixelBuffer, presentationTimeStamp: sourcePTS)
nextFrame = VTFrameProcessorFrame(buffer: nextPixelBuffer, presentationTimeStamp: nextPTS)
previousFrame = VTFrameProcessorFrame(buffer: prevPixelBuffer, presentationTimeStamp: prevPTS)
destinationFrame = VTFrameProcessorFrame(buffer: destPixelBuffer, presentationTimeStamp: sourcePTS)

guard let parameters = VTMotionBlurParameters(sourceFrame: currentFrame,
                                                nextFrame: nextFrame,
                                            previousFrame: previousFrame,
                                          nextOpticalFlow: nil,
                                      previousOpticalFlow: nil,
                                       motionBlurStrength: strength,
                                           submissionMode: .sequential,
                                         destinationFrame: destinationFrame) 
else {
    throw Fault.failedToCreateMotionBlurParameters
}

try await processor.process(parameters: parameters)

Resources