Dunfey · Hotel WWDC as data, est. 1983
Front desk everything
Years
Topics

2024 Photos & CameraSpatial Computing

WWDC24 · 22 min · Photos & Camera / Spatial Computing

Build compelling spatial photo and video experiences

Learn how to adopt spatial photos and videos in your apps. Explore the different types of stereoscopic media and find out how to capture spatial videos in your iOS app on iPhone 15 Pro. Discover the various ways to detect and present spatial media, including the new QuickLook Preview Application API in visionOS. And take a deep dive into the metadata and stereo concepts that make a photo or video spatial.

Watch at developer.apple.com ↗

Transcript all transcripts

Chapters

  • 0:00 — Introduction
  • 1:07 — Types of stereoscopic experiences
  • 4:13 — Tour of the new APIs
  • 13:14 — Deep dive into spatial media formats

Code shown on screen · 5 snippets

Spatial video capture on iPhone 15 Pro swift · at 6:19 ↗
class CaptureManager {
    var session: AVCaptureSession!
    var input: AVCaptureDeviceInput!
    var output: AVCaptureMovieFileOutput!

    func setupSession() throws -> Bool {
        session = AVCaptureSession()
        session.beginConfiguration()

        guard let videoDevice = AVCaptureDevice.default(
            .builtInDualWideCamera, for: .video, position: .back
        ) else { return false }

        var foundSpatialFormat = false
        for format in videoDevice.formats {
            if format.isSpatialVideoCaptureSupported {
                try videoDevice.lockForConfiguration()
                videoDevice.activeFormat = format
                videoDevice.unlockForConfiguration()
                foundSpatialFormat = true
                break
            }
        }
        guard foundSpatialFormat else { return false }

        let videoDeviceInput = try AVCaptureDeviceInput(device: videoDevice)
        guard session.canAddInput(videoDeviceInput) else { return false }
        session.addInput(videoDeviceInput)
        input = videoDeviceInput

        let movieFileOutput = AVCaptureMovieFileOutput()
        guard session.canAddOutput(movieFileOutput) else { return false }
        session.addOutput(movieFileOutput)
        output = movieFileOutput

        guard let connection = output.connection(with: .video) else { return false }
        guard connection.isVideoStabilizationSupported else { return false }
        connection.preferredVideoStabilizationMode = .cinematicExtendedEnhanced

        guard movieFileOutput.isSpatialVideoCaptureSupported else { return false }
        movieFileOutput.isSpatialVideoCaptureEnabled = true

        session.commitConfiguration()
        session.startRunning()
        return true
    }

}
Observing spatial capture discomfort reasons swift · at 9:13 ↗
let observation = videoDevice.observe(\.spatialCaptureDiscomfortReasons) { (device, change) in
    guard let newValue = change.newValue else { return }
    if newValue.contains(.subjectTooClose) {
        // Guide user to move back
    }
    if newValue.contains(.notEnoughLight) {
        // Guide user to find a brighter environment
    }
}
PhotosPicker swift · at 9:58 ↗
import SwiftUI
import PhotosUI

struct PickerView: View {
    @State var selectedItem: PhotosPickerItem?
    var body: some View {
        PhotosPicker(selection: $selectedItem, matching: .spatialMedia) {
            Text("Choose a spatial photo or video")
        }
    }
}
PhotoKit - all spatial assets swift · at 10:14 ↗
import Photos

func fetchSpatialAssets() {
    let fetchOptions = PHFetchOptions()
    fetchOptions.predicate = NSPredicate(
        format: "(mediaSubtypes & %d) != 0",
        argumentArray: [PHAssetMediaSubtype.spatialMedia.rawValue]
    )
    fetchResult = PHAsset.fetchAssets(with: fetchOptions)
}
AVAssetPlaybackAssistant swift · at 10:36 ↗
import AVFoundation

extension AVURLAsset {
    func isSpatialVideo() async -> Bool {
        let assistant = AVAssetPlaybackAssistant(asset: self)
        let options = await assistant.playbackConfigurationOptions
        return options.contains(.spatialVideo)
    }
}

Resources