Photos and the many other apps with video show playback correctly whatever the recording camera orientation. Is there a simple way to accomplish the correct rotation for playback?
After a lot of digging in stackOverflow and Apple docs I finally got the orientation of video playback to work – but it feels too difficult for something that is done all the time.
Here is my solution – is there a better way?
My app is locked to landscape full screen orientation only. Following the scheme for video to metal rendering with filters as outlined in the WWDC22 “Display HDR video in EDR with AVFoundation and Metal”, each frame of video should be converted into a CIImage for input into CIFilters.
There are two steps:
- Determine the orientation of the recording camera
- Apply a corrective rotation for the playback
StackOverFlow has multiple answers for orientation of an AVAsset – the best answer seems to be from How to detect if a video file was recorded in portrait orientation, or landscape in iOS where dhallman provides a function to answer both UIInterfaceOrientation and AVCaptureDevicePosition. Extract AVAsset Orientation and Camera Position
The need for an app function to determine UIInterfaceOrientation and AVCaptureDevicePosition seems wrong – Is this hidden in the metadata somewhere? One of the other issues that hobbled me is the iOS16 change to make loading of the videoTracks and the preferredTransform an async call. So in this extension of AVAsset a function
func videoOrientation() async -> PGLDevicePosition {
var orientation: UIInterfaceOrientation = .unknown
var device: AVCaptureDevice.Position = .unspecified
var myVideoTracks:[AVAssetTrack]?
var t: CGAffineTransform = CGAffineTransformIdentity
do {
myVideoTracks = try await loadTracks(withMediaType: .video)
}
catch {
/// return init values of .unknown and .unspecificed
return PGLDevicePosition(orientation: orientation, device: device)
}
if let videoTrack = myVideoTracks?.first {
do {
t = try await videoTrack.load(.preferredTransform)
}
catch {
/// return init values of .unknown and .unspecificed
return PGLDevicePosition(orientation: orientation, device: device)
}
The async function needs to wrapped into a task such as this
func getVideoPreferredTransform(callBack: @escaping (PGLDevicePosition) -> Void ) {
Task {
let devicePosition = await avPlayerItem.asset.videoOrientation()
callBack(devicePosition)
}
}
Now knowing the device .front or .back and orientation a switch statement is determine the correct CGImagePropertyOrientation for the AffineTransform for the CIImage. The switch statement is
var result = CGImagePropertyOrientation.up
// default
switch (imageOrientation.orientation, imageOrientation.device) {
case (.unknown,.unspecified) :
result = CGImagePropertyOrientation.up
case (.portrait, .front) :
result = CGImagePropertyOrientation.right
case (.portraitUpsideDown, .front):
result = CGImagePropertyOrientation.right
case (.landscapeLeft, .front) :
result = CGImagePropertyOrientation.up
case (.landscapeRight, .front) :
result = CGImagePropertyOrientation.up
case (.portrait, .back) :
result = CGImagePropertyOrientation.right
case (.portraitUpsideDown, .back):
result = CGImagePropertyOrientation.left
case (.landscapeLeft, .back) :
result = CGImagePropertyOrientation.down
case (.landscapeRight, .back) :
result = CGImagePropertyOrientation.up
default:
return result // default .up
}
return result
Finally, the correction to the ciImage can be made. First the CIImage is converted from the cvPixelBuffer as suggested in the WWDC22 workshop. Then transform with the correct CGImagePropertyOrientation is applied to the CIImage.
let sourceFrame = CIImage(cvPixelBuffer: buffer)
let neededTransform = sourceFrame.orientationTransform(for: videoPropertyOrientation)
videoCIFrame = sourceFrame.transformed(by: neededTransform)
And we are done… Seems way too complicated.. Isn’t there a simpler way???
