In recent weeks, we worked on improving various parts of the Jami user experience and one sticking point was video orientation on mobile devices. Until now, users had to manually press a button in order to change it, which broke their expectations and could take several seconds.
Our previous approach for managing video orientation was to change it directly on the recording device, requiring the whole media pipeline to be reinitialized. This caused glitches and frame drops in addition to creating an interruption in the users’ conversation which had a negative impact on their experience.
This technical limitation was due to the fact that decoders cannot change video resolution on the fly, and switching the orientation is a resolution change (width x height becomes height x width). Video rotation also requires significant resources, which are scarce on mobile devices especially during video calls. In some cases with hardware-accelerated encoding, the system provides an encoded H.264 or VP8 stream that cannot easily be rotated.
When looking at industry practices, we noticed that instead of the video being rotated on the recording device, orientation is increasingly stored as metadata which allows the video to be rotated at display time. This has the advantage of being very cost effective in terms of performance, thanks to a simple display matrix applied by the GPU. For instance, video files recorded with modern smartphones are saved with the native sensor’s orientation and include orientation metadata that allows the video to be properly displayed.
We applied this technique by sending orientation information to the peer during calls using SIP messages. At its core, this information is associated to each connected camera, so that the proper orientation is applied even when switching camera. Sending orientation metadata was implemented in Jami for both Android and iOS, and displaying rotated video is available on every supported platforms. This feature is exposed by Jami core APIs, allowing to easily build projects including rotated cameras.
For Jami users, this improvement provides a seamless experience on mobile when changing the device orientation.
By Adrien Béraud
