The landing of Swarm has allowed us to bring many new features to Jami users, most notably the synchronization of conversation histories, but it has also given us the opportunity to improve existing features such as file transfer.

In this article we will examine the older file transfer system and its limits, and then see how the new swarm file transfer improves upon it.

Limits of the older file transfer system

The previous file transfer system in Jami was far from perfect. To transfer a file to a peer, the sender's device would open a socket with all of the peer's devices, and the peer would need to accept or decline the sender's request on each. So, the strategy was to push the file onto the peer's devices, which had several drawbacks. First, because of the peer needing to approve the transfer request to receive the file and the fact that sending the file to the devices was done all at once, any device that was offline at that moment would never receive the file. Second, the sender would only have the file on one device (the one it was sent from), because the history was not synchronized across devices. Third, if there were any connectivity issues on the sender or the receiver side, the file transfer would completely fail. This made for a suboptimal user experience, because file transfer was not robust, hard to follow, and manual retries were necessary every so often.

The new approach

The new file transfer system being based on swarm, it brings several improvements by default over the previous approach:

the history is synchronized across all devices of conversation participants, so all devices in the conversation can see the transfer request; and
if the original sender is offline, any device that has received a copy of the file can send the file to the other devices when they become online.

Under the hood, the approaches and implementations changed quite a bit. Instead of pushing files to other devices, the other devices will now pull the file if they want to get a copy. Now based on swarm, file transfer still has a major difference with the other messages in a conversation: not every device may want to download all the files in a conversation. A practical example of this would be in conversations with large file transfers where some devices have small storage capacity.

In essence, the following is how the new file transfer process works:

To send a file, the sender will add a new message to the conversation to inform the other members that a file is available.
Peers will know that a new file is available for transfer/download upon receiving this new interaction, and they can then choose whether to download it or not. Users can always choose to not receive files, for instance to avoid filling up the storage space on their device.
After clicking on download, the device will ask other devices currently online in the conversation to send the file. If no other device is currently connected, it will wait until a member of the conversation comes back online and ask for the file again.
The sender then opens a channel with the receiver to transmit the file. If the transfer fails at this step — for instance due to overly restrictive firewalls or other network issues — the transfer will be retried with the next member(s) until the file is successfully transfered.

This approach allows us to:

transmit files to all other devices in a conversation, including other potential devices of the sender and any other devices that may not be online when the file is initially made available for transfer;
never have "unjoinable peers" or failed transfers, because the daemon automatically handles failures and retries transfers when necessary;
allow the original sender to go offline once the file has been transferred to at least one other device, because other peers will then be able to pull the file from the other device(s) that have already received a copy of the file; and
this method is also compatible with conversations with more than one peer, allowing us to support file transfer in swarms with multiple participants.

What's next for file transfers?

In this article we described Jami's new file transfer system, and the improvements it brings over the previous system and if you want to read more about this subject, the documentation is here. But this is not all, and there is yet more to come for file transfers in the future. Though this new approach is much better than the previous one, given the distributed nature and goals of Jami, the best case would be the scenario where multiple devices could interact with each other. For now, the new file transfer system works with one device transferring the file to one another. Our long-term goal and vision with the new file transfer system is to implement and support better transfer protocols and libraries — like torrents with libtorrent — for use in a distributed setting, where a file could be downloaded from multiple sources at the same time, by simultaneously downloading different fragments of it from multiple peers, for improved resiliency and potential increases in download speed.

By Sébastien Blin, Amin Bandali