A File Sharing Linux Application

1004891 Liang Junyi

1004890 Guo Ziniu

1005604 Zhang Chunjie

1004887 Liu Lingao

1005446 Shaun Hin

1005327 Yu Qin

Intro & problem statement

Based on our daily life experience and research, most existing file transfer apps do not support direct transfer data through the clipboard, which makes it difficult for users to transfer data conveniently and efficiently. Also, most existing FTP protocol-based apps lack a file recovery function, an unexpected network connection interruption can disrupt the entire downloading process and cause failure of download.

Hence we decided to design our file sharing system, with clipboard direct-transfer and file recovery features. With these enhancements, users of our app will be able to transfer data more efficiently, and without worrying about potential loss of progress of downloading bulk files due to an unstable network environment.

Related work

• Project scaffolding: electron-vite https://evite.netlify.app/

• FTP:

• • basic-ftp https://www.npmjs.com/package/basic-ftp
  • Ftp-srv https://www.npmjs.com/package/ftp-srv

• SFTP:

• • ssh2-sftp-client https://www.npmjs.com/package/ssh2-sftp-client
  • node-ftp-server https://www.npmjs.com/package/node-sftp-server

• Other:

• • Clipboardy https://www.npmjs.com/package/clipboardy

Approach / methodology

Clipboard & protocol switch

• Direct Transfer from Clipboard

Firstly, a listener is created when the application is running. The listener will periodically examine the clipboard every second. If any content is received on clipboard, an UDP socket will be created to transfer the content to the device with the IP address that it is connected with. Every device running the application should have a listener examined on the clipboard.

• Protocol Switch

When the application starts, FTP and SFTP server will be run together. At the same time, a global variable will be created to store the current protocol state, using FTP by default. Once the client clicks on the protocol switch, the current protocol state will be updated. Hence, every time a client tries to get a file, it will check the current protocol state, based on that, corresponding downloading channel will be selected.

Recovery

Details :

• Before network outage

• 1. Client : create a TCP and FTP connection with server
  2. Server : return file list to client
  3. Client : send download request to server by TCP
  4. Server : read file into byte array
  5. Server : slice the file into smaller parts
  6. Server : inform client about file slices by TCP (header information)
  7. Client : based on header information generate a log file to track the file slices
  8. Client : send file slices download request to server by FTP
  9. Server : return file slices based on client request
  10. Client : received file slices, update the log file

• After network outage

• 1. Client : want to send download request for the large file again
  2. Client : check log file first, check integrity
  3. Client : send not finished file slices download request to server by FTP
  4. Server : return file slices based on client request
  5. Client : received file slices, update the log file
  6. Client : received all file slices, delete the log file, merge the file slices together into original large file

Brief summary about Recovery mechanism :

Client requests the large file, server splits the file into many file slices, client uses FTP to get each file slice instead of the large file itself.

Example header info :

Example log file :

Q & A

1. Why do we choose TCP to send header info, but using FTP to send files since FTP is built on TCP, why not choose UDP since it is faster without hand shake?

2. • Answer :
   • TCP is a protocol that provides reliable, ordered, and error-checked delivery of data between applications running on hosts communicating over an IP network. It establishes a connection between two endpoints, handles data segmentation, and manages flow control and congestion control. TCP ensures that data is delivered to the destination without loss or duplication.
     FTP, on the other hand, is a protocol designed specifically for transferring files between hosts over a network. It runs on top of TCP and uses its reliable data delivery capabilities. FTP provides features like file upload and download, directory listing, and file management. It also includes security mechanisms like authentication and encryption to protect data during transmission.
     So, the reason why we use FTP to send files instead of TCP is that FTP is built on top of TCP and adds additional functionality to it. FTP provides a user-friendly interface for transferring files, manages data transfer and communication with the remote host, and handles file management tasks that TCP doesn’t provide.
     The reason why we use TCP to send header information instead of FTP is that header information is not so much so we do not need to write it into a file and transfer by FTP. Because the information in header information is important that we do not want it to miss any parts so we need to rely on TCP’s reliable, ordered data transfer. So we did not choose UDP.

Code base (if any)

Ftp

Our FTP transfer is based on node.js packages ftp-srv and basic-ftp. The server part is relatively simple, we wrap it up in a separate class and start it on a child process when starting the main application. For the client side code, we also encapsulated the library in our own class, such that we can invoke the api more easily, and it’s more convenient to integrate the recovery function, etc. Within the class, we implemented getFile() to download a file from ftp server, getFileList() to get the file information of the designated directory, and putFIle() to upload a file from client to server.

Those file transfer api are exposed from Electron’s main process to renderer process via the preload script. When a user enters a destination IP, clicks on a folder or on a file, the corresponding event handlers will be triggered, and invoke those functionalities provided in the main process, and do the file transfer operations. And, when a download task finishes, an event will be triggered and the downloaded files list will be refreshed.

Sftp

For our Secure File Transfer Protocol (SFTP) code base, we used a node.js repository ssh2-sftp-client for our client script functions, and node-sftp-server for our server script functions. Through our reading, we implemented and tested the sending of files over the local area network directly from one machine to another machine by private IP address.

Since the protocol runs over a Secure Shell (SSH) session, we deemed that this was secure for our purposes and as the protocol listens on the default port 22 which does not clash with the port 21 used by FTP, we believe SFTP is a suitable alternative to the FTP protocol that is the default mode of transfer.

Our implementation of the code is called through setting the UI radio button for secure transfer, however we were unable to test the speed of transferring over FTP compared to SFTP due to time constraints.

Evaluation / experiment results

Assume

Client requests a file, and this file is splitted by server into 5 parts.

The horizontal axis shows the file received by the client when a network outage happens.

The vertical axis shows the time to re-download this file.

1. The blue line is Without recovery, so no matter how much it downloads before network outage. Client will not check how much content has already been received but request the original file again.

2. The red line is Expected time for recovery, it means when client receives some files for example 2 out of 5, then when client re-download, it only needs re-download for the remaining 3 parts, so time is reduced to 3/5 .

3. The yellow line is Our recovery time, it is different compared to the red line because sending more requests needs more time then just one request. Also Server split files and Client update log file, merge file slices both need some time, just not so much.

4. • At first slower then red line
   • Gradually approx the red line (because we focus on the re-download time, less requests, less time)

Conclusion / future work

We implemented, and debugged the FTP and SFTP protocols for file transfer over local area network (LAN), direct transfer of clipboard over UDP, and a recovery mechanism for resuming the download of large files instead of restarting the download, then compiled the mechanisms into an Electron user interface to package our file transfer application Quick Share. Since our final application could reliably transfer files from one machine to another over LAN, we conclude that our application matches our expectations of making file transfer over LAN convenient and without central servers to facilitate the transfer.

Although we could not implement fixes for edge cases, further improvements can be made by using the hash of files to indicate if the sliced file parts have been transferred so as to mitigate file changes on the server after a network loss event. The cache of files on both the client and server can be cleared in the event of a cancelled download during the network loss event as well, so as to reduce the impact on storage that our application requires.

Github Repository: https://github.com/LiangJunyi-010/NetworkProject