mscp: multi-threaded scp

mscp, a variant of scp, copies files over multiple SSH (SFTP) connections by multiple threads. It enables transferring (1) multiple files simultaneously and (2) a large file in parallel, reducing the transfer time for a lot of/large files over networks.

You can use mscp like scp, for example:

$ mscp user@example.com:srcfile /tmp/dstfile

Remote hosts only need to run standard sshd supporting the SFTP subsystem (e.g. openssh-server), and you need to be able to ssh to the hosts as usual. mscp does not require anything else.

https://github.com/upa/mscp/assets/184632/19230f57-be7f-4ef0-98dd-cb4c460f570d

---

Differences from scp on usage:

• Remote-to-remote copy is not supported.
• -r option is not needed to transfer directories.
• and any other differences I have not implemented and noticed.

Paper:

• Ryo Nakamura and Yohei Kuga. 2023. Multi-threaded scp: Easy and Fast File Transfer over SSH. In Practice and Experience in Advanced Research Computing (PEARC '23). Association for Computing Machinery, New York, NY, USA, 320–323. https://doi.org/10.1145/3569951.3597582

Install

• macOS

brew install upa/tap/mscp

• Ubuntu

sudo add-apt-repository ppa:upaa/mscp
sudo apt-get install mscp

• Rocky 8.8

yum install https://github.com/upa/mscp/releases/latest/download/mscp_rocky-8.8-x86_64.rpm

• Alma 8.8

yum install https://github.com/upa/mscp/releases/latest/download/mscp_almalinux-8.8-x86_64.rpm

• Linux with single binary mscp (x86_64 only, and not optimal performance)

wget https://github.com/upa/mscp/releases/latest/download/mscp.linux.x86.static -O /usr/local/bin/mscp
chmod 755 /usr/local/bin/mscp

Build

mscp depends on a patched libssh. The patch introduces asynchronous SFTP Write, which is derived from https://github.com/limes-datentechnik-gmbh/libssh (see Re: SFTP Write async).

Currently macOS and Linux (Ubuntu, Rocky and Alma) are supported.

# clone this repository
git clone https://github.com/upa/mscp.git
cd mscp

# prepare patched libssh
git submodule update --init
patch -d libssh -p1 < patch/$(git --git-dir=./libssh/.git describe).patch

# install build dependency
bash ./scripts/install-build-deps.sh

# configure mscp
mkdir build && cd build
cmake ..

# in macOS, you may need OPENSSL_ROOT_DIR for cmake:
# cmake .. -DOPENSSL_ROOT_DIR=$(brew --prefix)/opt/openssl@1.1

# build
make

# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install

Source tar balls (mscp-X.X.X.tar.gz, not Source code) in Releases page contains the patched version of libssh. So you can start from cmake with it.

Documentation

manpage is available.