ec663cc966
We have already provided DEB packages in launchpad PPA and RPM packages in COPR. Thus, we need no more deb/rpm packages in Github releases. The single binary build of mscp is an execptio. Updated container image versions: - almalinux 8.8 -> 9.3 - rocky 8.8 -> 8.9, and 9.3 is added - alpine 3.17 -> 3.19
mscp: multi-threaded scp
mscp, a variant of scp, copies files over multiple SSH (SFTP)
connections by multiple threads. It enables transferring (1) multiple
files simultaneously and (2) a large file in parallel, reducing the
transfer time for a lot of/large files over networks.
You can use mscp like scp, for example:
$ mscp user@example.com:srcfile /tmp/dstfile
Remote hosts only need to run standard sshd supporting the SFTP
subsystem (e.g. openssh-server), and you need to be able to ssh to the
hosts as usual. mscp does not require anything else.
https://github.com/upa/mscp/assets/184632/19230f57-be7f-4ef0-98dd-cb4c460f570d
Differences from scp on usage:
- Remote-to-remote copy is not supported.
-roption is not needed to transfer directories.- and any other differences I have not implemented and noticed.
Paper:
- Ryo Nakamura and Yohei Kuga. 2023. Multi-threaded scp: Easy and Fast File Transfer over SSH. In Practice and Experience in Advanced Research Computing (PEARC '23). Association for Computing Machinery, New York, NY, USA, 320–323. https://doi.org/10.1145/3569951.3597582
Install
- macOS
brew install upa/tap/mscp
- Ubuntu
sudo add-apt-repository ppa:upaa/mscp
sudo apt-get install mscp
- RHEL-based distributions
sudo dnf copr enable upaaa/mscp
sudo dnf install mscp
- Single binary
mscpfor x86_64 (not optimal performance)
wget https://github.com/upa/mscp/releases/latest/download/mscp.linux.x86.static -O /usr/local/bin/mscp
chmod 755 /usr/local/bin/mscp
Build
mscp depends on a patched libssh. The patch introduces asynchronous SFTP Write, which is derived from https://github.com/limes-datentechnik-gmbh/libssh (see Re: SFTP Write async).
We test building mscp on Linux (Ubuntu, Rocky, Alma, and Alpine), macOS, and FreeBSD.
# clone this repository
git clone https://github.com/upa/mscp.git
cd mscp
# prepare patched libssh
git submodule update --init
patch -d libssh -p1 < patch/$(git --git-dir=./libssh/.git describe).patch
# install build dependency
bash ./scripts/install-build-deps.sh
# configure mscp
mkdir build && cd build
cmake ..
# in macOS, you may need OPENSSL_ROOT_DIR for cmake:
# cmake .. -DOPENSSL_ROOT_DIR=$(brew --prefix)/opt/openssl@1.1
# build
make
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install
Source tar balls (mscp-X.X.X.tar.gz, not Source code) in
Releases page contains the patched version
of libssh. So you can start from cmake with it.
Documentation
manpage is available.