mirror of
https://github.com/upa/mscp.git
synced 2026-02-14 09:14:47 +08:00
1479607efe
doc/mscp.rst is generate from mscp.1 by make generate-mscp-rst. README is also updateded to reference doc/mscp.rst.
108 lines
3.4 KiB
Markdown
108 lines
3.4 KiB
Markdown
# mscp: multi-threaded scp
|
||
|
||
[](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [](https://github.com/upa/mscp/actions/workflows/test.yml)
|
||
|
||
|
||
`mscp`, a variant of `scp`, copies files over multiple SSH (SFTP)
|
||
connections by multiple threads. It enables transferring (1) multiple
|
||
files simultaneously and (2) a large file in parallel, reducing the
|
||
transfer time for a lot of/large files over networks.
|
||
|
||
You can use `mscp` like `scp`, for example:
|
||
|
||
```shell-session
|
||
$ mscp user@example.com:srcfile /tmp/dstfile
|
||
```
|
||
|
||
Remote hosts only need to run standard `sshd` supporting the SFTP
|
||
subsystem (e.g. openssh-server), and you need to be able to ssh to the
|
||
hosts as usual. `mscp` does not require anything else.
|
||
|
||
|
||
https://github.com/upa/mscp/assets/184632/19230f57-be7f-4ef0-98dd-cb4c460f570d
|
||
|
||
--------------------------------------------------------------------
|
||
|
||
Differences from `scp` on usage:
|
||
|
||
- Remote-to-remote copy is not supported.
|
||
- `-r` option is not needed to transfer directories.
|
||
- and any other differences I have not implemented and noticed.
|
||
|
||
Paper:
|
||
- Ryo Nakamura and Yohei Kuga. 2023. Multi-threaded scp: Easy and Fast File Transfer over SSH. In Practice and Experience in Advanced Research Computing (PEARC '23). Association for Computing Machinery, New York, NY, USA, 320–323. https://doi.org/10.1145/3569951.3597582
|
||
|
||
## Install
|
||
|
||
- macOS
|
||
|
||
```console
|
||
brew install upa/tap/mscp
|
||
```
|
||
|
||
- Ubuntu
|
||
```console
|
||
sudo add-apt-repository ppa:upaa/mscp
|
||
sudo apt-get install mscp
|
||
```
|
||
|
||
- Rocky 8.8
|
||
```console
|
||
yum install https://github.com/upa/mscp/releases/latest/download/mscp_rocky-8.8-x86_64.rpm
|
||
```
|
||
|
||
- Alma 8.8
|
||
```console
|
||
yum install https://github.com/upa/mscp/releases/latest/download/mscp_almalinux-8.8-x86_64.rpm
|
||
```
|
||
|
||
- Linux with single binary `mscp` (x86_64 only, and not optimal performance)
|
||
```console
|
||
wget https://github.com/upa/mscp/releases/latest/download/mscp.linux.x86.static -O /usr/local/bin/mscp
|
||
chmod 755 /usr/local/bin/mscp
|
||
```
|
||
|
||
|
||
## Build
|
||
|
||
mscp depends on a patched [libssh](https://www.libssh.org/). The
|
||
patch introduces asynchronous SFTP Write, which is derived from
|
||
https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write
|
||
async](https://archive.libssh.org/libssh/2020-06/0000004.html)).
|
||
|
||
Currently macOS and Linux (Ubuntu, Rocky and Alma) are supported.
|
||
|
||
```console
|
||
# clone this repository
|
||
git clone https://github.com/upa/mscp.git
|
||
cd mscp
|
||
|
||
# prepare patched libssh
|
||
git submodule update --init
|
||
patch -d libssh -p1 < patch/$(git --git-dir=./libssh/.git describe).patch
|
||
|
||
# install build dependency
|
||
bash ./scripts/install-build-deps.sh
|
||
|
||
# configure mscp
|
||
mkdir build && cd build
|
||
cmake ..
|
||
|
||
# in macOS, you may need OPENSSL_ROOT_DIR for cmake:
|
||
# cmake .. -DOPENSSL_ROOT_DIR=$(brew --prefix)/opt/openssl@1.1
|
||
|
||
# build
|
||
make
|
||
|
||
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
|
||
make install
|
||
```
|
||
|
||
Source tar balls (`mscp-X.X.X.tar.gz`, not `Source code`) in
|
||
[Releases page](https://github.com/upa/mscp/releases) contains the patched version
|
||
of libssh. So you can start from cmake with it.
|
||
|
||
|
||
## Documentation
|
||
|
||
[manpage](/doc/mscp.rst) is available. |