Ma/mscp
1
0
Fork 0
mirror of https://github.com/upa/mscp.git synced 2026-02-04 03:24:58 +08:00
Code Issues Packages Projects Releases 1 Wiki Activity

add manpage for mscp

Browse Source 
doc/mscp.rst is generate from mscp.1 by make generate-mscp-rst.
README is also updateded to reference doc/mscp.rst.
This commit is contained in:
Ryo Nakamura
2024-01-13 19:06:56 +09:00
parent 6f4038a480
commit 1479607efe
5 changed files with 569 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
README.md
View File

@@ -3,17 +3,20 @@
[![build on ubuntu](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [![build on macOS](https://github.com/upa/mscp/actions/workflows/build-macos.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [![test](https://github.com/upa/mscp/actions/workflows/test.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/test.yml)
`mscp`, a variant of `scp`, copies files over multiple ssh (SFTP)
connections. Multiple threads and connections in mscp transfer (1)
multiple files simultaneously and (2) a large file in parallel. It
would shorten the waiting time for transferring a lot of/large files
over networks.
`mscp`, a variant of `scp`, copies files over multiple SSH (SFTP)
connections by multiple threads. It enables transferring (1) multiple
files simultaneously and (2) a large file in parallel, reducing the
transfer time for a lot of/large files over networks.
You can use `mscp` like `scp`, for example, `mscp
user@example.com:srcfile /tmp/dstfile`. Remote hosts only need to run
standard `sshd` supporting the SFTP subsystem (e.g. openssh-server),
and you need to be able to ssh to the hosts as usual. `mscp` does not
require anything else.
You can use `mscp` like `scp`, for example:
```shell-session
$ mscp user@example.com:srcfile /tmp/dstfile
```
Remote hosts only need to run standard `sshd` supporting the SFTP
subsystem (e.g. openssh-server), and you need to be able to ssh to the
hosts as usual. `mscp` does not require anything else.
https://github.com/upa/mscp/assets/184632/19230f57-be7f-4ef0-98dd-cb4c460f570d
@@ -62,7 +65,7 @@ chmod 755 /usr/local/bin/mscp
## Build
mscp depends on a patched [libssh](https://www.libssh.org/). The
mscp depends on a patched [libssh](https://www.libssh.org/). The
patch introduces asynchronous SFTP Write, which is derived from
https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write
async](https://archive.libssh.org/libssh/2020-06/0000004.html)).
@@ -94,105 +97,12 @@ make
# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install
```
Source tar balls (`mscp-X.X.X.tar.gz`, not `Source code`) in
[Releases page](https://github.com/upa/mscp/releases) contains the patched version
of libssh. So you can start from cmake with it.
## Run
- Usage
## Documentation
```console
$ mscp
mscp v0.0.8: copy files over multiple ssh connections
Usage: mscp [vqDHdNh] [-n nr_conns] [-m coremask] [-u max_startups]
[-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz]
[-l login_name] [-p port] [-i identity_file]
[-c cipher_spec] [-M hmac_spec] [-C compress] source ... target
```
- Example: copy a 15GB file on memory over a 100Gbps link
- Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs.
- Default `openssh-server` runs on the remote host.
```console
$ mscp /var/ram/test.img 10.0.0.1:/var/ram/
[======================================] 100% 15GB/15GB 1.7GB/s 00:00 ETA
```
```console
# with some optimizations. top speed reaches 3.0GB/s.
$ mscp -n 5 -m 0x1f -c aes128-gcm@openssh.com /var/ram/test.img 10.0.0.1:/var/ram/
[======================================] 100% 15GB/15GB 2.4GB/s 00:00 ETA
```
- `-v` option increments verbose output level.
```console
$ mscp test 10.0.0.1:
[=======================================] 100% 49B /49B 198.8B/s 00:00 ETA
```
```console
$ mscp -vv test 10.0.0.1:
file: test/test1 -> ./test/test1
file: test/testdir/asdf -> ./test/testdir/asdf
file: test/testdir/qwer -> ./test/testdir/qwer
file: test/test2 -> ./test/test2
we have only 4 chunk(s). set number of connections to 4
connecting to localhost for a copy thread...
connecting to localhost for a copy thread...
connecting to localhost for a copy thread...
copy start: test/test1
copy start: test/test2
copy start: test/testdir/asdf
copy start: test/testdir/qwer
copy done: test/test1
copy done: test/test2
copy done: test/testdir/qwer
copy done: test/testdir/asdf
[=======================================] 100% 49B /49B 198.1B/s 00:00 ETA
```
- Full usage
```console
$ mscp -h
mscp v0.0.9-11-g5802679: copy files over multiple ssh connections
Usage: mscp [vqDHdNh] [-n nr_conns] [-m coremask] [-u max_startups]
[-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz]
[-l login_name] [-p port] [-F ssh_config] [-i identity_file]
[-c cipher_spec] [-M hmac_spec] [-C compress] source ... target
-n NR_CONNECTIONS number of connections (default: floor(log(cores)*2)+1)
-m COREMASK hex value to specify cores where threads pinned
-u MAX_STARTUPS number of concurrent outgoing connections (default: 8)
-s MIN_CHUNK_SIZE min chunk size (default: 64MB)
-S MAX_CHUNK_SIZE max chunk size (default: filesize/nr_conn)
-a NR_AHEAD number of inflight SFTP commands (default: 32)
-b BUF_SZ buffer size for i/o and transfer
-v increment verbose output level
-q disable output
-D dry run. check copy destinations with -vvv
-r no effect
-l LOGIN_NAME login name
-p PORT port number
-F CONFIG path to user ssh config (default ~/.ssh/config)
-i IDENTITY identity file for public key authentication
-c CIPHER cipher spec
-M HMAC hmac spec
-C COMPRESS enable compression: yes, no, zlib, zlib@openssh.com
-H disable hostkey check
-d increment ssh debug output level
-N enable Nagle's algorithm (default disabled)
-h print this help
```
Note: mscp is still under development, and the author is not
responsible for any accidents due to mscp.
[manpage](/doc/mscp.rst) is available.