add README

2026-02-04 11:34:44 +08:00 · 2022-10-30 15:52:45 +09:00
parent b9f44bb030
commit 6e37331b4a
1 changed files with 119 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,119 @@
+# mscp
+
+`mscp`, a variant of `scp`, copies files over multiple ssh (sftp)
+sessions. Multiple threads in mscp transfer (1) multiple files
+simultaneously and (2) a large file in parallel.
+
+The usage of mscp is similar to scp, e.g., `mscp example.com:srcfile
+/tmp/dstfile`, except remote glob on remote shell expansion and `-r`
+is not needed on mscp.
+
+## Build
+
+mscp depends on [libssh](https://www.libssh.org/).
+
+- macOS
+
+```console
+brew install libssh
+```
+
+- ubuntu
+
+```console
+sudo apt-get install libssh-dev
+```
+
+Clone and build this repositoy.
+
+```console
+git clone https://github.com/upa/mscp.git
+
+cd mscp
+mkdir build
+cd build
+cmake ..
+make
+```
+
+
+## Run
+
+- Usage
+
+```shell-session
+./mscp -h
+mscp: copy files over multiple ssh connections
+
+Usage: mscp [CvqDdh] [-n nr_conns] [-s min_chunk_sz] [-S max_chunk_sz]
+            [-b sftp_buf_sz] [-B io_buf_sz]
+            [-l login_name] [-p port] [-i identity_file]
+            [-c cipher_spec] source ... target
+
+    -n NR_CONNECTIONS  number of connections (default: half of # of cpu cores)
+    -s MIN_CHUNK_SIZE  min chunk size (default: 64MB)
+    -S MAX_CHUNK_SIZE  max chunk size (default: filesize / nr_conn)
+    -b SFTP_BUF_SIZE   buf size for sftp_read/write (default 131072B)
+    -B IO_BUF_SIZE     buf size for read/write (default 131072B)
+                       Note that this value is derived from
+                       qemu/block/ssh.c. need investigation...
+    -v                 increment verbose output level
+    -q                 disable output
+    -D                 dry run
+
+    -l LOGIN_NAME      login name
+    -p PORT            port number
+    -i IDENTITY        identity file for publickey authentication
+    -c CIPHER          cipher spec, see `ssh -Q cipher`
+    -C                 enable compression on libssh
+    -d                 increment ssh debug output level
+    -h                 print this help
+```
+
+- Copy an 8GB file over a 100Gbls link
+  - Two Intel Xeon Gold 6130 machines directly connected with Intel E810 100Gbps NICs.
+
+```shell-session
+$ ./mscp /tmp/test.img 10.0.0.1:/tmp/
+[===============================================================] 100% 8GB/8GB 3.02GB/s 
+$
+```
+
+- `-v` options increment verbose output level.
+
+```shell-session
+$ ./mscp test 10.0.0.1:
+[===============================================================] 100% 13B/13B 2.41KB/s 
+
+$ ./mscp -v test 10.0.0.1:
+file test/test.txt (local) -> ./test/test.txt (remote) 9B
+file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
+file test/1.txt (local) -> ./test/1.txt (remote) 2B
+copy start: test/test.txt
+copy start: test/1.txt
+copy start: test/test2/2.txt
+copy done: test/1.txt
+copy done: test/test2/2.txt
+copy done: test/test.txt
+[===============================================================] 100% 13B/13B 2.51KB/s 
+
+$ ./mscp -vv -n 4 test 10.0.0.1:
+connecting to 10.0.0.1 for checking destinations...
+file test/test.txt (local) -> ./test/test.txt (remote) 9B
+file test/test2/2.txt (local) -> ./test/test2/2.txt (remote) 2B
+file test/1.txt (local) -> ./test/1.txt (remote) 2B
+connecting to 10.0.0.1 for a copy thread...
+connecting to 10.0.0.1 for a copy thread...
+connecting to 10.0.0.1 for a copy thread...
+connecting to 10.0.0.1 for a copy thread...
+copy start: test/test.txt
+copy start: test/1.txt
+copy start: test/test2/2.txt
+copy done: test/test.txt
+copy done: test/test2/2.txt
+copy done: test/1.txt
+[===============================================================] 100% 13B/13B 3.27KB/s
+```
+
+Note: mscp is still under development, and the author is not
+responsible for any accidents on mscp.