Ma/mscp
1
0
Fork 0
mirror of https://github.com/upa/mscp.git synced 2026-02-18 20:34:59 +08:00
Code Issues Packages Projects Releases 1 Wiki Activity
341 Commits 5 Branches 22 Tags
d6f437bcb165992deb32050f7d27f250ab9fe115
Commit Graph

176 Commits

Author SHA1 Message Date
Ryo Nakamura
3bce4ec277 set m->tid_prepare 0 to avoid duble join 2023-03-15 21:56:46 +09:00
Ryo Nakamura
a923d40ada mscp: add -u max_startups option. 
pymscp also accepts keyword 'max_startups' (int).
 2023-03-15 21:53:34 +09:00
Ryo Nakamura
24fef5f539 fix: when msg_fd is 0, use STDOUT_FILENO 2023-03-15 01:35:55 +09:00
Ryo Nakamura
4e80b05da7 do not fdopen(msg_fd) if msg_fd < 0 2023-03-15 00:39:56 +09:00
Ryo Nakamura
98eca409af introduce semaphore for concurrent connecting ssh 
instead of ssh_estab_queue (delay-based approach). MaxStartups in
sshd_config limits number of conccurent incoming ssh connections.
mscp_opts->max_startups adjusts this value.
 2023-03-15 00:35:48 +09:00
Ryo Nakamura
cf99a439cb cleanup message print functions 2023-03-15 00:00:23 +09:00
Ryo Nakamura
3077bb0856 rename ssh_connect_flag to ssh_estab_queue 2023-03-14 01:20:55 +09:00
Ryo Nakamura
72c27f16d6 implement ssh_connect_flag 
Each copy thread establishes SSH/SFTP connection to remote host.
A delay is inserted between SSH connecting to the remote.
 2023-03-14 00:43:53 +09:00
Ryo Nakamura
9b0eb668f9 cleanup mscp_prepare-related code 2023-03-14 00:11:13 +09:00
Ryo Nakamura
5f9f20f150 mscp_prepare() scans source paths in a thread. 
This commit runs mscp_prepare() in a pthread. mscp copy threads
run aysnchronously with mscp_prepare(). So, when mscp_prepare()
has not finished yet (due to too many source files), we can start
to copy files.
 2023-03-13 22:35:51 +09:00
Ryo Nakamura
ceb9ebd5a8 revise walk_src_path. 
In new walk_src_path, resolve dst path and resolve chunks are
invoked when adding a path.
 2023-03-13 21:02:26 +09:00
Ryo Nakamura
cafbefe48c fix mscp.py 2023-03-13 14:53:54 +09:00
Ryo Nakamura
597a7a8cba little cleanup for python bindings 2023-03-12 17:39:51 +09:00
Ryo Nakamura
2416b5f182 fix cleanup 2023-03-12 17:06:02 +09:00
Ryo Nakamura
10812071aa mscp copy via python bindings works. 
ToDo: memory for struct instance in pymscp.c is not released until
mscp.free() is called. It is memory leak in a typical pyhotn usage.
Use python extension refcnt instead.
 2023-03-12 00:05:39 +09:00
Ryo Nakamura
8ea05729c2 add mscp and pymscp python modules. 
pymscp is a C Python wrapper for libmscp functions. mscp module
provides simple (?) Python API.
 2023-03-11 22:49:19 +09:00
Ryo Nakamura
74d58e986a move direction from mscp_opts to mscp_init argument 2023-03-11 21:48:07 +09:00
Ryo Nakamura
7e7bc61ff2 start to implement pymscp.c 2023-03-11 20:54:45 +09:00
Ryo Nakamura
2477647a3b fix uninitialized dst_path_is_dir bool 2023-03-10 21:42:00 +09:00
Ryo Nakamura
309371ed75 now make install installs libmscp 2023-03-10 21:10:06 +09:00
Ryo Nakamura
d766b3a99e fix dryrun handling on main.c 2023-03-04 19:01:44 +09:00
Ryo Nakamura
d5a86292b7 add doxygen for mscp.h 2023-03-04 18:47:44 +09:00
Ryo Nakamura
cc18c74d32 remove sigalrm. integrate print messages and progress bar 2023-03-04 17:37:19 +09:00
Ryo Nakamura
205c7cf803 tiny fix for clang warning 2023-03-04 16:50:34 +09:00
Ryo Nakamura
e67b7468e5 use setitimer instead of alarm, and print message. 
print_stat now prints messages per interval.
ToDo:
- realtime message printing
- use timer_create instead of setitimer (mscOS has different one)
 2023-03-04 16:48:26 +09:00
Ryo Nakamura
f9c8dec389 compilable on ubuntu 2023-03-04 15:53:54 +09:00
Ryo Nakamura
9342c18f0e accidentaly swap min_chunk_sz and max_chunk_sz... 2023-03-04 15:53:42 +09:00
Ryo Nakamura
df2f922b0d remove pprint, use message (mpr_*) instead. 
ToDo:
main should use pipe to receive messages from libmscp.
 2023-03-04 15:44:10 +09:00
Ryo Nakamura
1e57e8fb2f implementing messaging. 
ToDo: remove pprint.
mscp should use mpr_* functions, and main.c should use
just fprintf(stdout, "\r\033[K" fmt, ...) for printing progress bar.
 2023-03-03 22:14:54 +09:00
Ryo Nakamura
1b9ae51974 add message.h and message.c, mscp_set|get_error() 
Instead of pr_err(), libmscp uses mscp_set_error() and
applications use mscp_get_errror() to get error message.
 2023-03-03 21:29:43 +09:00
Ryo Nakamura
c5aa70d9c9 tiny cleanup 2023-03-03 18:30:34 +09:00
Ryo Nakamura
a0b7482f66 add mscp_get_stats 
move progress bar-related functions from mscp.c to main.c.
 2023-03-03 18:27:14 +09:00
Ryo Nakamura
363296f499 add mscp_ssh_opts and change -C optarg 2023-03-03 16:50:06 +09:00
Ryo Nakamura
a8af79f9cf remove test.c 2023-02-27 10:57:10 +09:00
Ryo Nakamura
fc45fa2532 add comments to mscp.h 2023-02-26 23:56:57 +09:00
Ryo Nakamura
ca0ea3ee77 tiny fix on comment 2023-02-26 23:46:53 +09:00
Ryo Nakamura
c649742b3e fix dst path resolve 2023-02-26 23:42:25 +09:00
Ryo Nakamura
700d64b375 now mscp links libmscp 2023-02-26 23:18:39 +09:00
Ryo Nakamura
2bad21bdc2 set default params in mscp_init 2023-02-26 18:43:24 +09:00
Ryo Nakamura
89777032cd have written mscp.c 2023-02-26 18:17:58 +09:00
Ryo Nakamura
3d26cc2c18 add copy-related functions to path 2023-02-25 23:39:20 +09:00
Ryo Nakamura
1be9b70808 start to impliment mscp as a library 
this commit starts to refactor file.h|c to path.h|c and
add mscp.c|h. not completed yet.
 2023-02-25 22:17:29 +09:00
Ryo Nakamura
dbc96f9166 add -r option (but no effect) 2023-02-16 18:04:16 +09:00
Ryo Nakamura
fd5c6e971e add \n at the final output 2023-02-16 17:59:26 +09:00
Ryo Nakamura
c18fb6996f remove unnecessary pr_warn 2023-01-05 21:37:50 +09:00
Ryo Nakamura
3d98451bba set default nr_threads to floor(log(cores) * 2) + 1) 
This change prevents mscp from establishing too many ssh connections
on many-core machines in default.
 2022-12-11 14:01:52 +09:00
Ryo Nakamura
d27db01d8d use pthread_cleanup to acquire and release lock 
In chunk_prepare(), if multiple threads wait for acquiring f->lock,
and then pthread_cancel() is called, the waiting threads are never
canceled because pthread_mutex_lock() is not a cancellation point.
So, use pthread_cleanup_push/pop to release the lock.
 2022-12-11 13:23:41 +09:00
Ryo Nakamura
6ae3f0f9f1 set default NR_AHEAD to 32 2022-12-08 18:01:50 +09:00
Ryo Nakamura
847c80276a fix final progress output 2022-12-06 20:04:04 +09:00
Ryo Nakamura
c4ea9a1e78 add ssh_buffer_new_size and ssh_buffer_add_func to libssh 
sftp_async_write() with these functions reduces
  1. realloc_buffer by ssh_buffer_new_size()
  2. memcpy from read data to ssh buffer by ssh_buffer_add_func()
 2022-12-06 15:02:14 +09:00