26 Commits

Author SHA1 Message Date
Ryo Nakamura
2ef4267f72 bump version to 0.1.4 2024-02-07 16:16:40 +09:00
Ryo Nakamura
1d8d853b30 fix manpage to add -g option 2024-02-07 15:14:29 +09:00
Ryo Nakamura
b2628b54fb add -4 and -6 options to use either IPv4 or IPv6 2024-02-07 15:14:27 +09:00
Ryo Nakamura
a9c59f744a update man and doc for adjusting to -p option 2024-02-07 14:32:36 +09:00
Ryo Nakamura
96084004b6 cleanup error message handling
The top-level funtion in a thread should print errors using
priv_get_err(), while lower-level functions should set error messages
using priv_set_err() except that error mesesages should be printed
immediately, e.g., under walk_src_path().
2024-02-07 13:29:45 +09:00
Ryo Nakamura
5119d5ae26 remove unused FindGSSAPI.cmake module 2024-02-06 23:24:57 +09:00
Ryo Nakamura
93f155cffe add .clang-format from Linux kernel and format sources
The exception is that ColumnLimit is 90.
2024-02-06 23:09:59 +09:00
Ryo Nakamura
4f0669f8f8 refactor error message-related functions
split message print fuctions (mpr_*), strerrno, and mscp_get/set_error
into print.c/h and strerrno.c/h.

ToDo: revise usages of priv_set_errv and pr_* functions.
2024-02-06 21:54:04 +09:00
Ryo Nakamura
76892a69f9 do not build strlcat.c if HAVE_STRLCAT is true 2024-02-06 16:25:05 +09:00
Ryo Nakamura
a7f8ad948b add -p option, preserving file timestamps 2024-02-06 16:15:43 +09:00
Ryo Nakamura
ff45d9d71b add two env vars to pass password/keyphrase (#9)
MSCP_SSH_AUTH_PASSWORD passes a password, and MSCP_SSH_AUTH_PASSPHRASE
passes a passphrase for publickey auth. They enable avoiding interactive
password input. Test cases are also added.
2024-02-06 10:35:38 +09:00
Ryo Nakamura
9908fb309d passing options via pointers in mscp_ssh_opts
We do not need static buf because we have already dropped python biding
support.
2024-02-05 23:36:59 +09:00
Ryo Nakamura
c95e6a4fff do not passing msg_fp via mscp opts
instead, mpr_* functions print messages to stdout or stderr directly.
2024-02-04 20:23:08 +09:00
Ryo Nakamura
304e71d5a0 remove (accidentally added) build-srpm.Dockerfile 2024-02-04 17:37:57 +09:00
Ryo Nakamura
ec663cc966 bump up container image versions and drop using CPack
We have already provided DEB packages in launchpad PPA and RPM packages
in COPR. Thus, we need no more deb/rpm packages in Github releases. The
single binary build of mscp is an execptio.

Updated container image versions:
- almalinux	8.8  -> 9.3
- rocky		8.8  -> 8.9, and 9.3 is added
- alpine	3.17 -> 3.19
2024-02-04 17:22:31 +09:00
Ryo Nakamura
d57ed4149d drop python binding support 2024-02-04 16:18:27 +09:00
Ryo Nakamura
7f5fcc617c add doc/DEVELOP.md instead of doc/README.md 2024-02-04 12:05:18 +09:00
Ryo Nakamura
36e45cd5f3 change build target update-mscp-rst to update-rst 2024-02-04 11:50:14 +09:00
Ryo Nakamura
25207cf865 build deb sources inside a container
A new build target `build-deb` builds mscp deb inside a docker
container. All debuild -us -uc outputs are copied to build/debbuild
directory.
2024-02-04 11:35:20 +09:00
Ryo Nakamura
306a00ced5 build source rpm inside a container
A new build target `build-srpm` builds mscp src.rpm inside a docker
container. The src.rpm can be published at COPR.
2024-02-04 10:58:11 +09:00
Ryo Nakamura
65025b9022 update README for building on FreeBSD 2024-01-20 22:09:02 +09:00
Ryo Nakamura
bbc2fbdacf fix install-build-deps.sh to run on FreeBSD 2024-01-20 22:05:39 +09:00
Ryo Nakamura
331bab6b41 run build on freebsd action in dev and main branches 2024-01-20 21:42:44 +09:00
Ryo Nakamura
1e375abb88 add github actions to build mscp in FreeBSD 2024-01-21 05:56:53 +09:00
Ryo Nakamura
7ab5cc3ce9 make it buildable at FreeBSD 2024-01-21 04:50:34 +09:00
Ryo Nakamura
60977a8e7a update doc version
and remove --dirty from git-describe-based versioning
2024-01-18 13:52:26 +09:00
67 changed files with 2576 additions and 2831 deletions

745
.clang-format Normal file
View File

@@ -0,0 +1,745 @@
# SPDX-License-Identifier: GPL-2.0
#
# clang-format configuration file. Intended for clang-format >= 11.
#
# For more information, see:
#
# Documentation/process/clang-format.rst
# https://clang.llvm.org/docs/ClangFormat.html
# https://clang.llvm.org/docs/ClangFormatStyleOptions.html
#
# clang-format configuration for Linux kernel, except that ColumnLimit is 90
---
AccessModifierOffset: -4
AlignAfterOpenBracket: Align
AlignConsecutiveAssignments: false
AlignConsecutiveDeclarations: false
AlignEscapedNewlines: Left
AlignOperands: true
AlignTrailingComments: false
AllowAllParametersOfDeclarationOnNextLine: false
AllowShortBlocksOnASingleLine: false
AllowShortCaseLabelsOnASingleLine: false
AllowShortFunctionsOnASingleLine: None
AllowShortIfStatementsOnASingleLine: false
AllowShortLoopsOnASingleLine: false
AlwaysBreakAfterDefinitionReturnType: None
AlwaysBreakAfterReturnType: None
AlwaysBreakBeforeMultilineStrings: false
AlwaysBreakTemplateDeclarations: false
BinPackArguments: true
BinPackParameters: true
BraceWrapping:
AfterClass: false
AfterControlStatement: false
AfterEnum: false
AfterFunction: true
AfterNamespace: true
AfterObjCDeclaration: false
AfterStruct: false
AfterUnion: false
AfterExternBlock: false
BeforeCatch: false
BeforeElse: false
IndentBraces: false
SplitEmptyFunction: true
SplitEmptyRecord: true
SplitEmptyNamespace: true
BreakBeforeBinaryOperators: None
BreakBeforeBraces: Custom
BreakBeforeInheritanceComma: false
BreakBeforeTernaryOperators: false
BreakConstructorInitializersBeforeComma: false
BreakConstructorInitializers: BeforeComma
BreakAfterJavaFieldAnnotations: false
BreakStringLiterals: false
ColumnLimit: 90
CommentPragmas: '^ IWYU pragma:'
CompactNamespaces: false
ConstructorInitializerAllOnOneLineOrOnePerLine: false
ConstructorInitializerIndentWidth: 8
ContinuationIndentWidth: 8
Cpp11BracedListStyle: false
DerivePointerAlignment: false
DisableFormat: false
ExperimentalAutoDetectBinPacking: false
FixNamespaceComments: false
# Taken from:
# git grep -h '^#define [^[:space:]]*for_each[^[:space:]]*(' include/ tools/ \
# | sed "s,^#define \([^[:space:]]*for_each[^[:space:]]*\)(.*$, - '\1'," \
# | LC_ALL=C sort -u
ForEachMacros:
- '__ata_qc_for_each'
- '__bio_for_each_bvec'
- '__bio_for_each_segment'
- '__evlist__for_each_entry'
- '__evlist__for_each_entry_continue'
- '__evlist__for_each_entry_from'
- '__evlist__for_each_entry_reverse'
- '__evlist__for_each_entry_safe'
- '__for_each_mem_range'
- '__for_each_mem_range_rev'
- '__for_each_thread'
- '__hlist_for_each_rcu'
- '__map__for_each_symbol_by_name'
- '__pci_bus_for_each_res0'
- '__pci_bus_for_each_res1'
- '__pci_dev_for_each_res0'
- '__pci_dev_for_each_res1'
- '__perf_evlist__for_each_entry'
- '__perf_evlist__for_each_entry_reverse'
- '__perf_evlist__for_each_entry_safe'
- '__rq_for_each_bio'
- '__shost_for_each_device'
- '__sym_for_each'
- 'apei_estatus_for_each_section'
- 'ata_for_each_dev'
- 'ata_for_each_link'
- 'ata_qc_for_each'
- 'ata_qc_for_each_raw'
- 'ata_qc_for_each_with_internal'
- 'ax25_for_each'
- 'ax25_uid_for_each'
- 'bio_for_each_bvec'
- 'bio_for_each_bvec_all'
- 'bio_for_each_folio_all'
- 'bio_for_each_integrity_vec'
- 'bio_for_each_segment'
- 'bio_for_each_segment_all'
- 'bio_list_for_each'
- 'bip_for_each_vec'
- 'bond_for_each_slave'
- 'bond_for_each_slave_rcu'
- 'bpf_for_each'
- 'bpf_for_each_reg_in_vstate'
- 'bpf_for_each_reg_in_vstate_mask'
- 'bpf_for_each_spilled_reg'
- 'bpf_object__for_each_map'
- 'bpf_object__for_each_program'
- 'btree_for_each_safe128'
- 'btree_for_each_safe32'
- 'btree_for_each_safe64'
- 'btree_for_each_safel'
- 'card_for_each_dev'
- 'cgroup_taskset_for_each'
- 'cgroup_taskset_for_each_leader'
- 'cpu_aggr_map__for_each_idx'
- 'cpufreq_for_each_efficient_entry_idx'
- 'cpufreq_for_each_entry'
- 'cpufreq_for_each_entry_idx'
- 'cpufreq_for_each_valid_entry'
- 'cpufreq_for_each_valid_entry_idx'
- 'css_for_each_child'
- 'css_for_each_descendant_post'
- 'css_for_each_descendant_pre'
- 'damon_for_each_region'
- 'damon_for_each_region_from'
- 'damon_for_each_region_safe'
- 'damon_for_each_scheme'
- 'damon_for_each_scheme_safe'
- 'damon_for_each_target'
- 'damon_for_each_target_safe'
- 'damos_for_each_filter'
- 'damos_for_each_filter_safe'
- 'data__for_each_file'
- 'data__for_each_file_new'
- 'data__for_each_file_start'
- 'device_for_each_child_node'
- 'displayid_iter_for_each'
- 'dma_fence_array_for_each'
- 'dma_fence_chain_for_each'
- 'dma_fence_unwrap_for_each'
- 'dma_resv_for_each_fence'
- 'dma_resv_for_each_fence_unlocked'
- 'do_for_each_ftrace_op'
- 'drm_atomic_crtc_for_each_plane'
- 'drm_atomic_crtc_state_for_each_plane'
- 'drm_atomic_crtc_state_for_each_plane_state'
- 'drm_atomic_for_each_plane_damage'
- 'drm_client_for_each_connector_iter'
- 'drm_client_for_each_modeset'
- 'drm_connector_for_each_possible_encoder'
- 'drm_exec_for_each_locked_object'
- 'drm_exec_for_each_locked_object_reverse'
- 'drm_for_each_bridge_in_chain'
- 'drm_for_each_connector_iter'
- 'drm_for_each_crtc'
- 'drm_for_each_crtc_reverse'
- 'drm_for_each_encoder'
- 'drm_for_each_encoder_mask'
- 'drm_for_each_fb'
- 'drm_for_each_legacy_plane'
- 'drm_for_each_plane'
- 'drm_for_each_plane_mask'
- 'drm_for_each_privobj'
- 'drm_gem_for_each_gpuva'
- 'drm_gem_for_each_gpuva_safe'
- 'drm_gpuva_for_each_op'
- 'drm_gpuva_for_each_op_from_reverse'
- 'drm_gpuva_for_each_op_safe'
- 'drm_gpuvm_for_each_va'
- 'drm_gpuvm_for_each_va_range'
- 'drm_gpuvm_for_each_va_range_safe'
- 'drm_gpuvm_for_each_va_safe'
- 'drm_mm_for_each_hole'
- 'drm_mm_for_each_node'
- 'drm_mm_for_each_node_in_range'
- 'drm_mm_for_each_node_safe'
- 'dsa_switch_for_each_available_port'
- 'dsa_switch_for_each_cpu_port'
- 'dsa_switch_for_each_cpu_port_continue_reverse'
- 'dsa_switch_for_each_port'
- 'dsa_switch_for_each_port_continue_reverse'
- 'dsa_switch_for_each_port_safe'
- 'dsa_switch_for_each_user_port'
- 'dsa_tree_for_each_cpu_port'
- 'dsa_tree_for_each_user_port'
- 'dsa_tree_for_each_user_port_continue_reverse'
- 'dso__for_each_symbol'
- 'dsos__for_each_with_build_id'
- 'elf_hash_for_each_possible'
- 'elf_symtab__for_each_symbol'
- 'evlist__for_each_cpu'
- 'evlist__for_each_entry'
- 'evlist__for_each_entry_continue'
- 'evlist__for_each_entry_from'
- 'evlist__for_each_entry_reverse'
- 'evlist__for_each_entry_safe'
- 'flow_action_for_each'
- 'for_each_acpi_consumer_dev'
- 'for_each_acpi_dev_match'
- 'for_each_active_dev_scope'
- 'for_each_active_drhd_unit'
- 'for_each_active_iommu'
- 'for_each_active_route'
- 'for_each_aggr_pgid'
- 'for_each_and_bit'
- 'for_each_andnot_bit'
- 'for_each_available_child_of_node'
- 'for_each_bench'
- 'for_each_bio'
- 'for_each_board_func_rsrc'
- 'for_each_btf_ext_rec'
- 'for_each_btf_ext_sec'
- 'for_each_bvec'
- 'for_each_card_auxs'
- 'for_each_card_auxs_safe'
- 'for_each_card_components'
- 'for_each_card_dapms'
- 'for_each_card_pre_auxs'
- 'for_each_card_prelinks'
- 'for_each_card_rtds'
- 'for_each_card_rtds_safe'
- 'for_each_card_widgets'
- 'for_each_card_widgets_safe'
- 'for_each_cgroup_storage_type'
- 'for_each_child_of_node'
- 'for_each_clear_bit'
- 'for_each_clear_bit_from'
- 'for_each_clear_bitrange'
- 'for_each_clear_bitrange_from'
- 'for_each_cmd'
- 'for_each_cmsghdr'
- 'for_each_collection'
- 'for_each_comp_order'
- 'for_each_compatible_node'
- 'for_each_component_dais'
- 'for_each_component_dais_safe'
- 'for_each_conduit'
- 'for_each_console'
- 'for_each_console_srcu'
- 'for_each_cpu'
- 'for_each_cpu_and'
- 'for_each_cpu_andnot'
- 'for_each_cpu_or'
- 'for_each_cpu_wrap'
- 'for_each_dapm_widgets'
- 'for_each_dedup_cand'
- 'for_each_dev_addr'
- 'for_each_dev_scope'
- 'for_each_dma_cap_mask'
- 'for_each_dpcm_be'
- 'for_each_dpcm_be_rollback'
- 'for_each_dpcm_be_safe'
- 'for_each_dpcm_fe'
- 'for_each_drhd_unit'
- 'for_each_dss_dev'
- 'for_each_efi_memory_desc'
- 'for_each_efi_memory_desc_in_map'
- 'for_each_element'
- 'for_each_element_extid'
- 'for_each_element_id'
- 'for_each_endpoint_of_node'
- 'for_each_event'
- 'for_each_event_tps'
- 'for_each_evictable_lru'
- 'for_each_fib6_node_rt_rcu'
- 'for_each_fib6_walker_rt'
- 'for_each_free_mem_pfn_range_in_zone'
- 'for_each_free_mem_pfn_range_in_zone_from'
- 'for_each_free_mem_range'
- 'for_each_free_mem_range_reverse'
- 'for_each_func_rsrc'
- 'for_each_gpiochip_node'
- 'for_each_group_evsel'
- 'for_each_group_evsel_head'
- 'for_each_group_member'
- 'for_each_group_member_head'
- 'for_each_hstate'
- 'for_each_if'
- 'for_each_inject_fn'
- 'for_each_insn'
- 'for_each_insn_prefix'
- 'for_each_intid'
- 'for_each_iommu'
- 'for_each_ip_tunnel_rcu'
- 'for_each_irq_nr'
- 'for_each_lang'
- 'for_each_link_codecs'
- 'for_each_link_cpus'
- 'for_each_link_platforms'
- 'for_each_lru'
- 'for_each_matching_node'
- 'for_each_matching_node_and_match'
- 'for_each_media_entity_data_link'
- 'for_each_mem_pfn_range'
- 'for_each_mem_range'
- 'for_each_mem_range_rev'
- 'for_each_mem_region'
- 'for_each_member'
- 'for_each_memory'
- 'for_each_migratetype_order'
- 'for_each_missing_reg'
- 'for_each_mle_subelement'
- 'for_each_mod_mem_type'
- 'for_each_net'
- 'for_each_net_continue_reverse'
- 'for_each_net_rcu'
- 'for_each_netdev'
- 'for_each_netdev_continue'
- 'for_each_netdev_continue_rcu'
- 'for_each_netdev_continue_reverse'
- 'for_each_netdev_dump'
- 'for_each_netdev_feature'
- 'for_each_netdev_in_bond_rcu'
- 'for_each_netdev_rcu'
- 'for_each_netdev_reverse'
- 'for_each_netdev_safe'
- 'for_each_new_connector_in_state'
- 'for_each_new_crtc_in_state'
- 'for_each_new_mst_mgr_in_state'
- 'for_each_new_plane_in_state'
- 'for_each_new_plane_in_state_reverse'
- 'for_each_new_private_obj_in_state'
- 'for_each_new_reg'
- 'for_each_node'
- 'for_each_node_by_name'
- 'for_each_node_by_type'
- 'for_each_node_mask'
- 'for_each_node_state'
- 'for_each_node_with_cpus'
- 'for_each_node_with_property'
- 'for_each_nonreserved_multicast_dest_pgid'
- 'for_each_numa_hop_mask'
- 'for_each_of_allnodes'
- 'for_each_of_allnodes_from'
- 'for_each_of_cpu_node'
- 'for_each_of_pci_range'
- 'for_each_old_connector_in_state'
- 'for_each_old_crtc_in_state'
- 'for_each_old_mst_mgr_in_state'
- 'for_each_old_plane_in_state'
- 'for_each_old_private_obj_in_state'
- 'for_each_oldnew_connector_in_state'
- 'for_each_oldnew_crtc_in_state'
- 'for_each_oldnew_mst_mgr_in_state'
- 'for_each_oldnew_plane_in_state'
- 'for_each_oldnew_plane_in_state_reverse'
- 'for_each_oldnew_private_obj_in_state'
- 'for_each_online_cpu'
- 'for_each_online_node'
- 'for_each_online_pgdat'
- 'for_each_or_bit'
- 'for_each_path'
- 'for_each_pci_bridge'
- 'for_each_pci_dev'
- 'for_each_pcm_streams'
- 'for_each_physmem_range'
- 'for_each_populated_zone'
- 'for_each_possible_cpu'
- 'for_each_present_blessed_reg'
- 'for_each_present_cpu'
- 'for_each_prime_number'
- 'for_each_prime_number_from'
- 'for_each_probe_cache_entry'
- 'for_each_process'
- 'for_each_process_thread'
- 'for_each_prop_codec_conf'
- 'for_each_prop_dai_codec'
- 'for_each_prop_dai_cpu'
- 'for_each_prop_dlc_codecs'
- 'for_each_prop_dlc_cpus'
- 'for_each_prop_dlc_platforms'
- 'for_each_property_of_node'
- 'for_each_reg'
- 'for_each_reg_filtered'
- 'for_each_reloc'
- 'for_each_reloc_from'
- 'for_each_requested_gpio'
- 'for_each_requested_gpio_in_range'
- 'for_each_reserved_mem_range'
- 'for_each_reserved_mem_region'
- 'for_each_rtd_codec_dais'
- 'for_each_rtd_components'
- 'for_each_rtd_cpu_dais'
- 'for_each_rtd_dais'
- 'for_each_sband_iftype_data'
- 'for_each_script'
- 'for_each_sec'
- 'for_each_set_bit'
- 'for_each_set_bit_from'
- 'for_each_set_bit_wrap'
- 'for_each_set_bitrange'
- 'for_each_set_bitrange_from'
- 'for_each_set_clump8'
- 'for_each_sg'
- 'for_each_sg_dma_page'
- 'for_each_sg_page'
- 'for_each_sgtable_dma_page'
- 'for_each_sgtable_dma_sg'
- 'for_each_sgtable_page'
- 'for_each_sgtable_sg'
- 'for_each_sibling_event'
- 'for_each_sta_active_link'
- 'for_each_subelement'
- 'for_each_subelement_extid'
- 'for_each_subelement_id'
- 'for_each_sublist'
- 'for_each_subsystem'
- 'for_each_supported_activate_fn'
- 'for_each_supported_inject_fn'
- 'for_each_sym'
- 'for_each_test'
- 'for_each_thread'
- 'for_each_token'
- 'for_each_unicast_dest_pgid'
- 'for_each_valid_link'
- 'for_each_vif_active_link'
- 'for_each_vma'
- 'for_each_vma_range'
- 'for_each_vsi'
- 'for_each_wakeup_source'
- 'for_each_zone'
- 'for_each_zone_zonelist'
- 'for_each_zone_zonelist_nodemask'
- 'func_for_each_insn'
- 'fwnode_for_each_available_child_node'
- 'fwnode_for_each_child_node'
- 'fwnode_for_each_parent_node'
- 'fwnode_graph_for_each_endpoint'
- 'gadget_for_each_ep'
- 'genradix_for_each'
- 'genradix_for_each_from'
- 'genradix_for_each_reverse'
- 'hash_for_each'
- 'hash_for_each_possible'
- 'hash_for_each_possible_rcu'
- 'hash_for_each_possible_rcu_notrace'
- 'hash_for_each_possible_safe'
- 'hash_for_each_rcu'
- 'hash_for_each_safe'
- 'hashmap__for_each_entry'
- 'hashmap__for_each_entry_safe'
- 'hashmap__for_each_key_entry'
- 'hashmap__for_each_key_entry_safe'
- 'hctx_for_each_ctx'
- 'hists__for_each_format'
- 'hists__for_each_sort_list'
- 'hlist_bl_for_each_entry'
- 'hlist_bl_for_each_entry_rcu'
- 'hlist_bl_for_each_entry_safe'
- 'hlist_for_each'
- 'hlist_for_each_entry'
- 'hlist_for_each_entry_continue'
- 'hlist_for_each_entry_continue_rcu'
- 'hlist_for_each_entry_continue_rcu_bh'
- 'hlist_for_each_entry_from'
- 'hlist_for_each_entry_from_rcu'
- 'hlist_for_each_entry_rcu'
- 'hlist_for_each_entry_rcu_bh'
- 'hlist_for_each_entry_rcu_notrace'
- 'hlist_for_each_entry_safe'
- 'hlist_for_each_entry_srcu'
- 'hlist_for_each_safe'
- 'hlist_nulls_for_each_entry'
- 'hlist_nulls_for_each_entry_from'
- 'hlist_nulls_for_each_entry_rcu'
- 'hlist_nulls_for_each_entry_safe'
- 'i3c_bus_for_each_i2cdev'
- 'i3c_bus_for_each_i3cdev'
- 'idr_for_each_entry'
- 'idr_for_each_entry_continue'
- 'idr_for_each_entry_continue_ul'
- 'idr_for_each_entry_ul'
- 'in_dev_for_each_ifa_rcu'
- 'in_dev_for_each_ifa_rtnl'
- 'inet_bind_bucket_for_each'
- 'interval_tree_for_each_span'
- 'intlist__for_each_entry'
- 'intlist__for_each_entry_safe'
- 'kcore_copy__for_each_phdr'
- 'key_for_each'
- 'key_for_each_safe'
- 'klp_for_each_func'
- 'klp_for_each_func_safe'
- 'klp_for_each_func_static'
- 'klp_for_each_object'
- 'klp_for_each_object_safe'
- 'klp_for_each_object_static'
- 'kunit_suite_for_each_test_case'
- 'kvm_for_each_memslot'
- 'kvm_for_each_memslot_in_gfn_range'
- 'kvm_for_each_vcpu'
- 'libbpf_nla_for_each_attr'
- 'list_for_each'
- 'list_for_each_codec'
- 'list_for_each_codec_safe'
- 'list_for_each_continue'
- 'list_for_each_entry'
- 'list_for_each_entry_continue'
- 'list_for_each_entry_continue_rcu'
- 'list_for_each_entry_continue_reverse'
- 'list_for_each_entry_from'
- 'list_for_each_entry_from_rcu'
- 'list_for_each_entry_from_reverse'
- 'list_for_each_entry_lockless'
- 'list_for_each_entry_rcu'
- 'list_for_each_entry_reverse'
- 'list_for_each_entry_safe'
- 'list_for_each_entry_safe_continue'
- 'list_for_each_entry_safe_from'
- 'list_for_each_entry_safe_reverse'
- 'list_for_each_entry_srcu'
- 'list_for_each_from'
- 'list_for_each_prev'
- 'list_for_each_prev_safe'
- 'list_for_each_rcu'
- 'list_for_each_reverse'
- 'list_for_each_safe'
- 'llist_for_each'
- 'llist_for_each_entry'
- 'llist_for_each_entry_safe'
- 'llist_for_each_safe'
- 'lwq_for_each_safe'
- 'map__for_each_symbol'
- 'map__for_each_symbol_by_name'
- 'maps__for_each_entry'
- 'maps__for_each_entry_safe'
- 'mas_for_each'
- 'mci_for_each_dimm'
- 'media_device_for_each_entity'
- 'media_device_for_each_intf'
- 'media_device_for_each_link'
- 'media_device_for_each_pad'
- 'media_entity_for_each_pad'
- 'media_pipeline_for_each_entity'
- 'media_pipeline_for_each_pad'
- 'mlx5_lag_for_each_peer_mdev'
- 'msi_domain_for_each_desc'
- 'msi_for_each_desc'
- 'mt_for_each'
- 'nanddev_io_for_each_page'
- 'netdev_for_each_lower_dev'
- 'netdev_for_each_lower_private'
- 'netdev_for_each_lower_private_rcu'
- 'netdev_for_each_mc_addr'
- 'netdev_for_each_synced_mc_addr'
- 'netdev_for_each_synced_uc_addr'
- 'netdev_for_each_uc_addr'
- 'netdev_for_each_upper_dev_rcu'
- 'netdev_hw_addr_list_for_each'
- 'nft_rule_for_each_expr'
- 'nla_for_each_attr'
- 'nla_for_each_nested'
- 'nlmsg_for_each_attr'
- 'nlmsg_for_each_msg'
- 'nr_neigh_for_each'
- 'nr_neigh_for_each_safe'
- 'nr_node_for_each'
- 'nr_node_for_each_safe'
- 'of_for_each_phandle'
- 'of_property_for_each_string'
- 'of_property_for_each_u32'
- 'pci_bus_for_each_resource'
- 'pci_dev_for_each_resource'
- 'pcl_for_each_chunk'
- 'pcl_for_each_segment'
- 'pcm_for_each_format'
- 'perf_config_items__for_each_entry'
- 'perf_config_sections__for_each_entry'
- 'perf_config_set__for_each_entry'
- 'perf_cpu_map__for_each_cpu'
- 'perf_cpu_map__for_each_idx'
- 'perf_evlist__for_each_entry'
- 'perf_evlist__for_each_entry_reverse'
- 'perf_evlist__for_each_entry_safe'
- 'perf_evlist__for_each_evsel'
- 'perf_evlist__for_each_mmap'
- 'perf_hpp_list__for_each_format'
- 'perf_hpp_list__for_each_format_safe'
- 'perf_hpp_list__for_each_sort_list'
- 'perf_hpp_list__for_each_sort_list_safe'
- 'perf_tool_event__for_each_event'
- 'plist_for_each'
- 'plist_for_each_continue'
- 'plist_for_each_entry'
- 'plist_for_each_entry_continue'
- 'plist_for_each_entry_safe'
- 'plist_for_each_safe'
- 'pnp_for_each_card'
- 'pnp_for_each_dev'
- 'protocol_for_each_card'
- 'protocol_for_each_dev'
- 'queue_for_each_hw_ctx'
- 'radix_tree_for_each_slot'
- 'radix_tree_for_each_tagged'
- 'rb_for_each'
- 'rbtree_postorder_for_each_entry_safe'
- 'rdma_for_each_block'
- 'rdma_for_each_port'
- 'rdma_umem_for_each_dma_block'
- 'resort_rb__for_each_entry'
- 'resource_list_for_each_entry'
- 'resource_list_for_each_entry_safe'
- 'rhl_for_each_entry_rcu'
- 'rhl_for_each_rcu'
- 'rht_for_each'
- 'rht_for_each_entry'
- 'rht_for_each_entry_from'
- 'rht_for_each_entry_rcu'
- 'rht_for_each_entry_rcu_from'
- 'rht_for_each_entry_safe'
- 'rht_for_each_from'
- 'rht_for_each_rcu'
- 'rht_for_each_rcu_from'
- 'rq_for_each_bvec'
- 'rq_for_each_segment'
- 'rq_list_for_each'
- 'rq_list_for_each_safe'
- 'sample_read_group__for_each'
- 'scsi_for_each_prot_sg'
- 'scsi_for_each_sg'
- 'sctp_for_each_hentry'
- 'sctp_skb_for_each'
- 'sec_for_each_insn'
- 'sec_for_each_insn_continue'
- 'sec_for_each_insn_from'
- 'sec_for_each_sym'
- 'shdma_for_each_chan'
- 'shost_for_each_device'
- 'sk_for_each'
- 'sk_for_each_bound'
- 'sk_for_each_bound_bhash2'
- 'sk_for_each_entry_offset_rcu'
- 'sk_for_each_from'
- 'sk_for_each_rcu'
- 'sk_for_each_safe'
- 'sk_nulls_for_each'
- 'sk_nulls_for_each_from'
- 'sk_nulls_for_each_rcu'
- 'snd_array_for_each'
- 'snd_pcm_group_for_each_entry'
- 'snd_soc_dapm_widget_for_each_path'
- 'snd_soc_dapm_widget_for_each_path_safe'
- 'snd_soc_dapm_widget_for_each_sink_path'
- 'snd_soc_dapm_widget_for_each_source_path'
- 'strlist__for_each_entry'
- 'strlist__for_each_entry_safe'
- 'sym_for_each_insn'
- 'sym_for_each_insn_continue_reverse'
- 'symbols__for_each_entry'
- 'tb_property_for_each'
- 'tcf_act_for_each_action'
- 'tcf_exts_for_each_action'
- 'ttm_resource_manager_for_each_res'
- 'twsk_for_each_bound_bhash2'
- 'udp_portaddr_for_each_entry'
- 'udp_portaddr_for_each_entry_rcu'
- 'usb_hub_for_each_child'
- 'v4l2_device_for_each_subdev'
- 'v4l2_m2m_for_each_dst_buf'
- 'v4l2_m2m_for_each_dst_buf_safe'
- 'v4l2_m2m_for_each_src_buf'
- 'v4l2_m2m_for_each_src_buf_safe'
- 'virtio_device_for_each_vq'
- 'while_for_each_ftrace_op'
- 'xa_for_each'
- 'xa_for_each_marked'
- 'xa_for_each_range'
- 'xa_for_each_start'
- 'xas_for_each'
- 'xas_for_each_conflict'
- 'xas_for_each_marked'
- 'xbc_array_for_each_value'
- 'xbc_for_each_key_value'
- 'xbc_node_for_each_array_value'
- 'xbc_node_for_each_child'
- 'xbc_node_for_each_key_value'
- 'xbc_node_for_each_subkey'
- 'zorro_for_each_dev'
IncludeBlocks: Preserve
IncludeCategories:
- Regex: '.*'
Priority: 1
IncludeIsMainRegex: '(Test)?$'
IndentCaseLabels: false
IndentGotoLabels: false
IndentPPDirectives: None
IndentWidth: 8
IndentWrappedFunctionNames: false
JavaScriptQuotes: Leave
JavaScriptWrapImports: true
KeepEmptyLinesAtTheStartOfBlocks: false
MacroBlockBegin: ''
MacroBlockEnd: ''
MaxEmptyLinesToKeep: 1
NamespaceIndentation: None
ObjCBinPackProtocolList: Auto
ObjCBlockIndentWidth: 8
ObjCSpaceAfterProperty: true
ObjCSpaceBeforeProtocolList: true
# Taken from git's rules
PenaltyBreakAssignment: 10
PenaltyBreakBeforeFirstCallParameter: 30
PenaltyBreakComment: 10
PenaltyBreakFirstLessLess: 0
PenaltyBreakString: 10
PenaltyExcessCharacter: 100
PenaltyReturnTypeOnItsOwnLine: 60
PointerAlignment: Right
ReflowComments: false
SortIncludes: false
SortUsingDeclarations: false
SpaceAfterCStyleCast: false
SpaceAfterTemplateKeyword: true
SpaceBeforeAssignmentOperators: true
SpaceBeforeCtorInitializerColon: true
SpaceBeforeInheritanceColon: true
SpaceBeforeParens: ControlStatementsExceptForEachMacros
SpaceBeforeRangeBasedForLoopColon: true
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 1
SpacesInAngles: false
SpacesInContainerLiterals: false
SpacesInCStyleCastParentheses: false
SpacesInParentheses: false
SpacesInSquareBrackets: false
Standard: Cpp03
TabWidth: 8
UseTab: Always
...

27
.github/workflows/build-freebsd.yml vendored Normal file
View File

@@ -0,0 +1,27 @@
name: build on FreeBSD
on:
push:
branches: [ "main", "dev" ]
pull_request:
branches: [ "main", "dev" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Build in FreeBSD
uses: vmactions/freebsd-vm@v1
with:
prepare: |
pkg install -y git cmake
run: |
patch -d libssh -p1 < patch/libssh-0.10.6-2-g6f1b1e76.patch
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
build/mscp -h

View File

@@ -33,22 +33,14 @@ jobs:
- name: Test
run: make -C ${{github.workspace}}/build docker-test-all
- name: Retrieve packages from containers
run: make -C ${{github.workspace}}/build docker-pkg-all
- name: Copy mscp single binary built inside Alpine
run: cp ${{github.workspace}}/build/mscp_alpine-3.17-x86_64.static ${{github.workspace}}/build/mscp.linux.x86.static
- name: Build single binary mscp
run: make -C ${{github.workspace}}/build build-single-binary
- name: Release
uses: softprops/action-gh-release@v1
with:
files: |
${{github.workspace}}/build/mscp_ubuntu-20.04-x86_64.deb
${{github.workspace}}/build/mscp_ubuntu-22.04-x86_64.deb
${{github.workspace}}/build/mscp_rocky-8.8-x86_64.rpm
${{github.workspace}}/build/mscp_almalinux-8.8-x86_64.rpm
${{github.workspace}}/build/mscp_alpine-3.17-x86_64.static
${{github.workspace}}/build/mscp.linux.x86.static
${{github.workspace}}/build/mscp.linux.x86_64.static
source-release:
runs-on: ubuntu-latest

4
.gitignore vendored
View File

@@ -5,7 +5,3 @@ CMakeUserPresets.json
.*.swp
include/mscp_version.h
dist
*.egg-info
__pycache__

View File

@@ -11,7 +11,7 @@ find_package(Git)
if (Git_FOUND)
# based on https://github.com/nocnokneo/cmake-git-versioning-example
execute_process(
COMMAND ${GIT_EXECUTABLE} describe --tags --dirty --match "v*"
COMMAND ${GIT_EXECUTABLE} describe --tags --match "v*"
OUTPUT_VARIABLE GIT_DESCRIBE_VERSION
RESULT_VARIABLE GIT_DESCRIBE_ERROR_CODE
OUTPUT_STRIP_TRAILING_WHITESPACE)
@@ -52,8 +52,6 @@ if (BUILD_STATIC)
endif()
option(INSTALL_EXECUTABLE_ONLY OFF) # do not install libmscp
# add libssh static library
set(CMAKE_POLICY_DEFAULT_CMP0077 NEW)
@@ -71,12 +69,12 @@ add_subdirectory(libssh EXCLUDE_FROM_ALL)
# setup mscp compile options
set(MSCP_COMPILE_OPTS -iquote ${CMAKE_CURRENT_BINARY_DIR}/libssh/include)
set(MSCP_BUILD_INCLUDE_DIRS
list(APPEND MSCP_COMPILE_OPTS -iquote ${CMAKE_CURRENT_BINARY_DIR}/libssh/include)
list(APPEND MSCP_BUILD_INCLUDE_DIRS
${mscp_SOURCE_DIR}/src
${CMAKE_CURRENT_BINARY_DIR}/libssh/include)
set(MSCP_LINK_LIBS ssh-static)
list(APPEND MSCP_LINK_LIBS ssh-static)
if(BUILD_CONAN)
find_package(ZLIB REQUIRED)
find_package(OpenSSL REQUIRED)
@@ -85,33 +83,24 @@ if(BUILD_CONAN)
endif()
# generate version header file
configure_file(
${mscp_SOURCE_DIR}/include/mscp_version.h.in
${mscp_SOURCE_DIR}/include/mscp_version.h)
# libmscp.so
set(LIBMSCP_SRC
src/mscp.c src/ssh.c src/fileops.c src/path.c src/platform.c src/message.c)
add_library(mscp-shared SHARED ${LIBMSCP_SRC})
target_include_directories(mscp-shared
PUBLIC $<BUILD_INTERFACE:${mscp_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
PRIVATE ${MSCP_BUILD_INCLUDE_DIRS})
target_compile_options(mscp-shared PRIVATE ${MSCP_COMPILE_OPTS})
target_link_libraries(mscp-shared PRIVATE ${MSCP_LINK_LIBS})
set_target_properties(mscp-shared
PROPERTIES
OUTPUT_NAME mscp
PUBLIC_HEADER ${mscp_SOURCE_DIR}/include/mscp.h)
if(!INSTALL_EXECUTABLE_ONLY)
install(TARGETS mscp-shared)
# Symbol check
check_symbol_exists(strlcat string.h HAVE_STRLCAT)
if (NOT HAVE_STRLCAT)
list(APPEND OPENBSD_COMPAT_SRC src/openbsd-compat/strlcat.c)
endif()
# generate config.h in build dir
configure_file(
${mscp_SOURCE_DIR}/include/config.h.in
${CMAKE_CURRENT_BINARY_DIR}/include/config.h)
list(APPEND MSCP_BUILD_INCLUDE_DIRS ${CMAKE_CURRENT_BINARY_DIR}/include)
# libmscp.a
set(LIBMSCP_SRC
src/mscp.c src/ssh.c src/fileops.c src/path.c src/platform.c
src/print.c src/strerrno.c
${OPENBSD_COMPAT_SRC})
add_library(mscp-static STATIC ${LIBMSCP_SRC})
target_include_directories(mscp-static
PRIVATE ${MSCP_BUILD_INCLUDE_DIRS} ${mscp_SOURCE_DIR}/include)
@@ -121,11 +110,6 @@ set_target_properties(mscp-static
PROPERTIES
OUTPUT_NAME mscp)
if(!INSTALL_EXECUTABLE_ONLY)
install(TARGETS mscp-static)
endif()
# mscp executable
list(APPEND MSCP_LINK_LIBS m pthread)
@@ -147,7 +131,7 @@ configure_file(
${mscp_SOURCE_DIR}/doc/mscp.1.in
${PROJECT_BINARY_DIR}/mscp.1)
add_custom_target(update-mscp-rst
add_custom_target(update-rst
COMMENT "Update doc/mscp.rst from mscp.1.in"
WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
COMMAND
@@ -167,51 +151,13 @@ enable_testing()
# CPACK Rules
#set(CPACK_SET_DESTDIR true)
set(CPACK_PACKAGE_NAME ${PROJECT_NAME})
set(CPACK_PACKAGE_CONTACT "Ryo Nakamura <upa@haeena.net>")
set(CPACK_PACKAGE_DESCRIPTION
"mscp, copy files over multiple ssh connections")
execute_process(COMMAND uname -m
OUTPUT_VARIABLE ARCH OUTPUT_STRIP_TRAILING_WHITESPACE)
if(UNIX AND NOT APPLE) # on linux
execute_process(COMMAND
bash "-c" "cat /etc/os-release|grep '^ID='|cut -d '=' -f 2|tr -d '\"'"
OUTPUT_VARIABLE DIST_NAME OUTPUT_STRIP_TRAILING_WHITESPACE)
execute_process(COMMAND
bash "-c" "cat /etc/os-release|grep '^VERSION_ID='|cut -d '=' -f 2|tr -d '\"'"
OUTPUT_VARIABLE DIST_VER OUTPUT_STRIP_TRAILING_WHITESPACE)
execute_process(COMMAND
bash "-c" "${mscp_SOURCE_DIR}/scripts/print-install-deps.sh ${DIST_NAME}-${DIST_VER}"
OUTPUT_VARIABLE DIST_DEP OUTPUT_STRIP_TRAILING_WHITESPACE)
set(PACKAGE_FILE_NAME
${PROJECT_NAME}_${DIST_NAME}-${DIST_VER}-${ARCH})
set(CPACK_DEBIAN_FILE_NAME ${PACKAGE_FILE_NAME}.deb)
set(CPACK_DEBIAN_PACKAGE_DEPENDS ${DIST_DEP})
set(CPACK_DEBIAN_PACKAGE_HOMEPAGE "https://github.com/upa/mscp")
set(CPACK_RPM_FILE_NAME ${PACKAGE_FILE_NAME}.rpm)
set(CPACK_RPM_PACKAGE_REQUIRES ${DIST_DEP})
set(CPACK_RPM_PACKAGE_HOMEPAGE "https://github.com/upa/mscp")
set(CPACK_RPM_PACKAGE_DESCRIPTION ${CPACK_PACKAGE_DESCRIPTION})
set(CPACK_RPM_PACKAGE_LICENSE "GPLv3")
endif() # on linux
include(CPack)
# Custom targets to build and test mscp in docker containers.
# foreach(IN ZIP_LISTS) (cmake >= 3.17) can shorten the following lists.
# However, ubuntu 20.04 has cmake 3.16.3. So this is a roundabout trick.
list(APPEND DIST_NAMES ubuntu ubuntu rocky almalinux alpine)
list(APPEND DIST_VERS 20.04 22.04 8.8 8.8 3.17)
list(APPEND DIST_PKGS deb deb rpm rpm static)
list(APPEND DIST_NAMES ubuntu ubuntu rocky rocky almalinux alpine)
list(APPEND DIST_VERS 20.04 22.04 8.9 9.3 9.3 3.19)
list(APPEND DIST_PKGS deb deb rpm rpm rpm static)
list(LENGTH DIST_NAMES _DIST_LISTLEN)
math(EXPR DIST_LISTLEN "${_DIST_LISTLEN} - 1")
@@ -230,13 +176,13 @@ foreach(x RANGE ${DIST_LISTLEN})
COMMENT "Build mscp in ${DOCKER_IMAGE} container"
WORKING_DIRECTORY ${mscp_SOURCE_DIR}
COMMAND
docker build -t ${DOCKER_IMAGE} -f docker/${DOCKER_INDEX}.Dockerfile .)
docker build -t ${DOCKER_IMAGE} -f Dockerfile/${DOCKER_INDEX}.Dockerfile .)
add_custom_target(docker-build-${DOCKER_INDEX}-no-cache
COMMENT "Build mscp in ${DOCKER_IMAGE} container"
WORKING_DIRECTORY ${mscp_SOURCE_DIR}
COMMAND
docker build --no-cache -t ${DOCKER_IMAGE} -f docker/${DOCKER_INDEX}.Dockerfile .)
docker build --no-cache -t ${DOCKER_IMAGE} -f Dockerfile/${DOCKER_INDEX}.Dockerfile .)
add_custom_target(docker-test-${DOCKER_INDEX}
COMMENT "Test mscp in ${DOCKER_IMAGE} container"
@@ -245,20 +191,68 @@ foreach(x RANGE ${DIST_LISTLEN})
docker run --init --rm --sysctl net.ipv6.conf.all.disable_ipv6=0
${DOCKER_IMAGE} /mscp/scripts/test-in-container.sh)
add_custom_target(docker-pkg-${DOCKER_INDEX}
COMMENT "Retrieve mscp package from ${DOCKER_IMAGE} container"
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMAND
docker run --rm -v ${CMAKE_BINARY_DIR}:/out ${DOCKER_IMAGE}
cp /mscp/build/${PKG_FILE_NAME} /out/)
list(APPEND DOCKER_BUILDS docker-build-${DOCKER_INDEX})
list(APPEND DOCKER_BUILDS_NO_CACHE docker-build-${DOCKER_INDEX}-no-cache)
list(APPEND DOCKER_TESTS docker-test-${DOCKER_INDEX})
list(APPEND DOCKER_PKGS docker-pkg-${DOCKER_INDEX})
endforeach()
add_custom_target(docker-build-all DEPENDS ${DOCKER_BUILDS})
add_custom_target(docker-build-all-no-cache DEPENDS ${DOCKER_BUILDS_NO_CACHE})
add_custom_target(docker-test-all DEPENDS ${DOCKER_TESTS})
add_custom_target(docker-pkg-all DEPENDS ${DOCKER_PKGS})
### debuild-related definitions
set(DEBBUILDCONTAINER mscp-build-deb)
add_custom_target(build-deb
COMMENT "build mscp deb files inside a container"
WORKING_DIRECTORY ${mscp_SOURCE_DIR}
BYPRODUCTS ${CMAKE_BINARY_DIR}/debbuild
COMMAND
docker build -t ${DEBBUILDCONTAINER} -f Dockerfile/build-deb.Dockerfile .
COMMAND
docker run --rm -v ${CMAKE_BINARY_DIR}:/out ${DEBBUILDCONTAINER}
cp -r /debbuild /out/)
### rpmbuild-related definitions
# generate files for rpmbuild
configure_file(
${mscp_SOURCE_DIR}/rpm/mscp.spec.in
${mscp_SOURCE_DIR}/rpm/mscp.spec
@ONLY)
configure_file(
${mscp_SOURCE_DIR}/Dockerfile/build-srpm.Dockerfile.in
${mscp_SOURCE_DIR}/Dockerfile/build-srpm.Dockerfile
@ONLY)
# Custom target to build mscp as a src.rpm in docker.
set(RPMBUILDCONTAINER mscp-build-srpm)
set(SRPMFILE mscp-${MSCP_VERSION}-1.el9.src.rpm)
add_custom_target(build-srpm
COMMENT "Build mscp src.rpm inside a container"
WORKING_DIRECTORY ${mscp_SOURCE_DIR}
BYPRODUCTS ${CMAKE_BINARY_DIR}/${SRPMFILE}
COMMAND
docker build -t ${RPMBUILDCONTAINER} -f Dockerfile/build-srpm.Dockerfile .
COMMAND
docker run --rm -v ${CMAKE_BINARY_DIR}:/out ${RPMBUILDCONTAINER}
cp /root/rpmbuild/SRPMS/${SRPMFILE} /out/)
### single-binary-build-related definitions
# Custom target to get single binary mscp
set(SINGLEBINARYFILE mscp.linux.${CMAKE_SYSTEM_PROCESSOR}.static)
add_custom_target(build-single-binary
COMMENT "Build mscp as a single binary in alpine conatiner"
WORKING_DIRECTORY ${mscp_SOURCE_DIR}
BYPRODUCTS ${CMAKE_BINARY_DIR}/${SINGLEBINARYFILE}
DEPENDS docker-build-alpine-3.19
COMMAND
docker run --rm -v ${CMAKE_BINARY_DIR}:/out mscp-alpine:3.19
cp /mscp/build/mscp /out/${SINGLEBINARYFILE})
add_custom_target(build-pkg-all
DEPENDS build-deb build-srpm build-single-binary)

3
Dockerfile/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
# generated by cmake
build-srpm.Dockerfile

19
Dockerfile/README.md Normal file
View File

@@ -0,0 +1,19 @@
Dockerfiles for building and testing mscp.
Build container:
```
docker build -t mscp-DIST:VER -f docker/DIST-VER.Dockerfile .
```
Run test:
```
docker run --init --rm mscp-DST:VER /mscp/scripts/test-in-container.sh
```
Custom targets to build and test mscp in the containers are provided
via `cmake`. See `make docker-*` targets. `make docker-build-all`
builds all container images, and `make docker-test-all` runs the test
in all container images.

View File

@@ -0,0 +1,40 @@
FROM almalinux:9.3
# install pytest, sshd for test, and rpm-build
RUN set -ex && yum -y install \
python3 python3-pip python3-devel openssh openssh-server openssh-clients rpm-build
RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
# create test user
RUN useradd -m -d /home/test test \
&& echo "test:userpassword" | chpasswd \
&& mkdir -p /home/test/.ssh \
&& ssh-keygen -f /home/test/.ssh/id_rsa_test -N "keypassphrase" \
&& cat /home/test/.ssh/id_rsa_test.pub >> /home/test/.ssh/authorized_keys \
&& chown -R test:test /home/test \
&& chown -R test:test /home/test/.ssh
RUN rm -rf /run/nologin
ARG mscpdir="/mscp"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# build
RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make -j 2 \
&& make install

View File

@@ -0,0 +1,45 @@
FROM alpine:3.19
# Build mscp with conan to create single binary mscp
RUN apk add --no-cache \
gcc make cmake python3 py3-pip perl linux-headers libc-dev \
openssh bash python3-dev py3-pytest g++
RUN pip3 install --break-system-packages conan
# preparation for sshd
RUN ssh-keygen -A \
&& mkdir /var/run/sshd \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
# create test user
RUN addgroup -S test \
&& adduser -S test -G test \
&& echo "test:userpassword" | chpasswd \
&& mkdir -p /home/test/.ssh \
&& ssh-keygen -f /home/test/.ssh/id_rsa_test -N "keypassphrase" \
&& cat /home/test/.ssh/id_rsa_test.pub >> /home/test/.ssh/authorized_keys \
&& chown -R test:test /home/test \
&& chown -R test:test /home/test/.ssh
# Build mscp as a single binary
RUN conan profile detect --force
ARG mscpdir="/mscp"
COPY . ${mscpdir}
RUN cd ${mscpdir} \
&& rm -rf build \
&& conan install . --output-folder=build --build=missing \
&& cd ${mscpdir}/build \
&& cmake .. \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake \
-DBUILD_CONAN=ON -DBUILD_STATIC=ON \
&& make -j 2 \
&& make install

View File

@@ -0,0 +1,20 @@
FROM ubuntu:22.04
ARG DEBIAN_FRONTEND=noninteractive
RUN set -ex && apt-get update && apt-get install -y --no-install-recommends \
ca-certificates build-essential devscripts debhelper gcc make cmake
ARG mscpdir="/debbuild/mscp"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# build
RUN cd ${mscpdir} \
&& debuild -us -uc -S \
&& mv ${mscpdir} /
# Then all debuild output files exsit at /debbuild

View File

@@ -0,0 +1,22 @@
FROM rockylinux:9
# install pytest, sshd for test, and rpm-build
RUN set -ex && yum -y install rpm-build rpmdevtools
ARG mscpdir="/mscp-@MSCP_VERSION@"
ARG mscptgz="mscp-@MSCP_VERSION@.tar.gz"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# prepare rpmbuild
RUN rpmdev-setuptree \
&& rm -rf ${mscpdir}/build \
&& tar zcvf /${mscptgz} --exclude-vcs ${mscpdir} \
&& cp /${mscptgz} ~/rpmbuild/SOURCES/ \
&& cp ${mscpdir}/rpm/mscp.spec ~/rpmbuild/SPECS/
# build rpm and src.rpm
RUN rpmbuild -ba ~/rpmbuild/SPECS/mscp.spec

View File

@@ -0,0 +1,41 @@
FROM rockylinux:8.9
# install pytest, sshd for test, and rpm-build
RUN set -ex && yum -y install \
python3 python3-pip python3-devel openssh openssh-server openssh-clients rpm-build
RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
# create test user
RUN useradd -m -d /home/test test \
&& echo "test:userpassword" | chpasswd \
&& mkdir -p /home/test/.ssh \
&& ssh-keygen -f /home/test/.ssh/id_rsa_test -N "keypassphrase" \
&& cat /home/test/.ssh/id_rsa_test.pub >> /home/test/.ssh/authorized_keys \
&& chown -R test:test /home/test \
&& chown -R test:test /home/test/.ssh
RUN rm -rf /run/nologin
ARG mscpdir="/mscp"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# build
RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make -j 2 \
&& make install

View File

@@ -0,0 +1,41 @@
FROM rockylinux:9.3
# install pytest, sshd for test, and rpm-build
RUN set -ex && yum -y install \
python3 python3-pip python3-devel openssh openssh-server openssh-clients rpm-build
RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
# create test user
RUN useradd -m -d /home/test test \
&& echo "test:userpassword" | chpasswd \
&& mkdir -p /home/test/.ssh \
&& ssh-keygen -f /home/test/.ssh/id_rsa_test -N "keypassphrase" \
&& cat /home/test/.ssh/id_rsa_test.pub >> /home/test/.ssh/authorized_keys \
&& chown -R test:test /home/test \
&& chown -R test:test /home/test/.ssh
RUN rm -rf /run/nologin
ARG mscpdir="/mscp"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# build
RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make -j 2 \
&& make install

View File

@@ -12,9 +12,19 @@ RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
&& cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
# create test user
RUN useradd -m -d /home/test test \
&& echo "test:userpassword" | chpasswd \
&& mkdir -p /home/test/.ssh \
&& ssh-keygen -f /home/test/.ssh/id_rsa_test -N "keypassphrase" \
&& cat /home/test/.ssh/id_rsa_test.pub >> /home/test/.ssh/authorized_keys \
&& chown -R test:test /home/test \
&& chown -R test:test /home/test/.ssh
ARG mscpdir="/mscp"
@@ -30,11 +40,6 @@ RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make \
&& cpack -G DEB CPackConfig.cmake \
&& dpkg -i *.deb
&& make -j 2 \
&& make install
# install mscp python module
RUN cd ${mscpdir} \
&& python3 pysetup.py install --user \
&& ldconfig

View File

@@ -12,9 +12,20 @@ RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
# create test user
RUN useradd -m -d /home/test test \
&& echo "test:userpassword" | chpasswd \
&& mkdir -p /home/test/.ssh \
&& ssh-keygen -f /home/test/.ssh/id_rsa_test -N "keypassphrase" \
&& cat /home/test/.ssh/id_rsa_test.pub >> /home/test/.ssh/authorized_keys \
&& chown -R test:test /home/test \
&& chown -R test:test /home/test/.ssh
ARG mscpdir="/mscp"
@@ -29,12 +40,5 @@ RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make \
&& cpack -G DEB CPackConfig.cmake \
&& dpkg -i *.deb
# install mscp python module
RUN cd ${mscpdir} \
&& python3 pysetup.py install --user \
&& ldconfig
&& make -j 2 \
&& make install

View File

@@ -9,5 +9,5 @@ GENERATE_MAN = NO
SOURCE_BROWSER = YES
INPUT = src include mscp
INPUT = src include

View File

@@ -1,6 +1,10 @@
# mscp: multi-threaded scp
[![build on ubuntu](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml) [![build on macOS](https://github.com/upa/mscp/actions/workflows/build-macos.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-macos.yml) [![test](https://github.com/upa/mscp/actions/workflows/test.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/test.yml)
[![build on ubuntu](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-ubuntu.yml)
[![build on macOS](https://github.com/upa/mscp/actions/workflows/build-macos.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-macos.yml)
[![build on FreeBSD](https://github.com/upa/mscp/actions/workflows/build-freebsd.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/build-freebsd.yml)
[![test](https://github.com/upa/mscp/actions/workflows/test.yml/badge.svg)](https://github.com/upa/mscp/actions/workflows/test.yml)
`mscp`, a variant of `scp`, copies files over multiple SSH (SFTP)
@@ -46,17 +50,13 @@ sudo add-apt-repository ppa:upaa/mscp
sudo apt-get install mscp
```
- Rocky 8.8
- RHEL-based distributions
```console
yum install https://github.com/upa/mscp/releases/latest/download/mscp_rocky-8.8-x86_64.rpm
sudo dnf copr enable upaaa/mscp
sudo dnf install mscp
```
- Alma 8.8
```console
yum install https://github.com/upa/mscp/releases/latest/download/mscp_almalinux-8.8-x86_64.rpm
```
- Linux with single binary `mscp` (x86_64 only, and not optimal performance)
- Single binary `mscp` for x86_64 (not optimal performance)
```console
wget https://github.com/upa/mscp/releases/latest/download/mscp.linux.x86.static -O /usr/local/bin/mscp
chmod 755 /usr/local/bin/mscp
@@ -70,7 +70,9 @@ patch introduces asynchronous SFTP Write, which is derived from
https://github.com/limes-datentechnik-gmbh/libssh (see [Re: SFTP Write
async](https://archive.libssh.org/libssh/2020-06/0000004.html)).
Currently macOS and Linux (Ubuntu, Rocky and Alma) are supported.
We test building mscp on Linux (Ubuntu, Rocky, Alma, and Alpine),
macOS, and FreeBSD.
```console
# clone this repository

View File

@@ -1 +1 @@
0.1.3
0.1.4

View File

@@ -1,325 +0,0 @@
# - Try to find GSSAPI
# Once done this will define
#
# KRB5_CONFIG - Path to krb5-config
# GSSAPI_ROOT_DIR - Set this variable to the root installation of GSSAPI
#
# Read-Only variables:
# GSSAPI_FLAVOR_MIT - set to TURE if MIT Kerberos has been found
# GSSAPI_FLAVOR_HEIMDAL - set to TRUE if Heimdal Keberos has been found
# GSSAPI_FOUND - system has GSSAPI
# GSSAPI_INCLUDE_DIR - the GSSAPI include directory
# GSSAPI_LIBRARIES - Link these to use GSSAPI
# GSSAPI_DEFINITIONS - Compiler switches required for using GSSAPI
#
#=============================================================================
# Copyright (c) 2013 Andreas Schneider <asn@cryptomilk.org>
#
# Distributed under the OSI-approved BSD License (the "License");
# see accompanying file Copyright.txt for details.
#
# This software is distributed WITHOUT ANY WARRANTY; without even the
# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the License for more information.
#=============================================================================
#
find_path(GSSAPI_ROOT_DIR
NAMES
include/gssapi.h
include/gssapi/gssapi.h
HINTS
${_GSSAPI_ROOT_HINTS}
PATHS
${_GSSAPI_ROOT_PATHS}
)
mark_as_advanced(GSSAPI_ROOT_DIR)
if (UNIX)
find_program(KRB5_CONFIG
NAMES
krb5-config
PATHS
${GSSAPI_ROOT_DIR}/bin
/opt/local/bin)
mark_as_advanced(KRB5_CONFIG)
if (KRB5_CONFIG)
# Check if we have MIT KRB5
execute_process(
COMMAND
${KRB5_CONFIG} --vendor
RESULT_VARIABLE
_GSSAPI_VENDOR_RESULT
OUTPUT_VARIABLE
_GSSAPI_VENDOR_STRING)
if ((_GSSAPI_VENDOR_STRING MATCHES ".*Massachusetts.*") OR (_GSSAPI_VENDOR_STRING
MATCHES ".*MITKerberosShim.*"))
set(GSSAPI_FLAVOR_MIT TRUE)
else()
execute_process(
COMMAND
${KRB5_CONFIG} --libs gssapi
RESULT_VARIABLE
_GSSAPI_LIBS_RESULT
OUTPUT_VARIABLE
_GSSAPI_LIBS_STRING)
if (_GSSAPI_LIBS_STRING MATCHES ".*roken.*")
set(GSSAPI_FLAVOR_HEIMDAL TRUE)
endif()
endif()
# Get the include dir
execute_process(
COMMAND
${KRB5_CONFIG} --cflags gssapi
RESULT_VARIABLE
_GSSAPI_INCLUDE_RESULT
OUTPUT_VARIABLE
_GSSAPI_INCLUDE_STRING)
string(REGEX REPLACE "(\r?\n)+$" "" _GSSAPI_INCLUDE_STRING "${_GSSAPI_INCLUDE_STRING}")
string(REGEX REPLACE " *-I" "" _GSSAPI_INCLUDEDIR "${_GSSAPI_INCLUDE_STRING}")
endif()
if (NOT GSSAPI_FLAVOR_MIT AND NOT GSSAPI_FLAVOR_HEIMDAL)
# Check for HEIMDAL
find_package(PkgConfig)
if (PKG_CONFIG_FOUND)
pkg_check_modules(_GSSAPI heimdal-gssapi)
endif (PKG_CONFIG_FOUND)
if (_GSSAPI_FOUND)
set(GSSAPI_FLAVOR_HEIMDAL TRUE)
else()
find_path(_GSSAPI_ROKEN
NAMES
roken.h
PATHS
${GSSAPI_ROOT_DIR}/include
${_GSSAPI_INCLUDEDIR})
if (_GSSAPI_ROKEN)
set(GSSAPI_FLAVOR_HEIMDAL TRUE)
endif()
endif ()
endif()
endif (UNIX)
find_path(GSSAPI_INCLUDE_DIR
NAMES
gssapi.h
gssapi/gssapi.h
PATHS
${GSSAPI_ROOT_DIR}/include
${_GSSAPI_INCLUDEDIR}
)
if (GSSAPI_FLAVOR_MIT)
find_library(GSSAPI_LIBRARY
NAMES
gssapi_krb5
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(KRB5_LIBRARY
NAMES
krb5
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(K5CRYPTO_LIBRARY
NAMES
k5crypto
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(COM_ERR_LIBRARY
NAMES
com_err
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
if (GSSAPI_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${GSSAPI_LIBRARY}
)
endif (GSSAPI_LIBRARY)
if (KRB5_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${KRB5_LIBRARY}
)
endif (KRB5_LIBRARY)
if (K5CRYPTO_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${K5CRYPTO_LIBRARY}
)
endif (K5CRYPTO_LIBRARY)
if (COM_ERR_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${COM_ERR_LIBRARY}
)
endif (COM_ERR_LIBRARY)
endif (GSSAPI_FLAVOR_MIT)
if (GSSAPI_FLAVOR_HEIMDAL)
find_library(GSSAPI_LIBRARY
NAMES
gssapi
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(KRB5_LIBRARY
NAMES
krb5
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(HCRYPTO_LIBRARY
NAMES
hcrypto
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(COM_ERR_LIBRARY
NAMES
com_err
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(HEIMNTLM_LIBRARY
NAMES
heimntlm
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(HX509_LIBRARY
NAMES
hx509
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(ASN1_LIBRARY
NAMES
asn1
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(WIND_LIBRARY
NAMES
wind
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
find_library(ROKEN_LIBRARY
NAMES
roken
PATHS
${GSSAPI_ROOT_DIR}/lib
${_GSSAPI_LIBDIR}
)
if (GSSAPI_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${GSSAPI_LIBRARY}
)
endif (GSSAPI_LIBRARY)
if (KRB5_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${KRB5_LIBRARY}
)
endif (KRB5_LIBRARY)
if (HCRYPTO_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${HCRYPTO_LIBRARY}
)
endif (HCRYPTO_LIBRARY)
if (COM_ERR_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${COM_ERR_LIBRARY}
)
endif (COM_ERR_LIBRARY)
if (HEIMNTLM_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${HEIMNTLM_LIBRARY}
)
endif (HEIMNTLM_LIBRARY)
if (HX509_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${HX509_LIBRARY}
)
endif (HX509_LIBRARY)
if (ASN1_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${ASN1_LIBRARY}
)
endif (ASN1_LIBRARY)
if (WIND_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${WIND_LIBRARY}
)
endif (WIND_LIBRARY)
if (ROKEN_LIBRARY)
set(GSSAPI_LIBRARIES
${GSSAPI_LIBRARIES}
${WIND_LIBRARY}
)
endif (ROKEN_LIBRARY)
endif (GSSAPI_FLAVOR_HEIMDAL)
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(GSSAPI DEFAULT_MSG GSSAPI_LIBRARIES GSSAPI_INCLUDE_DIR)
if (GSSAPI_INCLUDE_DIRS AND GSSAPI_LIBRARIES)
set(GSSAPI_FOUND TRUE)
endif (GSSAPI_INCLUDE_DIRS AND GSSAPI_LIBRARIES)
# show the GSSAPI_INCLUDE_DIRS and GSSAPI_LIBRARIES variables only in the advanced view
mark_as_advanced(GSSAPI_INCLUDE_DIRS GSSAPI_LIBRARIES)

17
debian/changelog vendored
View File

@@ -1,4 +1,19 @@
mscp (0.1.3) UNRELEASED; urgency=medium
mscp (0.1.4) UNRELEASED; urgency=medium
* add a test for builds on FreeBSD
* updat container images for tests
* dropp support for Python bindings
* clean up error message handling
* add MSCP_SSH_AUTH_PASSWORD/PASSPHRASE environment variables
to pass passwords to MSCP without interactive input (issue #9)
* add a -p option to preserve timestamps
* add -4 and -6 options to use IPv4 or IPv6, respectively
* introduc .clang-format, which is derived from the Linux kernel
* fix the manpage
-- Ryo Nakamura <upa@haeena.net> Wed, 07 Feb 2024 15:56:58 +0900
mscp (0.1.3) unstable; urgency=medium
* add -I option for inserting intervals between SSH attempts (issue #7)
* add -P option, equivalent to -p (just for compatibility)

4
debian/rules vendored
View File

@@ -4,10 +4,6 @@
dh $@
override_dh_auto_configure:
dh_auto_configure -- \
-DINSTALL_EXECUTABLE_ONLY=ON
override_dh_auto_test:

View File

@@ -1,11 +0,0 @@
# Document
The base file of documents is `mscp.1.in`. The manpage of mscp and
`doc/mscp.rst` are generated from `mscp.1.in`.
When `mscp.1.in` is changed, update `doc/mscp.rst` by:
1. `cd build`
2. `cmake ..`
3. `make update-mscp-rst`

71
doc/RELEASE.md Normal file
View File

@@ -0,0 +1,71 @@
## Build mscp as deb package
`make build-deb` produces a mscp deb package and related files. This
target builds mscp with `debuild` inside a docker container
(Dockerfile is `docker/build-deb.Docerfile`).
```console
mkdir build && cd build && cmake ..
make build-deb
```
After that:
```console
$ ls debbuild
mscp_0.1.4.dsc mscp_0.1.4_source.buildinfo mscp_0.1.4.tar.xz
mscp_0.1.4_source.build mscp_0.1.4_source.changes
```
### To publush mscp in launchpad PPA:
1. write changes in `debian/changelog` at main branch (the date command needed here is `date -R`)
2. switch to `ppa-focal` or `ppa-jammy` branch
3. reblase to the `main` branch and modify `debian/changes`:
* change `UNRELEASED` to the release name (`focal` or `jammy`).
4. run `make build-deb` at the build directory and `cd debbuild`
5. sign the files with `debsign -k [GPGKEYID] mscp_X.X.X~X_source.changes`
5. upload the files with `dput ppa:upaa/mscp mscp_X.X.X~X_source.changes`
## Build mscp as (source) rpm package
`make build-srpm` produces a mscp src.rpm package. This target builts
mscp with `rpmbuild` inside a docker container (Dockerfile is
`docker/build-srpm.Dockerfile`, generated from
`build-srpm.Dockerfile.in` by cmake).
```console
mkdir build && cd build && cmake ..
make build-srpm
```
After that:
```console
$ ls *.rpm
mscp-0.1.3-1.el9.src.rpm
```
### To publish mscp in COPR:
1. update `changelog` in `rpm/mscp.spec` (the date command needed here is `date "+%a %b %d %Y"`)
2. run `make build-srpm`
3. download `mscp-X.X.X-1.yyy.src.rpm`
4. upload the src.rpm to Build page at COPR.
## Update Document
The docuemnt is `doc/mscp.rst` (at present). When `mscp.1.in` is
modified, run `make update-rst` to make it up to date.
```console
mkdir build cd build && cmake ..
make update-rst
```

View File

@@ -6,7 +6,7 @@ mscp \- copy files over multiple SSH connections
.SH SYNOPSIS
.B mscp
.RB [ \-vqDHdNh ]
.RB [ \-46vqDpHdNh ]
[\c
.BI \-n \ NR_CONNECTIONS\c
]
@@ -35,7 +35,6 @@ mscp \- copy files over multiple SSH connections
.BI \-l \ LOGIN_NAME\c
]
[\c
.BR \-p |\c
.BI \-P \ PORT\c
]
[\c
@@ -53,6 +52,9 @@ mscp \- copy files over multiple SSH connections
[\c
.BI \-C \ COMPRESS\c
]
[\c
.BI \-g \ CONGESTION\c
]
.I source ... target
.SH DESCRIPTION
@@ -151,6 +153,14 @@ Specifies the buffer size for I/O and transfer over SFTP. The default
value is 16384. Note that the SSH specification restricts buffer size
delivered over SSH. Changing this value is not recommended at present.
.TP
.B \-4
Uses IPv4 addresses only.
.TP
.B \-6
Uses IPv6 addresses only.
.TP
.B \-v
Increments the verbose output level.
@@ -180,7 +190,7 @@ Specifies the username to log in on the remote machine as with
.I ssh(1).
.TP
.B \-p,\-P \fIPORT\fR
.B \-P \fIPORT\fR
Specifies the port number to connect to on the remote machine as with
ssh(1) and scp(1).
@@ -217,6 +227,15 @@ none. See
libssh features
.UE .
.TP
.B \-g \fICONGESTION\fR
Specifies the TCP congestion control algorithm to use (Linux only).
.TP
.B \-p
Preserves modification times and access times (file mode bits are
preserved by default).
.TP
.B \-H
Disables hostkey checking.
@@ -236,6 +255,24 @@ Prints help.
.SH EXIT STATUS
Exit status is 0 on success, and >0 if an error occurs.
.SH ENVIRONMENT
.PP
.B mscp
recognizes the following environment variables.
.TP
.B MSCP_SSH_AUTH_PASSWORD
This environment variable passes a password for password
authentication to establish SSH connections.
.TP
.B MSCP_SSH_AUTH_PASSPHRASE
This environment variable passes a passphrase for public-key
authentication for establishing SSH connections.
.SH NOTES
.PP

View File

@@ -2,7 +2,7 @@
MSCP
====
:Date: v0.1.2-14-g24617d2
:Date: v0.1.3-23-ga9c59f7
NAME
====
@@ -12,13 +12,13 @@ mscp - copy files over multiple SSH connections
SYNOPSIS
========
**mscp** [**-vqDHdNh**] [ **-n**\ *NR_CONNECTIONS* ] [
**mscp** [**-46vqDpHdNh**] [ **-n**\ *NR_CONNECTIONS* ] [
**-m**\ *COREMASK* ] [ **-u**\ *MAX_STARTUPS* ] [ **-I**\ *INTERVAL* ] [
**-s**\ *MIN_CHUNK_SIZE* ] [ **-S**\ *MAX_CHUNK_SIZE* ] [
**-a**\ *NR_AHEAD* ] [ **-b**\ *BUF_SIZE* ] [ **-l**\ *LOGIN_NAME* ] [
**-p**\ \| **-P**\ *PORT* ] [ **-F**\ *CONFIG* ] [ **-i**\ *IDENTITY* ]
[ **-c**\ *CIPHER* ] [ **-M**\ *HMAC* ] [ **-C**\ *COMPRESS* ] *source
... target*
**-P**\ *PORT* ] [ **-F**\ *CONFIG* ] [ **-i**\ *IDENTITY* ] [
**-c**\ *CIPHER* ] [ **-M**\ *HMAC* ] [ **-C**\ *COMPRESS* ] *source ...
target*
DESCRIPTION
===========
@@ -87,6 +87,12 @@ OPTIONS
delivered over SSH. Changing this value is not recommended at
present.
**-4**
Uses IPv4 addresses only.
**-6**
Uses IPv6 addresses only.
**-v**
Increments the verbose output level.
@@ -106,7 +112,7 @@ OPTIONS
Specifies the username to log in on the remote machine as with
*ssh(1).*
**-p,-P PORT**
**-P PORT**
Specifies the port number to connect to on the remote machine as with
ssh(1) and scp(1).
@@ -130,6 +136,10 @@ OPTIONS
Enables compression: yes, no, zlib, zlib@openssh.com. The default is
none. See `libssh features <https://www.libssh.org/features/>`__.
**-p**
Preserves modification times and access times (file mode bits are
preserved by default).
**-H**
Disables hostkey checking.
@@ -147,6 +157,19 @@ EXIT STATUS
Exit status is 0 on success, and >0 if an error occurs.
ENVIRONMENT
===========
**mscp** recognizes the following environment variables.
**MSCP_SSH_AUTH_PASSWORD**
This environment variable passes a password for password
authentication to establish SSH connections.
**MSCP_SSH_AUTH_PASSPHRASE**
This environment variable passes a passphrase for public-key
authentication for establishing SSH connections.
NOTES
=====

View File

@@ -1,37 +0,0 @@
Build docker containers.
```console
cd ..
docker build -t mscp-ubuntu:20.04 -f docker/ubuntu-20.04.Dockerfile .
docker build -t mscp-ubuntu:22.04 -f docker/ubuntu-22.04.Dockerfile .
docker build -t mscp-rocky:8.8 -f docker/rocky-8.Dockerfile .
```
Test `mscp` in the containers.
```console
docker run --init --rm mscp-ubuntu:20.04 /mscp/scripts/test-in-container.sh
docker run --init --rm mscp-ubuntu:22.04 /mscp/scripts/test-in-container.sh
docker run --init --rm mscp-rocky:8.9 /mscp/scripts/test-in-container.sh
```
Retrieve deb/rpm packages.
```console
docker run --rm -v (pwd):/out mscp-ubuntu:20.04 \
cp /mscp/build/mscp_ubuntu-20.04-x86_64.deb /out/
docker run --rm -v (pwd):/out mscp-ubuntu:22.04 \
cp /mscp/build/mscp_ubuntu-22.04-x86_64.deb /out/
docker run --rm -v (pwd):/out mscp-rocky:8.8 \
cp /mscp/build/mscp_rocky-8.8-x86_64.rpm /out/
```
I don't know whether these are good way.

View File

@@ -1,39 +0,0 @@
FROM almalinux:8.8
# install pytest, sshd for test, and rpm-build
RUN set -ex && \
rpm --import https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux && \
yum -y install \
python3 python3-pip python3-devel openssh openssh-server openssh-clients rpm-build
RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
ARG mscpdir="/mscp"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# build
RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make \
&& cpack -G RPM CPackConfig.cmake \
&& rpm -iv *.rpm
# install mscp python module
RUN cd ${mscpdir} \
&& python3 pysetup.py install --user \
&& ldconfig

View File

@@ -1,40 +0,0 @@
FROM alpine:3.17
# Build mscp with conan to create single binary mscp
RUN apk add --no-cache \
gcc make cmake python3 py3-pip perl linux-headers libc-dev \
openssh bash python3-dev g++
RUN pip3 install conan pytest
# preparation for sshd
RUN ssh-keygen -A
RUN mkdir /var/run/sshd \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
# Build mscp as a single binary
RUN conan profile detect --force
ARG mscpdir="/mscp"
COPY . ${mscpdir}
RUN cd ${mscpdir} \
&& rm -rf build \
&& conan install . --output-folder=build --build=missing \
&& cd ${mscpdir}/build \
&& cmake .. \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake \
-DBUILD_CONAN=ON -DBUILD_STATIC=ON \
&& make \
&& cp mscp /usr/bin/ \
&& cp mscp /mscp/build/mscp_alpine-3.17-x86_64.static
# copy mscp to PKG FILE NAME because this build doesn't use CPACK
# install mscp python module
RUN cd ${mscpdir} \
&& python3 pysetup.py install --user

View File

@@ -1,36 +0,0 @@
FROM rockylinux:8.8
# install pytest, sshd for test, and rpm-build
RUN set -ex && yum -y install \
python3 python3-pip python3-devel openssh openssh-server openssh-clients rpm-build
RUN python3 -m pip install pytest
# preparation for sshd
RUN mkdir /var/run/sshd \
&& ssh-keygen -A \
&& ssh-keygen -f /root/.ssh/id_rsa -N "" \
&& mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
ARG mscpdir="/mscp"
COPY . ${mscpdir}
# install build dependency
RUN ${mscpdir}/scripts/install-build-deps.sh
# build
RUN cd ${mscpdir} \
&& rm -rf build \
&& cmake -B build \
&& cd ${mscpdir}/build \
&& make \
&& cpack -G RPM CPackConfig.cmake \
&& rpm -iv *.rpm
# install mscp python module
RUN cd ${mscpdir} \
&& python3 pysetup.py install --user \
&& ldconfig

3
examples/.gitignore vendored
View File

@@ -1,3 +0,0 @@
simple-copy-dest
*.img
.ipynb_checkpoints

View File

@@ -1,226 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ccda9e3a-35de-43fc-9b6e-02475c763f6b",
"metadata": {},
"source": [
"# mscp python binding example"
]
},
{
"cell_type": "code",
"execution_count": 60,
"id": "df04d655-a082-47eb-9a1e-154ebc2a5655",
"metadata": {},
"outputs": [],
"source": [
"import glob\n",
"import time\n",
"import os\n",
"\n",
"import mscp"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "e9ed4519-c3fd-4639-89a5-1c1cdffd9519",
"metadata": {},
"outputs": [],
"source": [
"this_dir = os.getcwd()"
]
},
{
"cell_type": "markdown",
"id": "fee75bf8-df40-45f4-81d1-113069c34f13",
"metadata": {},
"source": [
"## Simple copy"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "2b06e6d3-30cc-47be-bd4f-af27eb141c8c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['../src/ssh.c',\n",
" '../src/mscp.c',\n",
" '../src/platform.c',\n",
" '../src/pymscp.c',\n",
" '../src/main.c',\n",
" '../src/path.c',\n",
" '../src/message.c',\n",
" '../src/fileops.c']"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# preparing files to be transferred\n",
"c_sources = glob.glob(\"../src/*.c\")\n",
"c_sources"
]
},
{
"cell_type": "code",
"execution_count": 55,
"id": "89bb4558-9472-4d26-9af3-24f426b15edc",
"metadata": {},
"outputs": [],
"source": [
"# copy files using mscp\n",
"dst_dir = this_dir + \"/simple-copy-dest\"\n",
"m = mscp.mscp(\"localhost\", mscp.LOCAL2REMOTE)\n",
"m.copy(c_sources, dst_dir)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "6daf2c98-8905-4039-b82a-a593df3107fe",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['ssh.c',\n",
" 'mscp.c',\n",
" 'platform.c',\n",
" 'pymscp.c',\n",
" 'main.c',\n",
" 'path.c',\n",
" 'message.c',\n",
" 'fileops.c']"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"os.listdir(\"simple-copy-dest\")"
]
},
{
"cell_type": "markdown",
"id": "f4a3869a-878e-43b0-9758-a049eaf8b5bd",
"metadata": {},
"source": [
"## Simple Copy with Python Rich ProgressBar"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "e7cb7cd6-b845-4d26-93ed-aee8ed3983ab",
"metadata": {},
"outputs": [],
"source": [
"# make a 256MB file\n",
"src = \"example-256MB-src.img\"\n",
"with open(src, \"wb\") as f:\n",
" f.seek(128 * 1024 * 1024 -1, 0)\n",
" f.write(b'1')"
]
},
{
"cell_type": "code",
"execution_count": 69,
"id": "878607ed-5c06-4b15-81ac-9845dad0c9c6",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b700e9fc00464969a22a26300404dc35",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# copy the 256MB file while ploting progress bar using python rich\n",
"dst = this_dir + \"/example-256MB-dst.img\"\n",
"\n",
"kw = {\"nr_threads\": 1, \"nr_ahead\": 1} # slow mscp to watch the progress bar\n",
"\n",
"m = mscp.mscp(\"localhost\", mscp.LOCAL2REMOTE, **kw)\n",
"m.copy(src, dst, nonblock = True)\n",
"\n",
"# m.stats() returns total bytes to be transferred, bytes transferred (done), and finished (bool).\n",
"total, done, finished = m.stats()\n",
"with Progress() as progress:\n",
"\n",
" task = progress.add_task(f\"[green]Copying {src}\", total = total)\n",
"\n",
" while not progress.finished:\n",
" total, done, finished = m.stats()\n",
" progress.update(task, completed = done)\n",
" time.sleep(0.5)\n",
"\n",
"m.join()\n",
"m.cleanup()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

View File

@@ -1,63 +0,0 @@
#!/usr/bin/env python3
"""mscp.py
An example python script running mscp
"""
import argparse
import time
import sys
from rich.progress import Progress
import mscp
def main():
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--from", dest = "fr",
metavar = "REMOTE", default = None,
help = "copy a file from this remote host")
parser.add_argument("-t", "--to", metavar = "REMOTE", default = None,
help = "copy a file to this remote host")
parser.add_argument("source", help = "path to source file to be copied")
parser.add_argument("destination", help = "path of copy destination")
args = parser.parse_args()
if args.fr and args.to:
print("-f and -t are exclusive", file = sys.stderr)
sys.exit(1)
elif args.fr:
d = mscp.REMOTE2LOCAL
remote = args.fr
elif args.to:
d = mscp.LOCAL2REMOTE
remote = args.to
else:
print("-f or -t must be specified", file = sys.stderr)
sys.exit(1)
m = mscp.mscp(remote, d)
m.connect()
m.add_src_path(args.source)
m.set_dst_path(args.destination)
m.scan()
m.start()
total, done, finished = m.stats()
with Progress() as progress:
task = progress.add_task("[green]Copying...", total = total)
while not progress.finished:
total, done, finished = m.stats()
progress.update(task, completed = done)
time.sleep(0.5)
m.join()
m.cleanup()
if __name__ == "__main__":
main()

12
include/config.h.in Normal file
View File

@@ -0,0 +1,12 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _CONFIG_H_
#define _CONFIG_H_
#define MSCP_VERSION "@MSCP_VERSION@"
#define MSCP_BUILD_VERSION "@MSCP_BUILD_VERSION@"
/* Define to 1 if you have the strlcat function. */
#cmakedefine HAVE_STRLCAT 1
#endif /* _CONFIG_H_ */

View File

@@ -31,8 +31,6 @@
#define MSCP_DIRECTION_L2R 1 /** Indicates local to remote copy */
#define MSCP_DIRECTION_R2L 2 /** Indicates remote to local copy */
#define MSCP_MAX_COREMASK_STR 64
/**
* @struct mscp_opts
* @brief Structure configuring mscp.
@@ -43,24 +41,14 @@ struct mscp_opts {
size_t min_chunk_sz; /** minimum chunk size (default 64MB) */
size_t max_chunk_sz; /** maximum chunk size (default file size/nr_threads) */
size_t buf_sz; /** buffer size, default 16k. */
char coremask[MSCP_MAX_COREMASK_STR]; /** hex to specifiy usable cpu cores */
char *coremask; /** hex to specifiy usable cpu cores */
int max_startups; /** sshd MaxStartups concurrent connections */
int interval; /** interval between SSH connection attempts */
bool preserve_ts; /** preserve file timestamps */
int severity; /** messaging severity. set MSCP_SERVERITY_* */
int msg_fd; /** fd to output message. default STDOUT (0),
* and -1 disables output */
};
#define MSCP_SSH_MAX_LOGIN_NAME 64
#define MSCP_SSH_MAX_PORT_STR 32
#define MSCP_SSH_MAX_IDENTITY_PATH PATH_MAX
#define MSCP_SSH_MAX_CIPHER_STR 32
#define MSCP_SSH_MAX_HMAC_STR 32
#define MSCP_SSH_MAX_COMP_STR 32 /* yes, no, zlib, zlib@openssh.com, none */
#define MSCP_SSH_MAX_CCALGO_STR 16
#define MSCP_SSH_MAX_PASSWORD 128
#define MSCP_SSH_MAX_PASSPHRASE 128
/**
* @struct mscp_ssh_opts
@@ -68,23 +56,35 @@ struct mscp_opts {
*/
struct mscp_ssh_opts {
/* ssh options */
char login_name[MSCP_SSH_MAX_LOGIN_NAME]; /** ssh username */
char port[MSCP_SSH_MAX_PORT_STR]; /** ssh port */
char config[PATH_MAX]; /** path to ssh_config, default ~/.ssh/config*/
char identity[MSCP_SSH_MAX_IDENTITY_PATH]; /** path to private key */
char cipher[MSCP_SSH_MAX_CIPHER_STR]; /** cipher spec */
char hmac[MSCP_SSH_MAX_HMAC_STR]; /** hmacp spec */
char compress[MSCP_SSH_MAX_COMP_STR]; /** yes, no, zlib@openssh.com */
char ccalgo[MSCP_SSH_MAX_CCALGO_STR]; /** TCP cc algorithm */
char *login_name; /** ssh username */
char *port; /** ssh port */
int ai_family; /** address family */
char *config; /** path to ssh_config, default ~/.ssh/config*/
char *identity; /** path to private key */
char *cipher; /** cipher spec */
char *hmac; /** hmacp spec */
char *compress; /** yes, no, zlib@openssh.com */
char *ccalgo; /** TCP cc algorithm */
char password[MSCP_SSH_MAX_PASSWORD]; /** password auth passowrd */
char passphrase[MSCP_SSH_MAX_PASSPHRASE]; /** passphrase for private key */
char *password; /** password auth passowrd */
char *passphrase; /** passphrase for private key */
int debug_level; /** inclirement libssh debug output level */
bool no_hostkey_check; /** do not check host keys */
bool enable_nagle; /** enable Nagle's algorithm if true */
};
/** @def
* Environment variable that passes password for ssh password auth
*/
#define ENV_SSH_AUTH_PASSWORD "MSCP_SSH_AUTH_PASSWORD"
/** @def
* Environment vraible that passes passphrase for private key
*/
#define ENV_SSH_AUTH_PASSPHRASE "MSCP_SSH_AUTH_PASSPHRASE"
/**
* @struct mscp_stats
* @brief Structure to get mscp statistics
@@ -121,7 +121,6 @@ struct mscp *mscp_init(const char *remote_host, int direction,
* @param m mscp instance.
*
* @return 0 on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*/
int mscp_connect(struct mscp *m);
@@ -137,7 +136,6 @@ int mscp_connect(struct mscp *m);
* @param src_path source file path to be copied.
*
* @return 0 on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*/
int mscp_add_src_path(struct mscp *m, const char *src_path);
@@ -151,7 +149,6 @@ int mscp_add_src_path(struct mscp *m, const char *src_path);
* @param dst_path destination path to which source files copied.
*
* @return 0 on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*/
int mscp_set_dst_path(struct mscp *m, const char *dst_path);
@@ -166,7 +163,6 @@ int mscp_set_dst_path(struct mscp *m, const char *dst_path);
* @param m mscp instance.
*
* @return 0 on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*/
int mscp_scan(struct mscp *m);
@@ -177,7 +173,6 @@ int mscp_scan(struct mscp *m);
*
* @param m mscp instance.
* @return 0 on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*/
int mscp_scan_join(struct mscp *m);
@@ -190,7 +185,6 @@ int mscp_scan_join(struct mscp *m);
* @param m mscp instance.
*
* @return number of threads on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*
* @see mscp_join()
*/
@@ -212,7 +206,6 @@ void mscp_stop(struct mscp *m);
* @param m mscp instance.
*
* @return 0 on success, < 0 if an error occured.
* mscp_get_error() can be used to retrieve error message.
*/
int mscp_join(struct mscp *m);
@@ -262,15 +255,4 @@ enum {
};
/**
* @brief Get the recent error message from libmscp. Note that this
* function is not thread-safe.
*
* @return pointer to the message.
*/
const char *mscp_get_error(void);
#endif /* _MSCP_H_ */

View File

@@ -1,8 +0,0 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _MSCP_VERSION_H_
#define _MSCP_VERSION_H_
#define MSCP_VERSION "@MSCP_VERSION@"
#define MSCP_BUILD_VERSION "@MSCP_BUILD_VERSION@"
#endif /* _MSCP_VERSION_H_ */

View File

@@ -1 +0,0 @@
from mscp.mscp import *

View File

@@ -1,187 +0,0 @@
_retry_import_pymscp = False
try:
import pymscp
except ImportError:
_retry_import_pymscp = True
if _retry_import_pymscp:
""" libmscp.so is not installed on system library paths. So retry
to import libmscp.so installed on the mscp python module
directory.
"""
import os
import sys
import ctypes
if sys.platform == "linux":
libmscp = "libmscp.so"
elif sys.platform == "darwin":
libmscp = "libmscp.dylib"
mscp_dir = os.path.dirname(__file__)
ctypes.cdll.LoadLibrary("{}/{}".format(mscp_dir, libmscp))
import pymscp
# inherit static values from pymscp
LOCAL2REMOTE = pymscp.LOCAL2REMOTE
REMOTE2LOCAL = pymscp.REMOTE2LOCAL
SEVERITY_NONE = pymscp.SEVERITY_NONE
SEVERITY_ERR = pymscp.SEVERITY_ERR
SEVERITY_WARN = pymscp.SEVERITY_WARN
SEVERITY_NOTICE = pymscp.SEVERITY_NOTICE
SEVERITY_INFO = pymscp.SEVERITY_INFO
SEVERITY_DEBUG = pymscp.SEVERITY_DEBUG
STATE_INIT = 0
STATE_CONNECTED = 1
STATE_SCANNED = 2
STATE_RUNNING = 3
STATE_STOPPED = 4
STATE_JOINED = 5
STATE_CLEANED = 6
STATE_RELEASED = 7
_state_str = {
STATE_INIT: "init",
STATE_CONNECTED: "connected",
STATE_SCANNED: "scanned",
STATE_RUNNING: "running",
STATE_STOPPED: "stopped",
STATE_JOINED: "joined",
STATE_CLEANED: "cleaned",
STATE_RELEASED: "released",
}
class mscp:
def __init__(self, remote: str, direction: int, **kwargs):
self.remote = remote
self.direction = direction
kwargs["remote"] = remote
kwargs["direction"] = direction
self.m = pymscp.mscp_init(**kwargs)
self.src_paths = []
self.dst_path = None
self.state = STATE_INIT
def __str__(self):
if not hasattr(self, "state"):
# this instance failed on mscp_init
return "mscp:{}:init-failed"
return "mscp:{}:{}".format(self.remote, self.__state2str())
def __repr__(self):
return "<{}>".format(str(self))
def __del__(self):
if not hasattr(self, "state"):
return # this instance failed on mscp_init
if self.state == STATE_RUNNING:
self.stop()
if self.state == STATE_STOPPED:
self.join()
self.cleanup()
self.release()
def __state2str(self):
return _state_str[self.state]
def connect(self):
if not (self.state == STATE_INIT or state.state == STATE_CLEANED):
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
pymscp.mscp_connect(m = self.m)
self.state = STATE_CONNECTED
def add_src_path(self, src_path: str):
if type(src_path) != str:
raise ValueError("src_path must be str: {}".format(src_path))
self.src_paths.append(src_path)
pymscp.mscp_add_src_path(m = self.m, src_path = src_path)
def set_dst_path(self, dst_path: str):
if type(dst_path) != str:
raise ValueError("dst_path must be str: {}".format(dst_path))
self.dst_path = dst_path
pymscp.mscp_set_dst_path(m = self.m, dst_path = dst_path);
def scan(self):
if self.state == STATE_SCANNED:
return
if self.state != STATE_CONNECTED:
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
if not self.src_paths:
raise RuntimeError("src path list is empty")
if self.dst_path == None:
raise RuntimeError("dst path is not set")
pymscp.mscp_scan(m = self.m)
self.state = STATE_SCANNED
def start(self):
if self.state != STATE_SCANNED:
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
pymscp.mscp_start(m = self.m)
self.state = STATE_RUNNING
def stop(self):
if self.state != STATE_RUNNING:
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
pymscp.mscp_stop(m = self.m)
self.state = STATE_STOPPED
def join(self):
if self.state == STATE_JOINED:
return
if not (self.state == STATE_RUNNING or self.state == STATE_STOPPED):
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
pymscp.mscp_join(m = self.m)
self.state = STATE_JOINED
def stats(self):
return pymscp.mscp_get_stats(m = self.m)
def cleanup(self):
if self.state == STATE_RUNNING:
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
pymscp.mscp_cleanup(m = self.m)
self.state = STATE_CLEANED
def release(self):
if self.state != STATE_CLEANED:
raise RuntimeError("invalid mscp state: {}".format(self.__state2str()))
pymscp.mscp_free(m = self.m)
self.state = STATE_RELEASED
# Simple interface: mscp.copy(src, dst)
def copy(self, src, dst, nonblock = False):
if self.state < STATE_CONNECTED:
self.connect()
if type(src) == list:
for path in src:
self.add_src_path(path)
elif type(src) == str:
self.add_src_path(src)
else:
raise ValueError("src must be str of list: '{}'".format(src))
self.set_dst_path(dst)
self.scan()
self.start()
if nonblock:
return
self.join()
self.cleanup()

View File

@@ -37,10 +37,18 @@ index 1fce7b76..b64d1455 100644
int ssh_buffer_validate_length(struct ssh_buffer_struct *buffer, size_t len);
diff --git a/include/libssh/libssh.h b/include/libssh/libssh.h
index 669a0a96..b6a93ac7 100644
index 669a0a96..da5b4099 100644
--- a/include/libssh/libssh.h
+++ b/include/libssh/libssh.h
@@ -402,6 +402,7 @@ enum ssh_options_e {
@@ -368,6 +368,7 @@ enum ssh_options_e {
SSH_OPTIONS_HOST,
SSH_OPTIONS_PORT,
SSH_OPTIONS_PORT_STR,
+ SSH_OPTIONS_AI_FAMILY,
SSH_OPTIONS_FD,
SSH_OPTIONS_USER,
SSH_OPTIONS_SSH_DIR,
@@ -402,6 +403,7 @@ enum ssh_options_e {
SSH_OPTIONS_GSSAPI_AUTH,
SSH_OPTIONS_GLOBAL_KNOWNHOSTS,
SSH_OPTIONS_NODELAY,
@@ -48,7 +56,7 @@ index 669a0a96..b6a93ac7 100644
SSH_OPTIONS_PUBLICKEY_ACCEPTED_TYPES,
SSH_OPTIONS_PROCESS_CONFIG,
SSH_OPTIONS_REKEY_DATA,
@@ -833,6 +834,7 @@ LIBSSH_API const char* ssh_get_hmac_in(ssh_session session);
@@ -833,6 +835,7 @@ LIBSSH_API const char* ssh_get_hmac_in(ssh_session session);
LIBSSH_API const char* ssh_get_hmac_out(ssh_session session);
LIBSSH_API ssh_buffer ssh_buffer_new(void);
@@ -56,7 +64,7 @@ index 669a0a96..b6a93ac7 100644
LIBSSH_API void ssh_buffer_free(ssh_buffer buffer);
#define SSH_BUFFER_FREE(x) \
do { if ((x) != NULL) { ssh_buffer_free(x); x = NULL; } } while(0)
@@ -843,6 +845,8 @@ LIBSSH_API void *ssh_buffer_get(ssh_buffer buffer);
@@ -843,6 +846,8 @@ LIBSSH_API void *ssh_buffer_get(ssh_buffer buffer);
LIBSSH_API uint32_t ssh_buffer_get_len(ssh_buffer buffer);
LIBSSH_API int ssh_session_set_disconnect_message(ssh_session session, const char *message);
@@ -66,10 +74,18 @@ index 669a0a96..b6a93ac7 100644
#include "libssh/legacy.h"
#endif
diff --git a/include/libssh/session.h b/include/libssh/session.h
index 97936195..e4a7f80c 100644
index 97936195..e4fc4fce 100644
--- a/include/libssh/session.h
+++ b/include/libssh/session.h
@@ -258,6 +258,7 @@ struct ssh_session_struct {
@@ -249,6 +249,7 @@ struct ssh_session_struct {
unsigned long timeout; /* seconds */
unsigned long timeout_usec;
uint16_t port;
+ int ai_family;
socket_t fd;
int StrictHostKeyChecking;
char compressionlevel;
@@ -258,6 +259,7 @@ struct ssh_session_struct {
int flags;
int exp_flags;
int nodelay;
@@ -204,9 +220,27 @@ index 8991e006..e0414801 100644
* @brief Ensure the buffer has at least a certain preallocated size.
*
diff --git a/src/connect.c b/src/connect.c
index 15cae644..e7520f40 100644
index 15cae644..02ef43b4 100644
--- a/src/connect.c
+++ b/src/connect.c
@@ -114,7 +114,7 @@ static int ssh_connect_socket_close(socket_t s)
#endif
}
-static int getai(const char *host, int port, struct addrinfo **ai)
+static int getai(const char *host, int port, int ai_family, struct addrinfo **ai)
{
const char *service = NULL;
struct addrinfo hints;
@@ -123,7 +123,7 @@ static int getai(const char *host, int port, struct addrinfo **ai)
ZERO_STRUCT(hints);
hints.ai_protocol = IPPROTO_TCP;
- hints.ai_family = PF_UNSPEC;
+ hints.ai_family = ai_family > 0 ? ai_family : PF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
if (port == 0) {
@@ -156,6 +156,20 @@ static int set_tcp_nodelay(socket_t socket)
sizeof(opt));
}
@@ -228,6 +262,24 @@ index 15cae644..e7520f40 100644
/**
* @internal
*
@@ -173,7 +187,7 @@ socket_t ssh_connect_host_nonblocking(ssh_session session, const char *host,
struct addrinfo *ai = NULL;
struct addrinfo *itr = NULL;
- rc = getai(host, port, &ai);
+ rc = getai(host, port, session->opts.ai_family, &ai);
if (rc != 0) {
ssh_set_error(session, SSH_FATAL,
"Failed to resolve hostname %s (%s)",
@@ -199,7 +213,7 @@ socket_t ssh_connect_host_nonblocking(ssh_session session, const char *host,
SSH_LOG(SSH_LOG_PACKET, "Resolving %s", bind_addr);
- rc = getai(bind_addr, 0, &bind_ai);
+ rc = getai(bind_addr, 0, session->opts.ai_family, &bind_ai);
if (rc != 0) {
ssh_set_error(session, SSH_FATAL,
"Failed to resolve bind address %s (%s)",
@@ -256,6 +270,18 @@ socket_t ssh_connect_host_nonblocking(ssh_session session, const char *host,
}
}
@@ -248,7 +300,7 @@ index 15cae644..e7520f40 100644
rc = connect(s, itr->ai_addr, itr->ai_addrlen);
if (rc == -1 && (errno != 0) && (errno != EINPROGRESS)) {
diff --git a/src/options.c b/src/options.c
index b3ecffe1..fb966fa1 100644
index b3ecffe1..8de24ed6 100644
--- a/src/options.c
+++ b/src/options.c
@@ -217,6 +217,7 @@ int ssh_options_copy(ssh_session src, ssh_session *dest)
@@ -259,7 +311,17 @@ index b3ecffe1..fb966fa1 100644
new->opts.config_processed = src->opts.config_processed;
new->common.log_verbosity = src->common.log_verbosity;
new->common.callbacks = src->common.callbacks;
@@ -458,6 +459,10 @@ int ssh_options_set_algo(ssh_session session,
@@ -268,6 +269,9 @@ int ssh_options_set_algo(ssh_session session,
* - SSH_OPTIONS_PORT_STR:
* The port to connect to (const char *).
*
+ * - SSH_OPTIONS_AI_FAMILY:
+ * The address family for connecting (int *).
+ *
* - SSH_OPTIONS_FD:
* The file descriptor to use (socket_t).\n
* \n
@@ -458,6 +462,10 @@ int ssh_options_set_algo(ssh_session session,
* Set it to disable Nagle's Algorithm (TCP_NODELAY) on the
* session socket. (int, 0=false)
*
@@ -270,7 +332,29 @@ index b3ecffe1..fb966fa1 100644
* - SSH_OPTIONS_PROCESS_CONFIG
* Set it to false to disable automatic processing of per-user
* and system-wide OpenSSH configuration files. LibSSH
@@ -1017,6 +1022,20 @@ int ssh_options_set(ssh_session session, enum ssh_options_e type,
@@ -571,6 +579,21 @@ int ssh_options_set(ssh_session session, enum ssh_options_e type,
session->opts.port = i & 0xffffU;
}
break;
+ case SSH_OPTIONS_AI_FAMILY:
+ if (value == NULL) {
+ session->opts.ai_family = 0;
+ ssh_set_error_invalid(session);
+ return -1;
+ } else {
+ int *x = (int *) value;
+ if (*x < 0) {
+ session->opts.ai_family = 0;
+ ssh_set_error_invalid(session);
+ return -1;
+ }
+ session->opts.ai_family = *x;
+ }
+ break;
case SSH_OPTIONS_FD:
if (value == NULL) {
session->opts.fd = SSH_INVALID_SOCKET;
@@ -1017,6 +1040,20 @@ int ssh_options_set(ssh_session session, enum ssh_options_e type,
session->opts.nodelay = (*x & 0xff) > 0 ? 1 : 0;
}
break;
@@ -292,10 +376,14 @@ index b3ecffe1..fb966fa1 100644
if (value == NULL) {
ssh_set_error_invalid(session);
diff --git a/src/session.c b/src/session.c
index 8c509699..88602b6a 100644
index 8c509699..307388e5 100644
--- a/src/session.c
+++ b/src/session.c
@@ -108,6 +108,7 @@ ssh_session ssh_new(void)
@@ -105,9 +105,11 @@ ssh_session ssh_new(void)
/* OPTIONS */
session->opts.StrictHostKeyChecking = 1;
session->opts.port = 22;
+ session->opts.ai_family = 0;
session->opts.fd = -1;
session->opts.compressionlevel = 7;
session->opts.nodelay = 0;

View File

@@ -1,37 +0,0 @@
from setuptools import setup, Extension, find_packages
import sys
import os
mypackage_root_dir = os.path.dirname(__file__)
with open(os.path.join(mypackage_root_dir, 'VERSION')) as version_file:
version = version_file.read().strip()
if sys.platform == "linux":
libmscp = "libmscp.so"
elif sys.platform == "darwin":
libmscp = "libmscp.dylib"
data_dir = sys.prefix + "/lib"
libmscp = "build/" + libmscp
setup(
name='mscp',
version = version,
description = "libmscp python binding",
author = "Ryo Nakamura",
author_email = "upa@haeena.net",
url = "https://github.com/upa/mscp",
packages = find_packages("mscp"),
package_dir = {"": "mscp"},
data_files = [ (data_dir, [libmscp])],
py_modules = [ "mscp" ],
ext_modules = [
Extension(
'pymscp',
['src/pymscp.c'],
library_dirs = ['build'],
libraries = ['mscp'],
include_dirs = ['include']
)
]
)

3
rpm/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
# generated by cmake
mscp.spec

45
rpm/mscp.spec.in Normal file
View File

@@ -0,0 +1,45 @@
Name: mscp
Version: @MSCP_VERSION@
Release: 1%{?dist}
Summary: mscp, fast file transfer over multiple SSH connections
Group: Applications/Internet
License: GPLv3
URL: https://github.com/upa/mscp
Source0: %{name}-%{version}.tar.gz
BuildRequires: gcc make cmake zlib-devel openssl-devel
Requires: glibc crypto-policies krb5-libs openssl-libs libcom_err
%description
mscp transfers files over multiple SSH connections. Multiple threads
and connections in mscp transfer (1) multiple files simultaneously
and (2) a large file in parallel. It would shorten the waiting time
for transferring a lot of/large files over networks.
%global debug_package %{nil}
%prep
%setup -q
%build
cmake -S . -B build -DINSTALL_EXECUTABLE_ONLY=ON
make -C build %{?_smp_mflags}
%install
make -C build install DESTDIR=%{buildroot}
%files
/usr/local/bin/mscp
/usr/local/share/man/man1/mscp.1
%changelog
* Wed Feb 07 2024 Ryo nakamura <upa@haeena.net> - 0.1.4-0
- RPM release for v0.1.4
* Sat Feb 03 2024 Ryo nakamura <upa@haeena.net> - 0.1.3-0
- Initial release for rpm packaging

View File

@@ -1,7 +1,10 @@
#!/bin/bash -eu
#!/usr/bin/env bash
#
# Install build dpenedencies.
set -e
set -u
platform=$(uname -s)
if [ -e /etc/os-release ]; then
@@ -16,13 +19,16 @@ case $platform in
brew install openssl@1.1
;;
Linux-ubuntu*)
apt-get install -y \
apt-get install --no-install-recommends -y \
gcc make cmake zlib1g-dev libssl-dev libkrb5-dev
;;
Linux-centos* | Linux-rhel* | Linux-rocky* | Linux-almalinux)
yum install -y \
gcc make cmake zlib-devel openssl-devel rpm-build
;;
FreeBSD-freebsd)
pkg install cmake
;;
*)
echo "unsupported platform: $platform"
exit 1

View File

@@ -1,23 +0,0 @@
#!/bin/bash -e
#
# Print install dpenedencies on Linux. CMake runs this script to obtain deps for CPACK.
# mscp dependes on packages on which libssh depends.
source /etc/os-release
release=$1
case $release in
ubuntu-20.04*)
echo "libc6 (>= 2.27), libgssapi-krb5-2 (>= 1.17), libssl1.1 (>= 1.1.1), zlib1g (>= 1:1.1.4)"
;;
ubuntu-22.04*)
echo "libc6 (>= 2.33), libgssapi-krb5-2 (>= 1.17), libssl3 (>= 3.0.0~~alpha1), zlib1g (>= 1:1.1.4)"
;;
centos* | rhel* | rocky* | almalinux*)
echo "glibc crypto-policies krb5-libs openssl-libs libcom_err"
;;
*)
echo "$(basename $0): unsupported install dependency: $release"
exit 1
esac

View File

@@ -15,6 +15,8 @@ if [ ! -e /var/run/sshd.pid ]; then
fi
ssh-keyscan localhost >> ${HOME}/.ssh/known_hosts
ssh-keyscan 127.0.0.1 >> ${HOME}/.ssh/known_hosts
ssh-keyscan ::1 >> ${HOME}/.ssh/known_hosts
# Run test
python3 -m pytest ../test -v

View File

@@ -6,8 +6,6 @@
#include <assert.h>
#include <pthread.h>
#include <message.h>
typedef int refcnt;
static inline void refcnt_inc(refcnt *cnt)
@@ -20,7 +18,6 @@ static inline refcnt refcnt_dec(refcnt *cnt)
return __sync_sub_and_fetch(cnt, 1);
}
/* mutex */
typedef pthread_mutex_t lock;
@@ -47,14 +44,11 @@ static inline void lock_release_via_cleanup(void *l)
lock_release(l);
}
#define LOCK_ACQUIRE(l) \
lock_acquire(l); \
#define LOCK_ACQUIRE(l) \
lock_acquire(l); \
pthread_cleanup_push(lock_release_via_cleanup, l)
#define LOCK_RELEASE() \
pthread_cleanup_pop(1)
#define LOCK_RELEASE() pthread_cleanup_pop(1)
/* read/write lock */
typedef pthread_rwlock_t rwlock;
@@ -87,18 +81,14 @@ static inline void rwlock_release_via_cleanup(void *rw)
rwlock_release(rw);
}
#define RWLOCK_READ_ACQUIRE(rw) \
rwlock_read_acquire(rw); \
#define RWLOCK_READ_ACQUIRE(rw) \
rwlock_read_acquire(rw); \
pthread_cleanup_push(rwlock_release_via_cleanup, rw)
#define RWLOCK_WRITE_ACQUIRE(rw) \
rwlock_write_acquire(rw); \
#define RWLOCK_WRITE_ACQUIRE(rw) \
rwlock_write_acquire(rw); \
pthread_cleanup_push(rwlock_release_via_cleanup, rw)
#define RWLOCK_RELEASE() \
pthread_cleanup_pop(1)
#define RWLOCK_RELEASE() pthread_cleanup_pop(1)
#endif /* _ATOMIC_H_ */

View File

@@ -4,11 +4,13 @@
#include <errno.h>
#include <fcntl.h>
#include <dirent.h>
#include <sys/times.h>
#include <utime.h>
#include <fileops.h>
#include <ssh.h>
#include <message.h>
#include <print.h>
#include <platform.h>
sftp_session __thread tls_sftp;
/* tls_sftp is used *_wrapped() functions */
@@ -22,7 +24,7 @@ static void sftp_err_to_errno(sftp_session sftp)
{
int sftperr = sftp_get_error(sftp);
switch (sftperr){
switch (sftperr) {
case SSH_FX_OK:
case SSH_FX_EOF:
errno = 0;
@@ -60,11 +62,10 @@ static void sftp_err_to_errno(sftp_session sftp)
errno = ENODEV;
break;
default:
mpr_warn(stderr, "unkown SSH_FX response %d", sftperr);
pr_warn("unkown SSH_FX response %d", sftperr);
}
}
MDIR *mscp_opendir(const char *path, sftp_session sftp)
{
MDIR *md;
@@ -109,7 +110,6 @@ void mscp_closedir(MDIR *md)
free(md);
}
struct dirent __thread tls_dirent;
/* tls_dirent contains dirent converted from sftp_attributes returned
* from sftp_readdir(). This trick is derived from openssh's
@@ -158,14 +158,25 @@ int mscp_mkdir(const char *path, mode_t mode, sftp_session sftp)
return ret;
}
static void sftp_attr_to_stat(sftp_attributes attr, struct stat *st)
{
memset(st, 0, sizeof(*st));
st->st_size = attr->size;
st->st_uid = attr->uid;
st->st_gid = attr->gid;
st->st_mode = attr->permissions;
memset(st, 0, sizeof(*st));
st->st_size = attr->size;
st->st_uid = attr->uid;
st->st_gid = attr->gid;
st->st_mode = attr->permissions;
#if defined(__APPLE__)
#define st_atim st_atimespec
#define st_mtim st_mtimespec
#define st_ctim st_ctimespec
#endif
st->st_atim.tv_sec = attr->atime;
st->st_atim.tv_nsec = attr->atime_nseconds;
st->st_mtim.tv_sec = attr->mtime;
st->st_mtim.tv_nsec = attr->mtime_nseconds;
st->st_ctim.tv_sec = attr->createtime;
st->st_ctim.tv_nsec = attr->createtime_nseconds;
switch (attr->type) {
case SSH_FILEXFER_TYPE_REGULAR:
@@ -184,18 +195,17 @@ static void sftp_attr_to_stat(sftp_attributes attr, struct stat *st)
st->st_mode |= S_IFIFO; /* really? */
break;
default:
mpr_warn(stderr, "unkown SSH_FILEXFER_TYPE %d", attr->type);
pr_warn("unkown SSH_FILEXFER_TYPE %d", attr->type);
}
/* ToDo: convert atime, ctime, and mtime */
}
int mscp_stat(const char *path, struct stat *st, sftp_session sftp)
{
sftp_attributes attr;
int ret = 0;
memset(st, 0, sizeof(*st));
if (sftp) {
attr = sftp_stat(sftp, path);
sftp_err_to_errno(sftp);
@@ -241,7 +251,6 @@ int mscp_lstat_wrapped(const char *path, struct stat *st)
return mscp_lstat(path, st, tls_sftp);
}
mf *mscp_open(const char *path, int flags, mode_t mode, sftp_session sftp)
{
mf *f;
@@ -292,23 +301,34 @@ off_t mscp_lseek(mf *f, off_t off)
return ret;
}
int mscp_setstat(const char *path, mode_t mode, size_t size, sftp_session sftp)
int mscp_setstat(const char *path, struct stat *st, bool preserve_ts, sftp_session sftp)
{
int ret;
if (sftp) {
struct sftp_attributes_struct attr;
memset(&attr, 0, sizeof(attr));
attr.permissions = mode;
attr.size = size;
attr.flags = (SSH_FILEXFER_ATTR_PERMISSIONS|SSH_FILEXFER_ATTR_SIZE);
attr.permissions = st->st_mode;
attr.size = st->st_size;
attr.flags = (SSH_FILEXFER_ATTR_PERMISSIONS | SSH_FILEXFER_ATTR_SIZE);
if (preserve_ts) {
attr.atime = st->st_atim.tv_sec;
attr.atime_nseconds = st->st_atim.tv_nsec;
attr.mtime = st->st_mtim.tv_sec;
attr.mtime_nseconds = st->st_mtim.tv_nsec;
attr.flags |= (SSH_FILEXFER_ATTR_ACCESSTIME |
SSH_FILEXFER_ATTR_MODIFYTIME |
SSH_FILEXFER_ATTR_SUBSECOND_TIMES);
}
ret = sftp_setstat(sftp, path, &attr);
sftp_err_to_errno(sftp);
} else {
if ((ret = chmod(path, mode)) < 0)
if ((ret = chmod(path, st->st_mode)) < 0)
return ret;
if ((ret = truncate(path, size)) < 0)
if ((ret = truncate(path, st->st_size)) < 0)
return ret;
if (preserve_ts)
ret = setutimes(path, st->st_atim, st->st_mtim);
}
return ret;
@@ -329,9 +349,9 @@ int mscp_glob(const char *pattern, int flags, glob_t *pglob, sftp_session sftp)
#else
flags |= GLOB_ALTDIRFUNC;
set_tls_sftp_session(sftp);
#ifdef __APPLE__
#if defined(__APPLE__) || defined(__FreeBSD__)
pglob->gl_opendir = (void *(*)(const char *))mscp_opendir_wrapped;
pglob->gl_readdir = (struct dirent *(*)(void *))mscp_readdir;
pglob->gl_readdir = (struct dirent * (*)(void *)) mscp_readdir;
pglob->gl_closedir = (void (*)(void *))mscp_closedir;
pglob->gl_lstat = mscp_lstat_wrapped;
pglob->gl_stat = mscp_stat_wrapped;

View File

@@ -19,7 +19,6 @@ struct mdir_struct {
};
typedef struct mdir_struct MDIR;
MDIR *mscp_opendir(const char *path, sftp_session sftp);
MDIR *mscp_opendir_wrapped(const char *path);
void mscp_closedir(MDIR *md);
@@ -34,7 +33,6 @@ int mscp_stat_wrapped(const char *path, struct stat *st);
int mscp_lstat(const char *path, struct stat *st, sftp_session sftp);
int mscp_lstat_wrapped(const char *path, struct stat *st);
/* file operations */
struct mf_struct {
@@ -50,7 +48,7 @@ off_t mscp_lseek(mf *f, off_t off);
/* mscp_setstat() involves chmod and truncate. It executes both at
* once via a single SFTP command (sftp_setstat()).
*/
int mscp_setstat(const char *path, mode_t mode, size_t size, sftp_session sftp);
int mscp_setstat(const char *path, struct stat *st, bool preserve_ts, sftp_session sftp);
/* remote glob */
int mscp_glob(const char *pattern, int flags, glob_t *pglob, sftp_session sftp);

View File

@@ -9,20 +9,25 @@
#include <sys/time.h>
#include <sys/ioctl.h>
#include <poll.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <pthread.h>
#include <mscp.h>
#include <mscp_version.h>
#include <util.h>
#include <minmax.h>
#include <strerrno.h>
#include <print.h>
#include "config.h"
void usage(bool print_help) {
printf("mscp " MSCP_BUILD_VERSION ": copy files over multiple ssh connections\n"
void usage(bool print_help)
{
printf("mscp " MSCP_BUILD_VERSION ": copy files over multiple SSH connections\n"
"\n"
"Usage: mscp [vqDHdNh] [-n nr_conns] [-m coremask]\n"
"Usage: mscp [-46vqDpHdNh] [-n nr_conns] [-m coremask]\n"
" [-u max_startups] [-I interval]\n"
" [-s min_chunk_sz] [-S max_chunk_sz] [-a nr_ahead] [-b buf_sz]\n"
" [-l login_name] [-p port] [-F ssh_config] [-i identity_file]\n"
" [-l login_name] [-P port] [-F ssh_config] [-i identity_file]\n"
" [-c cipher_spec] [-M hmac_spec] [-C compress] [-g congestion]\n"
" source ... target\n"
"\n");
@@ -42,13 +47,15 @@ void usage(bool print_help) {
" -a NR_AHEAD number of inflight SFTP commands (default: 32)\n"
" -b BUF_SZ buffer size for i/o and transfer\n"
"\n"
" -4 use IPv4\n"
" -6 use IPv6\n"
" -v increment verbose output level\n"
" -q disable output\n"
" -D dry run. check copy destinations with -vvv\n"
" -r no effect\n"
"\n"
" -l LOGIN_NAME login name\n"
" -p/-P PORT port number\n"
" -P PORT port number\n"
" -F CONFIG path to user ssh config (default ~/.ssh/config)\n"
" -i IDENTITY identity file for public key authentication\n"
" -c CIPHER cipher spec\n"
@@ -56,6 +63,7 @@ void usage(bool print_help) {
" -C COMPRESS enable compression: "
"yes, no, zlib, zlib@openssh.com\n"
" -g CONGESTION specify TCP congestion control algorithm\n"
" -p preserve timestamps of files\n"
" -H disable hostkey check\n"
" -d increment ssh debug output level\n"
" -N enable Nagle's algorithm (default disabled)\n"
@@ -78,7 +86,7 @@ char *split_user_host_path(const char *s, char **userp, char **hostp, char **pat
bool inbrackets = false;
if (!(tmp = strdup(s))) {
fprintf(stderr, "stdrup: %s\n", strerror(errno));
pr_err("stdrup: %s", strerror(errno));
return NULL;
}
@@ -173,17 +181,19 @@ struct target *validate_targets(char **arg, int len)
int n;
if ((t = calloc(len, sizeof(struct target))) == NULL) {
fprintf(stderr, "calloc: %s\n", strerror(errno));
pr_err("calloc: %s", strerrno());
return NULL;
}
memset(t, 0, len * sizeof(struct target));
/* split remote:path into remote and path */
for (n = 0; n < len; n++) {
t[n].copy = split_user_host_path(arg[n], &t[n].user,
&t[n].host, &t[n].path);
if (!t[n].copy)
t[n].copy =
split_user_host_path(arg[n], &t[n].user, &t[n].host, &t[n].path);
if (!t[n].copy) {
pr_err("failed to parse '%s'", arg[n]);
goto free_target_out;
}
}
/* check all user@host are identical. t[len - 1] is destination,
@@ -197,23 +207,24 @@ struct target *validate_targets(char **arg, int len)
/* check inconsistent remote position in args */
if (t[0].host == NULL && t[len - 1].host == NULL) {
fprintf(stderr, "no remote host given\n");
pr_err("no remote host given");
goto free_split_out;
}
if (t[0].host != NULL && t[len - 1].host != NULL) {
fprintf(stderr, "no local path given\n");
pr_err("no local path given");
goto free_split_out;
}
return t;
invalid_remotes:
fprintf(stderr, "invalid remote host notation\n");
pr_err("invalid remote host notation");
free_split_out:
for (n = 0; n < len; n++)
if (t[n].copy) free(t[n].copy);
if (t[n].copy)
free(t[n].copy);
free_target_out:
free(t);
@@ -221,19 +232,20 @@ free_target_out:
}
struct mscp *m = NULL;
int msg_fd = 0;
pthread_t tid_stat = 0;
void sigint_handler(int sig)
{
mscp_stop(m);
if (tid_stat > 0)
pthread_cancel(tid_stat);
}
void *print_stat_thread(void *arg);
void print_cli(const char *fmt, ...)
{
va_list va;
va_list va;
va_start(va, fmt);
vfprintf(stdout, fmt, va);
fflush(stdout);
@@ -255,19 +267,18 @@ int main(int argc, char **argv)
memset(&o, 0, sizeof(o));
o.severity = MSCP_SEVERITY_WARN;
while ((ch = getopt(argc, argv,
"n:m:u:I:s:S:a:b:vqDrl:P:p:i:F:c:M:C:g:HdNh")) != -1) {
#define mscpopts "n:m:u:I:s:S:a:b:46vqDrl:P:i:F:c:M:C:g:pHdNh"
while ((ch = getopt(argc, argv, mscpopts)) != -1) {
switch (ch) {
case 'n':
o.nr_threads = atoi(optarg);
if (o.nr_threads < 1) {
fprintf(stderr, "invalid number of connections: %s\n",
optarg);
pr_err("invalid number of connections: %s", optarg);
return 1;
}
break;
case 'm':
strncpy(o.coremask, optarg, sizeof(o.coremask));
o.coremask = optarg;
break;
case 'u':
o.max_startups = atoi(optarg);
@@ -287,6 +298,12 @@ int main(int argc, char **argv)
case 'b':
o.buf_sz = atoi(optarg);
break;
case '4':
s.ai_family = AF_INET;
break;
case '6':
s.ai_family = AF_INET6;
break;
case 'v':
o.severity++;
break;
@@ -300,58 +317,31 @@ int main(int argc, char **argv)
/* for compatibility with scp */
break;
case 'l':
if (strlen(optarg) > MSCP_SSH_MAX_LOGIN_NAME - 1) {
fprintf(stderr, "long login name: %s\n", optarg);
return -1;
}
strncpy(s.login_name, optarg, MSCP_SSH_MAX_LOGIN_NAME - 1);
s.login_name = optarg;
break;
case 'P':
/* fallthough for compatibility with scp */
case 'p':
if (strlen(optarg) > MSCP_SSH_MAX_PORT_STR - 1) {
fprintf(stderr, "long port string: %s\n", optarg);
return -1;
}
strncpy(s.port, optarg, MSCP_SSH_MAX_PORT_STR);
s.port = optarg;
break;
case 'F':
strncpy(s.config, optarg, PATH_MAX - 1);
s.config = optarg;
break;
case 'i':
if (strlen(optarg) > MSCP_SSH_MAX_IDENTITY_PATH - 1) {
fprintf(stderr, "long identity path: %s\n", optarg);
return -1;
}
strncpy(s.identity, optarg, MSCP_SSH_MAX_IDENTITY_PATH);
s.identity = optarg;
break;
case 'c':
if (strlen(optarg) > MSCP_SSH_MAX_CIPHER_STR - 1) {
fprintf(stderr, "long cipher string: %s\n", optarg);
return -1;
}
strncpy(s.cipher, optarg, MSCP_SSH_MAX_CIPHER_STR);
s.cipher = optarg;
break;
case 'M':
if (strlen(optarg) > MSCP_SSH_MAX_HMAC_STR - 1) {
fprintf(stderr, "long hmac string: %s\n", optarg);
return -1;
}
strncpy(s.hmac, optarg, MSCP_SSH_MAX_HMAC_STR);
s.hmac = optarg;
break;
case 'C':
if (strlen(optarg) > MSCP_SSH_MAX_COMP_STR - 1) {
fprintf(stderr, "long compress string: %s\n", optarg);
return -1;
}
strncpy(s.compress, optarg, MSCP_SSH_MAX_COMP_STR);
s.compress = optarg;
break;
case 'g':
if (strlen(optarg) > MSCP_SSH_MAX_CCALGO_STR - 1) {
fprintf(stderr, "long ccalgo string: %s\n", optarg);
return -1;
}
strncpy(s.ccalgo, optarg, MSCP_SSH_MAX_CCALGO_STR);
s.ccalgo = optarg;
break;
case 'p':
o.preserve_ts = true;
break;
case 'H':
s.no_hostkey_check = true;
@@ -371,6 +361,9 @@ int main(int argc, char **argv)
}
}
s.password = getenv(ENV_SSH_AUTH_PASSWORD);
s.passphrase = getenv(ENV_SSH_AUTH_PASSPHRASE);
if (argc - optind < 2) {
/* mscp needs at lease 2 (src and target) argument */
usage(false);
@@ -385,50 +378,38 @@ int main(int argc, char **argv)
/* copy remote to local */
direction = MSCP_DIRECTION_R2L;
remote = t[0].host;
if (t[0].user != NULL && s.login_name[0] == '\0')
strncpy(s.login_name, t[0].user, MSCP_SSH_MAX_LOGIN_NAME - 1);
s.login_name = s.login_name ? s.login_name : t[0].user;
} else {
/* copy local to remote */
direction = MSCP_DIRECTION_L2R;
remote = t[i - 1].host;
if (t[i - 1].user != NULL && s.login_name[0] == '\0')
strncpy(s.login_name, t[i - 1].user,
MSCP_SSH_MAX_LOGIN_NAME - 1);
}
if (!dryrun) {
if (pipe(pipe_fd) < 0) {
fprintf(stderr, "pipe: %s\n", strerror(errno));
return -1;
}
msg_fd = pipe_fd[0];
o.msg_fd = pipe_fd[1];
s.login_name = s.login_name ? s.login_name : t[i - 1].user;
}
if ((m = mscp_init(remote, direction, &o, &s)) == NULL) {
fprintf(stderr, "mscp_init: %s\n", mscp_get_error());
pr_err("mscp_init: %s", priv_get_err());
return -1;
}
if (mscp_connect(m) < 0) {
fprintf(stderr, "mscp_connect: %s\n", mscp_get_error());
pr_err("mscp_connect: %s", priv_get_err());
return -1;
}
for (n = 0; n < i - 1; n++) {
if (mscp_add_src_path(m, t[n].path) < 0) {
fprintf(stderr, "mscp_add_src_path: %s\n", mscp_get_error());
pr_err("mscp_add_src_path: %s", priv_get_err());
return -1;
}
}
}
if (mscp_set_dst_path(m, t[i - 1].path) < 0) {
fprintf(stderr, "mscp_set_dst_path: %s\n", mscp_get_error());
pr_err("mscp_set_dst_path: %s", priv_get_err());
return -1;
}
if (mscp_scan(m) < 0) {
fprintf(stderr, "mscp_scan: %s\n", mscp_get_error());
pr_err("mscp_scan: %s", priv_get_err());
return -1;
}
@@ -438,22 +419,20 @@ int main(int argc, char **argv)
}
if (pthread_create(&tid_stat, NULL, print_stat_thread, NULL) < 0) {
fprintf(stderr, "pthread_create: %s\n", strerror(errno));
pr_err("pthread_create: %s", strerror(errno));
return -1;
}
if (signal(SIGINT, sigint_handler) == SIG_ERR) {
fprintf(stderr, "signal: %s\n", strerror(errno));
pr_err("signal: %s", strerror(errno));
return -1;
}
ret = mscp_start(m);
if (ret < 0)
fprintf(stderr, "mscp_start: %s\n", mscp_get_error());
pr_err("mscp_start: %s", priv_get_err());
ret = mscp_join(m);
if (ret != 0)
fprintf(stderr, "mscp_join: %s\n", mscp_get_error());
pthread_cancel(tid_stat);
pthread_join(tid_stat, NULL);
@@ -465,52 +444,49 @@ out:
return ret;
}
/* progress bar-related functions */
double calculate_timedelta(struct timeval *b, struct timeval *a)
{
double sec, usec;
double sec, usec;
if (a->tv_usec < b->tv_usec) {
a->tv_usec += 1000000;
a->tv_sec--;
}
if (a->tv_usec < b->tv_usec) {
a->tv_usec += 1000000;
a->tv_sec--;
}
sec = a->tv_sec - b->tv_sec;
usec = a->tv_usec - b->tv_usec;
sec += usec / 1000000;
sec = a->tv_sec - b->tv_sec;
usec = a->tv_usec - b->tv_usec;
sec += usec / 1000000;
return sec;
return sec;
}
double calculate_bps(size_t diff, struct timeval *b, struct timeval *a)
{
return (double)diff / calculate_timedelta(b, a);
return (double)diff / calculate_timedelta(b, a);
}
char *calculate_eta(size_t remain, size_t diff, struct timeval *b, struct timeval *a,
bool final)
{
static char buf[16];
static char buf[16];
#define bps_window_size 16
static double bps_window[bps_window_size];
static size_t sum, idx, count;
double elapsed = calculate_timedelta(b, a);
double elapsed = calculate_timedelta(b, a);
double bps = diff / elapsed;
double avg, eta;
/* early return when diff == 0 (stalled) or final output */
if (diff == 0) {
snprintf(buf, sizeof(buf), "--:-- ETA");
if (diff == 0) {
snprintf(buf, sizeof(buf), "--:-- ETA");
return buf;
}
if (final) {
snprintf(buf, sizeof(buf), "%02d:%02d ",
(int)(floor(elapsed / 60)), (int)round(elapsed) % 60);
snprintf(buf, sizeof(buf), "%02d:%02d ", (int)(floor(elapsed / 60)),
(int)round(elapsed) % 60);
return buf;
}
@@ -524,127 +500,113 @@ char *calculate_eta(size_t remain, size_t diff, struct timeval *b, struct timeva
/* calcuate ETA from avg of recent bps values */
avg = sum / min(count, bps_window_size);
eta = remain / avg;
snprintf(buf, sizeof(buf), "%02d:%02d ETA",
(int)floor(eta / 60), (int)round(eta) % 60);
snprintf(buf, sizeof(buf), "%02d:%02d ETA", (int)floor(eta / 60),
(int)round(eta) % 60);
return buf;
return buf;
}
void print_progress_bar(double percent, char *suffix)
{
int n, thresh, bar_width;
struct winsize ws;
char buf[128];
int n, thresh, bar_width;
struct winsize ws;
char buf[128];
/*
/*
* [=======> ] XX% SUFFIX
*/
buf[0] = '\0';
buf[0] = '\0';
if (ioctl(STDOUT_FILENO, TIOCGWINSZ, &ws) < 0)
return; /* XXX */
bar_width = min(sizeof(buf), ws.ws_col) - strlen(suffix) - 7;
if (ioctl(STDOUT_FILENO, TIOCGWINSZ, &ws) < 0)
return; /* XXX */
bar_width = min(sizeof(buf), ws.ws_col) - strlen(suffix) - 7;
memset(buf, 0, sizeof(buf));
if (bar_width > 8) {
thresh = floor(bar_width * (percent / 100)) - 1;
memset(buf, 0, sizeof(buf));
if (bar_width > 8) {
thresh = floor(bar_width * (percent / 100)) - 1;
for (n = 1; n < bar_width - 1; n++) {
if (n <= thresh)
buf[n] = '=';
else
buf[n] = ' ';
}
buf[thresh] = '>';
buf[0] = '[';
buf[bar_width - 1] = ']';
snprintf(buf + bar_width, sizeof(buf) - bar_width,
" %3d%% ", (int)floor(percent));
}
for (n = 1; n < bar_width - 1; n++) {
if (n <= thresh)
buf[n] = '=';
else
buf[n] = ' ';
}
buf[thresh] = '>';
buf[0] = '[';
buf[bar_width - 1] = ']';
snprintf(buf + bar_width, sizeof(buf) - bar_width, " %3d%% ",
(int)floor(percent));
}
print_cli("\r\033[K" "%s%s", buf, suffix);
print_cli("\r\033[K"
"%s%s",
buf, suffix);
}
void print_progress(struct timeval *b, struct timeval *a,
size_t total, size_t last, size_t done, bool final)
void print_progress(struct timeval *b, struct timeval *a, size_t total, size_t last,
size_t done, bool final)
{
char *bps_units[] = { "B/s ", "KB/s", "MB/s", "GB/s" };
char *byte_units[] = { "B ", "KB", "MB", "GB", "TB", "PB" };
char suffix[128];
int bps_u, byte_tu, byte_du;
double total_round, done_round;
int percent;
double bps;
char *bps_units[] = { "B/s ", "KB/s", "MB/s", "GB/s" };
char *byte_units[] = { "B ", "KB", "MB", "GB", "TB", "PB" };
char suffix[128];
int bps_u, byte_tu, byte_du;
double total_round, done_round;
int percent;
double bps;
#define array_size(a) (sizeof(a) / sizeof(a[0]))
if (total <= 0) {
print_cli("\r\033[K" "total 0 byte transferred");
return; /* copy 0-byte file(s) */
}
if (total <= 0) {
print_cli("\r\033[K"
"total 0 byte transferred");
return; /* copy 0-byte file(s) */
}
total_round = total;
for (byte_tu = 0; total_round > 1000 && byte_tu < array_size(byte_units) - 1;
byte_tu++)
total_round /= 1024;
total_round = total;
for (byte_tu = 0; total_round > 1000 && byte_tu < array_size(byte_units) - 1;
byte_tu++)
total_round /= 1024;
bps = calculate_bps(done - last, b, a);
for (bps_u = 0; bps > 1000 && bps_u < array_size(bps_units); bps_u++)
bps /= 1000;
bps = calculate_bps(done - last, b, a);
for (bps_u = 0; bps > 1000 && bps_u < array_size(bps_units); bps_u++)
bps /= 1000;
percent = floor(((double)(done) / (double)total) * 100);
percent = floor(((double)(done) / (double)total) * 100);
done_round = done;
for (byte_du = 0; done_round > 1024 && byte_du < array_size(byte_units) - 1;
byte_du++)
done_round /= 1024;
done_round = done;
for (byte_du = 0; done_round > 1024 && byte_du < array_size(byte_units) - 1;
byte_du++)
done_round /= 1024;
snprintf(suffix, sizeof(suffix), "%4.1lf%s/%.1lf%s %6.1f%s %s",
done_round, byte_units[byte_du], total_round, byte_units[byte_tu],
bps, bps_units[bps_u],
calculate_eta(total - done, done - last, b, a, final));
snprintf(suffix, sizeof(suffix), "%4.1lf%s/%.1lf%s %6.1f%s %s", done_round,
byte_units[byte_du], total_round, byte_units[byte_tu], bps,
bps_units[bps_u], calculate_eta(total - done, done - last, b, a, final));
print_progress_bar(percent, suffix);
print_progress_bar(percent, suffix);
}
struct xfer_stat {
struct timeval start, before, after;
size_t total;
size_t last;
size_t done;
struct timeval start, before, after;
size_t total;
size_t last;
size_t done;
};
struct xfer_stat x;
void print_stat(bool final)
{
struct pollfd pfd = { .fd = msg_fd, .events = POLLIN };
struct mscp_stats s;
char buf[8192];
int timeout;
if (poll(&pfd, 1, !final ? 100 : 0) < 0) {
fprintf(stderr, "poll: %s\n", strerror(errno));
return;
}
if (pfd.revents & POLLIN) {
memset(buf, 0, sizeof(buf));
if (read(msg_fd, buf, sizeof(buf)) < 0) {
fprintf(stderr, "read: %s\n", strerror(errno));
return;
}
print_cli("\r\033[K" "%s", buf);
}
gettimeofday(&x.after, NULL);
if (calculate_timedelta(&x.before, &x.after) > 1 || final) {
mscp_get_stats(m, &s);
x.total = s.total;
x.done = s.done;
print_progress(!final ? &x.before : &x.start, &x.after,
x.total, !final ? x.last : 0, x.done, final);
print_progress(!final ? &x.before : &x.start, &x.after, x.total,
!final ? x.last : 0, x.done, final);
x.before = x.after;
x.last = x.done;
}
@@ -658,20 +620,20 @@ void print_stat_thread_cleanup(void *arg)
void *print_stat_thread(void *arg)
{
struct pollfd pfd = { .fd = msg_fd, .events = POLLIN };
struct mscp_stats s;
char buf[8192];
memset(&x, 0, sizeof(x));
gettimeofday(&x.start, NULL);
x.before = x.start;
gettimeofday(&x.start, NULL);
x.before = x.start;
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL);
pthread_cleanup_push(print_stat_thread_cleanup, NULL);
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL);
pthread_cleanup_push(print_stat_thread_cleanup, NULL);
while (true) {
print_stat(false);
sleep(1);
}
pthread_cleanup_pop(1);

View File

@@ -1,50 +0,0 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#include <stdio.h>
#include <stdarg.h>
#include <string.h>
#include <limits.h>
#include <pthread.h>
#include <util.h>
#include <message.h>
/* strerror_r wrapper */
__thread char thread_strerror[128];
/* mscp error message buffer */
#define MSCP_ERRMSG_SIZE (PATH_MAX * 2)
static char errmsg[MSCP_ERRMSG_SIZE];
void _mscp_set_error(const char *fmt, ...)
{
va_list va;
memset(errmsg, 0, sizeof(errmsg));
va_start(va, fmt);
vsnprintf(errmsg, sizeof(errmsg) - 1, fmt, va);
va_end(va);
}
const char *mscp_get_error()
{
return errmsg;
}
/* message print functions */
static int mprint_severity = MSCP_SEVERITY_WARN;
void mprint_set_severity(int serverity)
{
if (serverity < 0)
mprint_severity = -1; /* no print */
mprint_severity = serverity;
}
int mprint_get_severity()
{
return mprint_severity;
}

View File

@@ -1,57 +0,0 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _MESSAGE_H_
#define _MESSAGE_H_
#include <libgen.h>
#include <stdio.h>
#include <mscp.h>
/* message print. printed messages are passed to application via msg_fd */
void mprint_set_severity(int severity);
int mprint_get_severity();
#define mprint(fp, severity, fmt, ...) \
do { \
if (fp && severity <= mprint_get_severity()) { \
fprintf(fp, fmt, ##__VA_ARGS__); \
fflush(fp); \
} \
} while (0)
#define mpr_err(fp, fmt, ...) \
mprint(fp, MSCP_SEVERITY_ERR, fmt, ##__VA_ARGS__)
#define mpr_warn(fp, fmt, ...) \
mprint(fp, MSCP_SEVERITY_WARN, fmt, ##__VA_ARGS__)
#define mpr_notice(fp, fmt, ...) \
mprint(fp, MSCP_SEVERITY_NOTICE, fmt, ##__VA_ARGS__)
#define mpr_info(fp, fmt, ...) \
mprint(fp, MSCP_SEVERITY_INFO, fmt, ##__VA_ARGS__)
#define mpr_debug(fp, fmt, ...) \
mprint(fp, MSCP_SEVERITY_DEBUG, fmt, ##__VA_ARGS__)
/* errorno wrapper */
extern __thread char thread_strerror[128];
#ifdef _GNU_SOURCE
/* GNU strerror_r */
#define strerrno() \
strerror_r(errno, thread_strerror, sizeof(thread_strerror))
#else
/* this macro assumes that strerror_r never fails. any good way? */
#define strerrno() \
(strerror_r(errno, thread_strerror, sizeof(thread_strerror)) \
? thread_strerror : thread_strerror)
#endif
/* error message buffer */
#define mscp_set_error(fmt, ...) \
_mscp_set_error("%s:%d:%s: " fmt "\0", \
basename(__FILE__), __LINE__, __func__, ##__VA_ARGS__)
void _mscp_set_error(const char *fmt, ...);
#endif /* _MESSAGE_H_ */

8
src/minmax.h Normal file
View File

@@ -0,0 +1,8 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _MINMAX_H_
#define _MINMAX_H_
#define min(a, b) (((a) > (b)) ? (b) : (a))
#define max(a, b) (((a) > (b)) ? (a) : (b))
#endif /* _MINMAX_H_ */

View File

@@ -6,69 +6,68 @@
#include <semaphore.h>
#include <sys/time.h>
#include <list.h>
#include <util.h>
#include <ssh.h>
#include <list.h>
#include <minmax.h>
#include <ssh.h>
#include <path.h>
#include <fileops.h>
#include <atomic.h>
#include <atomic.h>
#include <platform.h>
#include <message.h>
#include <print.h>
#include <strerrno.h>
#include <mscp.h>
#include <openbsd-compat/openbsd-compat.h>
struct mscp {
char *remote; /* remote host (and uername) */
int direction; /* copy direction */
struct mscp_opts *opts;
struct mscp_ssh_opts *ssh_opts;
char *remote; /* remote host (and uername) */
int direction; /* copy direction */
struct mscp_opts *opts;
struct mscp_ssh_opts *ssh_opts;
FILE *msg_fp; /* writer fd for message pipe */
int *cores; /* usable cpu cores by COREMASK */
int nr_cores; /* length of array of cores */
int *cores; /* usable cpu cores by COREMASK */
int nr_cores; /* length of array of cores */
sem_t *sem; /* semaphore for concurrent
sem_t *sem; /* semaphore for concurrent
* connecting ssh sessions */
sftp_session first; /* first sftp session */
sftp_session first; /* first sftp session */
char dst_path[PATH_MAX];
struct list_head src_list;
struct list_head path_list;
struct chunk_pool cp;
char dst_path[PATH_MAX];
struct list_head src_list;
struct list_head path_list;
struct chunk_pool cp;
pthread_t tid_scan; /* tid for scan thread */
int ret_scan; /* return code from scan thread */
pthread_t tid_scan; /* tid for scan thread */
int ret_scan; /* return code from scan thread */
size_t total_bytes; /* total bytes to be transferred */
size_t total_bytes; /* total bytes to be transferred */
struct list_head thread_list;
rwlock thread_rwlock;
struct list_head thread_list;
rwlock thread_rwlock;
};
struct mscp_thread {
struct list_head list; /* mscp->thread_list */
struct list_head list; /* mscp->thread_list */
struct mscp *m;
int id;
sftp_session sftp;
pthread_t tid;
int cpu;
size_t done;
bool finished;
int ret;
struct mscp *m;
int id;
sftp_session sftp;
pthread_t tid;
int cpu;
size_t done;
bool finished;
int ret;
};
struct src {
struct list_head list; /* mscp->src_list */
struct list_head list; /* mscp->src_list */
char *path;
};
#define DEFAULT_MIN_CHUNK_SZ (64 << 20) /* 64MB */
#define DEFAULT_NR_AHEAD 32
#define DEFAULT_BUF_SZ 16384
#define DEFAULT_MIN_CHUNK_SZ (64 << 20) /* 64MB */
#define DEFAULT_NR_AHEAD 32
#define DEFAULT_BUF_SZ 16384
/* XXX: we use 16384 byte buffer pointed by
* https://api.libssh.org/stable/libssh_tutor_sftp.html. The larget
* read length from sftp_async_read is 65536 byte. Read sizes larger
@@ -76,87 +75,86 @@ struct src {
* sftp_async_read returns 0.
*/
#define DEFAULT_MAX_STARTUPS 8
#define DEFAULT_MAX_STARTUPS 8
#define non_null_string(s) (s[0] != '\0')
static int expand_coremask(const char *coremask, int **cores, int *nr_cores)
{
int n, *core_list, core_list_len = 0, nr_usable, nr_all;
char c[2] = { 'x', '\0' };
const char *_coremask;
long v, needle;
int ncores = nr_cpus();
int n, *core_list, core_list_len = 0, nr_usable, nr_all;
char c[2] = { 'x', '\0' };
const char *_coremask;
long v, needle;
int ncores = nr_cpus();
/*
/*
* This function returns array of usable cores in `cores` and
* returns the number of usable cores (array length) through
* nr_cores.
*/
if (strncmp(coremask, "0x", 2) == 0)
_coremask = coremask + 2;
else
_coremask = coremask;
if (strncmp(coremask, "0x", 2) == 0)
_coremask = coremask + 2;
else
_coremask = coremask;
core_list = realloc(NULL, sizeof(int) * 64);
if (!core_list) {
mscp_set_error("failed to realloc: %s", strerrno());
return -1;
}
core_list = realloc(NULL, sizeof(int) * 64);
if (!core_list) {
priv_set_errv("realloc: %s", strerrno());
return -1;
}
nr_usable = 0;
nr_all = 0;
for (n = strlen(_coremask) - 1; n >=0; n--) {
c[0] = _coremask[n];
v = strtol(c, NULL, 16);
if (v == LONG_MIN || v == LONG_MAX) {
mscp_set_error("invalid coremask: %s", coremask);
return -1;
}
nr_usable = 0;
nr_all = 0;
for (n = strlen(_coremask) - 1; n >= 0; n--) {
c[0] = _coremask[n];
v = strtol(c, NULL, 16);
if (v == LONG_MIN || v == LONG_MAX) {
priv_set_errv("invalid coremask: %s", coremask);
return -1;
}
for (needle = 0x01; needle < 0x10; needle <<= 1) {
nr_all++;
if (nr_all > ncores)
break; /* too long coremask */
if (v & needle) {
nr_usable++;
core_list = realloc(core_list, sizeof(int) * nr_usable);
if (!core_list) {
mscp_set_error("realloc: %s", strerrno());
return -1;
}
core_list[nr_usable - 1] = nr_all - 1;
}
}
}
for (needle = 0x01; needle < 0x10; needle <<= 1) {
nr_all++;
if (nr_all > ncores)
break; /* too long coremask */
if (v & needle) {
nr_usable++;
core_list = realloc(core_list, sizeof(int) * nr_usable);
if (!core_list) {
priv_set_errv("realloc: %s", strerrno());
return -1;
}
core_list[nr_usable - 1] = nr_all - 1;
}
}
}
if (nr_usable < 1) {
mscp_set_error("invalid core mask: %s", coremask);
return -1;
}
if (nr_usable < 1) {
priv_set_errv("invalid core mask: %s", coremask);
return -1;
}
*cores = core_list;
*nr_cores = nr_usable;
return 0;
*cores = core_list;
*nr_cores = nr_usable;
return 0;
}
static int default_nr_threads()
{
return (int)(floor(log(nr_cpus()) * 2) + 1);
return (int)(floor(log(nr_cpus()) * 2) + 1);
}
static int validate_and_set_defaut_params(struct mscp_opts *o)
{
if (o->nr_threads < 0) {
mscp_set_error("invalid nr_threads: %d", o->nr_threads);
priv_set_errv("invalid nr_threads: %d", o->nr_threads);
return -1;
} else if (o->nr_threads == 0)
o->nr_threads = default_nr_threads();
if (o->nr_ahead < 0) {
mscp_set_error("invalid nr_ahead: %d", o->nr_ahead);
priv_set_errv("invalid nr_ahead: %d", o->nr_ahead);
return -1;
} else if (o->nr_ahead == 0)
o->nr_ahead = DEFAULT_NR_AHEAD;
@@ -166,9 +164,9 @@ static int validate_and_set_defaut_params(struct mscp_opts *o)
else {
if (o->min_chunk_sz < getpagesize() ||
o->min_chunk_sz % getpagesize() != 0) {
mscp_set_error("min chunk size must be "
"larget than and multiple of page size %d: %lu",
getpagesize(), o->min_chunk_sz);
priv_set_errv("min chunk size must be "
"larget than and multiple of page size %d: %lu",
getpagesize(), o->min_chunk_sz);
return -1;
}
}
@@ -176,14 +174,14 @@ static int validate_and_set_defaut_params(struct mscp_opts *o)
if (o->max_chunk_sz) {
if (o->max_chunk_sz < getpagesize() ||
o->max_chunk_sz % getpagesize() != 0) {
mscp_set_error("min chunk size must be larget than and "
"multiple of page size %d: %lu",
getpagesize(), o->max_chunk_sz);
priv_set_errv("min chunk size must be larget than and "
"multiple of page size %d: %lu",
getpagesize(), o->max_chunk_sz);
}
if (o->min_chunk_sz > o->max_chunk_sz) {
mscp_set_error("smaller max chunk size than "
"min chunk size: %lu < %lu",
o->max_chunk_sz, o->min_chunk_sz);
priv_set_errv("smaller max chunk size than "
"min chunk size: %lu < %lu",
o->max_chunk_sz, o->min_chunk_sz);
return -1;
}
}
@@ -191,14 +189,14 @@ static int validate_and_set_defaut_params(struct mscp_opts *o)
if (o->buf_sz == 0)
o->buf_sz = DEFAULT_BUF_SZ;
else if (o->buf_sz == 0) {
mscp_set_error("invalid buf size: %lu", o->buf_sz);
priv_set_errv("invalid buf size: %lu", o->buf_sz);
return -1;
}
if (o->max_startups == 0)
o->max_startups = DEFAULT_MAX_STARTUPS;
else if (o->max_startups < 0) {
mscp_set_error("invalid max_startups: %d", o->max_startups);
priv_set_errv("invalid max_startups: %d", o->max_startups);
return -1;
}
@@ -207,30 +205,26 @@ static int validate_and_set_defaut_params(struct mscp_opts *o)
o->max_startups = 1;
}
if (o->msg_fd == 0)
o->msg_fd = STDOUT_FILENO;
return 0;
}
struct mscp *mscp_init(const char *remote_host, int direction,
struct mscp_opts *o, struct mscp_ssh_opts *s)
struct mscp *mscp_init(const char *remote_host, int direction, struct mscp_opts *o,
struct mscp_ssh_opts *s)
{
struct mscp *m;
int n;
if (!remote_host) {
mscp_set_error("empty remote host");
priv_set_errv("empty remote host");
return NULL;
}
if (!(direction == MSCP_DIRECTION_L2R ||
direction == MSCP_DIRECTION_R2L)) {
mscp_set_error("invalid copy direction: %d", direction);
if (!(direction == MSCP_DIRECTION_L2R || direction == MSCP_DIRECTION_R2L)) {
priv_set_errv("invalid copy direction: %d", direction);
return NULL;
}
mprint_set_severity(o->severity);
set_print_severity(o->severity);
if (validate_and_set_defaut_params(o) < 0) {
return NULL;
@@ -238,7 +232,7 @@ struct mscp *mscp_init(const char *remote_host, int direction,
m = malloc(sizeof(*m));
if (!m) {
mscp_set_error("failed to allocate memory: %s", strerrno());
priv_set_errv("malloc: %s", strerrno());
return NULL;
}
@@ -251,32 +245,28 @@ struct mscp *mscp_init(const char *remote_host, int direction,
rwlock_init(&m->thread_rwlock);
if ((m->sem = sem_create(o->max_startups)) == NULL) {
mscp_set_error("sem_create: %s", strerrno());
priv_set_errv("sem_create: %s", strerrno());
goto free_out;
}
m->remote = strdup(remote_host);
if (!m->remote) {
mscp_set_error("failed to allocate memory: %s", strerrno());
priv_set_errv("strdup: %s", strerrno());
goto free_out;
}
m->direction = direction;
if (o->msg_fd > -1) {
m->msg_fp = fdopen(o->msg_fd, "a");
if (!m->msg_fp) {
mscp_set_error("fdopen failed: %s", strerrno());
goto free_out;
}
} else
m->msg_fp = NULL;
if (strlen(o->coremask) > 0) {
if (o->coremask) {
if (expand_coremask(o->coremask, &m->cores, &m->nr_cores) < 0)
goto free_out;
mpr_notice(m->msg_fp, "usable cpu cores:");
for (n = 0; n < m->nr_cores; n++)
mpr_notice(m->msg_fp, " %d", m->cores[n]);
mpr_notice(m->msg_fp, "\n");
char b[512], c[8];
memset(b, 0, sizeof(b));
for (n = 0; n < m->nr_cores; n++) {
memset(c, 0, sizeof(c));
snprintf(c, sizeof(c) - 1, " %d", m->cores[n]);
strlcat(b, c, sizeof(b));
}
pr_notice("usable cpu cores:%s", b);
}
m->opts = o;
@@ -304,14 +294,14 @@ int mscp_add_src_path(struct mscp *m, const char *src_path)
s = malloc(sizeof(*s));
if (!s) {
mscp_set_error("failed to allocate memory: %s", strerrno());
priv_set_errv("malloc: %s", strerrno());
return -1;
}
memset(s, 0, sizeof(*s));
s->path = strdup(src_path);
if (!s->path) {
mscp_set_error("failed to allocate memory: %s", strerrno());
priv_set_errv("malloc: %s", strerrno());
free(s);
return -1;
}
@@ -323,7 +313,7 @@ int mscp_add_src_path(struct mscp *m, const char *src_path)
int mscp_set_dst_path(struct mscp *m, const char *dst_path)
{
if (strlen(dst_path) + 1 >= PATH_MAX) {
mscp_set_error("too long dst path: %s", dst_path);
priv_set_errv("too long dst path: %s", dst_path);
return -1;
}
@@ -337,16 +327,16 @@ int mscp_set_dst_path(struct mscp *m, const char *dst_path)
static int get_page_mask(void)
{
long page_sz = sysconf(_SC_PAGESIZE);
size_t page_mask = 0;
int n;
long page_sz = sysconf(_SC_PAGESIZE);
size_t page_mask = 0;
int n;
for (n = 0; page_sz > 0; page_sz >>= 1, n++) {
page_mask <<= 1;
page_mask |= 1;
}
for (n = 0; page_sz > 0; page_sz >>= 1, n++) {
page_mask <<= 1;
page_mask |= 1;
}
return page_mask >> 1;
return page_mask >> 1;
}
static void mscp_stop_copy_thread(struct mscp *m)
@@ -384,7 +374,7 @@ void *mscp_scan_thread(void *arg)
struct stat ss, ds;
glob_t pglob;
int n;
m->ret_scan = 0;
switch (m->direction) {
@@ -397,13 +387,12 @@ void *mscp_scan_thread(void *arg)
dst_sftp = NULL;
break;
default:
mscp_set_error("invalid copy direction: %d", m->direction);
pr_err("invalid copy direction: %d", m->direction);
goto err_out;
}
/* initialize path_resolve_args */
memset(&a, 0, sizeof(a));
a.msg_fp = m->msg_fp;
a.total_bytes = &m->total_bytes;
if (list_count(&m->src_list) > 1)
@@ -420,19 +409,19 @@ void *mscp_scan_thread(void *arg)
a.max_chunk_sz = m->opts->max_chunk_sz;
a.chunk_align = get_page_mask();
mpr_info(m->msg_fp, "start to walk source path(s)\n");
pr_info("start to walk source path(s)");
/* walk a src_path recusively, and resolve path->dst_path for each src */
list_for_each_entry(s, &m->src_list, list) {
memset(&pglob, 0, sizeof(pglob));
if (mscp_glob(s->path, GLOB_NOCHECK, &pglob, src_sftp) < 0) {
mscp_set_error("mscp_glob: %s", strerrno());
pr_err("mscp_glob: %s", strerrno());
goto err_out;
}
for (n = 0; n < pglob.gl_pathc; n++) {
if (mscp_stat(pglob.gl_pathv[n], &ss, src_sftp) < 0) {
mscp_set_error("stat: %s %s", s->path, strerrno());
pr_err("stat: %s %s", s->path, strerrno());
goto err_out;
}
@@ -453,7 +442,7 @@ void *mscp_scan_thread(void *arg)
mscp_globfree(&pglob);
}
mpr_info(m->msg_fp, "walk source path(s) done\n");
pr_info("walk source path(s) done");
chunk_pool_set_filled(&m->cp);
m->ret_scan = 0;
return NULL;
@@ -468,7 +457,7 @@ int mscp_scan(struct mscp *m)
{
int ret = pthread_create(&m->tid_scan, NULL, mscp_scan_thread, m);
if (ret < 0) {
mscp_set_error("pthread_create_error: %d", ret);
priv_set_err("pthread_create: %d", ret);
m->tid_scan = 0;
mscp_stop(m);
return -1;
@@ -496,8 +485,6 @@ int mscp_scan_join(struct mscp *m)
return 0;
}
static void *mscp_copy_thread(void *arg);
static struct mscp_thread *mscp_copy_thread_spawn(struct mscp *m, int id)
@@ -506,8 +493,8 @@ static struct mscp_thread *mscp_copy_thread_spawn(struct mscp *m, int id)
int ret;
t = malloc(sizeof(*t));
if (!t){
mscp_set_error("malloc: %s,", strerrno());
if (!t) {
priv_set_errv("malloc: %s,", strerrno());
return NULL;
}
@@ -515,13 +502,13 @@ static struct mscp_thread *mscp_copy_thread_spawn(struct mscp *m, int id)
t->m = m;
t->id = id;
if (m->cores == NULL)
t->cpu = -1; /* not pinned to cpu */
t->cpu = -1; /* not pinned to cpu */
else
t->cpu = m->cores[id % m->nr_cores];
ret = pthread_create(&t->tid, NULL, mscp_copy_thread, t);
if (ret < 0) {
mscp_set_error("pthread_create error: %d", ret);
priv_set_errv("pthread_create: %d", ret);
free(t);
return NULL;
}
@@ -529,28 +516,25 @@ static struct mscp_thread *mscp_copy_thread_spawn(struct mscp *m, int id)
return t;
}
int mscp_start(struct mscp *m)
{
struct mscp_thread *t;
int n, ret = 0;
if ((n = chunk_pool_size(&m->cp)) < m->opts->nr_threads) {
mpr_notice(m->msg_fp, "we have only %d chunk(s). "
"set number of connections to %d\n", n, n);
pr_notice("we have %d chunk(s), set number of connections to %d", n, n);
m->opts->nr_threads = n;
}
for (n = 0; n < m->opts->nr_threads; n++) {
t = mscp_copy_thread_spawn(m, n);
if (!t) {
mpr_err(m->msg_fp, "failed to spawn copy thread\n");
if (!t)
break;
}
RWLOCK_WRITE_ACQUIRE(&m->thread_rwlock);
list_add_tail(&t->list, &m->thread_list);
RWLOCK_RELEASE();
}
}
return n;
}
@@ -565,12 +549,12 @@ int mscp_join(struct mscp *m)
/* waiting for scan thread joins... */
ret = mscp_scan_join(m);
/* waiting for copy threads join... */
/* waiting for copy threads join... */
RWLOCK_READ_ACQUIRE(&m->thread_rwlock);
list_for_each_entry(t, &m->thread_list, list) {
pthread_join(t->tid, NULL);
done += t->done;
if (t->ret < 0)
if (t->ret != 0)
ret = t->ret;
if (t->sftp) {
ssh_sftp_close(t->sftp);
@@ -579,8 +563,8 @@ int mscp_join(struct mscp *m)
}
RWLOCK_RELEASE();
if (m->first) {
ssh_sftp_close(m->first);
if (m->first) {
ssh_sftp_close(m->first);
m->first = NULL;
}
@@ -592,8 +576,8 @@ int mscp_join(struct mscp *m)
}
}
mpr_notice(m->msg_fp, "%lu/%lu bytes copied for %lu/%lu files\n",
done, m->total_bytes, nr_copied, nr_tobe_copied);
pr_notice("%lu/%lu bytes copied for %lu/%lu files", done, m->total_bytes,
nr_copied, nr_tobe_copied);
return ret;
}
@@ -617,8 +601,8 @@ static void wait_for_interval(int interval)
static void mscp_copy_thread_cleanup(void *arg)
{
struct mscp_thread *t = arg;
t->finished = true;
struct mscp_thread *t = arg;
t->finished = true;
}
void *mscp_copy_thread(void *arg)
@@ -629,78 +613,85 @@ void *mscp_copy_thread(void *arg)
struct chunk *c;
bool nomore;
pthread_cleanup_push(mscp_copy_thread_cleanup, t);
/* when error occurs, each thread prints error messages
* immediately with pr_* functions. */
if (t->cpu > -1) {
if (set_thread_affinity(pthread_self(), t->cpu) < 0)
pthread_cleanup_push(mscp_copy_thread_cleanup, t);
if (t->cpu > -1) {
if (set_thread_affinity(pthread_self(), t->cpu) < 0) {
pr_err("set_thread_affinity: %s", priv_get_err());
goto err_out;
}
}
pr_notice("thread[%d]: pin to cpu core %d", t->id, t->cpu);
}
if (sem_wait(m->sem) < 0) {
mscp_set_error("sem_wait: %s", strerrno());
mpr_err(m->msg_fp, "%s", mscp_get_error());
pr_err("sem_wait: %s", strerrno());
goto err_out;
}
if (!(nomore = chunk_pool_is_empty(&m->cp))) {
if (m->opts->interval > 0)
wait_for_interval(m->opts->interval);
mpr_notice(m->msg_fp, "thread:%d connecting to %s\n", t->id, m->remote);
pr_notice("thread[%d]: connecting to %s", t->id, m->remote);
t->sftp = ssh_init_sftp_session(m->remote, m->ssh_opts);
}
if (sem_post(m->sem) < 0) {
mscp_set_error("sem_post: %s", strerrno());
mpr_err(m->msg_fp, "%s", mscp_get_error());
pr_err("sem_post: %s", strerrno());
goto err_out;
}
if (nomore) {
mpr_notice(m->msg_fp, "thread:%d no more connections needed\n", t->id);
pr_notice("thread[%d]: no more connections needed", t->id);
goto out;
}
if (!t->sftp) {
mpr_err(m->msg_fp, "thread:%d: %s\n", t->id, mscp_get_error());
pr_err("thread[%d]: %s", t->id, priv_get_err());
goto err_out;
}
switch (m->direction) {
case MSCP_DIRECTION_L2R:
src_sftp = NULL;
dst_sftp = t->sftp;
break;
case MSCP_DIRECTION_R2L:
src_sftp = t->sftp;
dst_sftp = NULL;
break;
default:
return NULL; /* not reached */
}
switch (m->direction) {
case MSCP_DIRECTION_L2R:
src_sftp = NULL;
dst_sftp = t->sftp;
break;
case MSCP_DIRECTION_R2L:
src_sftp = t->sftp;
dst_sftp = NULL;
break;
default:
assert(false);
goto err_out; /* not reached */
}
while (1) {
c = chunk_pool_pop(&m->cp);
while (1) {
c = chunk_pool_pop(&m->cp);
if (c == CHUNK_POP_WAIT) {
usleep(100); /* XXX: hard code */
continue;
}
if (!c)
break; /* no more chunks */
if (!c)
break; /* no more chunks */
if ((t->ret = copy_chunk(m->msg_fp,
c, src_sftp, dst_sftp, m->opts->nr_ahead,
m->opts->buf_sz, &t->done)) < 0)
if ((t->ret = copy_chunk(c, src_sftp, dst_sftp, m->opts->nr_ahead,
m->opts->buf_sz, m->opts->preserve_ts,
&t->done)) < 0)
break;
}
}
pthread_cleanup_pop(1);
pthread_cleanup_pop(1);
if (t->ret < 0)
mpr_err(m->msg_fp, "thread:%d copy failed: %s 0x%010lx-0x%010lx\n",
t->id, c->p->path, c->off, c->off + c->len);
if (t->ret < 0) {
pr_err("thread[%d]: copy failed: %s -> %s, 0x%010lx-0x%010lx, %s", t->id,
c->p->path, c->p->dst_path, c->off, c->off + c->len,
priv_get_err());
}
return NULL;
return NULL;
err_out:
t->finished = true;
@@ -712,7 +703,6 @@ out:
return NULL;
}
/* cleanup-related functions */
static void list_free_src(struct list_head *list)
@@ -739,8 +729,8 @@ static void list_free_thread(struct list_head *list)
void mscp_cleanup(struct mscp *m)
{
if (m->first) {
ssh_sftp_close(m->first);
if (m->first) {
ssh_sftp_close(m->first);
m->first = NULL;
}

View File

@@ -0,0 +1,10 @@
#ifndef _OPENBSD_COMPAT_H
#define _OPENBSD_COMPAT_H
#include "config.h"
#ifndef HAVE_STRLCAT
size_t strlcat(char *dst, const char *src, size_t siz);
#endif
#endif /* _OPENBSD_COMPAT_H_ */

View File

@@ -0,0 +1,62 @@
/* $OpenBSD: strlcat.c,v 1.13 2005/08/08 08:05:37 espie Exp $ */
/*
* Copyright (c) 1998 Todd C. Miller <Todd.Miller@courtesan.com>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
/* OPENBSD ORIGINAL: lib/libc/string/strlcat.c */
#include "config.h"
#ifndef HAVE_STRLCAT
#include <sys/types.h>
#include <string.h>
/*
* Appends src to string dst of size siz (unlike strncat, siz is the
* full size of dst, not space left). At most siz-1 characters
* will be copied. Always NUL terminates (unless siz <= strlen(dst)).
* Returns strlen(src) + MIN(siz, strlen(initial dst)).
* If retval >= siz, truncation occurred.
*/
size_t
strlcat(char *dst, const char *src, size_t siz)
{
char *d = dst;
const char *s = src;
size_t n = siz;
size_t dlen;
/* Find the end of dst and adjust bytes left but don't go past end */
while (n-- != 0 && *d != '\0')
d++;
dlen = d - dst;
n = siz - dlen;
if (n == 0)
return(dlen + strlen(s));
while (*s != '\0') {
if (n != 1) {
*d++ = *s;
n--;
}
s++;
}
*d = '\0';
return(dlen + (s - src)); /* count does not include NUL */
}
#endif /* !HAVE_STRLCAT */

View File

@@ -7,16 +7,17 @@
#include <assert.h>
#include <ssh.h>
#include <util.h>
#include <minmax.h>
#include <fileops.h>
#include <list.h>
#include <atomic.h>
#include <path.h>
#include <message.h>
#include <strerrno.h>
#include <print.h>
/* chunk pool operations */
#define CHUNK_POOL_STATE_FILLING 0
#define CHUNK_POOL_STATE_FILLED 1
#define CHUNK_POOL_STATE_FILLING 0
#define CHUNK_POOL_STATE_FILLED 1
void chunk_pool_init(struct chunk_pool *cp)
{
@@ -82,9 +83,9 @@ struct chunk *chunk_pool_pop(struct chunk_pool *cp)
static void chunk_free(struct list_head *list)
{
struct chunk *c;
c = list_entry(list, typeof(*c), list);
free(c);
struct chunk *c;
c = list_entry(list, typeof(*c), list);
free(c);
}
void chunk_pool_release(struct chunk_pool *cp)
@@ -95,17 +96,17 @@ void chunk_pool_release(struct chunk_pool *cp)
/* paths of copy source resoltion */
static char *resolve_dst_path(const char *src_file_path, struct path_resolve_args *a)
{
char copy[PATH_MAX + 1], dst_file_path[PATH_MAX + 1];
char *prefix;
int offset;
char copy[PATH_MAX + 1], dst_file_path[PATH_MAX + 1];
char *prefix;
int offset;
int ret;
strncpy(copy, a->src_path, PATH_MAX);
prefix = dirname(copy);
if (!prefix) {
mscp_set_error("dirname: %s", strerrno());
return NULL;
}
strncpy(copy, a->src_path, PATH_MAX);
prefix = dirname(copy);
if (!prefix) {
pr_err("dirname: %s", strerrno());
return NULL;
}
offset = strlen(prefix) + 1;
if (strlen(prefix) == 1) { /* corner cases */
@@ -119,93 +120,93 @@ static char *resolve_dst_path(const char *src_file_path, struct path_resolve_arg
}
}
if (!a->src_path_is_dir && !a->dst_path_is_dir) {
/* src path is file. dst path is (1) file, or (2) does not exist.
if (!a->src_path_is_dir && !a->dst_path_is_dir) {
/* src path is file. dst path is (1) file, or (2) does not exist.
* In the second case, we need to put src under the dst.
*/
if (a->dst_path_should_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s",
a->dst_path, a->src_path + offset);
else
ret = snprintf(dst_file_path, PATH_MAX, "%s", a->dst_path);
}
if (a->dst_path_should_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s", a->dst_path,
a->src_path + offset);
else
ret = snprintf(dst_file_path, PATH_MAX, "%s", a->dst_path);
}
/* src is file, and dst is dir */
if (!a->src_path_is_dir && a->dst_path_is_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s",
a->dst_path, a->src_path + offset);
/* src is file, and dst is dir */
if (!a->src_path_is_dir && a->dst_path_is_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s", a->dst_path,
a->src_path + offset);
/* both are directory */
if (a->src_path_is_dir && a->dst_path_is_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s",
a->dst_path, src_file_path + offset);
/* both are directory */
if (a->src_path_is_dir && a->dst_path_is_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s", a->dst_path,
src_file_path + offset);
/* dst path does not exist. change dir name to dst_path */
if (a->src_path_is_dir && !a->dst_path_is_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s",
a->dst_path, src_file_path + strlen(a->src_path) + 1);
/* dst path does not exist. change dir name to dst_path */
if (a->src_path_is_dir && !a->dst_path_is_dir)
ret = snprintf(dst_file_path, PATH_MAX, "%s/%s", a->dst_path,
src_file_path + strlen(a->src_path) + 1);
if (ret >= PATH_MAX) {
mpr_warn(a->msg_fp, "Too long path: %s\n", dst_file_path);
pr_warn("Too long path: %s", dst_file_path);
return NULL;
}
mpr_debug(a->msg_fp, "file: %s -> %s\n", src_file_path, dst_file_path);
pr_debug("file: %s -> %s", src_file_path, dst_file_path);
return strndup(dst_file_path, PATH_MAX);
return strndup(dst_file_path, PATH_MAX);
}
/* chunk preparation */
static struct chunk *alloc_chunk(struct path *p)
{
struct chunk *c;
struct chunk *c;
if (!(c = malloc(sizeof(*c)))) {
mscp_set_error("malloc %s", strerrno());
return NULL;
}
memset(c, 0, sizeof(*c));
if (!(c = malloc(sizeof(*c)))) {
pr_err("malloc %s", strerrno());
return NULL;
}
memset(c, 0, sizeof(*c));
c->p = p;
c->off = 0;
c->len = 0;
refcnt_inc(&p->refcnt);
return c;
c->p = p;
c->off = 0;
c->len = 0;
refcnt_inc(&p->refcnt);
return c;
}
static int resolve_chunk(struct path *p, struct path_resolve_args *a)
{
struct chunk *c;
size_t chunk_sz;
size_t size;
struct chunk *c;
size_t chunk_sz;
size_t size;
if (p->size <= a->min_chunk_sz)
chunk_sz = p->size;
else if (a->max_chunk_sz)
chunk_sz = a->max_chunk_sz;
else {
chunk_sz = (p->size - (p->size % a->nr_conn)) / a->nr_conn;
chunk_sz &= ~a->chunk_align; /* align with page_sz */
if (chunk_sz <= a->min_chunk_sz)
chunk_sz = a->min_chunk_sz;
}
if (p->size <= a->min_chunk_sz)
chunk_sz = p->size;
else if (a->max_chunk_sz)
chunk_sz = a->max_chunk_sz;
else {
chunk_sz = (p->size - (p->size % a->nr_conn)) / a->nr_conn;
chunk_sz &= ~a->chunk_align; /* align with page_sz */
if (chunk_sz <= a->min_chunk_sz)
chunk_sz = a->min_chunk_sz;
}
/* for (size = f->size; size > 0;) does not create a file
/* for (size = f->size; size > 0;) does not create a file
* (chunk) when file size is 0. This do {} while (size > 0)
* creates just open/close a 0-byte file.
*/
size = p->size;
do {
c = alloc_chunk(p);
if (!c)
return -1;
c->off = p->size - size;
c->len = size < chunk_sz ? size : chunk_sz;
size -= c->len;
chunk_pool_add(a->cp, c);
} while (size > 0);
size = p->size;
do {
c = alloc_chunk(p);
if (!c)
return -1;
c->off = p->size - size;
c->len = size < chunk_sz ? size : chunk_sz;
size -= c->len;
chunk_pool_add(a->cp, c);
} while (size > 0);
return 0;
return 0;
}
void free_path(struct path *p)
@@ -223,15 +224,17 @@ static int append_path(sftp_session sftp, const char *path, struct stat st,
struct path *p;
if (!(p = malloc(sizeof(*p)))) {
mscp_set_error("failed to allocate memory: %s", strerrno());
priv_set_errv("failed to allocate memory: %s", strerrno());
return -1;
}
memset(p, 0, sizeof(*p));
INIT_LIST_HEAD(&p->list);
p->path = strndup(path, PATH_MAX);
if (!p->path)
if (!p->path) {
pr_err("strndup: %s", strerrno());
goto free_out;
}
p->size = st.st_size;
p->mode = st.st_mode;
p->state = FILE_STATE_INIT;
@@ -256,13 +259,13 @@ free_out:
}
static bool check_path_should_skip(const char *path)
{
int len = strlen(path);
if ((len == 1 && strncmp(path, ".", 1) == 0) ||
(len == 2 && strncmp(path, "..", 2) == 0)) {
return true;
}
return false;
{
int len = strlen(path);
if ((len == 1 && strncmp(path, ".", 1) == 0) ||
(len == 2 && strncmp(path, "..", 2) == 0)) {
return true;
}
return false;
}
static int walk_path_recursive(sftp_session sftp, const char *path,
@@ -275,7 +278,7 @@ static int walk_path_recursive(sftp_session sftp, const char *path,
int ret;
if (mscp_stat(path, &st, sftp) < 0) {
mpr_warn(a->msg_fp, "%s: %s\n", strerrno(), path);
pr_err("stat: %s: %s", path, strerrno());
return -1;
}
@@ -289,23 +292,27 @@ static int walk_path_recursive(sftp_session sftp, const char *path,
/* ok, this path is a directory. walk through it. */
if (!(d = mscp_opendir(path, sftp))) {
mpr_warn(a->msg_fp, "%s: %s\n", strerrno(), path);
pr_err("opendir: %s: %s", path, strerrno());
return -1;
}
for (e = mscp_readdir(d); e; e = mscp_readdir(d)) {
if (check_path_should_skip(e->d_name))
continue;
ret = snprintf(next_path, PATH_MAX, "%s/%s", path, e->d_name);
if (ret >= PATH_MAX) {
mpr_warn(a->msg_fp, "Too long path: %s/%s\n", path, e->d_name);
pr_warn("Too long path: %s/%s", path, e->d_name);
continue;
}
walk_path_recursive(sftp, next_path, path_list, a);
/* do not stop even when walk_path_recursive returns
* -1 due to an unreadable file. go to a next file. */
* -1 due to an unreadable file. go to a next
* file. Thus, do not pass error messages via
* priv_set_err() under walk_path_recursive. Print
* the error with pr_err immediately.
*/
}
mscp_closedir(d);
@@ -319,69 +326,59 @@ int walk_src_path(sftp_session src_sftp, const char *src_path,
return walk_path_recursive(src_sftp, src_path, path_list, a);
}
void path_dump(struct list_head *path_list)
{
struct path *p;
list_for_each_entry(p, path_list, list) {
printf("src: %s %lu-byte\n", p->path, p->size);
printf("dst: %s\n", p->dst_path);
}
}
/* based on
* https://stackoverflow.com/questions/2336242/recursive-mkdir-system-call-on-unix */
static int touch_dst_path(struct path *p, sftp_session sftp)
{
/* XXX: should reflect the permission of the original directory? */
mode_t mode = S_IRWXU | S_IRWXG | S_IRWXO;
/* XXX: should reflect the permission of the original directory? */
mode_t mode = S_IRWXU | S_IRWXG | S_IRWXO;
struct stat st;
char path[PATH_MAX];
char *needle;
int ret;
char path[PATH_MAX];
char *needle;
int ret;
mf *f;
strncpy(path, p->dst_path, sizeof(path));
strncpy(path, p->dst_path, sizeof(path));
/* mkdir -p.
/* mkdir -p.
* XXX: this may be slow when dst is the remote side. need speed-up. */
for (needle = strchr(path + 1, '/'); needle; needle = strchr(needle + 1, '/')) {
*needle = '\0';
for (needle = strchr(path + 1, '/'); needle; needle = strchr(needle + 1, '/')) {
*needle = '\0';
if (mscp_stat(path, &st, sftp) == 0) {
if (S_ISDIR(st.st_mode))
goto next; /* directory exists. go deeper */
else
else {
priv_set_errv("mscp_stat %s: not a directory", path);
return -1; /* path exists, but not directory. */
}
}
if (errno == ENOENT) {
/* no file on the path. create directory. */
if (mscp_mkdir(path, mode, sftp) < 0) {
mscp_set_error("mscp_mkdir %s: %s", path, strerrno());
priv_set_errv("mscp_mkdir %s: %s", path, strerrno());
return -1;
}
}
next:
*needle = '/';
}
next:
*needle = '/';
}
/* Do not set O_TRUNC here. Instead, do mscp_setstat() at the
* end. see https://bugzilla.mindrot.org/show_bug.cgi?id=3431 */
f = mscp_open(p->dst_path, O_WRONLY|O_CREAT, S_IRUSR|S_IWUSR, sftp);
f = mscp_open(p->dst_path, O_WRONLY | O_CREAT, S_IRUSR | S_IWUSR, sftp);
if (!f) {
mscp_set_error("mscp_open %s: %s\n", p->dst_path, strerrno());
priv_set_errv("mscp_open %s: %s\n", p->dst_path, strerrno());
return -1;
}
mscp_close(f);
return 0;
return 0;
}
static int prepare_dst_path(FILE *msg_fp, struct path *p, sftp_session dst_sftp)
static int prepare_dst_path(struct path *p, sftp_session dst_sftp)
{
int ret = 0;
@@ -392,7 +389,7 @@ static int prepare_dst_path(FILE *msg_fp, struct path *p, sftp_session dst_sftp)
goto out;
}
p->state = FILE_STATE_OPENED;
mpr_info(msg_fp, "copy start: %s\n", p->path);
pr_info("copy start: %s", p->path);
}
out:
@@ -400,161 +397,157 @@ out:
return ret;
}
/* functions for copy */
static ssize_t read_to_buf(void *ptr, size_t len, void *userdata)
{
int fd = *((int *)userdata);
return read(fd, ptr, len);
int fd = *((int *)userdata);
return read(fd, ptr, len);
}
static int copy_chunk_l2r(struct chunk *c, int fd, sftp_file sf,
int nr_ahead, int buf_sz, size_t *counter)
static int copy_chunk_l2r(struct chunk *c, int fd, sftp_file sf, int nr_ahead, int buf_sz,
size_t *counter)
{
ssize_t read_bytes, remaind, thrown;
int idx, ret;
struct {
uint32_t id;
ssize_t len;
} reqs[nr_ahead];
ssize_t read_bytes, remaind, thrown;
int idx, ret;
struct {
uint32_t id;
ssize_t len;
} reqs[nr_ahead];
if (c->len == 0)
return 0;
if (c->len == 0)
return 0;
remaind = thrown = c->len;
for (idx = 0; idx < nr_ahead && thrown > 0; idx++) {
reqs[idx].len = min(thrown, buf_sz);
reqs[idx].len = sftp_async_write(sf, read_to_buf, reqs[idx].len, &fd,
&reqs[idx].id);
if (reqs[idx].len < 0) {
mscp_set_error("sftp_async_write: %s or %s",
sftp_get_ssh_error(sf->sftp), strerrno());
return -1;
}
thrown -= reqs[idx].len;
}
remaind = thrown = c->len;
for (idx = 0; idx < nr_ahead && thrown > 0; idx++) {
reqs[idx].len = min(thrown, buf_sz);
reqs[idx].len = sftp_async_write(sf, read_to_buf, reqs[idx].len, &fd,
&reqs[idx].id);
if (reqs[idx].len < 0) {
priv_set_errv("sftp_async_write: %s or %s",
sftp_get_ssh_error(sf->sftp), strerrno());
return -1;
}
thrown -= reqs[idx].len;
}
for (idx = 0; remaind > 0; idx = (idx + 1) % nr_ahead) {
ret = sftp_async_write_end(sf, reqs[idx].id, 1);
if (ret != SSH_OK) {
mscp_set_error("sftp_async_write_end: %s",
sftp_get_ssh_error(sf->sftp));
return -1;
}
for (idx = 0; remaind > 0; idx = (idx + 1) % nr_ahead) {
ret = sftp_async_write_end(sf, reqs[idx].id, 1);
if (ret != SSH_OK) {
priv_set_errv("sftp_async_write_end: %s",
sftp_get_ssh_error(sf->sftp));
return -1;
}
*counter += reqs[idx].len;
remaind -= reqs[idx].len;
*counter += reqs[idx].len;
remaind -= reqs[idx].len;
if (remaind <= 0)
break;
if (remaind <= 0)
break;
if (thrown <= 0)
continue;
if (thrown <= 0)
continue;
reqs[idx].len = min(thrown, buf_sz);
reqs[idx].len = sftp_async_write(sf, read_to_buf, reqs[idx].len, &fd,
&reqs[idx].id);
if (reqs[idx].len < 0) {
mscp_set_error("sftp_async_write: %s or %s",
sftp_get_ssh_error(sf->sftp), strerrno());
return -1;
}
thrown -= reqs[idx].len;
}
reqs[idx].len = min(thrown, buf_sz);
reqs[idx].len = sftp_async_write(sf, read_to_buf, reqs[idx].len, &fd,
&reqs[idx].id);
if (reqs[idx].len < 0) {
priv_set_errv("sftp_async_write: %s or %s",
sftp_get_ssh_error(sf->sftp), strerrno());
return -1;
}
thrown -= reqs[idx].len;
}
if (remaind < 0) {
mscp_set_error("invalid remaind bytes %ld. "
"last async_write_end bytes %lu.",
remaind, reqs[idx].len);
return -1;
}
return 0;
if (remaind < 0) {
priv_set_errv("invalid remaind bytes %ld. "
"last async_write_end bytes %lu.",
remaind, reqs[idx].len);
return -1;
}
return 0;
}
static int copy_chunk_r2l(struct chunk *c, sftp_file sf, int fd,
int nr_ahead, int buf_sz, size_t *counter)
static int copy_chunk_r2l(struct chunk *c, sftp_file sf, int fd, int nr_ahead, int buf_sz,
size_t *counter)
{
ssize_t read_bytes, write_bytes, remaind, thrown;
char buf[buf_sz];
int idx;
struct {
int id;
ssize_t len;
} reqs[nr_ahead];
ssize_t read_bytes, write_bytes, remaind, thrown;
char buf[buf_sz];
int idx;
struct {
int id;
ssize_t len;
} reqs[nr_ahead];
if (c->len == 0)
return 0;
if (c->len == 0)
return 0;
remaind = thrown = c->len;
remaind = thrown = c->len;
for (idx = 0; idx < nr_ahead && thrown > 0; idx++) {
reqs[idx].len = min(thrown, sizeof(buf));
reqs[idx].id = sftp_async_read_begin(sf, reqs[idx].len);
if (reqs[idx].id < 0) {
mscp_set_error("sftp_async_read_begin: %d",
sftp_get_error(sf->sftp));
return -1;
}
thrown -= reqs[idx].len;
}
for (idx = 0; idx < nr_ahead && thrown > 0; idx++) {
reqs[idx].len = min(thrown, sizeof(buf));
reqs[idx].id = sftp_async_read_begin(sf, reqs[idx].len);
if (reqs[idx].id < 0) {
priv_set_errv("sftp_async_read_begin: %d",
sftp_get_error(sf->sftp));
return -1;
}
thrown -= reqs[idx].len;
}
for (idx = 0; remaind > 0; idx = (idx + 1) % nr_ahead) {
read_bytes = sftp_async_read(sf, buf, reqs[idx].len, reqs[idx].id);
if (read_bytes == SSH_ERROR) {
mscp_set_error("sftp_async_read: %d",
sftp_get_error(sf->sftp));
return -1;
}
for (idx = 0; remaind > 0; idx = (idx + 1) % nr_ahead) {
read_bytes = sftp_async_read(sf, buf, reqs[idx].len, reqs[idx].id);
if (read_bytes == SSH_ERROR) {
priv_set_errv("sftp_async_read: %d", sftp_get_error(sf->sftp));
return -1;
}
if (thrown > 0) {
reqs[idx].len = min(thrown, sizeof(buf));
reqs[idx].id = sftp_async_read_begin(sf, reqs[idx].len);
thrown -= reqs[idx].len;
}
if (thrown > 0) {
reqs[idx].len = min(thrown, sizeof(buf));
reqs[idx].id = sftp_async_read_begin(sf, reqs[idx].len);
thrown -= reqs[idx].len;
}
write_bytes = write(fd, buf, read_bytes);
if (write_bytes < 0) {
mscp_set_error("write: %s", strerrno());
return -1;
}
write_bytes = write(fd, buf, read_bytes);
if (write_bytes < 0) {
priv_set_errv("write: %s", strerrno());
return -1;
}
if (write_bytes < read_bytes) {
mscp_set_error("failed to write full bytes");
return -1;
}
if (write_bytes < read_bytes) {
priv_set_errv("failed to write full bytes");
return -1;
}
*counter += write_bytes;
remaind -= read_bytes;
}
*counter += write_bytes;
remaind -= read_bytes;
}
if (remaind < 0) {
mscp_set_error("invalid remaind bytes %ld. last async_read bytes %ld. "
"last write bytes %ld",
remaind, read_bytes, write_bytes);
return -1;
}
if (remaind < 0) {
priv_set_errv("invalid remaind bytes %ld. last async_read bytes %ld. "
"last write bytes %ld",
remaind, read_bytes, write_bytes);
return -1;
}
return 0;
return 0;
}
static int _copy_chunk(struct chunk *c, mf *s, mf *d,
int nr_ahead, int buf_sz, size_t *counter)
static int _copy_chunk(struct chunk *c, mf *s, mf *d, int nr_ahead, int buf_sz,
size_t *counter)
{
if (s->local && d->remote) /* local to remote copy */
return copy_chunk_l2r(c, s->local, d->remote, nr_ahead, buf_sz, counter);
else if (s->remote && d->local) /* remote to local copy */
else if (s->remote && d->local) /* remote to local copy */
return copy_chunk_r2l(c, s->remote, d->local, nr_ahead, buf_sz, counter);
assert(false);
return -1; /* not reached */
}
int copy_chunk(FILE *msg_fp, struct chunk *c,
sftp_session src_sftp, sftp_session dst_sftp,
int nr_ahead, int buf_sz, size_t *counter)
int copy_chunk(struct chunk *c, sftp_session src_sftp, sftp_session dst_sftp,
int nr_ahead, int buf_sz, bool preserve_ts, size_t *counter)
{
mode_t mode;
int flags;
@@ -563,44 +556,41 @@ int copy_chunk(FILE *msg_fp, struct chunk *c,
assert((src_sftp && !dst_sftp) || (!src_sftp && dst_sftp));
if (prepare_dst_path(msg_fp, c->p, dst_sftp) < 0)
if (prepare_dst_path(c->p, dst_sftp) < 0)
return -1;
/* open src */
flags = O_RDONLY;
mode = S_IRUSR;
flags = O_RDONLY;
mode = S_IRUSR;
s = mscp_open(c->p->path, flags, mode, src_sftp);
if (!s) {
mscp_set_error("mscp_open: %s: %s", c->p->path, strerrno());
priv_set_errv("mscp_open: %s: %s", c->p->path, strerrno());
return -1;
}
if (mscp_lseek(s, c->off) < 0) {
mscp_set_error("mscp_lseek: %s: %s", c->p->path, strerrno());
priv_set_errv("mscp_lseek: %s: %s", c->p->path, strerrno());
return -1;
}
/* open dst */
flags = O_WRONLY;
mode = S_IRUSR|S_IWUSR;
flags = O_WRONLY;
mode = S_IRUSR | S_IWUSR;
d = mscp_open(c->p->dst_path, flags, mode, dst_sftp);
if (!d) {
mscp_close(s);
mscp_set_error("mscp_open: %s: %s", c->p->dst_path, strerrno());
priv_set_errv("mscp_open: %s: %s", c->p->dst_path, strerrno());
return -1;
}
if (mscp_lseek(d, c->off) < 0) {
mscp_set_error("mscp_lseek: %s: %s", c->p->dst_path, strerrno());
priv_set_errv("mscp_lseek: %s: %s", c->p->dst_path, strerrno());
return -1;
}
mpr_debug(msg_fp, "copy chunk start: %s 0x%lx-0x%lx\n",
c->p->path, c->off, c->off + c->len);
pr_debug("copy chunk start: %s 0x%lx-0x%lx", c->p->path, c->off, c->off + c->len);
ret = _copy_chunk(c, s, d, nr_ahead, buf_sz, counter);
mpr_debug(msg_fp, "copy chunk done: %s 0x%lx-0x%lx\n",
c->p->path, c->off, c->off + c->len);
pr_debug("copy chunk done: %s 0x%lx-0x%lx", c->p->path, c->off, c->off + c->len);
mscp_close(d);
mscp_close(s);
@@ -608,11 +598,19 @@ int copy_chunk(FILE *msg_fp, struct chunk *c,
return ret;
if (refcnt_dec(&c->p->refcnt) == 0) {
struct stat st;
c->p->state = FILE_STATE_DONE;
if (mscp_setstat(c->p->dst_path, c->p->mode, c->p->size, dst_sftp) < 0)
mpr_err(msg_fp, "failed to chmod and truncate %s: %s\n",
c->p->path, strerrno());
mpr_info(msg_fp, "copy done: %s\n", c->p->path);
/* sync stat */
if (mscp_stat(c->p->path, &st, src_sftp) < 0) {
priv_set_errv("mscp_stat: %s: %s", c->p->path, strerrno());
return -1;
}
if (mscp_setstat(c->p->dst_path, &st, preserve_ts, dst_sftp) < 0) {
priv_set_errv("mscp_setstat: %s: %s", c->p->path, strerrno());
return -1;
}
pr_info("copy done: %s", c->p->path);
}
return ret;

View File

@@ -10,42 +10,40 @@
#include <list.h>
#include <atomic.h>
#include <ssh.h>
#include <message.h>
struct path {
struct list_head list; /* mscp->path_list */
struct list_head list; /* mscp->path_list */
char *path; /* file path */
size_t size; /* size of file on this path */
mode_t mode; /* permission */
char *path; /* file path */
size_t size; /* size of file on this path */
mode_t mode; /* permission */
char *dst_path; /* copy dst path */
char *dst_path; /* copy dst path */
int state;
lock lock;
refcnt refcnt;
int state;
lock lock;
refcnt refcnt;
};
#define FILE_STATE_INIT 0
#define FILE_STATE_OPENED 1
#define FILE_STATE_DONE 2
#define FILE_STATE_INIT 0
#define FILE_STATE_OPENED 1
#define FILE_STATE_DONE 2
struct chunk {
struct list_head list; /* chunk_pool->list */
struct list_head list; /* chunk_pool->list */
struct path *p;
size_t off; /* offset of this chunk on the file on path p */
size_t len; /* length of this chunk */
size_t done; /* copied bytes for this chunk by a thread */
size_t off; /* offset of this chunk on the file on path p */
size_t len; /* length of this chunk */
size_t done; /* copied bytes for this chunk by a thread */
};
struct chunk_pool {
struct list_head list; /* list of struct chunk */
size_t count;
lock lock;
int state;
struct list_head list; /* list of struct chunk */
size_t count;
lock lock;
int state;
};
/* initialize chunk pool */
void chunk_pool_init(struct chunk_pool *cp);
@@ -54,7 +52,7 @@ void chunk_pool_init(struct chunk_pool *cp);
* added, or pointer to chunk.
*/
struct chunk *chunk_pool_pop(struct chunk_pool *cp);
#define CHUNK_POP_WAIT ((void *) -1)
#define CHUNK_POP_WAIT ((void *)-1)
/* set and check fillingchunks to this pool has finished */
void chunk_pool_set_filled(struct chunk_pool *cp);
@@ -69,25 +67,22 @@ bool chunk_pool_is_empty(struct chunk_pool *cp);
/* free chunks in the chunk_pool */
void chunk_pool_release(struct chunk_pool *cp);
struct path_resolve_args {
FILE *msg_fp;
size_t *total_bytes;
size_t *total_bytes;
/* args to resolve src path to dst path */
const char *src_path;
const char *dst_path;
bool src_path_is_dir;
bool dst_path_is_dir;
bool dst_path_should_dir;
/* args to resolve src path to dst path */
const char *src_path;
const char *dst_path;
bool src_path_is_dir;
bool dst_path_is_dir;
bool dst_path_should_dir;
/* args to resolve chunks for a path */
struct chunk_pool *cp;
int nr_conn;
size_t min_chunk_sz;
size_t max_chunk_sz;
size_t chunk_align;
/* args to resolve chunks for a path */
struct chunk_pool *cp;
int nr_conn;
size_t min_chunk_sz;
size_t max_chunk_sz;
size_t chunk_align;
};
/* recursivly walk through src_path and fill path_list for each file */
@@ -98,9 +93,8 @@ int walk_src_path(sftp_session src_sftp, const char *src_path,
void free_path(struct path *p);
/* copy a chunk. either src_sftp or dst_sftp is not null, and another is null */
int copy_chunk(FILE *msg_fp, struct chunk *c,
sftp_session src_sftp, sftp_session dst_sftp,
int nr_ahead, int buf_sz, size_t *counter);
int copy_chunk(struct chunk *c, sftp_session src_sftp, sftp_session dst_sftp,
int nr_ahead, int buf_sz, bool preserve_ts, size_t *counter);
/* just print contents. just for debugging */
void path_dump(struct list_head *path_list);

View File

@@ -2,19 +2,29 @@
#ifdef __APPLE__
#include <stdlib.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/sysctl.h>
#elif linux
#define _GNU_SOURCE
#include <sched.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sched.h>
#elif __FreeBSD__
#include <stdlib.h>
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread_np.h>
#else
#error unsupported platform
#endif
#include <util.h>
#include <platform.h>
#include <message.h>
#include <strerrno.h>
#include <print.h>
#ifdef __APPLE__
int nr_cpus()
@@ -23,7 +33,7 @@ int nr_cpus()
size_t size = sizeof(n);
if (sysctlbyname("machdep.cpu.core_count", &n, &size, NULL, 0) != 0) {
mscp_set_error("failed to get number of cpu cores: %s", strerrno());
priv_set_errv("failed to get number of cpu cores: %s", strerrno());
return -1;
}
@@ -32,21 +42,35 @@ int nr_cpus()
int set_thread_affinity(pthread_t tid, int core)
{
pr_warn("setting thread afinity is not implemented on apple\n");
pr_warn("setting thread afinity is not implemented on apple");
return 0;
}
int setutimes(const char *path, struct timespec atime, struct timespec mtime)
{
struct timeval tv[2] = {
{
.tv_sec = atime.tv_sec,
.tv_usec = atime.tv_nsec * 1000,
},
{
.tv_sec = mtime.tv_sec,
.tv_usec = mtime.tv_nsec * 1000,
},
};
return utimes(path, tv);
}
static void random_string(char *buf, size_t size)
{
char chars[] = "abcdefhijklmnopqrstuvwxyz1234567890";
int n, x;
char chars[] = "abcdefhijklmnopqrstuvwxyz1234567890";
int n, x;
for (n = 0; n < size - 1; n++) {
x = arc4random() % (sizeof(chars) - 1);
buf[n] = chars[x];
}
buf[size - 1] = '\0';
for (n = 0; n < size - 1; n++) {
x = arc4random() % (sizeof(chars) - 1);
buf[n] = chars[x];
}
buf[size - 1] = '\0';
}
sem_t *sem_create(int value)
@@ -55,10 +79,10 @@ sem_t *sem_create(int value)
sem_t *sem;
int n;
n = strlen(sem_name);
random_string(sem_name + n, sizeof(sem_name) - n - 1);
if ((sem = sem_open(sem_name, O_CREAT, 600, value)) == SEM_FAILED)
return NULL;
n = strlen(sem_name);
random_string(sem_name + n, sizeof(sem_name) - n - 1);
if ((sem = sem_open(sem_name, O_CREAT, 600, value)) == SEM_FAILED)
return NULL;
return sem;
}
@@ -78,6 +102,17 @@ int nr_cpus()
return CPU_COUNT(&cpu_set);
return -1;
}
#endif
#ifdef __FreeBSD__
int nr_cpus()
{
long nr_cpus = sysconf(_SC_NPROCESSORS_ONLN);
return nr_cpus;
}
#endif
#if defined(linux) || defined(__FreeBSD__)
int set_thread_affinity(pthread_t tid, int core)
{
@@ -88,8 +123,21 @@ int set_thread_affinity(pthread_t tid, int core)
CPU_SET(core, &target_cpu_set);
ret = pthread_setaffinity_np(tid, sizeof(target_cpu_set), &target_cpu_set);
if (ret < 0)
mscp_set_error("failed to set thread/cpu affinity for core %d: %s",
core, strerrno());
priv_set_errv("failed to set thread/cpu affinity for core %d: %s", core,
strerrno());
return ret;
}
int setutimes(const char *path, struct timespec atime, struct timespec mtime)
{
struct timespec ts[2] = { atime, mtime };
int fd = open(path, O_WRONLY);
int ret;
if (fd < 0)
return -1;
ret = futimens(fd, ts);
close(fd);
return ret;
}
@@ -115,4 +163,3 @@ int sem_release(sem_t *sem)
}
#endif

View File

@@ -7,6 +7,7 @@
int nr_cpus(void);
int set_thread_affinity(pthread_t tid, int core);
int setutimes(const char *path, struct timespec atime, struct timespec mtime);
/*
* macOS does not support sem_init(). macOS (seems to) releases the

18
src/print.c Normal file
View File

@@ -0,0 +1,18 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#include <print.h>
/* message print functions */
static int __print_severity = MSCP_SEVERITY_WARN;
void set_print_severity(int serverity)
{
if (serverity < 0)
__print_severity = -1; /* no print */
__print_severity = serverity;
}
int get_print_severity()
{
return __print_severity;
}

30
src/print.h Normal file
View File

@@ -0,0 +1,30 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _PRINT_H_
#define _PRINT_H_
#include <libgen.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <mscp.h>
/* message print. printed messages are passed to application via msg_fd */
void set_print_severity(int severity);
int get_print_severity();
#define __print(fp, severity, fmt, ...) \
do { \
if (severity <= get_print_severity()) { \
fprintf(fp, "\r\033[K" fmt "\n", ##__VA_ARGS__); \
fflush(fp); \
} \
} while (0)
#define pr_err(fmt, ...) __print(stderr, MSCP_SEVERITY_ERR, fmt, ##__VA_ARGS__)
#define pr_warn(fmt, ...) __print(stderr, MSCP_SEVERITY_WARN, fmt, ##__VA_ARGS__)
#define pr_notice(fmt, ...) __print(stdout, MSCP_SEVERITY_NOTICE, fmt, ##__VA_ARGS__)
#define pr_info(fmt, ...) __print(stdout, MSCP_SEVERITY_INFO, fmt, ##__VA_ARGS__)
#define pr_debug(fmt, ...) __print(stdout, MSCP_SEVERITY_DEBUG, fmt, ##__VA_ARGS__)
#endif /* _PRINT_H_ */

View File

@@ -1,499 +0,0 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <mscp.h>
/*
* This is a wrapper for python binding of libmscp. setup.py builds
* pymscp.c after libmscp was built, and setup.py installs pymscp
* modlue and mscp python module (mscp/mscp.py), which is a warpper
* for pymscp.
*/
#define MAX_MSCP_INSTS 64
/* XXX: cut corners */
struct instance {
struct mscp_opts mo;
struct mscp_ssh_opts so;
struct mscp *m;
};
struct instance *insts[MAX_MSCP_INSTS];
static int add_instance(struct instance *i)
{
int n;
for (n = 0; n < MAX_MSCP_INSTS; n++) {
if (insts[n] == NULL) {
insts[n] = i;
return 0;
}
}
return -1; /* full of mscp instances */
}
static struct instance *get_instance(unsigned long long addr)
{
int n;
for (n = 0; n < MAX_MSCP_INSTS; n++) {
if (insts[n] == (void *)addr)
return insts[n];
}
return NULL;
}
static struct mscp *get_mscp(unsigned long long addr)
{
struct instance *i = get_instance(addr);
if (!i)
return NULL;
return i->m;
}
static int release_instance(struct instance *i)
{
int n;
for (n = 0; n < MAX_MSCP_INSTS; n++) {
if (insts[n] == i) {
insts[n] = NULL;
return 0;
}
}
free(i);
return -1;
}
/* wrapper functions */
static PyObject *wrap_mscp_init(PyObject *self, PyObject *args, PyObject *kw)
{
/*
* Initialize struct mscp with options. wrap_mscp_init
* receives all the arguments with keywords.
*/
char *remote;
char *keywords[] = {
"remote", /* const char * */
"direction", /* int, MSCP_DIRECTION_L2R or MSCP_DIRECTION_R2L */
/* mscp_opts */
"nr_threads", /* int */
"nr_ahead", /* int */
"min_chunk_sz", /* unsigned long */
"max_chunk_sz", /* unsigned long */
"buf_sz", /* unsigned long */
"coremask", /* const char * */
"max_startups", /* int */
"interval", /* int */
"severity", /* int, MSCP_SERVERITY_* */
"msg_fd", /* int */
/* mscp_ssh_opts */
"login_name", /* const char * */
"port", /* const char * */
"config", /* const char * */
"identity", /* const char * */
"cipher", /* const char * */
"hmac", /* const char * */
"compress", /* const char * */
"ccalgo", /* const char * */
"password", /* const char * */
"passphrase", /* const char * */
"debug_level", /* int */
"no_hostkey_check", /* bool */
"enable_nagle", /* bool */
NULL,
};
const char *fmt = "si" "|" "ii" "kkk" "s" "iiii" "ssss" "ssssss" "ipp";
char *coremask = NULL;
char *login_name = NULL, *port = NULL, *config = NULL, *identity = NULL;
char *cipher = NULL, *hmac = NULL, *compress = NULL, *ccalgo = NULL;
char *password = NULL, *passphrase = NULL;
struct instance *i;
int direction;
int ret;
i = malloc(sizeof(*i));
if (!i) {
PyErr_Format(PyExc_RuntimeError, strerror(errno));
return NULL;
}
memset(i, 0, sizeof(*i));
ret = PyArg_ParseTupleAndKeywords(args, kw, fmt, keywords,
&remote,
&direction,
&i->mo.nr_threads,
&i->mo.nr_ahead,
&i->mo.min_chunk_sz,
&i->mo.max_chunk_sz,
&i->mo.buf_sz,
&coremask,
&i->mo.max_startups,
&i->mo.interval,
&i->mo.severity,
&i->mo.msg_fd,
&login_name,
&port,
&config,
&identity,
&cipher,
&hmac,
&compress,
&ccalgo,
&password,
&passphrase,
&i->so.debug_level,
&i->so.no_hostkey_check,
&i->so.enable_nagle);
if (!ret)
return NULL;
if (coremask)
strncpy(i->mo.coremask, coremask, MSCP_MAX_COREMASK_STR - 1);
if (login_name)
strncpy(i->so.login_name, login_name, MSCP_SSH_MAX_LOGIN_NAME - 1);
if (port)
strncpy(i->so.port, port, MSCP_SSH_MAX_PORT_STR - 1);
if (config)
strncpy(i->so.config, config, PATH_MAX - 1);
if (identity)
strncpy(i->so.identity, identity, MSCP_SSH_MAX_IDENTITY_PATH - 1);
if (cipher)
strncpy(i->so.cipher, cipher, MSCP_SSH_MAX_CIPHER_STR - 1);
if (hmac)
strncpy(i->so.hmac, hmac, MSCP_SSH_MAX_HMAC_STR - 1);
if (compress)
strncpy(i->so.compress, compress, MSCP_SSH_MAX_COMP_STR - 1);
if (ccalgo)
strncpy(i->so.ccalgo, ccalgo, MSCP_SSH_MAX_CCALGO_STR - 1);
if (password)
strncpy(i->so.password, password, MSCP_SSH_MAX_PASSWORD - 1);
if (passphrase)
strncpy(i->so.passphrase, passphrase, MSCP_SSH_MAX_PASSPHRASE - 1);
i->m = mscp_init(remote, direction, &i->mo, &i->so);
if (!i->m) {
PyErr_Format(PyExc_RuntimeError, "%s", mscp_get_error());
free(i);
return NULL;
}
if (add_instance(i) < 0) {
PyErr_Format(PyExc_RuntimeError, "too many mscp isntances");
mscp_free(i->m);
free(i);
return NULL;
}
return Py_BuildValue("K", (unsigned long long)i);
}
static PyObject *wrap_mscp_connect(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
if (mscp_connect(m) < 0) {
PyErr_Format(PyExc_RuntimeError, mscp_get_error());
return NULL;
}
return Py_BuildValue("");
}
static PyObject *wrap_mscp_add_src_path(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", "src_path", NULL };
unsigned long long addr;
char *src_path;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "Ks", keywords, &addr, &src_path))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
if (mscp_add_src_path(m, src_path) < 0) {
PyErr_Format(PyExc_RuntimeError, mscp_get_error());
return NULL;
}
return Py_BuildValue("");
}
static PyObject *wrap_mscp_set_dst_path(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", "dst_path", NULL };
unsigned long long addr;
char *dst_path;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "Ks", keywords, &addr, &dst_path))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
if (mscp_set_dst_path(m, dst_path) < 0) {
PyErr_Format(PyExc_RuntimeError, mscp_get_error());
return NULL;
}
return Py_BuildValue("");
}
static PyObject *wrap_mscp_scan(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
if (mscp_scan(m) < 0) {
PyErr_Format(PyExc_RuntimeError, mscp_get_error());
return NULL;
}
return Py_BuildValue("");
}
static PyObject *wrap_mscp_start(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
if (mscp_start(m) < 0) {
PyErr_Format(PyExc_RuntimeError, mscp_get_error());
return NULL;
}
return Py_BuildValue("");
}
static PyObject *wrap_mscp_stop(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
mscp_stop(m);
return Py_BuildValue("");
}
static PyObject *wrap_mscp_join(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
if (mscp_join(m) < 0) {
PyErr_Format(PyExc_RuntimeError, mscp_get_error());
return NULL;
}
return Py_BuildValue("");
}
static PyObject *wrap_mscp_get_stats(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp_stats s;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
mscp_get_stats(m, &s);
return Py_BuildValue("KKO", s.total, s.done, PyBool_FromLong(s.finished));
}
static PyObject *wrap_mscp_cleanup(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct mscp *m;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
m = get_mscp(addr);
if (!m) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
mscp_cleanup(m);
return Py_BuildValue("");
}
static PyObject *wrap_mscp_free(PyObject *self, PyObject *args, PyObject *kw)
{
char *keywords[] = { "m", NULL };
unsigned long long addr;
struct instance *i;
if (!PyArg_ParseTupleAndKeywords(args, kw, "K", keywords, &addr))
return NULL;
i = get_instance(addr);
if (!i) {
PyErr_Format(PyExc_RuntimeError, "invalid mscp instance address");
return NULL;
}
mscp_free(i->m);
release_instance(i);
return Py_BuildValue("");
}
static PyMethodDef pymscpMethods[] = {
{
"mscp_init", (PyCFunction)wrap_mscp_init,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_connect", (PyCFunction)wrap_mscp_connect,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_add_src_path", (PyCFunction)wrap_mscp_add_src_path,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_set_dst_path", (PyCFunction)wrap_mscp_set_dst_path,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_scan", (PyCFunction)wrap_mscp_scan,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_start", (PyCFunction)wrap_mscp_start,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_stop", (PyCFunction)wrap_mscp_stop,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_join", (PyCFunction)wrap_mscp_join,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_get_stats", (PyCFunction)wrap_mscp_get_stats,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_cleanup", (PyCFunction)wrap_mscp_cleanup,
METH_VARARGS | METH_KEYWORDS, NULL
},
{
"mscp_free", (PyCFunction)wrap_mscp_free,
METH_VARARGS | METH_KEYWORDS, NULL
},
{ NULL, NULL, 0, NULL },
};
static PyModuleDef pymscpModule = {
PyModuleDef_HEAD_INIT, "pymscp", NULL, -1, pymscpMethods,
};
PyMODINIT_FUNC PyInit_pymscp(void) {
PyObject *mod = PyModule_Create(&pymscpModule);
PyModule_AddIntConstant(mod, "LOCAL2REMOTE", MSCP_DIRECTION_L2R);
PyModule_AddIntConstant(mod, "REMOTE2LOCAL", MSCP_DIRECTION_R2L);
PyModule_AddIntConstant(mod, "SEVERITY_NONE", MSCP_SEVERITY_NONE);
PyModule_AddIntConstant(mod, "SEVERITY_ERR", MSCP_SEVERITY_ERR);
PyModule_AddIntConstant(mod, "SEVERITY_WARN", MSCP_SEVERITY_WARN);
PyModule_AddIntConstant(mod, "SEVERITY_NOTICE", MSCP_SEVERITY_NOTICE);
PyModule_AddIntConstant(mod, "SEVERITY_INFO", MSCP_SEVERITY_INFO);
PyModule_AddIntConstant(mod, "SEVERITY_DEBUG", MSCP_SEVERITY_DEBUG);
return mod;
}

115
src/ssh.c
View File

@@ -7,67 +7,68 @@
#include "libssh/callbacks.h"
#include <ssh.h>
#include <util.h>
#include <message.h>
#include <mscp.h>
#include <strerrno.h>
static int ssh_verify_known_hosts(ssh_session session);
#define is_specified(s) (strlen(s) > 0)
static int ssh_set_opts(ssh_session ssh, struct mscp_ssh_opts *opts)
{
ssh_set_log_level(opts->debug_level);
if (is_specified(opts->login_name) &&
if (opts->login_name &&
ssh_options_set(ssh, SSH_OPTIONS_USER, opts->login_name) < 0) {
mscp_set_error("failed to set login name");
priv_set_errv("failed to set login name");
return -1;
}
if (is_specified(opts->port) &&
ssh_options_set(ssh, SSH_OPTIONS_PORT_STR, opts->port) < 0) {
mscp_set_error("failed to set port number");
if (opts->port && ssh_options_set(ssh, SSH_OPTIONS_PORT_STR, opts->port) < 0) {
priv_set_errv("failed to set port number");
return -1;
}
if (is_specified(opts->identity) &&
if (opts->ai_family &&
ssh_options_set(ssh, SSH_OPTIONS_AI_FAMILY, &opts->ai_family) < 0) {
priv_set_errv("failed to set address family");
return -1;
}
if (opts->identity &&
ssh_options_set(ssh, SSH_OPTIONS_IDENTITY, opts->identity) < 0) {
mscp_set_error("failed to set identity");
priv_set_errv("failed to set identity");
return -1;
}
if (is_specified(opts->cipher)) {
if (opts->cipher) {
if (ssh_options_set(ssh, SSH_OPTIONS_CIPHERS_C_S, opts->cipher) < 0) {
mscp_set_error("failed to set cipher for client to server");
priv_set_errv("failed to set cipher for client to server");
return -1;
}
if (ssh_options_set(ssh, SSH_OPTIONS_CIPHERS_S_C, opts->cipher) < 0) {
mscp_set_error("failed to set cipher for server to client");
priv_set_errv("failed to set cipher for server to client");
return -1;
}
}
if (is_specified(opts->hmac)) {
if (opts->hmac) {
if (ssh_options_set(ssh, SSH_OPTIONS_HMAC_C_S, opts->hmac) < 0) {
mscp_set_error("failed to set hmac for client to server");
priv_set_errv("failed to set hmac for client to server");
return -1;
}
if (ssh_options_set(ssh, SSH_OPTIONS_HMAC_S_C, opts->hmac) < 0) {
mscp_set_error("failed to set hmac for server to client");
priv_set_errv("failed to set hmac for server to client");
return -1;
}
}
if (is_specified(opts->compress) &&
if (opts->compress &&
ssh_options_set(ssh, SSH_OPTIONS_COMPRESSION, opts->compress) < 0) {
mscp_set_error("failed to enable ssh compression");
priv_set_errv("failed to enable ssh compression");
return -1;
}
if (is_specified(opts->ccalgo) &&
ssh_options_set(ssh, SSH_OPTIONS_CCALGO, opts->ccalgo) < 0) {
mscp_set_error("failed to set cclago");
if (opts->ccalgo && ssh_options_set(ssh, SSH_OPTIONS_CCALGO, opts->ccalgo) < 0) {
priv_set_errv("failed to set cclago");
return -1;
}
@@ -75,14 +76,13 @@ static int ssh_set_opts(ssh_session ssh, struct mscp_ssh_opts *opts)
if (!opts->enable_nagle) {
int v = 1;
if (ssh_options_set(ssh, SSH_OPTIONS_NODELAY, &v) < 0) {
mscp_set_error("failed to set TCP_NODELAY");
priv_set_errv("failed to set TCP_NODELAY");
return -1;
}
}
if (is_specified(opts->config) &&
ssh_options_parse_config(ssh, opts->config) < 0) {
mscp_set_error("failed to parse ssh_config: %s", opts->config);
if (opts->config && ssh_options_parse_config(ssh, opts->config) < 0) {
priv_set_errv("failed to parse ssh_config: %s", opts->config);
return -1;
}
@@ -106,15 +106,20 @@ static int ssh_authenticate(ssh_session ssh, struct mscp_ssh_opts *opts)
return 0;
if (auth_bit_mask & SSH_AUTH_METHOD_PUBLICKEY) {
char *p = is_specified(opts->passphrase) ? opts->passphrase : NULL;
char *p = opts->passphrase ? opts->passphrase : NULL;
if (ssh_userauth_publickey_auto(ssh, NULL, p) == SSH_AUTH_SUCCESS)
return 0;
}
if (auth_bit_mask & SSH_AUTH_METHOD_PASSWORD) {
if (!is_specified(opts->password)) {
if (ssh_getpass("Password: ", opts->password,
MSCP_SSH_MAX_PASSWORD, 0, 0) < 0) {
if (!opts->password) {
char buf[128] = {};
if (ssh_getpass("Password: ", buf, sizeof(buf), 0, 0) < 0) {
priv_set_errv("ssh_getpass failed");
return -1;
}
if (!(opts->password = strndup(buf, sizeof(buf)))) {
priv_set_errv("strndup: %s", strerrno());
return -1;
}
}
@@ -136,15 +141,21 @@ static int ssh_cache_passphrase(const char *prompt, char *buf, size_t len, int e
* second time or after because cached passphrase is passed
* to ssh_userauth_publickey_auto(). */
/* ToDo: use
* ssh_userauth_publickey_auto_get_current_identity() to print
* id for which we ask passphrase */
if (ssh_getpass("Passphrase: ", buf, len, echo, verify) < 0)
return -1;
/* cache the passphrase */
if (strlen(buf) > MSCP_SSH_MAX_PASSPHRASE - 1) {
pr_warn("sorry, passphrase is too long to cache...\n");
return 0;
if (opts->passphrase)
free(opts->passphrase);
if (!(opts->passphrase = strndup(buf, len))) {
priv_set_errv("strndup: %s", strerrno());
return -1;
}
strncpy(opts->passphrase, buf, MSCP_SSH_MAX_PASSPHRASE);
return 0;
}
@@ -163,7 +174,7 @@ static ssh_session ssh_init_session(const char *sshdst, struct mscp_ssh_opts *op
ssh_set_callbacks(ssh, &cb);
if (ssh_options_set(ssh, SSH_OPTIONS_HOST, sshdst) != SSH_OK) {
mscp_set_error("failed to set destination host");
priv_set_errv("failed to set destination host");
goto free_out;
}
@@ -171,16 +182,17 @@ static ssh_session ssh_init_session(const char *sshdst, struct mscp_ssh_opts *op
goto free_out;
if (ssh_connect(ssh) != SSH_OK) {
mscp_set_error("failed to connect ssh server: %s", ssh_get_error(ssh));
priv_set_errv("failed to connect ssh server: %s", ssh_get_error(ssh));
goto free_out;
}
if (ssh_authenticate(ssh, opts) != 0) {
mscp_set_error("authentication failed: %s", ssh_get_error(ssh));
priv_set_errv("authentication failed: %s", ssh_get_error(ssh));
goto disconnect_out;
}
if (!opts->no_hostkey_check && ssh_verify_known_hosts(ssh) != 0) {
priv_set_errv("ssh_veriy_known_hosts failed");
goto disconnect_out;
}
@@ -198,20 +210,18 @@ sftp_session ssh_init_sftp_session(const char *sshdst, struct mscp_ssh_opts *opt
sftp_session sftp;
ssh_session ssh = ssh_init_session(sshdst, opts);
if (!ssh) {
if (!ssh)
return NULL;
}
sftp = sftp_new(ssh);
if (!sftp) {
mscp_set_error("failed to allocate sftp session: %s",
ssh_get_error(ssh));
priv_set_errv("failed to allocate sftp session: %s", ssh_get_error(ssh));
goto err_out;
}
if (sftp_init(sftp) != SSH_OK) {
mscp_set_error("failed to initialize sftp session: err code %d",
sftp_get_error(sftp));
priv_set_errv("failed to initialize sftp session: err code %d",
sftp_get_error(sftp));
goto err_out;
}
@@ -222,7 +232,6 @@ err_out:
return NULL;
}
/* copied from https://api.libssh.org/stable/libssh_tutor_guided_tour.html*/
static int ssh_verify_known_hosts(ssh_session session)
{
@@ -241,10 +250,7 @@ static int ssh_verify_known_hosts(ssh_session session)
return -1;
}
rc = ssh_get_publickey_hash(srv_pubkey,
SSH_PUBLICKEY_HASH_SHA1,
&hash,
&hlen);
rc = ssh_get_publickey_hash(srv_pubkey, SSH_PUBLICKEY_HASH_SHA1, &hash, &hlen);
ssh_key_free(srv_pubkey);
if (rc < 0) {
return -1;
@@ -265,8 +271,9 @@ static int ssh_verify_known_hosts(ssh_session session)
return -1;
case SSH_KNOWN_HOSTS_OTHER:
fprintf(stderr, "The host key for this server was not found but an other"
"type of key exists.\n");
fprintf(stderr, "An attacker might change the default server key to"
"type of key exists.\n");
fprintf(stderr,
"An attacker might change the default server key to"
"confuse your client into thinking the key does not exist\n");
ssh_clean_pubkey_hash(&hash);
@@ -274,7 +281,7 @@ static int ssh_verify_known_hosts(ssh_session session)
case SSH_KNOWN_HOSTS_NOT_FOUND:
fprintf(stderr, "Could not find known host file.\n");
fprintf(stderr, "If you accept the host key here, the file will be"
"automatically created.\n");
"automatically created.\n");
/* FALL THROUGH to SSH_SERVER_NOT_KNOWN behavior */
@@ -297,13 +304,13 @@ static int ssh_verify_known_hosts(ssh_session session)
rc = ssh_session_update_known_hosts(session);
if (rc < 0) {
fprintf(stderr, "Error %s\n", strerror(errno));
priv_set_errv("%s", ssh_get_error(session));
return -1;
}
break;
case SSH_KNOWN_HOSTS_ERROR:
fprintf(stderr, "Error %s", ssh_get_error(session));
fprintf(stderr, "known hosts error: %s", ssh_get_error(session));
ssh_clean_pubkey_hash(&hash);
return -1;
}

36
src/strerrno.c Normal file
View File

@@ -0,0 +1,36 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <stdarg.h>
#include <strerrno.h>
#define STRERRNO_TLS_BUFSIZ 128
__thread char tls_strerrno_buf[STRERRNO_TLS_BUFSIZ];
const char *strerrno(void)
{
snprintf(tls_strerrno_buf, sizeof(tls_strerrno_buf), "%s", "strerror_r error");
strerror_r(errno, tls_strerrno_buf, sizeof(tls_strerrno_buf));
return tls_strerrno_buf;
}
#define PRIV_ERR_BUFSIZ (1 << 12)
__thread char priv_err_buf[PRIV_ERR_BUFSIZ], internal[PRIV_ERR_BUFSIZ];
void priv_set_err(const char *fmt, ...)
{
va_list va;
memset(internal, 0, sizeof(internal));
va_start(va, fmt);
vsnprintf(internal, sizeof(internal), fmt, va);
va_end(va);
snprintf(priv_err_buf, sizeof(priv_err_buf), "%s", internal);
}
const char *priv_get_err()
{
return priv_err_buf;
}

36
src/strerrno.h Normal file
View File

@@ -0,0 +1,36 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _STRERRNO_
#define _STRERRNO_
#include <libgen.h> /* basename() */
/**
* strerrno() returns error message string corresponding to errno.
* strerrno() is thread safe.
*/
const char *strerrno(void);
/**
* priv_set_err() sets an error message into a thread-local private
* buffer. This error message can be accessed via priv_get_err().
*
* The top-level function in a thread should print errors using
* priv_get_err(), while lower-level functions should set error
* messages using priv_set_err().
*/
void priv_set_err(const char *fmt, ...);
/**
* priv_set_errv(), a wrapper for priv_set_err(), just adds filename,
* line, and function name to the error message.
*/
#define priv_set_errv(fmt, ...) \
priv_set_err("[%s:%d:%s] " fmt "\0", basename(__FILE__), __LINE__, __func__, \
##__VA_ARGS__)
/**
* priv_get_err() gets the error message sotred in the thread-local private buffer.
*/
const char *priv_get_err();
#endif /* _STRERRNO_ */

View File

@@ -1,39 +0,0 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#ifndef _UTIL_H_
#define _UTIL_H_
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <libgen.h>
#define likely(x) __builtin_expect(!!(x), 1)
#define unlikely(x) __builtin_expect(!!(x), 0)
#define pr(fmt, ...) fprintf(stderr, fmt, ##__VA_ARGS__)
#define pr_info(fmt, ...) fprintf(stderr, "INFO:%s(): " fmt, \
__func__, ##__VA_ARGS__)
#define pr_warn(fmt, ...) fprintf(stderr, "\x1b[1m\x1b[33m" \
"WARN:%s():\x1b[0m " fmt, \
__func__, ##__VA_ARGS__)
#define pr_err(fmt, ...) fprintf(stderr, "\x1b[1m\x1b[31m" \
"ERR:%s:%d:%s():\x1b[0m " fmt, \
basename(__FILE__), __LINE__, __func__, ##__VA_ARGS__)
#ifdef DEBUG
#define pr_debug(fmt, ...) fprintf(stderr, "\x1b[1m\x1b[33m" \
"DEBUG:%s():\x1b[0m " fmt, \
__func__, ##__VA_ARGS__);
#else
#define pr_debug(fmt, ...)
#endif
#define min(a, b) (((a) > (b)) ? (b) : (a))
#define max(a, b) (((a) > (b)) ? (a) : (b))
#endif /* _UTIL_H_ */

View File

@@ -12,16 +12,16 @@ from subprocess import check_call, CalledProcessError, PIPE
from util import File, check_same_md5sum
def run2ok(args):
def run2ok(args, env = None):
cmd = list(map(str, args))
print("cmd: {}".format(" ".join(cmd)))
check_call(cmd)
check_call(cmd, env = env)
def run2ng(args):
def run2ng(args, env = None):
cmd = list(map(str, args))
print("cmd: {}".format(" ".join(cmd)))
with pytest.raises(CalledProcessError) as e:
check_call(cmd)
check_call(cmd, env = env)
""" usage test """
@@ -324,6 +324,38 @@ def test_set_port(mscp, src_prefix, dst_prefix, src, dst):
run2ng([mscp, "-H", "-vvv", "-P", 21, src_prefix + src.path, dst_prefix + dst.path])
src.cleanup()
def test_v4only(mscp):
src = File("src", size = 1024).make()
dst = File("dst")
dst_prefix = "localhost:{}/".format(os.getcwd())
run2ok([mscp, "-H", "-vvv", "-4", src.path, dst_prefix + dst.path])
assert check_same_md5sum(src, dst)
src.cleanup()
dst.cleanup()
def test_v6only(mscp):
src = File("src", size = 1024).make()
dst = File("dst")
dst_prefix = "localhost:{}/".format(os.getcwd())
run2ok([mscp, "-H", "-vvv", "-6", src.path, dst_prefix + dst.path])
assert check_same_md5sum(src, dst)
src.cleanup()
dst.cleanup()
def test_v4_to_v6_should_fail(mscp):
src = File("src", size = 1024).make()
dst = File("dst")
dst_prefix = "[::1]:{}/".format(os.getcwd())
run2ng([mscp, "-H", "-vvv", "-4", src.path, dst_prefix + dst.path])
src.cleanup()
def test_v6_to_v4_should_fail(mscp):
src = File("src", size = 1024).make()
dst = File("dst")
dst_prefix = "127.0.0.1:{}/".format(os.getcwd())
run2ng([mscp, "-H", "-vvv", "-6", src.path, dst_prefix + dst.path])
src.cleanup()
@pytest.mark.parametrize("src_prefix, dst_prefix", param_remote_prefix)
def test_set_conn_interval(mscp, src_prefix, dst_prefix):
srcs = []
@@ -401,3 +433,45 @@ def test_config_ng(mscp, src_prefix, dst_prefix):
os.remove(config)
src.cleanup()
dst.cleanup()
# username test assumes that this test runs inside a container, see Dockerfiles
def test_specify_passphrase_via_env(mscp):
src = File(os.getcwd() + "/src", size = 1024).make()
dst = File("/home/test/dst")
env = os.environ
env["MSCP_SSH_AUTH_PASSPHRASE"] = "keypassphrase"
run2ok([mscp, "-H", "-vvv", "-l", "test", "-i", "/home/test/.ssh/id_rsa_test",
src.path, "localhost:" + dst.path], env = env)
assert check_same_md5sum(src, dst)
src.cleanup()
dst.cleanup()
def test_specify_invalid_passphrase_via_env(mscp):
src = File(os.getcwd() + "/src", size = 1024).make()
dst = File("/home/test/dst")
env = os.environ
env["MSCP_SSH_AUTH_PASSPHRASE"] = "invalid-keypassphrase"
run2ng([mscp, "-H", "-vvv", "-l", "test", "-i", "/home/test/.ssh/id_rsa_test",
src.path, "localhost:" + dst.path], env = env)
src.cleanup()
def test_specify_password_via_env(mscp):
src = File(os.getcwd() + "/src", size = 1024).make()
dst = File("/home/test/dst")
env = os.environ
env["MSCP_SSH_AUTH_PASSWORD"] = "userpassword"
run2ok([mscp, "-H", "-vvv", "-l", "test",
src.path, "localhost:" + dst.path], env = env)
assert check_same_md5sum(src, dst)
src.cleanup()
dst.cleanup()
def test_specify_invalid_password_via_env(mscp):
src = File(os.getcwd() + "/src", size = 1024).make()
dst = File("/home/test/dst")
env = os.environ
env["MSCP_SSH_AUTH_PASSWORD"] = "invalid-userpassword"
run2ng([mscp, "-H", "-vvv", "-l", "test",
src.path, "localhost:" + dst.path], env = env)
src.cleanup()

View File

@@ -1,131 +0,0 @@
"""
test_python.py: Testing libmscp through the mscp python binding.
"""
import pytest
import mscp
import os
from util import File, check_same_md5sum
def test_create_and_release():
m = mscp.mscp("localhost", mscp.LOCAL2REMOTE)
m.cleanup()
""" copy test """
remote = "localhost"
remote_prefix = "{}/".format(os.getcwd()) # use current dir
param_remote_prefix_and_direction = [
("", remote_prefix, mscp.LOCAL2REMOTE), (remote_prefix, "", mscp.REMOTE2LOCAL)
]
param_single_copy = [
(File("src", size = 64), File("dst")),
(File("src", size = 4096 * 1), File("dst")),
(File("src", size = 128 * 1024 * 1024), File("dst")),
]
@pytest.mark.parametrize("src_prefix, dst_prefix, direction",
param_remote_prefix_and_direction)
@pytest.mark.parametrize("src, dst", param_single_copy)
def test_single_copy(src_prefix, dst_prefix, direction, src, dst):
src.make()
m = mscp.mscp(remote, direction)
m.copy(src_prefix + src.path, dst_prefix + dst.path)
assert check_same_md5sum(src, dst)
src.cleanup()
dst.cleanup()
param_double_copy = [
(File("src1", size = 1024 * 1024), File("src2", size = 1024 * 1024),
File("dst/src1"), File("dst/src2")
)
]
@pytest.mark.parametrize("src_prefix, dst_prefix, direction",
param_remote_prefix_and_direction)
@pytest.mark.parametrize("s1, s2, d1, d2", param_double_copy)
def test_double_copy(src_prefix, dst_prefix, direction, s1, s2, d1, d2):
s1.make()
s2.make()
mscp.mscp(remote, direction).copy([src_prefix + s1.path, src_prefix + s2.path],
dst_prefix + "dst")
assert check_same_md5sum(s1, d1)
assert check_same_md5sum(s2, d2)
s1.cleanup()
s2.cleanup()
d1.cleanup()
d2.cleanup()
param_single_copy = [
(File("src", size = 1024 * 1024 * 4), File("dst")),
]
param_kwargs = [
{ "nr_threads": 6 },
{ "nr_ahead": 64 },
{ "min_chunk_sz": 1 * 1024 * 1024 },
{ "max_chunk_sz": 64 * 1024 * 1024 },
{ "coremask": "0x0f" },
{ "max_startups": 5 },
{ "severity": mscp.SEVERITY_NONE },
{ "cipher": "aes128-gcm@openssh.com" },
{ "compress": "yes" },
{ "no_hostkey_check": True },
{ "enable_nagle": True },
]
@pytest.mark.parametrize("src_prefix, dst_prefix, direction",
param_remote_prefix_and_direction)
@pytest.mark.parametrize("src, dst", param_single_copy)
@pytest.mark.parametrize("kw", param_kwargs)
def test_kwargs(src_prefix, dst_prefix, direction, src, dst, kw):
src.make()
m = mscp.mscp(remote, direction, **kw)
m.copy(src_prefix + src.path, dst_prefix + dst.path)
assert check_same_md5sum(src, dst)
src.cleanup()
dst.cleanup()
def test_login_failed():
m = mscp.mscp("asdfasdf@" + remote, mscp.LOCAL2REMOTE)
with pytest.raises(RuntimeError) as e:
m.connect()
m = mscp.mscp(remote, mscp.LOCAL2REMOTE, login_name = "asdfadsf")
with pytest.raises(RuntimeError) as e:
m.connect()
m = mscp.mscp(remote, mscp.LOCAL2REMOTE, port = "65534")
with pytest.raises(RuntimeError) as e:
m.connect()
def test_get_stat_before_copy_start():
m = mscp.mscp("localhost", mscp.LOCAL2REMOTE)
m.connect()
(total, done, finished) = m.stats()
assert total == 0 and done == 0
param_invalid_kwargs = [
{ "nr_threads": -1 },
{ "nr_ahead": -1 },
{ "min_chunk_sz": 1 },
{ "max_chunk_sz": 1 },
{ "coremask": "xxxxx" },
{ "max_startups": -1 },
{ "cipher": "invalid" },
{ "hmac": "invalid"},
{ "compress": "invalid"},
{ "ccalgo": "invalid"},
]
@pytest.mark.parametrize("kw", param_invalid_kwargs)
def test_invalid_options(kw):
with pytest.raises(RuntimeError) as e:
m = mscp.mscp("localhost", mscp.LOCAL2REMOTE, **kw)
m.connect()