Increase SFTP chunk size to increase the SFTP throughput in both directions

[Jakuje]

SUMMARY

This is built on top of #638 so it depends on that change

Previously, the 1024b chunk was used, which resulted in large traffic overhead as each of the chunk needs to be wrapped in SFTP packet, as well as large overhead on high-latency links as the SFTP is now synchronous, waiting for every packet confirmation.

The libssh documentation recommends to use 16kB chunks, but using 32kB chunks should be safe as it still fits to the recommended limits in the SFTP specification. Using large reads could cause issues such as curl/curl#11804

ISSUE TYPE

• Bugfix Pull Request

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo yum install -y dnf-plugins-core on RHEL 8
• sudo dnf install -y dnf-plugins-core on Fedora
• dnf copr enable packit/ansible-pylibssh-664
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[Jakuje]

The 🐍 3.9@macos-13 job is weird. It fails the first time (with some internal error -- not sure if we can do anything about that), but works on maximum verbosity ...

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[NilashishC]

@Jakuje When trying to build a wheel/sdist with the combined changes in PR #638 and this one, I get the following error:

Error compiling Cython file:
------------------------------------------------------------
...
#
from posix.types cimport mode_t

from libc cimport stdint

from pylibsshext.includes.libssh cimport ssh_channel, ssh_session, ssh_string
^
------------------------------------------------------------

src/pylibsshext/includes/sftp.pxd:22:0: 'pylibsshext/includes/libssh/ssh_string.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
        stdint.uint64_t createtime
        stdint.uint32_t createtime_nseconds
        stdint.uint64_t mtime64
        stdint.uint32_t mtime
        stdint.uint32_t mtime_nseconds
        ssh_string acl
        ^
------------------------------------------------------------

src/pylibsshext/includes/sftp.pxd:53:8: 'ssh_string' is not a type identifier

Error compiling Cython file:
------------------------------------------------------------
...
        stdint.uint64_t mtime64
        stdint.uint32_t mtime
        stdint.uint32_t mtime_nseconds
        ssh_string acl
        stdint.uint32_t extended_count
        ssh_string extended_type
        ^
------------------------------------------------------------

src/pylibsshext/includes/sftp.pxd:55:8: 'ssh_string' is not a type identifier

Error compiling Cython file:
------------------------------------------------------------
...
        stdint.uint32_t mtime
        stdint.uint32_t mtime_nseconds
        ssh_string acl
        stdint.uint32_t extended_count
        ssh_string extended_type
        ssh_string extended_data
        ^
------------------------------------------------------------

src/pylibsshext/includes/sftp.pxd:56:8: 'ssh_string' is not a type identifier

[fedora@libssh-fedora-41 pylibssh]$ rpm -qa | grep libssh
libssh-config-0.10.6-8.fc41.noarch
libssh-0.10.6-8.fc41.x86_64
libssh-devel-0.10.6-8.fc41.x86_64

Any ideas?

[Jakuje]

@Jakuje When trying to build a wheel/sdist with the combined changes in PR #638 and this one, I get the following error:

How did you combine these? This PR has all the commits from the #638.

This sounds like somehow the change to src/pylibsshext/includes/libssh.pxd did not land in your code:

https://github.com/ansible/pylibssh/pull/664/files#diff-fcc18a6d0552824c2398f687496ca9bc8db6eadb690f6093826daf73fdc83230

[NilashishC]

@Jakuje You're right. I might have missed something while applying the patches. The PR works out great. I've verified it to significantly improve SFTP performance compared to the existing one.

[webknjaz]

The 🐍 3.9@macos-13 job is weird. It fails the first time (with some internal error -- not sure if we can do anything about that), but works on maximum verbosity ...

@Jakuje py3.10 too. One of the differences is that the rerun disables the coverage plugin. And the first run seems to be crashing in one of the pytest-cov's teardown hook.

I've restarted the CI to see if that happens again. But will probably have to look into what deps need to be pinned or upgraded.

[webknjaz]

Error: This request has been automatically failed because it uses a deprecated version of actions/download-artifact: v3. Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

Urgh.. Need to upgrade that too.

[webknjaz]

@NilashishC adding merge commits to topic branches causes foxtrot merges and inconveniences for contributors, it's best to select the rebase mode or let the contributor handle such updates unless they stated that it's okay to mess with their branch. Let's not be careless like this.