kick-off

[s-m-e]

Continuing fusepy#134.

[billziss-gh]

Adding myself.

[lavdnone]

Thank you @s-m-e
Guess we can try switching from fusepy to refuse

[mxmlnkn]

Hello, I'm looking for alternatives for FUSE Python bindings supporting FUSE 3. So, as this issue is continuing the one in fusepy, what is the project status? It looks like it has fizzled out shortly after it has been forked?

[s-m-e]

I got stuck at (CI) testing, hence so little progress. I did not want to make major changes without the ability to just test them against a test suit of some sort across all supported operating systems, see #13. I can test for Linux via my own filesystem - passing the pjdfstest test-suite, a descendant of FreeBSD's fstest for POSIX compliance as well as stress tests with fsx-linux based on the fsx-flavor released by the Linux Test Project - but any other operating system became tricky. Just making the tests work for Linux told me a lot about edge cases that needed to be covered - yet another point for having proper tests on the other operating systems as well. With the CI tools at the time, nobody was able to properly run FreeBSD. OS X was possible but a pain and Windows ... Anyway, the CI landscape has changed since then and it appears to be worth looking into it again. I have long been considering to reevaluate the situation and I'd appreciative any help.

[billziss-gh]

For Windows CI I use AppVeyor or GitHub. Both will work for your purposes.

(For advanced Windows testing I prefer AppVeyor, because I have far more control over the test environment. For example, for WinFsp I need to disable test-signing, run the driver verifier, etc. and I can only control such options in AppVeyor.)

[mxmlnkn]

I have long been considering to reevaluate the situation and I'd appreciative any help.

I'd be willing to help. CI pipelines are always a difficult topic though. I do have Windows and macOS test pipelines for ratarmount, which also execute integration tests by actually FUSE mounting archives. Windows is always tricky though because there are so many environments like native (power)shell, WSL 1,2, cygwin, and msys.

[yarikoptic]

IMHO the choice of CI, or looking for specific CI solution (jenkins/buildbot) is not as important as to have a nice suite of tests which could be easily executed. By easily I mean to not require lots of ad-hoc setup needed, and most if not all setup done in the tests fixtures. May be with formalization within tox or nox. Then, it would be easily to start testing on some CI of preference or "availability" even if it is a service.

At some point for our https://github.com/datalad/datalad project I have used buildbot, worked out integration with github to report back on PRs etc, but it was just extra burden to maintain etc all those environments. Now we just use appveyor, github actions, and travis (historically happened -- was the first). Appveyor is preferred by my colleague for providing easy means to login into an instance if so needed. Using some basic scripting and templating we established on github actions quite extended building/testing of git-annex not only across Windows/Linux/OSX , and not only against downstream projects (which is what refuse seems to rely somewhat on ATM), but also on some client systems: see https://github.com/datalad/git-annex/ and setup there in. And then do even more downstream testing across extensions in https://github.com/datalad/datalad-extensions -- again relying on their consistent and simple testing.

So IMHO the current problem is not absence of the choice of CI solution but absence of the tests! And some tests are better than no tests, so may be the starting point could be composing some rudimentary tests here and trying to get them going e.g. on github actions across a range of OSes and Python versions?

Also extra could be adding workflows to test those downstream projects mentioned in the README right in this repo (we also do that in datalad and other projects to ensure that we aren't breaking downstream).

Here is some pieces from which I could try to initiate some testing if sounds useful:

• pytest based testing, and constructs like we have for datalad-fuse in https://github.com/datalad/datalad-fuse/blob/master/datalad_fuse/tests/test_fuse.py#L1 for really just to get going
• basic github action to sweep python/OSes as in https://github.com/dandi/dandi-cli/blob/master/.github/workflows/test.yml

[s-m-e]

So IMHO the current problem is not absence of the choice of CI solution but absence of the tests!

I can probably easily formalize my tests on Linux, i.e. make them part of this repo as a point to start.

The thing with FUSE in general is that it's hard to come up with "simple" tests. It's an operating system kernel interface, after all, that fusepy tries to wrap around in Python. One (wrong) API call alone usually does not yield any issues. It's rather a wrong sequence of events that causes issues (e.g. mistakes in handling internal states). This is why I concluded that the best path forward is to test through real file systems. The most reasonable approach, AFAIC, is to implement a clean and minimal in-RAM file system (similar to tmpfs) that can run both in RO and RW mode (and a few other "in-between things" that people are using fusepy apparently for). This theoretically covers most (but still not all) edge cases in FUSE.

For each test or set of tests, you ideally want to have a clean initial state of FUSE (i.e. the kernel and libfuse). For my own little file system, I did not do this - which was hell, bluntly put. Getting clean states via containers (docker, podman, etc) was my next bad idea - turns out that containers are "too thin", i.e. they rely too much on the actual "host"'s kernel. This is where I ended up even deeper in the rabbit hole: VMs. vagrant became an interesting solution with stripped-down, minimal VMs (doable for at least Linux and BSD). It's still within reason for pytest fixtures, but building the VM images and configuring this correctly is rather time-consuming and not every CI for instance can run this. But behold, there is one more thing: fusepy and FUSE support more than x86, platform-specific behavior included, which makes sense in a modern world, but now we're getting onto CPU emulation territory: qemu.

Back to the simple side: Any tests that can reasonably test fusepy behavior without requiring an FS are welcome, of course. The question is: What is a reasonable approach to work on this repo without loosing any of its features and abilities.

[yarikoptic]

I am yet to grasp the need for going to VM or even containers per se. Again -- some testing is better than no testing. CI environments are VMs or containers already. Building images -- no need to rebuild base images -- can sweep through a few existing/pre-built when needed. FWIW here is our session level pytest fixture(s) for starting docker-compose instance to be reused by tests -- might come handy here: https://github.com/dandi/dandi-cli/blob/master/dandi/tests/fixtures.py#L401

Indeed running some IO intensive tasks on the FUSE'd filesystem is what could provide nice integration testing! The problem is that often might be difficult to discern what elementary problem leads to some misbehavior.

FWIW -- we already test git annex under NFS and vfat in https://github.com/datalad/git-annex/blob/master/.github/workflows/build-ubuntu.yaml#L245 . Could have also tested on refuse-mounted partition if we knew that some fuse-specific behavior is needed to be supported in git-annex.

But may be instead here in this repo there could be a collection of CI tests based on external tools, like you already mention doing using pjdfstest and fsx for starters. Do you have some CI for that already?

The question is: What is a reasonable approach to work on this repo without loosing any of its features and abilities.

Sorry, I am not following in how addition of tests would require loosing anything what works?

[s-m-e]

Sorry, I am not following in how addition of tests would require loosing anything what works?

A bit of a misunderstanding here: Any test is better than no test, you are absolutely right. If you have an idea what to test and how to test it, go for it. I'd be happy to support it.

I tried to make a different point: At the moment, fusepy (and this fork) support a wide range of platforms and architectures - without a test suite, "battle-tested in the wild". Third-party projects rely on it, its exact behavior and (after digging through tons of code) its bugs (!). Which is cool, until you want to refactor. Which is necessary to get stuff like e.g. support for a more recent version of libfuse done. How do I reliably maintain the level of quality and reliability that the project already has (despite its other deficiencies)?

The problem is that often might be difficult to discern what elementary problem leads to some misbehavior.

Exactly, this is what I observed when I began running pjdfstest against my own file system back in the day. Layers of complicated, convoluted bugs - with combined root causes in both my code and in fusepy and (in one instance) even in libfuse itself. I spent multiple days on individual bugs each until I had figured out the interaction of all layers, the CPython interpreter, libfuse and the OS kernel. I suspected bugs in the latter (Linux), but thankfully I was wrong. The fact that FUSE is not entirely state-less makes it even "more fun".

I want to maintain at least current code quality / features / reliability. This is about file systems - users relying on them can lose (valuable) data. Any test therefore fundamentally helps. Making a judgement along the lines of "this code does what it is supposed to do without loosing or unexpectedly altering data" is, however, from my perspective a fundamentally more tricky thing to proof. This is what got me thinking about containers, VMs and stuff.

At the end of the day, when I invoke e.g. pytest, I want it to run against all target platforms and architectures and test the entire stack, from the a call via fusepy info libfuse and down to the kernel's FUSE sys calls (and back) - ideally not just for single calls but sequences of calls, i.e. for common file system operations. For me at least, it boils down to the following two questions:

1. How can I guarantee a clean initial state of the entire stack for each individual test?
2. How can I do this across all target platforms and architectures (and their specific behaviors) in an easy enough manner?

[yarikoptic]

How do I reliably maintain the level of quality and reliability that the project already has (despite its other deficiencies)?

I would start with some regression testing to establish baseline. I did that for https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/ at some point and it gave a healthy push forward in development since some fears were put behind ;)

[yarikoptic]

In #37 we can start getting some pytest'ing oing. Getting back to my question above -- do you have some CI or script you use to test using pjdfstest and fsx?

[s-m-e]

do you have some CI or script you use to test using pjdfstest and fsx?

Yes. The init*, destroy* and test* targets in my makefile control it, the tests module contains the entire implementation and the (outdated) Travis configuration uses it.