Add +LINUX and +WINDOWS doctest options (#2507)

* Add `+LINUX` and `+WINDOWS` doctest options

This allows to selectively run tests only on a single platform. We can add `# doctest: +LINUX` comments to tests that cannot work on Windows and the other way around.

To easily skip a lot of tests the `doctest_additional_flags` global variable can be defined in a `testsetup`.

This is achieved by monkey patching sphinx doctest's DocTestBuilder to use our own DocTestRunner which removes examples from the tests that have flags that don't match the platform we're running on.

* Limit Sphinx version to secure platform patches

Avoid major versions which might change the API. We have to check if the platform optionflags still work on newer versions once they are available.

* CI: Run doctests with coverage on Windows

Disable all non-trivial tests on Windows for now. The goal is to reduce the amount of linux-only tests.

* Only apply platform patch on Python 3

* Disable uploading coverage on Windows

The handrolled coveralls upload cannot handle mixed operating systems.

Refs #2480

* Use threading.Timer for doctest timeout

To interrupt the code running on the main thread, we send a signal using `_thread.interrupt_main()`. By default this causes a KeyboardInterrupt exception, which might be handled explicitly.

To raise an explicit EndlessLoop exception inside the code that is taking too long, register a SIGABRT signal handler which raises the EndlessLoop exception. The exception from the signal handler is added to the call stack and handled by the code currently running. This allows to print a better stack trace on timeout.

It is the same concept as the old implementation using `signal.alarm` but platform agnostic.

https://anonbadger.wordpress.com/2018/12/15/python-signal-handlers-and-exceptions/

* Add POSIX optionflag

Run test on other UNIX systems too if they don't use Linux specifics.

Add a TODO optionflag too to mark platform restrictions that might be too strict and should be looked at.

* Enable tube and tube/sockets tests on Windows

* Use `signal.alarm` for timeouts if it's available

* Update CHANGELOG

.github/workflows/ci.yml

@@ -284,10 +284,25 @@ jobs:
         pip install --upgrade pip
         pip install --upgrade --editable .
     
+    - name: Install documentation dependencies
+      run: pip install -r docs/requirements.txt
+
     - name: Sanity checks
       run: |
         python -bb -c 'from pwn import *'
         python -bb examples/text.py
+    
+    - name: Coverage doctests
+      run: |
+        python -bb -m coverage run -m sphinx -b doctest docs/source docs/build/doctest
+
+    # FIXME: Paths are broken when uploading coverage on ubuntu
+    # coverage.exceptions.NoSource: No source for code: '/home/runner/work/pwntools/pwntools/D:\a\pwntools\pwntools\pwn\__init__.py'.
+    # - uses: actions/upload-artifact@v4
+    #   with:
+    #     name: coverage-windows
+    #     path: .coverage*
+    #     include-hidden-files: true
 
   upload-coverage:
     runs-on: ubuntu-latest

CHANGELOG.md

@@ -74,6 +74,9 @@ The table below shows which release corresponds to each branch, and what date th
 
 ## 5.0.0 (`dev`)
 
+- [#2507][2507] Add `+LINUX` and `+WINDOWS` doctest options and start proper testing on Windows
+
+[2507]: https://github.com/Gallopsled/pwntools/pull/2507
 
 ## 4.15.0 (`beta`)
 - [#2508][2508] Ignore a warning when compiling with asm on nix

docs/requirements.txt

@@ -19,6 +19,6 @@ psutil
 requests>=2.5.1
 ropgadget>=5.3
 sphinx==1.8.6; python_version<'3'
-sphinx>=7.0.0; python_version>='3'
+sphinx>=8.1.3, <9; python_version>='3'
 sphinx_rtd_theme
 sphinxcontrib-autoprogram<=0.1.5

docs/source/adb.rst

@@ -4,6 +4,9 @@
    from pwn import *
    adb = pwnlib.adb
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.adb` --- Android Debug Bridge
 =====================================================
 

docs/source/asm.rst

@@ -4,6 +4,10 @@
    import subprocess
    from pwn import *
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.asm` --- Assembler functions
 =========================================
 

docs/source/conf.py

@@ -398,6 +398,15 @@ def dont_skip_any_doctests(app, what, name, obj, skip, options):
 
 class _DummyClass(object): pass
 
+# doctest optionflags for platform-specific tests
+# they are skipped on other platforms
+WINDOWS = doctest.register_optionflag('WINDOWS')
+LINUX = doctest.register_optionflag('LINUX')
+POSIX = doctest.register_optionflag('POSIX')
+
+# doctest optionflag for tests that haven't been looked at yet
+TODO = doctest.register_optionflag('TODO')
+
 class Py2OutputChecker(_DummyClass, doctest.OutputChecker):
     def check_output(self, want, got, optionflags):
         sup = super(Py2OutputChecker, self).check_output
@@ -425,27 +434,86 @@ def check_output(self, want, got, optionflags):
                 return False
         return True
 
+import sphinx.ext.doctest
+
+class PlatformDocTestRunner(sphinx.ext.doctest.SphinxDocTestRunner):
+    def run(self, test, compileflags=None, out=None, clear_globs=True):
+        original_optionflags = self.optionflags | test.globs.get('doctest_additional_flags', 0)
+        def filter_platform(example):
+            optionflags = original_optionflags
+            if example.options:
+                for (optionflag, val) in example.options.items():
+                    if val:
+                        optionflags |= optionflag
+                    else:
+                        optionflags &= ~optionflag
+
+            if (optionflags & WINDOWS) == WINDOWS and sys.platform != 'win32':
+                return False
+            if (optionflags & LINUX) == LINUX and sys.platform != 'linux':
+                return False
+            if (optionflags & POSIX) == POSIX and os.name != 'posix':
+                return False
+            return True
+
+        test.examples[:] = [example for example in test.examples if filter_platform(example)]
+
+        return super(PlatformDocTestRunner, self).run(test, compileflags, out, clear_globs)
+
+class PlatformDocTestBuilder(sphinx.ext.doctest.DocTestBuilder):
+    _test_runner = None
+
+    @property
+    def test_runner(self):
+        return self._test_runner
+
+    @test_runner.setter
+    def test_runner(self, value):
+        self._test_runner = PlatformDocTestRunner(value._checker, value._verbose, value.optionflags)
+
 def py2_doctest_init(self, checker=None, verbose=None, optionflags=0):
     if checker is None:
         checker = Py2OutputChecker()
     doctest.DocTestRunner.__init__(self, checker, verbose, optionflags)
 
 if 'doctest' in sys.argv:
-    def setup(app):
-        pass # app.connect('autodoc-skip-member', dont_skip_any_doctests)
 
     if sys.version_info[:1] < (3,):
-        import sphinx.ext.doctest
         sphinx.ext.doctest.SphinxDocTestRunner.__init__ = py2_doctest_init
     else:
+        def setup(app):
+            app.add_builder(PlatformDocTestBuilder, override=True)
+            # app.connect('autodoc-skip-member', dont_skip_any_doctests)
         # monkey patching paramiko due to https://github.com/paramiko/paramiko/pull/1661
         import paramiko.client
         import binascii
         paramiko.client.hexlify = lambda x: binascii.hexlify(x).decode()
         paramiko.util.safe_string = lambda x: '' # function result never *actually used*
     class EndlessLoop(Exception): pass
-    def alrm_handler(sig, frame):
-        signal.alarm(180) # three minutes
-        raise EndlessLoop()
-    signal.signal(signal.SIGALRM, alrm_handler)
-    signal.alarm(600) # ten minutes
+    if hasattr(signal, 'alarm'):
+        def alrm_handler(sig, frame):
+            signal.alarm(180) # three minutes
+            raise EndlessLoop()
+        signal.signal(signal.SIGALRM, alrm_handler)
+        signal.alarm(600) # ten minutes
+    else:
+        def sigabrt_handler(signum, frame):
+            raise EndlessLoop()
+        # thread.interrupt_main received the signum parameter in Python 3.10
+        if sys.version_info >= (3, 10):
+            signal.signal(signal.SIGABRT, sigabrt_handler)
+        def alrm_handler():
+            try:
+                import thread
+            except ImportError:
+                import _thread as thread
+            # pre Python 3.10 this raises a KeyboardInterrupt in the main thread.
+            # it might not show a traceback in that case, but it will stop the endless loop.
+            thread.interrupt_main(signal.SIGABRT)
+            timer = threading.Timer(interval=180, function=alrm_handler) # three minutes
+            timer.daemon = True
+            timer.start()
+        import threading
+        timer = threading.Timer(interval=600, function=alrm_handler) # ten minutes
+        timer.daemon = True
+        timer.start()

docs/source/elf/corefile.rst

@@ -18,6 +18,9 @@
    # Set the environment here so it's not in the middle of our tests.
    os.environ.setdefault('SHELL', '/bin/sh')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 
 :mod:`pwnlib.elf.corefile` --- Core Files
 ===========================================================

docs/source/elf/elf.rst

@@ -5,6 +5,10 @@
    from pwnlib.elf.maps import CAT_PROC_MAPS_EXIT
    import shutil
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.elf.elf` --- ELF Files
 ===========================================================
 

docs/source/encoders.rst

@@ -1,6 +1,10 @@
 .. testsetup:: *
 
    from pwn import *
+
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
 
 :mod:`pwnlib.encoders` --- Encoding Shellcode
 ===============================================

docs/source/filesystem.rst

@@ -6,6 +6,10 @@
     from pwnlib.tubes.ssh import ssh
     from pwnlib.filesystem import *
 
+    # TODO: Remove global POSIX flag
+    import doctest
+    doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.filesystem` --- Manipulating Files Locally and Over SSH
 ====================================================================
 

docs/source/gdb.rst

@@ -4,6 +4,10 @@
     context.arch = 'amd64'
     context.terminal = [os.path.join(os.path.dirname(pwnlib.__file__), 'gdb_faketerminal.py')]
 
+    # TODO: Test on cygwin too
+    import doctest
+    doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.gdb` --- Working with GDB
 ======================================
 

docs/source/intro.rst

@@ -2,6 +2,9 @@
 
    from pwn import *
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 Getting Started
 ========================
 

docs/source/libcdb.rst

@@ -3,6 +3,10 @@
    from pwn import *
    from pwnlib.libcdb import *
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.libcdb` --- Libc Database
 ===========================================
 

docs/source/qemu.rst

@@ -2,6 +2,10 @@
 
    from pwn import *
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 
 :mod:`pwnlib.qemu` --- QEMU Utilities
 ==========================================

docs/source/rop/rop.rst

@@ -19,6 +19,10 @@
 
    context.clear()
 
+   # TODO: Remove global LINUX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 
 :mod:`pwnlib.rop.rop` --- Return Oriented Programming
 ==========================================================

docs/source/rop/srop.rst

@@ -7,6 +7,9 @@
    from pwnlib.elf import ELF
    from pwnlib.tubes.process import process
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.rop.srop` --- Sigreturn Oriented Programming
 ==========================================================
 

docs/source/runner.rst

@@ -3,6 +3,10 @@
    from pwnlib.runner import *
    from pwnlib.asm import asm
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.runner` --- Running Shellcode
 ===========================================
 

docs/source/shellcraft.rst

@@ -2,6 +2,10 @@
 
    from pwnlib import shellcraft
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.shellcraft` --- Shellcode generation
 =================================================
 

docs/source/shellcraft/aarch64.rst

@@ -3,6 +3,9 @@
    from pwn import *
    context.clear(arch='aarch64')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.shellcraft.aarch64` --- Shellcode for AArch64
 ===========================================================
 

docs/source/shellcraft/amd64.rst

@@ -3,6 +3,10 @@
    from pwn import *
    context.clear(arch='amd64')
 
+   # TODO: POSIX/WINDOWS shellcode test
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.shellcraft.amd64` --- Shellcode for AMD64
 ===========================================================
 

docs/source/shellcraft/arm.rst

@@ -3,6 +3,9 @@
    from pwn import *
    context.clear(arch='arm')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.shellcraft.arm` --- Shellcode for ARM
 ===========================================================
 

docs/source/shellcraft/i386.rst

@@ -3,6 +3,9 @@
    from pwn import *
    context.clear(arch='i386')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.shellcraft.i386` --- Shellcode for Intel 80386
 ===========================================================
 

docs/source/shellcraft/mips.rst

@@ -12,6 +12,9 @@
 
    context.clear(arch='mips')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.shellcraft.mips` --- Shellcode for MIPS
 ===========================================================
 

docs/source/shellcraft/riscv64.rst

@@ -3,6 +3,9 @@
    from pwn import *
    context.clear(arch='riscv64')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.shellcraft.riscv64` --- Shellcode for RISCV64
 ==========================================================
 

docs/source/shellcraft/thumb.rst

@@ -3,6 +3,9 @@
    from pwn import *
    context.clear(arch='thumb')
 
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['LINUX']
+
 :mod:`pwnlib.shellcraft.thumb` --- Shellcode for Thumb Mode
 ===========================================================
 

docs/source/tubes/processes.rst

@@ -2,6 +2,10 @@
 
    from pwn import *
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.tubes.process` --- Processes
 ===========================================================
 

docs/source/tubes/serial.rst

@@ -2,6 +2,10 @@
 
    from pwn import *
 
+   # TODO: Remove global POSIX flag
+   import doctest
+   doctest_additional_flags = doctest.OPTIONFLAGS_BY_NAME['POSIX']
+
 :mod:`pwnlib.tubes.serialtube` --- Serial Ports
 ===========================================================