Screen: Autoenable Vulkan display backend on macOS ARM in various sit…

…uations.

On Apple Silicon Macs under macOS, visual stimulus presentation timing and timestamping is
utterly broken when using Apples proprietary OpenGL only on Apples proprietary gpu. Only our
Vulkan display backend allows proper timing.

Therefore, in a typical fullscreen display configuration, the Vulkan display backend should always
be used, so that scripts which don't explicitely request UseVulkanDisplay via PsychImaging still
get timing of similar precision as on Intel based Macs.

This commit adds auto-enable support for Vulkan on macOS on ARM:

1) PsychImaging('OpenWindow', ...) will auto-opt-in to Vulkan in typical usage scenarios.
2) To deal with legacy scripts that use Screen('OpenWindow', ...) without PsychImaging,
 add some detection logic so Screen('OpenWindow') can detect if it wasn't called through
PsychImaging. In that case it will call PsychImaging('OpenWindow', ...) to trigger case 1)
and thereby get those legacy scripts also promoted to use of the Vulkan display backend.
The new kPsychBackendDecisionMade specialflag is used by PsychImaging to signal to
Screen when it is in charge of controlling display backend choice.

PsychSourceGL/Source/Common/Screen/SCREENOpenWindow.c

@@ -326,30 +326,54 @@ PsychError SCREENOpenWindow(void)
         psych_bool isARM;
 
         PsychGetOSXMinorVersion(&isARM);
-        if (isARM && !(specialflags & kPsychExternalDisplayMethod) && !dontCaptureScreen) {
+        if (isARM && !(specialflags & kPsychExternalDisplayMethod) && !(specialflags & kPsychBackendDecisionMade) && !dontCaptureScreen) {
             // M1 SoC or later, Apple proprietary gpu with OpenGL emulated on top of Metal + CoreAnimation.
             // This does not work at all with OpenGL CGL low-level fullscreen display mode, only through
             // Cocoa+NSOpenGL+NSWindow on top of CoreAnimation. Not using Cocoa will simply error abort with
-            // a "CGLSetFullScreenOnDisplay failed: invalid fullscreen drawable" error. So we switch to Cocoa
-            // voluntarily. Ofc. with this, OpenGL display timing/timestamping is utterly broken, but it may
-            // allow users to limp along on their new shiny expensive M1 iToy. We take specialflags setting
-            // kPsychExternalDisplayMethod as a sign that the user requested Vulkan backend display or similar
-            // to try to workaround this issue, so we spare them extra warnings and actions, etc.:
-
-            // Need to take action. Request Quartz composition / Cocoa / NSOpenGL backend:
-            PsychPrefStateSet_ConserveVRAM(PsychPrefStateGet_ConserveVRAM() | kPsychUseAGLCompositorForFullscreenWindows);
-
-            if (PsychPrefStateGet_Verbosity() > 1) {
-                printf("PTB-WARNING: This is a Apple silicon based ARM M1 SoC or later with Apple proprietary gpu.\n");
-                printf("PTB-WARNING: All of Psychtoolbox own timing and timestamping mechanisms will not work on\n");
-                printf("PTB-WARNING: such a machine, leading to disastrously bad visual stimulus presentation timing\n");
-                printf("PTB-WARNING: and timestamping. Do not trust or use this machine if timing is of any concern!\n");
-                printf("PTB-WARNING: You may want to try enabling Psychtoolbox Vulkan display backend, after proper\n");
-                printf("PTB-WARNING: configuration. See 'help PsychImaging' the section about the 'UseVulkanDisplay'\n");
-                printf("PTB-WARNING: task, and 'help PsychHDR' for some more setup instructions for MoltenVK on macOS.\n");
-                printf("PTB-WARNING: Note that this approach is completely unsupported by us in case of any problems, and\n");
-                printf("PTB-WARNING: may just be as bad performance and timing-wise. It is completely untested on M1.\n");
+            // a "CGLSetFullScreenOnDisplay failed: invalid fullscreen drawable" error. So we have to switch
+            // to Cocoa voluntarily. Proper visual stimulus presentation timing and timestamping also requires
+            // use of our Vulkan display backend, which itself requires complex setup by PsychImaging.m and
+            // PsychVulkan.m. If Screen('Openwindow', ...) is called directly, instead of high-level wrapped
+            // via PsychImaging('OpenWindow', ...), this crucial setup can't happen. Therefore we detect this
+            // direct call by the absence of the specialflags flag kPsychBackendDecisionMade, and call
+            // PsychImaging('OpenWindow', ...) on behalf of the users script, essentially rewriting the call to
+            // Screen('OpenWindow', ...) into an equivalent PsychImaging('OpenWindow', ...), so PsychImaging
+            // can make the Vulkan vs. OpenGL decision and possibly perform needed Vulkan setup, then recursively
+            // call back into us. This allows legacy user scripts which don't use PsychImaging to continue to work
+            // unmodified on macOS for Apple Silicon Macs.
+            if (PsychPrefStateGet_Verbosity() > 2)
+                printf("PTB-INFO: Running on a macOS Apple Silicon system: Checking if Vulkan display backend should be used.\n");
+
+            // Array with PsychImaging return arguments [win, winRect]:
+            PsychGenericScriptType *outputs[2];
+
+            // Prepare/Assing PsychImaging('OpenWindow', ...); call arguments:
+            int nrInputs = PsychGetNumInputArgs() + 1;
+            PsychGenericScriptType *inputs[nrInputs];
+            for (int i = 0; i < nrInputs; i++) {
+                inputs[i] = (PsychGenericScriptType*) PsychGetInArgPtr(i);
             }
+
+            // Call [win, winRect] = PsychImaging('OpenWindow', ...); and error out on error:
+            // PsychImaging('OpenWindow', ...); itself will decide on a display backend, OpenGL or Vulkan,
+            // set things up in case of Vulkan, and then recursively call us, ie. Screen('OpenWindow', ...);
+            // with potentially tweaked parameters and the specialflags setting kPsychBackendDecisionMade
+            // again, so this code branch gets skipped and the regular Screen('OpenWindow', ...) will run.
+            // Its return arguments will be post-processed by PsychImaging and then PsychImaging returns
+            // final [win, winRect] = PsychImaging('OpenWindow', ...); [win, winRect] arguments to us
+            // when returning from this call, and we will return those return args to our caller.
+            if (Psych_mexCallMATLAB(2, outputs, nrInputs, inputs, "PsychImaging"))
+                PsychErrorExitMsg(PsychError_user, "Error in PsychImaging('OpenWindow', ...) redirected call on Apple Silicon system!");
+
+            // Worked! Return the window index and the rect argument from [win, winRect] = PsychImaging('OpenWindow', ...):
+            for (int i = 0; i < PsychGetNumOutputArgs(); i++) {
+                if (PsychIsArgPresent(PsychArgOut, i + 1)) {
+                    *PsychGetOutArgMxPtr(i + 1) = outputs[i];
+                }
+            }
+
+            // Back to caller of [win, winRect] = Screen('OpenWindow', ...);
+            return(PsychError_none);
         }
     }
     #endif

PsychSourceGL/Source/Common/Screen/WindowBank.h

@@ -168,15 +168,17 @@
 #define kPsychSkipWaitForFlipOnce           (1 << 30) // 'specialflags': Perform next flip on this window without waiting until the 'when' target time for the flip.
 #define kPsychNeedVBODouble12Workaround     (1ULL << 31) // 'specialflags': Gfx driver bug makes < 2 component vertex attribute buffers problematic if GL_DOUBLE is used for submission.
 #define kPsychExternalDisplayMethod         (1ULL << 32) // 'specialflags': This window is not used for actual visual stimulation, as some external display mechanism is used, e.g., Vulkan or VR compositor.
-#define kPsychDontAutoResetOneshotFlags     (1ULL << 33) // 'specialFlags': Do not auto-reset the "one-shot" flip flags after a flip, ie. don't clear kPsych*ForFlipOnce flags.
-#define kPsychDontUseFlipperThread          (1ULL << 34) // 'specialFlags': Do not allow use of the background flipper thread, because it conflicts with some external display method.
+#define kPsychDontAutoResetOneshotFlags     (1ULL << 33) // 'specialflags': Do not auto-reset the "one-shot" flip flags after a flip, ie. don't clear kPsych*ForFlipOnce flags.
+#define kPsychDontUseFlipperThread          (1ULL << 34) // 'specialflags': Do not allow use of the background flipper thread, because it conflicts with some external display method.
 #define kPsychSkipSecondaryVsyncForFlip     (1ULL << 35) // 'specialflags': Perform flips on this windows associated secondary/slavewindow without VSYNC, e.g., for mirror mode.
+#define kPsychBackendDecisionMade           (1ULL << 36) // 'specialflags': Decision wrt. use of display backend has been intentionally made by high-level 'OpenWindow' caller.
 
 // The following numbers are allocated to imagingMode flag above: A (S) means, shared with specialFlags:
 // 1,2,4,8,16,32,64,128,256,512,1024,S-2048,4096,S-8192,16384,32768,S-65536,2^17,2^18,2^19,2^20,2^21,2^22,2^23,2^24,S-2^25. --> Flags of 2^26 and higher are available...
 
 // The following numbers are allocated to specialFlags flag above: A (S) means, shared with imagingMode:
-// 1,2,4,8,16,32,64,128,256,512,1024,S-2048,4096,S-8192, 16384, 32768, S-65536,2^17,2^18,2^19,2^20,2^21,2^22,2^23,2^24,S-2^25,2^26,2^27,2^28,2^29,2^30,2^31,2^32,2^33,2^34,2^35. --> Flags of 2^36 and higher are available...
+// 1,2,4,8,16,32,64,128,256,512,1024,S-2048,4096,S-8192, 16384, 32768, S-65536,2^17,2^18,2^19,2^20,2^21,2^22,2^23,2^24,S-2^25,2^26,2^27,2^28,2^29,2^30,
+// 2^31,2^32,2^33,2^34,2^35,2^36. --> Flags of 2^37 and higher are available...
 
 // Definition of a single hook function spec:
 typedef struct PsychHookFunction*   PtrPsychHookFunction;

Psychtoolbox/PsychGLImageProcessing/PsychImaging.m

@@ -1755,7 +1755,7 @@
 
 % This global variable signals if a GPGPU compute api is enabled, and which
 % one. 0 = None, 1 = GPUmat.
-global psych_gpgpuapi;
+global psych_gpgpuapi; %#ok<*GVMIS>
 
 % These flags are global - needed in subfunctions as well (ugly ugly coding):
 global ptb_outputformatter_icmAware;
@@ -1946,6 +1946,26 @@
 
         % Compute special OpenWindow overrides for winRect, framebuffer rect, specialflags and MSAA, as needed:
         [winRect, ovrfbOverrideRect, ovrSpecialFlags, multiSample, screenid] = hmd.driver('OpenWindowSetup', hmd, screenid, winRect, ovrfbOverrideRect, ovrSpecialFlags, multiSample);
+    else
+        % No VR/AR/XR operation, but displaying on a more conventional display.
+
+        % Force the UseVulkanDisplay task if it doesn't exist already if running
+        % on macOS for Apple Silicon, as that is the only display backend which
+        % can ensure proper visual stimulus presentation timing and timestamping:
+        % TODO ASE: Add conditions like "windowed" or some ConserveVRAM flags
+        % etc. to prevent forced use of Vulkan...
+        if IsOSX && IsARM && isempty(find(mystrcmp(reqs, 'UseVulkanDisplay')))
+            % Request Vulkan display backend:
+            reqs = AddTask(reqs, 'General', 'UseVulkanDisplay');
+
+            % Set ovrSpecialFlags to signal that the backend decision has been
+            % made intentionally and explicitely for Screen() by PsychImaging():
+            if isempty(ovrSpecialFlags)
+                ovrSpecialFlags = 0;
+            end
+
+            ovrSpecialFlags = mor(ovrSpecialFlags, kPsychBackendDecisionMade);
+        end
     end
 
     % If multiSample is still "use default" choice, then override it to our default of 0 for "no MSAA":

Psychtoolbox/PsychGLImageProcessing/kPsychBackendDecisionMade.m

@@ -0,0 +1,37 @@
+function rval = kPsychBackendDecisionMade
+% rval = kPsychBackendDecisionMade
+%
+% This flag can be passed to the optional 'specialFlags' parameter of
+% Screen('OpenWindow', ...) or PsychImaging('OpenWindow', ...).
+%
+% This flag tells Screen('OpenWindow', ...) that a higher level calling
+% function of Screen('OpenWindow', ...), e.g., PsychImaging('OpenWindow', ...),
+% has made an appropriate, explicit, and intentional choice of the display
+% backend that Screen() should use for visual stimulus presentation for the
+% given onscreen window, and performed / performs all needed setup, so
+% Screen() should not take any initiative on its own, but simply obey
+% whatever the calling code has decided for it. Absence of the flag is an
+% indicator that Screen('OpenWindow', ...) is called by user written
+% experiment scripts directly, which are unaware of specific display
+% backend requirements on a given setup, so Screen() may have to make
+% backend selection decisions on its own.
+%
+% In practice, as of November 2024, this flag matters for Screen() running
+% on top of macOS on a Apple Silicon Mac with Apples own proprietary gpu
+% and display engine on top of Apples proprietary OpenGL implementation,
+% which requires use of the Vulkan display backend in most cases for proper
+% visual stimulus presentation timing and timestamping. If Screen() detects
+% it is running on macOS for Apple Silicon and the flag is present, then it
+% knows that high-level code like PsychImaging('OpenWindow', ...) has taken
+% care of proper setup. Absence of the flag signals that Screen was likely
+% called directly by a users legacy experiment script that was unaware of
+% the special requirements of the macOS + Apple Silicon platform. In this
+% case, Screen('Openwindow') will recursively call PsychImaging('OpenWindow')
+% so PsychImaging can then do the needed appropriate backend decisions and
+% itself call back into Screen() with the flag set, to perform actual
+% proper configuration. It is essentially a backwards compatibility measure
+% to make old legacy scripts work transparently on this special snowflake
+% platform.
+
+rval = 2^36;
+return