Update 'end2endTest' from e3-opcua/test @ 5aa3fc5

From remote https://gitlab.esss.lu.se/e3/wrappers/communication/e3-opcua.git
updated the 'end2endTest' subtree from the 'test' subdirectory

end2endTest/README.md

@@ -30,7 +30,7 @@ pip3 install pytest opcua pyepics
 pip3 install run-iocsh -i https://artifactory.esss.lu.se/artifactory/api/pypi/pypi-virtual/simple
 ```
 
-You must configure EPICS environment variables before running the test suite. 
+You must configure the EPICS environment before running the test suite.
 For the E3 environment, this requires you to ``source setE3Env.bash``.
 
 At least `EPICS_BASE` and `EPICS_HOST_ARCH` need to be set.
@@ -55,7 +55,7 @@ signals are available in OPC UA namespace 2.
 For further information on the server configuration, see [simulation server](test/server/README.md).
 
 ### IOC
-A test IOC is provided that translates the OPC UA variables from the test server. 
+A test IOC is provided that translates the OPC UA variables from the test server.
 The following records are defined:
 
 | Record         | EPICS Type | OPC-UA Type | Record            | EPICS Type | OPC-UA Type |
@@ -86,10 +86,10 @@ cmd/ and db/ subdirectories.
 ## Python Test Files
 
 The pytest framework [2] is used to implement the test cases. Individual test cases are provided
-as python functions (defs) in [\(opcua_test_cases.py\)](test/opcua_test_cases.py). Under the hood, 
+as python functions (defs) in [\(opcua_test_cases.py\)](test/opcua_test_cases.py). Under the hood,
 run_iocsh [3] and pyepics [4] are used for communication with the test IOC.
 
-To add a new test case, simply add a new funtion (def) to [\(opcua_test_cases.py\)](test/opcua_test_cases.py), 
+To add a new test case, simply add a new funtion (def) to [\(opcua_test_cases.py\)](test/opcua_test_cases.py),
 ensuring that the function name begins with the prefix ``test_``
 
 ### Test Classes and Test Cases
@@ -98,54 +98,54 @@ The test points are split into four classes:
 
 #### Connection tests (TestConnectionTests)
 
- 1. **_test_connect_disconnect_**: start and stop the test IOC 5 times. Parse the IOC output, 
+ 1. **_test_connect_disconnect_**: start and stop the test IOC 5 times. Parse the IOC output,
       and check it connects and disconnects to the OPC-UA server successfully.
 
- 2. **_test_connect_reconnect_**: Start the server, start the IOC. Stop the server, 
+ 2. **_test_connect_reconnect_**: Start the server, start the IOC. Stop the server,
       check for appropriate messaging. Start the server, check that the IOC reconnects.
 
  3. **_test_no_connection_**: Start an IOC with no server running. Check the module reports appropriately.
 
- 4. **_test_shutdown_on_ioc_reboot_**: Start the server. Start an IOC and ensure connection 
-      is made to the server. Shutdown the IOC and endure that the subscriptions and sessions are 
+ 4. **_test_shutdown_on_ioc_reboot_**: Start the server. Start an IOC and ensure connection
+      is made to the server. Shutdown the IOC and endure that the subscriptions and sessions are
       cleanly disconnected.
 
 
 #### Variable tests (TestVariableTests)
 
- 1. **_test_server_status_**: Check the informational values provided by the server are being 
+ 1. **_test_server_status_**: Check the informational values provided by the server are being
       translated via the module.
 
- 2. **_test_variable_pvget_**: Start the test IOC and use pvget to read the ``TstRamp`` PV 
+ 2. **_test_variable_pvget_**: Start the test IOC and use pvget to read the ``TstRamp`` PV
       value multiple times (every second). Check that it is incrementing as a ramp.
 
- 3. **_test_read_variable_**: Read the deafult value of a variable from the opcua server and 
-      check it matches the expected value. Parametrised for all supported datatypes 
+ 3. **_test_read_variable_**: Read the deafult value of a variable from the opcua server and
+      check it matches the expected value. Parametrised for all supported datatypes
       (boolean, sbyte, byte, int16, uint16, int32, uint32, int64, uint64, float, double, string.)
 
- 4. **_test_write_variable_**: Write a known value to the opcua server via the output PV linked 
-      to the variable. Read back via the input PV and check the values match. Parametrised for all 
+ 4. **_test_write_variable_**: Write a known value to the opcua server via the output PV linked
+      to the variable. Read back via the input PV and check the values match. Parametrised for all
       supported datatypes.
       (boolean, sbyte, byte, int16, uint16, int32, uint32, int64, uint64, float, double, string.)
 
- 5. **_test_timestamps_**:  Start the test server in a shell session with with a fake time in 
-      the past, using libfaketime [5]. Check that the timestamp for the PV read matches the 
-      known fake time given to the server. If they match, the OPCUA EPICS module is correctly 
+ 5. **_test_timestamps_**:  Start the test server in a shell session with with a fake time in
+      the past, using libfaketime [5]. Check that the timestamp for the PV read matches the
+      known fake time given to the server. If they match, the OPCUA EPICS module is correctly
       pulling the timestamps from the OPCUA server (and not using a local timestamp).
 
 
 #### Performance tests (TestPerformanceTests)
 
- 1. **_test_write_performance_**: Write 5000 variable values and measure time and memory 
+ 1. **_test_write_performance_**: Write 5000 variable values and measure time and memory
       consumption before and after. Repeat 10 times
 
- 2. **_test_read_performance_**: Read 5000 variable values and measure time and memory 
+ 2. **_test_read_performance_**: Read 5000 variable values and measure time and memory
       consumption before and after. Repeat 10 times
 
- 
+
 #### Negative tests (TestNegativeTests)
 
- 1. **_test_no_server_**: Start an OPC-UA IOC with no server running. 
+ 1. **_test_no_server_**: Start an OPC-UA IOC with no server running.
       Check the module reports this correctly.
 
  2. **_test_bad_var_name_**:  Specify an incorrect variable name in a db record.
@@ -158,22 +158,22 @@ The test points are split into four classes:
 ### Running the test suite
 You can run the test suite from the root of the repository wuth the following command:
 ```
-pytest -v test/opcua_test_cases.py
+pytest -v
 ```
 
 To view the stdout output from the tests in real-time, you can provide the ``-s`` flag:
 ```
-pytest -v -s test/opcua_test_cases.py
+pytest -v -s
 ```
 
 To run all tests in a class:
 ```
-pytest -v test/opcua_test_cases.py::TestConnectionTests
+pytest -v -k TestConnectionTests
 ```
 
 To run an individual test point:
 ```
-pytest -v test/opcua_test_cases.py::TestConnectionTests::test_connect_disconnect
+pytest -v -k test_connect_disconnect
 ```
 
 

end2endTest/cmds/test_pv.cmd

@@ -16,4 +16,3 @@ iocshLoad("$(opcua_DIR)/opcua.iocsh", "P=OPC:,SESS=$(SESSION),SUBS=$(SUBSCRIPT),
 dbLoadRecords("test_pv.db", "OPCSUB=$(SUBSCRIPT), NS=$(OPCNAMESPACE)")
 
 iocInit()
-

end2endTest/cmds/test_pv_neg.cmd

@@ -16,4 +16,3 @@ iocshLoad("$(opcua_DIR)/opcua.iocsh", "P=OPC:,SESS=$(SESSION),SUBS=$(SUBSCRIPT),
 dbLoadRecords("test_pv_neg.db", "OPCSUB=$(SUBSCRIPT), NS=$(OPCNAMESPACE)")
 
 iocInit()
-

end2endTest/opcua_test_cases.py → end2endTest/test_opcua.py

@@ -1,14 +1,15 @@
-from epics import PV
-from time import sleep
-from run_iocsh import IOC
-from os import environ
-from datetime import datetime
 import os
-import pytest
-import subprocess
 import resource
-import time
 import signal
+import subprocess
+import time
+from datetime import datetime
+from os import environ
+from time import sleep
+
+import pytest
+from epics import PV, ca
+from run_iocsh import IOC
 
 
 class opcuaTestHarness:
@@ -197,6 +198,9 @@ def test_inst():
     yield the harness handle to the test,
     close the server on test end / failure
     """
+    # Initialize channel access
+    ca.initialize_libca()
+
     # Create handle to Test Harness
     test_inst = opcuaTestHarness()
     # Poll to see if the server is running
@@ -208,11 +212,16 @@ def test_inst():
     test_inst.start_server()
     # Drop to test
     yield test_inst
+
     # Shutdown server by sending terminate signal
     test_inst.stop_server()
     # Check server is stopped
     assert not test_inst.isServerRunning
 
+    # Shut down channel access
+    ca.flush_io()
+    ca.clear_cache()
+
 
 # test fixture for use with timezone server
 @pytest.fixture(scope="function")
@@ -222,6 +231,9 @@ def test_inst_TZ():
     yield the harness handle to the test,
     close the server on test end / failure
     """
+    # Initialize channel access
+    ca.initialize_libca()
+
     # Create handle to Test Harness
     test_inst_TZ = opcuaTestHarness()
     # Poll to see if the server is running
@@ -233,11 +245,16 @@ def test_inst_TZ():
     test_inst_TZ.start_server_with_faketime()
     # Drop to test
     yield test_inst_TZ
+
     # Shutdown server by sending terminate signal
     test_inst_TZ.stop_server_group()
     # Check server is stopped
     assert not test_inst_TZ.isServerRunning
 
+    # Shut down channel access
+    ca.flush_io()
+    ca.clear_cache()
+
 
 class TestConnectionTests:
     nRuns = 5
@@ -281,8 +298,6 @@ def test_stop_and_restart_server(self, test_inst):
             ioc.start()
             assert ioc.is_running()
 
-            sleep(test_inst.sleepTime)
-
             test_inst.stop_server()
             assert ioc.is_running()
 
@@ -444,7 +459,6 @@ def test_variable_pvget(self, test_inst):
         with IOC(
             *test_inst.TestArgs,
             test_inst.cmd,
-            ioc_executable=test_inst.IOCSH_PATH,
         ):
             # PV name
             pvName = "TstRamp"
@@ -609,6 +623,7 @@ def test_timestamps(self, test_inst_TZ):
 
 
 class TestPerformanceTests:
+    @pytest.mark.xfail("CI" in environ, reason="GitLab runner performance issues")
     def test_write_performance(self, test_inst):
         """
         Write 5000 variable values and measure
@@ -632,7 +647,7 @@ def test_write_performance(self, test_inst):
             writeperrun = 5000
 
             # Run test 10 times
-            for j in range(testruns):
+            for j in range(1, testruns):
 
                 # Get time and memory conspumtion before test
                 r0 = resource.getrusage(resource.RUSAGE_THREAD)
@@ -645,10 +660,7 @@ def test_write_performance(self, test_inst):
                 # Get delta time and delta memory
                 dt = time.perf_counter() - t0
                 r1 = resource.getrusage(resource.RUSAGE_THREAD)
-                dr = (
-                    resource.getrusage(resource.RUSAGE_THREAD).ru_maxrss
-                    - r0.ru_maxrss  # NoQA: E501
-                )
+                dr = r1.ru_maxrss - r0.ru_maxrss  # NoQA: E501
 
                 # Collect data for statistics
                 if dt > maxt:
@@ -661,6 +673,7 @@ def test_write_performance(self, test_inst):
                 print("Memory incr: ", dr)
                 print("     before: ", r0.ru_maxrss)
                 print("      after: ", r1.ru_maxrss)
+
             avgt = tott / testruns
 
             print("Max time: ", "{:.3f} s".format(maxt))
@@ -669,9 +682,9 @@ def test_write_performance(self, test_inst):
             print("Total memory increase: ", totr)
 
             assert maxt < 17
-            assert mint > 1
-            assert avgt < 12
-            assert totr < 1000
+            assert mint > 0.8
+            assert avgt < 5
+            assert totr < 3000
 
     def test_read_performance(self, test_inst):
         """
@@ -696,7 +709,7 @@ def test_read_performance(self, test_inst):
             writeperrun = 5000
 
             # Run test 10 times
-            for j in range(testruns):
+            for j in range(1, testruns):
 
                 # Get time and memory conspumtion before test
                 r0 = resource.getrusage(resource.RUSAGE_SELF)
@@ -709,10 +722,7 @@ def test_read_performance(self, test_inst):
                 # Get delta time and delta memory
                 dt = time.perf_counter() - t0
                 r1 = resource.getrusage(resource.RUSAGE_SELF)
-                dr = (
-                    resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
-                    - r0.ru_maxrss  # NoQA: E501
-                )
+                dr = r1.ru_maxrss - r0.ru_maxrss  # NoQA: E501
 
                 # Collect data for statistics
                 if dt > maxt:
@@ -725,6 +735,7 @@ def test_read_performance(self, test_inst):
                 print("Memory incr: ", dr)
                 print("     before: ", r0.ru_maxrss)
                 print("      after: ", r1.ru_maxrss)
+
             avgt = tott / testruns
 
             print("Max time: ", "{:.3f} s".format(maxt))
@@ -734,11 +745,39 @@ def test_read_performance(self, test_inst):
 
             assert maxt < 10
             assert mint > 0.01
-            assert avgt < 5
+            assert avgt < 7
             assert totr < 1000
 
 
 class TestNegativeTests:
+    def test_no_server(self, test_inst):
+        """
+        Start an OPC-UA IOC with no server running.
+        Check the module reports this correctly.
+        """
+
+        ioc = test_inst.IOC
+
+        # Stop the running server
+        test_inst.stop_server()
+
+        # Start the IOC
+        ioc.start()
+        assert ioc.is_running()
+
+        # Check that PVs have SEVR INVALID (=3)
+        pv = PV("VarCheckSByte")
+        pv.get(timeout=test_inst.getTimeout)
+        assert pv.severity == 3
+
+        # Stop IOC, and check output
+        ioc.exit()
+        assert not ioc.is_running()
+
+        ioc.check_output()
+        output = ioc.outs
+        print(output)
+
     def test_bad_var_name(self, test_inst):
         """
         Specify an incorrect variable name in a db record.
@@ -753,9 +792,6 @@ def test_bad_var_name(self, test_inst):
         ioc.start()
         assert ioc.is_running()
 
-        # Wait some time
-        sleep(1)
-
         # Stop IOC, and check output
         ioc.exit()
         assert not ioc.is_running()
@@ -782,9 +818,6 @@ def test_wrong_datatype(self, test_inst):
         ioc.start()
         assert ioc.is_running()
 
-        # Wait some time
-        sleep(1)
-
         # Stop IOC, and check output
         ioc.exit()
         assert not ioc.is_running()