Disposition options, resend and MDN handling enhancements (#388)

* Use the LTS releases of Java in testing.

* Avoiding cleaning up files if there is a processing error allowing retry
to handle cleanup.

* Enhance processDMN method to return boolean.

Boolean allows downstream processing to decided if it should cleanup
files avoiding file not found errors.

* Support checking for resend state as a method on the message object.

* Make the SQL statement compliant with SQL 92 standard.

* Make log message multi line for readability.

* Add processing states

* Avoid calling cleanup of files if message is in resend state.

* Provide a random serial ID

* |Updated documentation

* Update supporting library versions to latest.

* Enhance handling of MDN issues.

* Add debug logging to display received Disposition-Notification-Options

* Enhance handling of MDN processing issues and resending.

* Enhance parsing of Disposition options to be more flexible and
independent of parameter order changes from partners.

* Add a message state for issues when using Async MDN.

* Version update and library upgrades

* Stay on 3.* for jersey/grizzly for Java 11 compatibility

* Update yarn.lock to avoid CVE warnings
Set note to indicate the WEB UI will be removed in the future.

* Disable all command processors by default.

* Update documentation about Java compatibility.

* Rollback H2 library to support Java 8

* Add release notes

.github/workflows/actions.yml

@@ -6,7 +6,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        java_version: [8, 11, 13, 17]
+        java_version: [8, 11, 17, 21]
         os: [windows-latest, ubuntu-latest]
     steps:
       - name: Checkout

RELEASE-NOTES.md

@@ -1,22 +1,22 @@
 #              OpenAS2 Server
-#              Version 3.10.1
+#              Version 3.11.0
 #              RELEASE NOTES
 -----
-The OpenAS2 project is pleased to announce the release of OpenAS2 3.10.1
+The OpenAS2 project is pleased to announce the release of OpenAS2 3.11.0
 
-The release download file is: OpenAS2Server-3.10.1.zip
+The release download file is: OpenAS2Server-3.11.0.zip
 
 The zip file contains a PDF document (OpenAS2HowTo.pdf) providing information on installing and using the application.
-## NOTE: Testing covers Java 8 to 17. The application should work for older versions down to Java 7 but they are not tested as part of the CI/CD pipeline.
+## NOTE: Testing covers Java 8 to 17. See Java Compatibility section of the OpenAS2 documentation for using Java 8.
 
-Version 3.10.1 - 2024-03-17
+Version 3.11.0 - 2024-08-22
 This is a minor bugfix release:
        **IMPORTANT NOTE**: Please review upgrade notes below if you are upgrading
 
- 1. Fix sending the Content-Length header correctly on MDN response when chinked transfer is not supported by the receiving software.
- 2. Fix log message to indicate SYNC or ASYNC mode when receiving an MDN.
- 3. Change the default Java package to Eclipse Temurin for the docker creation.
- 4. Upgrade the encryprion packages to fix a low severity security issue.
+1. Enhance the handling of MDN processing and asociated resend handling to provide moer reliable error handling.
+2. Enhance the Disposition-Notification-Options handler to properly support the standard.
+3. Use the LTS releases of Java for testing
+4. Use a SQL-92 compliant SQL format for accessing the database.
 
 ##Upgrade Notes
  See the openAS2HowTo appendix for the general process on upgrading OpenAS2.

Remote/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>net.sf.openas2</groupId>
         <artifactId>OpenAS2</artifactId>
-        <version>3.10.1</version>
+        <version>3.11.0</version>
     </parent>
 
     <modelVersion>4.0.0</modelVersion>

Server/pom.xml

@@ -7,7 +7,7 @@
         <!-- DO NOT CHANGE THIS "groupId" WITHOUT CHANGING XMLSession.getManifestAttributes.MANIFEST_VENDOR_ID_ATTRIB -->
         <groupId>net.sf.openas2</groupId>
         <artifactId>OpenAS2</artifactId>
-        <version>3.10.1</version>
+        <version>3.11.0</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

Server/src/config/config.xml

@@ -13,9 +13,9 @@
                email.logger.enabled="false"
                email.logger.only_active_msg_transfer_errors="false"
                email.logger.properties.log_exception_trace="false"
-               console.command.processor.enabled="true"
+               console.command.processor.enabled="false"
                socket.command.processor.enabled="false"
-               restapi.command.processor.enabled="true"
+               restapi.command.processor.enabled="false"
                restapi.command.processor.baseuri="http://localhost:8080"
                restapi.command.processor.userid="userID"
                restapi.command.processor.password="pWd"

Server/src/main/java/org/openas2/OpenAS2Exception.java

@@ -8,12 +8,9 @@
 
 
 public class OpenAS2Exception extends Exception {
+    private static final long serialVersionUID = -2266872772193560354L;
     public static final String SOURCE_MESSAGE = "message";
     public static final String SOURCE_FILE = "file";
-    /**
-     *
-     */
-    private static final long serialVersionUID = 1L;
     private Map<String, Object> sources = new HashMap<String, Object>();
     private Log logger = LogFactory.getLog(OpenAS2Exception.class.getSimpleName());
 

Server/src/main/java/org/openas2/message/BaseMessage.java

@@ -102,6 +102,11 @@ public void setStatus(String status) {
         this.status = status;
     }
 
+    public boolean isResend() {
+        // Determines if message is currently in resend phase
+        return Message.MSG_STATUS_MSG_RESEND.equals(getStatus());
+    }
+
     public Map<String, String> getCustomOuterMimeHeaders() {
         return customOuterMimeHeaders;
     }

Server/src/main/java/org/openas2/message/Message.java

@@ -41,7 +41,9 @@ public interface Message extends Serializable {
     String MSG_STATE_MDN_RECEIVE_START = "mdn_receive_start";
     String MSG_STATE_MDN_ASYNC_RECEIVE_FAIL = "mdn_asyn_receive_fail";
     String MSG_STATE_MSG_SENT_MDN_RECEIVED_ERROR = "msg_sent_mdn_received_error";
+    String MSG_STATE_MSG_SENT_MDN_PROCESSING_ERROR = "msg_sent_mdn_processing_error";
     String MSG_STATE_MSG_SENT_MDN_RECEIVED_OK = "msg_sent_mdn_received_ok";
+    String MSG_STATE_MSG_SENT_AWAIT_ASYNC_MDN_RESPONSE = "msg_sent_await_async_mdn_response";
     String MSG_STATE_MSG_RXD_MDN_SENDING_FAIL = "msg_rxd_mdn_sending_fail";
     String MSG_STATE_MSG_RXD_MDN_SENT_OK = "msg_rxd_mdn_sent_ok";
     String MSG_STATE_MSG_RXD_MDN_NOT_REQUESTED = "msg_rxd_mdn_not_requested_ok";
@@ -65,9 +67,13 @@ public interface Message extends Serializable {
             put(MSG_STATE_MDN_SEND_START, "Message received. MDN sending started");
             put(MSG_STATE_MDN_RECEIVE_START, "Message sent. MDN receiving started");
             put(MSG_STATE_MSG_SENT_MDN_RECEIVED_ERROR, "Message sent. Message MDN received indicates an error. Resend queued");
+            put(MSG_STATE_MSG_SENT_MDN_PROCESSING_ERROR, "Message sent and MDN received but error occurred processing MDN.");
             put(MSG_STATE_MSG_SENT_MDN_RECEIVED_OK, "Message sent. Message MDN success response received.");
+            put(MSG_STATE_MSG_SENT_AWAIT_ASYNC_MDN_RESPONSE, "Message sent. await async MDN response.");
             put(MSG_STATE_MSG_RXD_MDN_SENDING_FAIL, "Message was received but failed to successfully send an MDN response to partner");
+            put(MSG_STATE_MIC_MISMATCH, "Message sent successfully but MDN from partner indicates MIC mismatch.");
             put(MSG_STATE_MSG_RXD_MDN_NOT_REQUESTED, "Message received successfully but no MDN requested.");
+            put(MSG_STATE_MSG_RXD_MDN_SENT_OK, "Message received successfully and MDN succesfully sent to partner.");
         }
     };
 
@@ -161,6 +167,8 @@ public interface Message extends Serializable {
 
     void setFileCleanupCompleted(boolean cleanupDone);
 
+    boolean isResend();
+
     String getSubject();
 
     void setSubject(String subject);

Server/src/main/java/org/openas2/processor/msgtracking/DbTrackingModule.java

@@ -291,8 +291,9 @@ public ArrayList<HashMap<String, String>> getDataCharts(HashMap<String, String>
 
             Statement s = conn.createStatement();
             ResultSet rs = s.executeQuery("SELECT " + FIELDS.MSG_ID + ",STATE,STATUS,CREATE_DT FROM " + tableName
-                    + " WHERE CREATE_DT BETWEEN TIMESTAMP '" + map.get("startDate").toString()
-                    + " 00:00:00' AND TIMESTAMP '" + map.get("endDate").toString() + " 23:59:59'");
+                    + " WHERE CREATE_DT BETWEEN CAST('" + map.get("startDate").toString()
+                    + " 00:00:00' as TIMESTAMP) AND CAST('" + map.get("endDate").toString()
+                    + " 23:59:59' as TIMESTAMP)");
             ResultSetMetaData meta = rs.getMetaData();
             while (rs.next()) {
                 HashMap<String, String> row = new HashMap<String, String>();

Server/src/main/java/org/openas2/processor/receiver/AS2MDNReceiverHandler.java

@@ -3,19 +3,15 @@
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.openas2.OpenAS2Exception;
-import org.openas2.lib.util.MimeUtil;
 import org.openas2.message.AS2Message;
 import org.openas2.message.AS2MessageMDN;
 import org.openas2.message.Message;
 import org.openas2.message.MessageMDN;
 import org.openas2.partner.Partnership;
 import org.openas2.processor.msgtracking.BaseMsgTrackingModule.FIELDS;
 import org.openas2.util.AS2Util;
-import org.openas2.util.ByteArrayDataSource;
 import org.openas2.util.HTTPUtil;
 
-import javax.activation.DataHandler;
-import javax.mail.internet.ContentType;
 import javax.mail.internet.MimeBodyPart;
 import java.io.IOException;
 import java.net.HttpURLConnection;
@@ -72,75 +68,77 @@ public void handle(NetModule owner, Socket s) {
                     return;
                 }
             }
-            // check if the requested URL is defined in attribute "as2_receipt_option"
-            // in one of partnerships, if yes, then process incoming AsyncMDN
             if (logger.isInfoEnabled()) {
                 logger.info("incoming connection for receiving AsyncMDN" + " [" + getClientInfo(s) + "]" + msg.getLogMsgID());
             }
             if (logger.isTraceEnabled()) {
                 logger.trace("Incoming ASYNC MDN message - Message struct: " + msg.toString());
             }
-            ContentType receivedContentType;
             MimeBodyPart receivedPart = new MimeBodyPart(msg.getHeaders(), data);
-            receivedContentType = new ContentType(receivedPart.getContentType());
+            // ContentType receivedContentType = new ContentType(receivedPart.getContentType());
             msg.setData(receivedPart);
 
-            // Switch the msg headers since the original message went in the opposite direction
+            /* Switch the msg headers for To and From to standardise processing for SYNC and ASYNC MDN
+             * since the original message this MDN is responding to went in the opposite direction
+             */
             String to = msg.getHeader("AS2-To");
             msg.setHeader("AS2-To", msg.getHeader("AS2-From"));
             msg.setHeader("AS2-From", to);
             msg.getPartnership().setSenderID(Partnership.PID_AS2, msg.getHeader("AS2-From"));
             msg.getPartnership().setReceiverID(Partnership.PID_AS2, msg.getHeader("AS2-To"));
-            getModule().getSession().getPartnershipFactory().updatePartnership(msg, true);
+            try {
+                getModule().getSession().getPartnershipFactory().updatePartnership(msg, true);
+            } catch (OpenAS2Exception e) {
+                // Partnership not found so log and exit
+                try {
+                    HTTPUtil.sendHTTPResponse(s.getOutputStream(), HttpURLConnection.HTTP_BAD_REQUEST, null);
+                } catch (IOException e1) {
+                }
+                if (logger.isInfoEnabled()) {
+                    logger.info("Partnership lookup failed for MDN received from: " + msg.getHeader("AS2-To")
+                        + "  MDN is targeting partner: " + msg.getHeader("AS2-From"));
+                }
+                return;
+            }
 
             // Create a MessageMDN
             MessageMDN mdn = new AS2MessageMDN(msg, true);
 
             if (logger.isTraceEnabled()) {
                 logger.trace("Incoming ASYNC MDN message - MDN struct: " + mdn.toString());
             }
-            /*
-            // Log significant msg state
-            options.put("STATE", Message.MSG_STATE_MDN_RECEIVE_START);
-            options.put("STATE_MSG", "MDN response received. Message processing started.");
-            msg.trackMsgState(getModule().getSession(), options);
-            */
-            AS2Util.processMDN(msg, data, s.getOutputStream(), true, getModule().getSession(), this.getClass());
-            // Log significant msg state
-            msg.setOption("STATE", Message.MSG_STATE_MSG_SENT_MDN_RECEIVED_OK);
-            msg.trackMsgState(getModule().getSession());
+            try {
+                boolean partnerIdentificationProblem = AS2Util.processMDN(msg, data, s.getOutputStream(), true, getModule().getSession(), this.getClass());
+                // Assume that appropriate logging and state handling was done upstream if an error occurred so only log state change for success
+                if (!partnerIdentificationProblem) {
+                    // Log  state
+                    msg.setOption("STATE", Message.MSG_STATE_MSG_SENT_MDN_RECEIVED_OK);
+                    msg.trackMsgState(getModule().getSession());
+                }
+            } catch (Exception e) {
+                /* Processing of the MDN would have done extensive error handling so only log an error if the error
+                 * is an not OpenAS2 custom error.
+                 */
+                if (!(e instanceof OpenAS2Exception)){
+                    /*
+                     * Something unexpected (assumes a resend was not successfully initiated)
+                     */
+                    msg.setLogMsg("Unhandled error condition processing synchronous MDN. Message and associated files cleanup will be attempted but may be in an unknown state.");
+                    logger.error(msg, e);
+                }
+                // Log significant msg state
+                msg.setOption("STATE", Message.MSG_STATE_MSG_SENT_MDN_PROCESSING_ERROR);
+                msg.trackMsgState(getModule().getSession());
+                AS2Util.cleanupFiles(msg, true);
+            }
 
         } catch (Exception e) {
-            if (Message.MSG_STATUS_MDN_PROCESS_INIT.equals(msg.getStatus()) || Message.MSG_STATUS_MDN_PARSE.equals(msg.getStatus()) || !(e instanceof OpenAS2Exception)) {
-                /*
-                 * Cannot identify the target if in init or parse state so not sure what the
-                 * best course of action is apart from do nothing
-                 */
-                msg.setLogMsg("Unhandled error condition receiving asynchronous MDN. Message and associated files cleanup will be attempted but may be in an unknown state.");
+                msg.setLogMsg("Unhandled error condition receiving asynchronous MDN. Processing will be aborted.");
                 logger.error(msg.getLogMsg(), e);
                 try {
                     HTTPUtil.sendHTTPResponse(s.getOutputStream(), HttpURLConnection.HTTP_BAD_REQUEST, null);
                 } catch (IOException e1) {
                 }
-            } else {
-                /*
-                 * Most likely a resend abort of max resend reached if
-                 * OpenAS2Exception so do not log as should have been logged
-                 * upstream ... just clean up the mess
-                 */
-                // Must have received MDN successfully so must respond with OK
-                try {
-                    HTTPUtil.sendHTTPResponse(s.getOutputStream(), HttpURLConnection.HTTP_OK, null);
-                } catch (IOException e1) { // What to do ....
-                }
-                msg.setLogMsg("Exception receiving asynchronous MDN. Message and asociated files cleanup will be attempted but may be in an unknown state.");
-                logger.error(msg, e);
-
-            }
-            // Log significant msg state
-            msg.setOption("STATE", Message.MSG_STATE_SEND_FAIL);
-            msg.trackMsgState(getModule().getSession());
-            AS2Util.cleanupFiles(msg, true);
         }
 
     }

Server/src/main/java/org/openas2/processor/receiver/AS2ReceiverHandler.java

@@ -436,6 +436,9 @@ protected String decryptAndVerify(AS2Message msg) throws OpenAS2Exception {
          * RFC4130 section 7.3.1 for details)
          */
         DispositionOptions dispOptions = new DispositionOptions(msg.getHeader("Disposition-Notification-Options"));
+        if (LOG.isDebugEnabled()) {
+            LOG.debug("Received Disposition-Notification-Options header value: " + dispOptions);
+        }
         if (dispOptions.getMicalg() != null) {
             try {
                 mic = ch.calculateMIC(msg.getData(), dispOptions.getMicalg(), (msg.isRxdMsgWasSigned() || msg.isRxdMsgWasEncrypted()), msg.getPartnership().isPreventCanonicalization());

Server/src/main/java/org/openas2/processor/receiver/MessageBuilderModule.java

@@ -231,7 +231,8 @@ protected Message processDocument(File pendingFile, Message msg) throws OpenAS2E
             msg.setStatus(Message.MSG_STATUS_MSG_SEND);
             // Transmit the message
             getSession().getProcessor().handle(SenderModule.DO_SEND, msg, options);
-            if (!msg.isConfiguredForAsynchMDN()) {
+            // Cleanup files only if sending was successful and an MDN was already received
+            if (!msg.isResend() && !msg.isConfiguredForAsynchMDN()) {
                 AS2Util.cleanupFiles(msg, false);
             }
         } catch (Exception e) {

Server/src/main/java/org/openas2/processor/resender/DirectoryResenderModule.java

@@ -178,7 +178,11 @@ protected void processFile(File file) throws OpenAS2Exception {
                 }
                 if (logger.isTraceEnabled()) {
                     try {
-                        logger.trace("Reconstituted Message object in resender. Content-Disposition: " + msg.getContentDisposition() + "\n      Content-Type : " + msg.getContentType() + "\n      HEADERS : " + AS2Util.printHeaders(msg.getData().getAllHeaders()) + "\n      Content-Disposition in MSG getData() MIMEPART: " + msg.getData().getContentType() + "\n      ATTRIBUTES : " + msg.getAttributes() + msg.getLogMsgID());
+                        logger.trace("Reconstituted Message object in resender. Content-Disposition: " + msg.getContentDisposition()
+                            + "\n      Content-Type : " + msg.getContentType()
+                            + "\n      HEADERS : " + AS2Util.printHeaders(msg.getData().getAllHeaders())
+                            + "\n      Content-Disposition in MSG getData() MIMEPART: " + msg.getData().getContentType()
+                            + "\n      ATTRIBUTES : " + msg.getAttributes() + msg.getLogMsgID());
                     } catch (Exception e) {
                     }
                 }