Task/add ngsiv2 ngsild measures payload

CHANGES_NEXT_RELEASE

@@ -1,3 +1,4 @@
+- Add: allow to send NGSIv2 and NGSILD as measures depending on payloadType (#778)
 - Fix: binary data representation when sending data through HTTP & MQTT (#690)
 - Fix: ensure service and subservice from device in logs about error proccesing message
 - Remove: legacy code about unused parsedMessageError flag

docs/usermanual.md

@@ -87,7 +87,7 @@ It is possible to send a single measure to IoT Platform using an HTTP POST reque
 `/iot/json/attrs/<attributeName>` and the previously explained query parameters.
 
 In this case, sending a single measure, there is possible to send other kinds of payloads like `text/plain` and
-`application/octet-stream`, not just `application/json`. In case of using `application/octet-stream`, data will be 
+`application/octet-stream`, not just `application/json`. In case of using `application/octet-stream`, data will be
 treated as binary data, saved in the attribute maped as hex string. I.E:
 
 For a measure sent to `POST /iot/json/attrs/attrHex` with content-type: application/octet-stream and binary value
@@ -98,17 +98,152 @@ hello
 
 then the resulting attribute sent to ContextBroker:
 
+```
 {
    ...
    "attrHex": {
-     "value": "68656c6c6f"
+     "value": "68656c6c6f",
      "type": "<the one used at provisiong time for attrHex attribute>"
    }
 }
+```
+
+Note that every group of 2 character (I.E, the first group, `68`) corresponds to a single ASCII character or byte
+received in the payload (in this case, the value `0x68` corresponds to `h` in ASCII). You can use one of the multiple
+tools available online like [this one](https://string-functions.com/string-hex.aspx)
+
+##### NGSI-v2 and NGSI-LD Measure reporting
+
+It is possible report as a measure a NGSI-v2 or NGSI-LD payload when related device/group is configured with
+`payloadType` `ngsiv2` or `ngsild`. In these cases payload is ingested as measure where entity attributes are measure
+attributes and `id` and `type` are ignored, since `id` and `type` from device/group configuration provisioned are used,
+as well as `actionType`.
+
+Examples of these `ngsiv2` payloads are the following ones:
+
+(1) NGSI-v2 batch update format:
+
+```
+ {
+     "actionType": "append",
+     "entities": [
+        {
+          "id": "MyEntityId1",
+          "type": "MyEntityType1",
+          "attr1": { type: "Text", "value": "MyAttr1Value"},
+          ...
+        },
+        ...
+    ]
+ }
+```
+
+(2) NGSI-v2 plain entities array format:
+
+```
+ [
+    {
+          "id": "MyEntityId1",
+          "type": "MyEntityType1",
+          "attr1": { "type": "Text", "value": "MyAttr1Value"},
+          ...
+    },
+    ...
+ ]
+```
 
-Note that every group of 2 character (I.E, the first group, `68`) corresponds to a single ASCII character or byte received in 
-the payload (in this case, the value `0x68` corresponds to `h` in ASCII). You can use one of the multiple tools available 
-online like [this one](https://string-functions.com/string-hex.aspx)
+(3) NGSI-v2 plain single entity format:
+
+```
+{
+    "id": "MyEntityId1",
+    "type": "MyEntityType1",
+    "attr1": { "type": "Text", "value": "MyAttr1Value"},
+    ...
+}
+```
+```
+
+Example of these `ngsild` payloads are the following ones:
+
+(1) NGSI-LD entities array format:
+
+```
+ [
+         {
+                "id": "urn:ngsi-ld:ParkingSpot:santander:daoiz_velarde_1_5:3",
+                "type": "ParkingSpot",
+                "status": {
+                    "type": "Property",
+                    "value": "free",
+                    "observedAt": "2018-09-21T12:00:00Z"
+                },
+                "category": {
+                    "type": "Property",
+                    "value": ["onstreet"]
+                },
+                "refParkingSite": {
+                    "type": "Relationship",
+                    "object": "urn:ngsi-ld:ParkingSite:santander:daoiz_velarde_1_5"
+                },
+                "name": {
+                    "type": "Property",
+                    "value": "A-13"
+                },
+                "location": {
+                    "type": "GeoProperty",
+                    "value": {
+                        "type": "Point",
+                        "coordinates": [-3.80356167695194, 43.46296641666926]
+                    }
+                },
+                "@context": [
+                    "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld",
+                    "https://schema.lab.fiware.org/ld/context"
+                ]
+          },
+          ...
+ ]
+```
+
+(2) NGSI-LD single entity format:
+
+```
+{
+    "id": "urn:ngsi-ld:ParkingSpot:santander:daoiz_velarde_1_5:3",
+    "type": "ParkingSpot",
+    "status": {
+        "type": "Property",
+        "value": "free",
+        "observedAt": "2018-09-21T12:00:00Z"
+    },
+    "category": {
+        "type": "Property",
+        "value": ["onstreet"]
+    },
+    "refParkingSite": {
+        "type": "Relationship",
+        "object": "urn:ngsi-ld:ParkingSite:santander:daoiz_velarde_1_5"
+    },
+    "name": {
+        "type": "Property",
+        "value": "A-13"
+    },
+    "location": {
+        "type": "GeoProperty",
+        "value": {
+            "type": "Point",
+            "coordinates": [-3.80356167695194, 43.46296641666926]
+        }
+    },
+    "@context": [
+        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld",
+        "https://schema.lab.fiware.org/ld/context"
+    ]
+}
+```
+
+Note that array of entities are handled as a multiple measure, each entity is a measure.
 
 #### Configuration retrieval
 
@@ -316,29 +451,37 @@ attribute IDs `h` and `t`, then humidity measures are reported this way:
 $ mosquitto_pub -t /json/ABCDEF/id_sen1/attrs/h -m 70 -h <mosquitto_broker> -p <mosquitto_port> -u <user> -P <password>
 ```
 
-In the single measure case, when the published data is not a valid JSON, it is interpreted as binary content. For instance,
-if the following is published to `/json/ABCDEF/id_sen1/attrs/attrHex` topic:
+In the single measure case, when the published data is not a valid JSON, it is interpreted as binary content. For
+instance, if the following is published to `/json/ABCDEF/id_sen1/attrs/attrHex` topic:
 
 ```
 hello
 ```
 
 then the resulting attribute sent to ContextBroker:
 
+```
 {
    ...
    "attrHex": {
-     "value": "68656c6c6f"
+     "value": "68656c6c6f",
      "type": "<the one used at provisiong time for attrHex attribute>"
    }
 }
+```
+
+Note that every group of 2 character (I.E, the first group, `68`) corresponds to a single ASCII character or byte
+received in the payload (in this case, the value `0x68` corresponds to `h` in ASCII). You can use one of the multiple
+tools available online like [this one](https://string-functions.com/string-hex.aspx).
+
+Note this works differently that in HTTP transport. In HTTP the JSON vs. binary decission is based on
+`application/octed-stream` `content-type` header. Given that in MQTT we don't have anything equivalent to HTTP headers,
+we apply the heuristics of checking for JSON format.
 
-Note that every group of 2 character (I.E, the first group, `68`) corresponds to a single ASCII character or byte received in 
-the payload (in this case, the value `0x68` corresponds to `h` in ASCII). You can use one of the multiple tools available 
-online like [this one](https://string-functions.com/string-hex.aspx).
+##### NGSI-v2 and NGSI-LD Measure reporting
 
-Note this works differently that in HTTP transport. In HTTP the JSON vs. binary decission is based on `application/octed-stream` `content-type` header.
-Given that in MQTT we don't have anything equivalent to HTTP headers, we apply the heuristics of checking for JSON format.
+Using topics for multiple measure reporting its possible also ingest `ngsiv2` and `ngsild` payloads in the same way that
+was described for http binding.
 
 #### Configuration retrieval
 

lib/bindings/HTTPBinding.js

@@ -311,16 +311,22 @@ function handleIncomingMeasure(req, res, next) {
             } else {
                 payloadDataArr = req.jsonPayload;
             }
+
             if (req.jsonPayload) {
+                config.getLogger().debug(context, 'Parsing payloadDataArr %j for device %j', payloadDataArr, device);
                 for (const i in payloadDataArr) {
-                    values = commonBindings.extractAttributes(device, payloadDataArr[i]);
-                    attributeArr.push(values);
+                    values = commonBindings.extractAttributes(device, payloadDataArr[i], device.payloadType);
+                    if (values[0][0]) {
+                        // Check multimeasure from a ngsiv2/ngsild entities array
+                        attributeArr = attributeArr.concat(values);
+                    } else {
+                        attributeArr.push(values);
+                    }
                 }
             } else {
                 attributeArr = [];
             }
         }
-
         if (attributeArr.length === 0) {
             finishSouthBoundTransaction(next);
         } else {

lib/commonBindings.js

@@ -92,15 +92,67 @@ function guessType(attribute, device) {
     return constants.DEFAULT_ATTRIBUTE_TYPE;
 }
 
-function extractAttributes(device, current) {
-    const values = [];
-    for (const k in current) {
-        if (current.hasOwnProperty(k)) {
-            values.push({
-                name: k,
-                type: guessType(k, device),
-                value: current[k]
-            });
+function extractAttributes(device, current, payloadType) {
+    let values = [];
+
+    const ctxt = fillService(context, device);
+    config.getLogger().debug(ctxt, 'extractAttributes current %j payloadType %j', current, payloadType);
+
+    if (payloadType && [constants.PAYLOAD_NGSIv2, constants.PAYLOAD_NGSILD].includes(payloadType.toLowerCase())) {
+        let arrayEntities = [];
+        if (current.hasOwnProperty('actionType') && current.hasOwnProperty('entities')) {
+            arrayEntities = current.entities;
+        } else {
+            arrayEntities = [current];
+        }
+        for (const entity of arrayEntities) {
+            const valuesEntity = [];
+            for (const k in entity) {
+                // skip id and type
+                if (entity.hasOwnProperty(k) && !['id', 'type'].includes(k)) {
+                    if (payloadType.toLowerCase() === constants.PAYLOAD_NGSIv2) {
+                        valuesEntity.push({
+                            name: k,
+                            type: entity[k].type,
+                            value: entity[k].value,
+                            metadata: entity[k].metadata ? entity[k].metadata : undefined
+                        });
+                    } else if (payloadType.toLowerCase() === constants.PAYLOAD_NGSILD) {
+                        const ent = {
+                            name: k
+                        };
+                        if (k.toLowerCase() === '@context') {
+                            ent.type = '@context';
+                            ent.value = entity[k];
+                        } else {
+                            if (entity[k].type) {
+                                ent.type = entity[k].type;
+                                if (['property', 'geoproperty'].includes(entity[k].type.toLowerCase())) {
+                                    ent.value = entity[k].value;
+                                } else if (entity[k].type.toLowerCase() === 'relationship') {
+                                    ent.value = entity[k].object;
+                                }
+                            }
+                        }
+                        valuesEntity.push(ent);
+                    }
+                }
+            }
+            if (arrayEntities.length > 1) {
+                values.push(valuesEntity); // like a multimeasure
+            } else {
+                values = valuesEntity;
+            }
+        }
+    } else {
+        for (const k in current) {
+            if (current.hasOwnProperty(k)) {
+                values.push({
+                    name: k,
+                    type: guessType(k, device),
+                    value: current[k]
+                });
+            }
         }
     }
     return values;
@@ -205,38 +257,46 @@ function multipleMeasures(apiKey, deviceId, device, messageObj) {
 
     for (let j = 0; j < messageObj.length; j++) {
         measure = messageObj[j];
-        const values = [];
-        for (const i in measure) {
-            if (measure.hasOwnProperty(i)) {
-                values.push({
-                    name: i,
-                    type: guessType(i, device),
-                    value: measure[i]
-                });
-            }
+        let attributesArray = [];
+        const values = extractAttributes(device, measure, device.payloadType);
+        if (values[0][0]) {
+            // multi measure is extracted due to payloadType is ngsi
+            attributesArray = values;
+        } else {
+            attributesArray = [values];
         }
-
-        iotAgentLib.update(device.name, device.type, '', values, device, function (error) {
-            if (error) {
-                config.getLogger().error(
-                    ctxt,
-                    /*jshint quotmark: double */
-                    "MEASURES-002: Couldn't send the updated values to the Context Broker due to an error: %j",
-                    /*jshint quotmark: single */
-                    error
+        for (const k in attributesArray) {
+            config
+                .getLogger()
+                .debug(
+                    context,
+                    'Processing multiple measures for device %s with apiKey %s values %j',
+                    deviceId,
+                    apiKey,
+                    attributesArray[k]
                 );
-            } else {
-                config
-                    .getLogger()
-                    .info(
+            iotAgentLib.update(device.name, device.type, '', attributesArray[k], device, function (error) {
+                if (error) {
+                    config.getLogger().error(
                         ctxt,
-                        'Multiple measures for device %s with apiKey %s successfully updated',
-                        deviceId,
-                        apiKey
+                        /*jshint quotmark: double */
+                        "MEASURES-002: Couldn't send the updated values to the Context Broker due to an error: %j",
+                        /*jshint quotmark: single */
+                        error
                     );
-            }
-            finishSouthBoundTransaction(null);
-        });
+                } else {
+                    config
+                        .getLogger()
+                        .info(
+                            ctxt,
+                            'Multiple measures for device %s with apiKey %s successfully updated',
+                            deviceId,
+                            apiKey
+                        );
+                }
+                finishSouthBoundTransaction(null);
+            });
+        }
     }
 }