[BREAKING] Device Factory enhancements

docs/source/getting-started.md

@@ -332,8 +332,8 @@ Here is the configuration of the HTTP device factory, where we declare a periodi
     "utc_offset_seconds": 0
   }
   ```
-* maps the response JSON as follows:
-   * the result is associated to a fixed provider name: `grenoble-weather`. `@Provider` is a special placeholder to name the provider that holds the data we received, this is the only mandatory field in a mapping definition.
+* will map the response JSON as follows:
+   * the result is associated to a fixed provider name: `grenoble_weather`. `@Provider` is a special placeholder to name the provider that holds the data we received, this is the only mandatory field in a mapping definition. Note that model, provider, service and resource names are normalized before being transmitted to the Eclipse sensiNact model. See [here](./southbound/device-factory/core.md#names-handling) for more information.
    * the provider friendly (human readable) name will be: `Grenoble Weather`. It is stored using the `@Name` placeholder.
    * the exact location of the weather data point is given by its latitude, longitude and elevation.
    * the weather was measured/computed at the time given in the `time` entry of the `current_weather` object. The timestamp is given as an [ISO 8601 date time](https://en.wikipedia.org/wiki/ISO_8601) (in UTC timezone), that can be given to the device factory using `@datetime` placeholder. Other time-related placeholders are: `@Date`, `@Time` and `@Timestamp` (for Unix timestamps).
@@ -353,7 +353,7 @@ Here is the configuration of the HTTP device factory, where we declare a periodi
             "parser": "json",
             "mapping": {
               "@provider": {
-                "literal": "grenoble-weather"
+                "literal": "grenoble_weather"
               },
               "@name": {
                 "literal": "Grenoble Weather"
@@ -366,15 +366,15 @@ Here is the configuration of the HTTP device factory, where we declare a periodi
                 "path": "current_weather/temperature",
                 "type": "float"
               },
-              "weather/wind-speed": {
+              "weather/wind_speed": {
                 "path": "current_weather/windspeed",
                 "type": "float"
               },
-              "weather/wind-direction": {
+              "weather/wind_direction": {
                 "path": "current_weather/winddirection",
                 "type": "int"
               },
-              "weather/weather-code": "current_weather/weathercode"
+              "weather/weather_code": "current_weather/weathercode"
             }
           }
         }
@@ -391,21 +391,19 @@ You can (re)start the configured sensiNact instance, using `./start.sh`
 :::{note}
 On Windows, that would be:
 ```powershell
-
 & java -D"sensinact.config.dir=configuration" -jar launch\launcher.jar
 ```
 :::
 
 The current temperature at Grenoble should now be accessible using:
 
 ```bash
-curl http://localhost:8082/sensinact/providers/grenoble-weather/services/weather/resources/temperature/GET
+curl http://localhost:8082/sensinact/providers/grenoble_weather/services/weather/resources/temperature/GET
 ```
 
 And the result will have the following format:
 
 ```json
-
 {
   "response": {
     "name": "temperature",
@@ -415,6 +413,6 @@ And the result will have the following format:
   },
   "statusCode": 200,
   "type": "GET_RESPONSE",
-  "uri": "/grenoble-weather/weather/temperature"
+  "uri": "/grenoble_weather/weather/temperature"
 }
 ```

docs/source/southbound/device-factory/core.md

@@ -79,6 +79,35 @@ The key of the mapping entry can be in the following formats:
 * `$xxx`: definition of a variable that can be reused in or as a mapping value. The variable can be used in other mapping key or in record paths using the `${xxx}` syntax.
 * `"svc/rc"`: the parser value will be stored in a the resource `rc` of the service `svc`
 
+#### Names handling
+
+Eclipse sensiNact is based on EMF to manage its model, which requires all the names it handles to be valid Java identifiers.
+As a result, the Device Factory Core normalizes all model, provider, service and resource names before forwarding data to sensiNact.
+
+By default, the normalization process allows all Java identifier characters, including non-Latin characters and diacritics.
+Invalid characters (space, ...) are replaced with underscores.
+
+The mapping options include a `names.ascii` flag that can be set to true to restrict names to digits, underscores and Latin letters without diacritics.
+The device factory first normalizes the input in NKFD form to remove diacritics, then replaces all rejected characters with underscores.
+If the entire name is rejected, it will be replaced by a string of underscores that has the same length as the original name.
+
+In both cases, if the normalized name doesn't begin with a valid Java identifier start character or is a reserved Java keyword, then it is prefixed with an underscore.
+
+Here are some examples of transformation:
+
+| Input             | Default Normalization | ASCII Normalization |
+|-------------------|-----------------------|---------------------|
+| `état`            | `état`                | `etat`              |
+| `État`            | `État`                | `Etat`              |
+| `int/3.14`        | `_int/_3_14`          | `_int/_3_14`        |
+| `-Greek/Γαλαξιας` | `_Greek/Γαλαξιας`     | `_Greek/________`   |
+
+
+:::{warning}
+Before [pull request #297](https://github.com/eclipse/org.eclipse.sensinact.gateway/pull/297), the normalization was performed in ASCII mode, and the replacement character was the hyphen (`-`).
+As this character is invalid in a Java identifier, it is now replaced with an underscore (`_`).
+:::
+
 ### Mapping value
 
 The value of a mapping can be defined in many different formats.
@@ -294,10 +323,10 @@ For example:
 ```json
 {
     // ...
-    "${svcName}/${rcName}-txt": "${rcValuePath}", // service/rc-txt = "data"
-    "${svcName}/${rcName}-value":{
+    "${svcName}/${rcName}_txt": "${rcValuePath}", // service/rc_txt = "data"
+    "${svcName}/${rcName}_value":{
         "path": "${rcValuePath}"
-        // service/rc-value = <value in the "data" path>
+        // service/rc_value = <value in the "data" path>
     }
 }
 ```
@@ -317,3 +346,7 @@ The following entries are supported:
   Dates without timezone are considered to be in UTC.
 * Number inputs:
   * `numbers.locale`: the name of the locale to use to parse numbers, *e.g.* `fr`, `en_us`, `zh_Hand_TW`.
+* Name handling:
+  * `names.ascii`: if set to true, all model, provider, service and resource names will be normalized to contain only ASCII letters, digits and underscores. If set to false (default), those names will be normalized to be valid Java identifiers.
+* Model name handling:
+  * `model.raw`: if set to true, the explicit model names won't be escaped. This allows to use URLs of existing EMF models.

docs/source/southbound/device-factory/csv.md

@@ -38,8 +38,8 @@ Here are the paths that can be used:
 
 The CSV parser has the ID `csv`.
 It accepts the following options:
-* `encoding`: the payload encoding, as supported by [`java.nio.charset.Charset`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/charset/Charset.html), *e.g.* `"UTF-8"`, `"latin-1"`.
-* `delimiter`: the CSV delimiter character, *e.g.* ",", ";", "\t".
+* `encoding`: the payload encoding, as supported by [`java.nio.charset.Charset`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/charset/Charset.html), *e.g.* `"UTF-8"` (default), `"latin-1"`.
+* `delimiter`: the CSV delimiter character, *e.g.* `,` (default), `;` or `\t`.
 * `header`: a boolean flag to indicate if the CSV payload has a header
 
 ## Example
@@ -55,6 +55,8 @@ Here is an example configuration to parse that payload:
 {
   "parser": "csv",
   "parser.options": {
+    "encoding": "UTF-8",
+    "delimiter": ",",
     "header": true
   },
   "mapping": {

docs/source/southbound/device-factory/tuto-parser.md

@@ -617,7 +617,7 @@ Then we configure the MQTT device factory and tell it to use our parser by a con
     "mapping": {
       "$name": "provider",
       "@provider": {
-          "literal": "sensor-${name}"
+          "literal": "sensor_${name}"
         },
         "@name": {
           "literal": "Sensor ${name}"
@@ -653,18 +653,18 @@ With those configurations, the parser will be called by the device factory core
    **Note:** Each mapping configuration must at least contain the `@Provider` placeholder. Each record must therefore handle a path that returns a provider name.
 
    The result will be the update/creation of the following providers:
-   * `sensor-a`
-   * `admin`
+   * `sensor_a`
+     * `admin`
        * `location`: `null` @ 2023-06-20T13:40:00+0200
        * `friendlyName`: `"Sensor a"` @ 2023-06-20T13:40:00+0200
-   * `sensor`:
+     * `sensor`:
        * `temperature`: 40.0 @ 2023-06-20T13:40:00+0200
        * `humidity`: 0.0 @ 2023-06-20T13:40:00+0200
-   * `sensor-c`
-   * `admin`
+   * `sensor_c`
+     * `admin`
        * `location`: `null` @ 2023-06-20T13:40:00+0200
        * `friendlyName`: `"Sensor c"` @ 2023-06-20T13:40:00+0200
-   * `sensor`:
+     * `sensor`:
        * `temperature`: 38.0 @ 2023-06-20T13:40:00+0200
        * `humidity`: 15.0 @ 2023-06-20T13:40:00+0200
 7. Now consider a second payload:
@@ -677,21 +677,21 @@ With those configurations, the parser will be called by the device factory core
    ```
 
    Following the same steps as before, the providers will be updated as:
-   * `sensor-a`
+   * `sensor_a`
      * `admin`
        * `location`: `null` @ 2023-06-20T13:40:00+0200
        * `friendlyName`: `"Sensor a"` @ 2023-06-20T13:50:00+0200
      * `sensor`:
        * `temperature`: 42.0 @ 2023-06-20T13:50:00+0200
        * `humidity`: 0.0 @ 2023-06-20T13:50:00+0200
-   * `sensor-b`
+   * `sensor_b`
      * `admin`
        * `location`: `null` @ 2023-06-20T13:50:00+0200
        * `friendlyName`: `"Sensor b"` @ 2023-06-20T13:50:00+0200
      * `sensor`:
        * `temperature`: 21.0 @ 2023-06-20T13:50:00+0200
        * `humidity`: 30.0 @ 2023-06-20T13:50:00+0200
-   * `sensor-c`
+   * `sensor_c`
      * `admin`
        * `location`: `null` 2023-06-20T13:40:00+0200
        * `friendlyName`: `"Sensor c"` @ 2023-06-20T13:50:00+0200
@@ -740,7 +740,7 @@ This will ensure it is visible by other Maven projects and ease the creation of
           "mapping": {
             "$name": "provider",
             "@provider": {
-              "literal": "sensor-${name}"
+              "literal": "sensor_${name}"
             },
             "@name": {
               "literal": "Sensor ${name}"
@@ -802,7 +802,7 @@ After all those steps, the configuration file `${SENSINACT_HOME}/configuration/c
       "mapping": {
         "$name": "provider",
         "@provider": {
-          "literal": "sensor-${name}"
+          "literal": "sensor_${name}"
         },
         "@name": {
           "literal": "Sensor ${name}"
@@ -830,10 +830,10 @@ After all those steps, the configuration file `${SENSINACT_HOME}/configuration/c
 * To check the value of a resource, you can use any HTTP client (`curl`, `httPIE`, a browser...).
   With the payload we will send via MQTT, below are the URLs where we will find our resources values.
   Note that accessing those URLs before sending any data will return 404 errors.
-  * <http://localhost:8080/sensinact/providers/sensor-a/services/sensor/resources/temperature/GET>
-  * <http://localhost:8080/sensinact/providers/sensor-a/services/sensor/resources/humidity/GET>
-  * <http://localhost:8080/sensinact/providers/sensor-b/services/sensor/resources/temperature/GET>
-  * <http://localhost:8080/sensinact/providers/sensor-b/services/sensor/resources/humidity/GET>
+  * <http://localhost:8080/sensinact/providers/sensor_a/services/sensor/resources/temperature/GET>
+  * <http://localhost:8080/sensinact/providers/sensor_a/services/sensor/resources/humidity/GET>
+  * <http://localhost:8080/sensinact/providers/sensor_b/services/sensor/resources/temperature/GET>
+  * <http://localhost:8080/sensinact/providers/sensor_b/services/sensor/resources/humidity/GET>
 
 * To send messages, we will use the [MQTT dashboard websocket client](https://www.hivemq.com/demos/websocket-client/):
   1. Connect the broker using the default configuration

docs/source/southbound/http/http-device-factory.md

@@ -58,7 +58,7 @@ Here is an example of configuration for HTTP device factory that will get the de
         "mapping": {
           "$station_id": "station_id",
           "@provider": {
-            "literal": "cycling-${station_id}"
+            "literal": "cycling_${station_id}"
           },
           "@name": "name",
           "@latitude": "lat",
@@ -80,7 +80,7 @@ Here is an example of configuration for HTTP device factory that will get the de
         "mapping": {
           "$station_id": "station_id",
           "@provider": {
-            "literal": "cycling-${station_id}"
+            "literal": "cycling_${station_id}"
           },
           "station/active": "is_installed",
           "station/stationCode": "stationCode",

southbound/device-factory/device-factory-core/src/main/java/org/eclipse/sensinact/gateway/southbound/device/factory/IResourceMapping.java

@@ -64,8 +64,10 @@ IResourceMapping fillInVariables(final Map<String, String> variables)
      * Returns a copy of this mapping with a path that meets the sensiNact naming
      * requirements
      *
+     * @param asciiOnly If true, the path must only contain letters, digits and
+     *                  underscores
      * @return This object if the key was valid or a new one with a new path
      * @throws InvalidResourcePathException Invalid new resource path
      */
-    IResourceMapping ensureValidPath() throws InvalidResourcePathException;
+    IResourceMapping ensureValidPath(final boolean asciiOnly) throws InvalidResourcePathException;
 }

southbound/device-factory/device-factory-core/src/main/java/org/eclipse/sensinact/gateway/southbound/device/factory/dto/DeviceMappingOptionsDTO.java

@@ -38,4 +38,10 @@ public class DeviceMappingOptionsDTO {
 
     @JsonProperty("null.action")
     public NullAction nullAction = NullAction.UPDATE;
+
+    @JsonProperty("names.ascii")
+    public boolean asciiNames = false;
+
+    @JsonProperty("model.raw")
+    public boolean useRawModel = false;
 }

southbound/device-factory/device-factory-core/src/main/java/org/eclipse/sensinact/gateway/southbound/device/factory/impl/AbstractResourceMapping.java

@@ -100,46 +100,56 @@ public AbstractResourceMapping(final String rcPath) throws InvalidResourcePathEx
     /**
      * Returns the name of the service
      */
+    @Override
     public String getService() {
         return service;
     }
 
     /**
      * Returns the name of the resource
      */
+    @Override
     public String getResource() {
         return resource;
     }
 
     /**
      * Returns the name of the metadata, if any
      */
+    @Override
     public String getMetadata() {
         return metadata;
     }
 
     /**
      * Checks if this mapping targets a metadata
      */
+    @Override
     public boolean isMetadata() {
         return metadata != null;
     }
 
     /**
      * Returns the resource path
      */
+    @Override
     public String getResourcePath() {
         return path;
     }
 
     @Override
-    public IResourceMapping ensureValidPath() throws InvalidResourcePathException {
-        if (service == null) {
+    public IResourceMapping ensureValidPath(final boolean asciiOnly) throws InvalidResourcePathException {
+        if (path == null) {
             // Not a path
             return this;
         }
 
-        final String cleaned = NamingUtils.sanitizeName(path, true);
+        final String cleaned;
+        if (asciiOnly) {
+            cleaned = NamingUtils.asciiSanitizeName(path, true);
+        } else {
+            cleaned = NamingUtils.sanitizeName(path, true);
+        }
         if (cleaned.equals(path)) {
             // Nothing to do
             return this;

southbound/device-factory/device-factory-core/src/main/java/org/eclipse/sensinact/gateway/southbound/device/factory/impl/FactoryParserHandler.java

@@ -268,14 +268,23 @@ private List<GenericDto> handleRecord(final DeviceMappingConfigurationDTO config
         if (rawProvider == null || rawProvider.isBlank()) {
             throw new ParserException("Empty provider field");
         }
-        final String provider = NamingUtils.sanitizeName(rawProvider, false);
+        final String provider;
+        if (configuration.mappingOptions.asciiNames) {
+            provider = NamingUtils.asciiSanitizeName(rawProvider, false);
+        } else {
+            provider = NamingUtils.sanitizeName(rawProvider, false);
+        }
 
         // Extract the model
         final String model;
         if (recordState.placeholders.containsKey(KEY_MODEL)) {
             final String rawModel = getFieldString(record, recordState.placeholders.get(KEY_MODEL), options);
             if (rawModel == null || rawModel.isBlank()) {
                 throw new ParserException("Empty model field for " + provider);
+            } else if (configuration.mappingOptions.useRawModel) {
+                model = rawModel;
+            } else if (configuration.mappingOptions.asciiNames) {
+                model = NamingUtils.asciiSanitizeName(rawModel, false);
             } else {
                 model = NamingUtils.sanitizeName(rawModel, false);
             }
@@ -437,6 +446,8 @@ private RecordState computeRecordState(final DeviceMappingConfigurationDTO confi
             final RecordState initialState, final IDeviceMappingRecord record)
             throws InvalidResourcePathException, ParserException, VariableNotFoundException {
 
+        final boolean asciiPaths = configuration.mappingOptions.asciiNames;
+
         final RecordState state = new RecordState();
 
         // Resolve variables
@@ -447,12 +458,14 @@ private RecordState computeRecordState(final DeviceMappingConfigurationDTO confi
 
         state.rcMappings = new ArrayList<>(initialState.rcMappings.size());
         for (final ResourceRecordMapping rcMapping : initialState.rcMappings) {
-            state.rcMappings.add((ResourceRecordMapping) rcMapping.fillInVariables(state.variables).ensureValidPath());
+            state.rcMappings.add(
+                    (ResourceRecordMapping) rcMapping.fillInVariables(state.variables).ensureValidPath(asciiPaths));
         }
 
         state.rcLiterals = new ArrayList<>(initialState.rcLiterals.size());
         for (final ResourceLiteralMapping rcMapping : initialState.rcLiterals) {
-            state.rcLiterals.add((ResourceLiteralMapping) rcMapping.fillInVariables(state.variables).ensureValidPath());
+            state.rcLiterals.add(
+                    (ResourceLiteralMapping) rcMapping.fillInVariables(state.variables).ensureValidPath(asciiPaths));
         }
         return state;
     }