storing application data

[pljakobs]

so with all this wonderful json database code, I was thinking that I could move a bit of data into the configuration that is not quite as static as the "normal" config but also not totally changing either.
I have had the concept of light presets, groups and group presets for a while. Currently, only light presets are implemented and are stored in files. If I moved them into the schema, I could lose the code that writes them to files, too.

Presets can have three different formats: RAW, RGB or HSV. (the firmware differentiates only between HSV, where it does the full mixing of all three, four or five available channels) and RAW, where every channel is driven separately and no color calucations are made.
An RGB preset would look somewhat like this:

{
    "name":"Red",
    "ts":0,
    "id":0,
    "color":{ "r":100, "g":0, "b":0 } 
}

HSV is

{
    "name":"Red",
    "ts":0,
    "id":0,
    "color":{ "h":0, "s":100, "v":100 }
}

and lastly, RAW is

{
    "name":"Setting 1",
    "ts": 0,
    "id":0,
    "color":{ "r":800, "g":500, "b":200, "ww":1023, "cw":0 }
}

my first thought is: an array can only contain objects that all follow the same schema, right? It would be easier to have name, timestamp and id for all, then a type filed and the color field syntax according to the defined type. From what I understand, I would have to have three arrays, though, or does json / jsonschema allow variants within an array?

{
  "presets":
  {
    "hsv":
    [
      (all hsv presets)
    ].
    "rgb":
    [
      (all rgb presets)
    ]
    ....
  }
}

I would probably also want to store an array of controllers (found by mDNS)
Scenes could then be stored as an array of (controller_id, preset_id) tupels.

"controllers":[
    {
        "name": "controller1",
        "last-seen":0,
        "reachable":true,
        "ip": "0.0.0.0",
        "id":0
    }
],

groups would be groups of controllers that I can switch with a single ui action

"groups":[
    {
        "name": "group1",
        "ts":0,
        "ids":[
            0,
            3,
            8
        ]
    }
],

and lastly, scenes or group presets would combine controllers and presets, I'm not yet 100% sure if I will store the color values with the scene or if I will write a preset and link to it by id.

"scenes":
[
      {
          "name": "scene1",
          "ts":0,
          "id":0,
          "presets":[
              {
                  "controller":0,
                  "preset":0
              }
          ]
      }
  ]

In order to be able to operate on those IDs sensibly, I guess most of those object should be in the same store to prevent excessive read / thrashing.

• The exportToStream works on objects as well, right? so I should be able to stream the presets object when it's requested from the API?
• I should be able to use presets.hsv[id=xyz] to pick the preset with the corresponding id
• I'm including ts (timestamp) as I hope to be able to use it to synchronize part of the database across multiple controllers (using stream export / import edpoints)

The easiest way to do this is all in the front end, there's enough RAM, parsing json in JavaScript is very natural and it does make sense to control it from there. In this case, I might just pull those bits of the configuration into a structure and do all the api calls from there and write back the changes. That would generate the least RAM load on the controller as well as not require it to re-load the store over and over. So I would probably put all of this into stores according to how I structure it in JavaScript (currently Controllers, presets, groups and scenes- so four separate stores) and, in firmware, deal with each of them as a single unit.

(I notice, as usual, my thoughts have become clearer as I write this down :) )

Any thoughts on this?

[mikee47]

so with all this wonderful json database code, I was thinking that I could move a bit of data into the configuration that is not quite as static as the "normal" config but also not totally changing either. I have had the concept of light presets, groups and group presets for a while. Currently, only light presets are implemented and are stored in files. If I moved them into the schema, I could lose the code that writes them to files, too.

You could even use a separate database for this...

Presets can have three different formats: RAW, RGB or HSV. (the firmware differentiates only between HSV, where it does the full mixing of all three, four or five available channels) and RAW, where every channel is driven separately and no color calucations are made. An RGB preset would look somewhat like this:

my first thought is: an array can only contain objects that all follow the same schema, right? It would be easier to have name, timestamp and id for all, then a type filed and the color field syntax according to the defined type. From what I understand, I would have to have three arrays, though, or does json / jsonschema allow variants within an array?

JSON and JSONSchema do not restrict the content of arrays, so you can mix anything at all (integers, strings, objects, etc.). The file schema.json in the USB library is probably a useful example to refer to. There's a definition for Interfaces using the oneOf keyword, which is how we might describe the ItemType for such a variant array.

At present, there is a single ObjectInfo definition for an ObjectArray since all items are of the same time. There's no reason we cannot have multiple definitions, but we'd need some way to associate each array item with its object type information. That's not too hard: we'd just include an identifier (enumerated uint8_t would suffice) as the first field of an ObjectArray Item. That tag then tells everyone which type is actually used.

This same information can tell the user which object type an array entry is. Handily, as we're not using virtual classes we can define a union for these variant types so no casting is required.

Note that we also need to include support in ConfigDB for re-useable type definitions, via the $ref keyword. This would avoid duplication in the schema.

* The exportToStream works on objects as well, right? so I should be able to stream the presets object when it's requested from the API?

Yep.

[mikee47]

Just re-reading your initial question and noticed I haven't addressed the issue of variant parts of an Object.

Might be easier thinking about the C++ code we'd like to generate and work back.
If the options can be expressed as a union then that is do-able.
As with variant arrays we'd need a tag to resolve it at runtime:

class Color: public ConfigDB::UnionTemplate<...> {
public:
    struct Struct {
        enum class Tag: uint8_t {
            RGB,
            HSV,
            RAW,
        };
        Tag tag;
        union Content {
            struct RGB {
                uint8_t r;
                uint8_t g;
                uint8_t b;
            };
            struct HSV {
                uint8_t h;
                uint8_t s;
                uint8_t v;
            };
            struct RAW {
                uint8_t r;
                uint8_t g;
                uint8_t b;
                uint16_t ww;
                uint16_t cw;
            };
        };
    };

    const RGB& getRGB() const {
        assert(tag == Tag::RGB);
        return ...
    }

    const RGB& getHSV() const {
        assert(tag == Tag::HSV);
        return ...
    }

    const RGB& getRAW() const {
        assert(tag == Tag::RAW);
        return ...
    }
};

This would be a member of a regular Preset definition accessed via getColor().

In the schema the Variant field would be defined using:

"color": {
    "oneOf": [
        "RGB": {
            ...
        },
        "HSV": {
            ...
        },
        "RAW": {
            ...
        }
    ]
}

Once we have $ref support it could be restructured differently, for example:

"color": {
    "$ref": "#/$defs/Color"
},
"$defs": {
    "Color": {
        "oneOf": [
            "RGB": {
                "$ref": "#/$defs/RGB"
            },
            "HSV": {
                "$ref": "#/$defs/HSV"
            },
            "RAW": {
                "$ref": "#/$defs/RAW"
            }
        ]
    },
    "RGB": {
        ...
    },
    "HSV": {
        ...
    },
    "RAW": {
        ...
    }
}

[pljakobs]

You could even use a separate database for this...

I thought about this, but the result would be more RAM usage as basically there would be two cached stores. That may or may not be desireable.

btw, what happens if I open the same database twice? should we guard against that?

[mikee47]

btw, what happens if I open the same database twice? should we guard against that?

We may legitmately want two copies of the same database, albeit point to different paths, for backups, etc.

The Database instance should be an application global, or constrained to a single unit which manages access to it. It's not something to create and destroy dynamically, IMHO.

We could guard using a static to track all open instances but I'm not sure it's worth the overhead.