Currently SolarIn/MQTT posts data as CBOR "map" objects, which are like JSON objects, that include all datum property names and values. NET-279 proposed a way to use Protobuf to encode the messages, which would save significant bandwidth in part because the datum property names would not be included in the messages. SolarNetwork will already store datum using a "V2 datum format" that omits the property names in the table that stores the datum stream and instead stores just arrays of numbers, relying on a secondary "metadata" table to define what the property names are as an array of strings whose order defines what the property values represent in the datum stream table.

SolarIn/MQTT could post datum in a similar "V2 datum format" if SolarNode had access to the datum metadata that defined what the properties were. If the metadata didn't exist, or SolarNode discovered new properties that are not part of the metadata yet, it could revert to posting the datum using the current format that includes the property names, and when SolarIn stores that datum it will update the metadata for that datum stream. SolarNode could then download the updated metadata and then resume/start posing datum in the V2 format.

The benefits of this approach would be:

1. the network bandwidth required to post datum to SolarIn would be significantly reduced, as the property metadata does not change frequently so most datum messages could use the V2 format
2. users would not have to define a custom datum encoding (like Protobuf) in order to realize the bandwidth savings

V2 datum format

The "V2 datum format" proposed here would still use CBOR to encode each datum message. The contents would change from the current form that looks like this (expressed as JSON):

{
  "created" : 1234567890,
  "sourceId" : "/test/foo/bar",
  "i" : {
    "watts" : 1234,
    "voltage" : 249.0,
    "powerFactor" : 0.999987
  },
  "a" : {
    "wattHours" : 1234512345,
    "wattHoursReverse" : 54321
  },
  "s" : {
    "status" : "OK"
  }
  "t" : [
    "dr-event"
  ]
}

The new form would look more like this (expressed as JSON):

[
  1234567890,
  -39146522915747961,
  -8199130050457739548,
  [
    1234,
    249.0,
    0.999987
  ],
  [
    1234512345,
    54321
  ],
  [
    "OK"
  ],
  [
    "dr-event"
  ]
]

This is essentially an array of elements representing:

1. datum timestamp (millisecond epoch)
2. datum stream UUID (upper 64 bits)
3. datum stream UUID (lower 64 bits)
4. array of instantaneous property values
5. array of accumulating property values
6. array of status property values
7. array of tag values

The stream UUID and order of the various property value arrays would be defined by the stream metadata provided via an API query. Given the metadata changes infrequently, and properties can only be added to the metadata, then it is trivial for SolarNode to know if it has the proper metadata to encode each datum as, as long as every property in the datum is found in the metadata it has downloaded.