define a common dataset for a node

[SvenRoederer]

With the reborn OWM there started some work on the clients again, e.g.

• documentation and all_docs endpoint #12
• owm: add "hardwareinfo" node for modelname and system-platform openwrt-packages#48
• RFC: Replacement of owm.lua freifunk-berlin/falter-packages#159

This raises the issue of using different models of a node-dataset. technically this is no problem on the OWM-API / OWM-database, but might become a mightmare to the frontends using the nodes dataset. Having a web-statistic which will only work for data uploaded with certain client, is very likely nothing we want to have.
Example:

{"node": "some node", "latitude":52.436757800214,"longitude":13.546518087387, "height": 20, "script":"luci-app-owm"}

vs.

{"node": "some node", position: {"latitude":52.436757800214,"longitude":13.546518087387, "height": 20}, "script":"fancy-owm"}

I suggest we make a small definition of what data and a suggested format

• minimal data required
• common data that should be supplied
• optional data that can be provided

[Akira25]

I've crated a draft for a api-revision owm-api v2 in json-schema. A minimal subset for api revision v1 could be derived from this fairly easy. @sarumpaet could you please have a look on this?

One can pretty easy validate a specific json-file with the python-programm jsonschema (available via pip):

jsonschema -i anfrage_lua.json owm-schema.json

owmapi_v2_spec.zip

[sarumpaet]

For the general OWM use case ("a map of APs that are freely accessable"), only longitude, latitude, and hostname are required. I'd also make id a requirement as otherwise the document isn't self contained (owm-api adds that field based on the POST request path).

All other fields are nice but not really needed, and we shouldn't make them mandatory.

The freifunk/contact thing I'd rather name "contact" directly (there's not only Freifunk).

It might also be a good time to clean up freifunk/community which is full of cruft and/or stuff that probably never really worked.

I also feel that the whole current approach mixes static and dynamic data (i.e., configuration, and things like uptime+current links), and that shouldn't happen, but not sure about this.

But anyways I'd rather use another approach for this:

1. Collect current examples of data (i.e., select a number of nodes in the existing network that have somewhat complex setup that we might want to make more transparent using OWM data)
2. Collect requirements in text form
3. Work on a JSON schema based on 1 and 2

[Akira25]

For the general OWM use case ("a map of APs that are freely accessable"), only longitude, latitude, and hostname are required. I'd also make id a requirement as otherwise the document isn't self contained (owm-api adds that field based on the POST request path).

That sounds good. I tend to make the array links mandatory, but it could be empty too. But if owm just wants to show some hotspots that's right.

All other fields are nice but not really needed, and we shouldn't make them mandatory.

Thats right. With the specification I tried to get a minimum superset of the basic owm-data, that would support the statistics section in hopglass. Maybe I will add a filed on wifichannels.

My goal was to have mandatory data for hopglass, that every berlin node should at least send to owm. Maybe we should have a little minimum api for owm/world and a more specific, compatible api for berlin nodes?

[SvenRoederer]

I agree with sarumpaet that only the really required fields should be mandatory, to allow any kind of node to appear on the map (the unencrypted guest network of a FritzBox or similar). In addition the SSID and the Accesstype might be mandatory candidates.

As suggested this issue should result in a list of objects and where to place them (JSON scheme) to refer to. Collecting the list of objects and then structure them might work best in e.g. an etherpad (https://pad.freifunk.net/p/owm-datastructur)?

Btw. I think the API is still V1, as the database-interface is the same. It's "only" the dataset that's changed which uploads via the same API

[Akira25]

Okay, then lets do this on a pad. @sarumpaet would it be okay to have a superset for use with berlin firmware? I'd really like to use as much hopglass-features as we could, what requires those data for berlin-firmware.

I strongly disagree on the api version. It doesn't describe the (in this case trivial) db-interface, but the format of the data. If we decide to restructure the data we should also bump the api-revision. Otherwise it will get really complicate to parse different versions of owm-data. Have a look at owm2ffmap. It's huge and its really complicated!

Having a api-revision as distinct identifier is absolutely necessary here.

[SvenRoederer]

We should distinguish between API of the database and Revision of the data transferred by this API. The database is independent of the data, so we should non mix it up and keep the "data-scheme-version" separate in the data itself.

[SvenRoederer]

I've posted this for comments also on the WLAN-Talk List (https://lists.freifunk.net/pipermail/wlantalk-freifunk.net/2021-July/001352.html)

[andibraeu]

It would be great to have an optional object, maybe communityData within the schema that allows communities to add community specific data without being defined here

[SvenRoederer]

@andibraeu this is the idea of the custom-data node. "communities" assign an identifier, e.g. their name, and append any data there. So any community can add, or even a member of a community can add it's own data on top. As long as there is no clash in the "id" it should work fine.

{'custom-data': 
 {'community-a': 'community-greeting-text'}, 
 {'community-b': 'nodes last log-messages as JSON'},
 {'community-a_user_x': '<encrypted message of users nodes that only he decrypts to check something>'}
}

[Akira25]

Did we get any progress in this? Does anyone feel called to do the final work on standardizing this?

[SvenRoederer]

Just pushed #14 as proposal. it also incorporates usage of NETJson as suggested by feedback from the mailinglist.

[andibraeu]

looks good to me!

[Akira25]

sorry, I had still no time to have a detailed look on that. But on my first impression it looks very neat.