RFC: Replacement of owm.lua #159

Akira25 · 2021-03-10T21:38:51Z

Current Situtation

Since the deprecation of OpenWifiMap, there is an replacement called hopglass. It needs the data in ffmap-format, which is really different from owm-format. But it holds much less information also. By (old) standard owm contained almost every bit of information on the router including clients mac-addresses, interfaces and so on.

The current solution for showing the owm-data of (sometimes really old) nodes in hopglass is a converter script in python, which picks the data from the owm-files. Some information gets derived from the node-data. (i.e. a router acting as hotspot would have at least one unencrypted AP-Wifi, etc).

Proposal

Instead of pushing the node data in deprecated owm-format and converting them to ffmap, we should generate the ffmap-json-string directly on the router itself. Thus we don't need to expose any data on interfaces etc, to determine, if a router has AP-Wifi or not. The router itself checks that and just sends the answer.

On server side it will be really easy, to process this data: Data of the new format only needs to be concatenated, to form a valid input for hopglass. Some values needs to be determined by the hopglass-server though, like the first-seen-date.

In regard of OpenWrt's aim to drop lua-interpreter, we should consider implementing that new software in sh, C or anything else of that stuff. (My personal preference is sh).

For sending the data to the server, it might be the best, to abide with the current POST-method over HTTP.

Format

Example of a converted node in ffmap-format

{
      "firstseen": "2018-12-27T18:57:01",
      "flags": {
        "online": true,
        "uplink": true
      },
      "lastseen": "2021-03-09T16:06:00.877562",
      "nodeinfo": {
        "hardware": {
          "model": "TP-Link TL-WR1043ND v3",
          "nproc": 1
        },
        "hostname": "fffw-parkclub",
        "location": {
          "latitude": 52.367222818795,
          "longitude": 14.048248076896
        },
        "node_id": "fffw-parkclub.olsr",
        "owner": {
          "contact": "obsfucated-Email-or-plain-text,if-no-email-address"
        },
        "software": {
          "firmware": {
            "base": "0968abf",
            "release": "Freifunk Falter 1.1.0"
          }
        },
        "system": {
          "role": "node",
          "site_code": "hotspot"
        }
      }

format-Definition from https://github.com/ffnord/ffmap-backend

{ 'nodes': {
    node_id: { 'flags': { flags },
               'firstseen': isoformat,
               'lastseen': isoformat,
               'nodeinfo': {...},         # copied from alfred type 158
               'statistics': {
                  'uptime': double,       # seconds
                  'memory_usage': double, # 0..1
                  'clients': double,
                  'rootfs_usage': double, # 0..1
                  'loadavg': double,
                  'gateway': mac
                }
             },
    ...
  }
  'timestamp': isoformat
}

Notes

There must be a way for owm2ffmap.py to distinguish between owm-format and ffmap-format. I'm not sure, how to indicate that.

The text was updated successfully, but these errors were encountered:

pmelange · 2021-03-11T10:44:11Z

I think it would be better to write it in a script instead of compiled code. With a script it's a lot easier to test changes (no compilation necessary). Also, performance is not a huge issue as this command is only run once per hour.

There must be a way for owm2ffmap.py to distinguish between owm-format and ffmap-format. I'm not sure, how to indicate that.

Maybe it makes sense to have a seperate listener for the new ffmap-format script (a different URL or port or whatever makes sense).

Noki · 2021-03-12T10:33:39Z

Maybe it makes sense to have a seperate listener for the new ffmap-format script (a different URL or port or whatever makes sense).

Yes, please move it to a new URL/endpoint so nobody has to write code for format detection to deal with nodes still using the old format. Maybe also give that URL a version string/parameter. In addition to pushing the data please also keep it accessible under a new URL on the router itself so if someone wants to build something based on that data it can be collected from the routers.

btw.: The hopglass map updates every 30 minutes so sending the data every 30 minutes would also be nice. To avoid request spikes on the endpoint by crons run at the same second you can also add a random number of delay seconds when the script is started if this is an issue.

stargieg · 2021-03-12T14:53:30Z

Hi
Is there a testing env. for the https://github.com/ffnord/ffmap-backend format?
I think i can rewrite this in sh with libubox/jshn.sh like this https://github.com/stargieg/luci2-ffwizard/blob/master/luci-app-olsr2/root/usr/libexec/rpcd/status.olsrd2

Akira25 · 2021-03-12T17:34:59Z

@stargieg
That looks really interesting. As far I can tell there is no testing env. But maybe we can set up one. Actually Berlin Map doesn't use the ffmap-backend, but only the hopglass frontend, which uses the ffmap-format.

The current setup of our hopglass is in that repository.

sarumpaet · 2021-03-15T18:15:11Z

I'm not 100% sure that OWM is gone for good. Reimplementing its backend is only one or two days of work. Switching to ffmap style data formats will kill this option. However, if you want to reimplement the functionality of owm.lua anyways... 🤷 otherwise I'd say "it's not broken so it doesn't need fixing".

Akira25 · 2021-03-20T23:39:12Z

Its right that owm isn't broken too much. I also liked the idea of owm to provide an extensive api for mesh-communities worldwide. On the other hand, sending nearly every information on the router and its clients, does not match with current privacy and security expectations. Additionally, in the last 4 to 5 years, only Freifunk-Berlin firmwares did use that format. Most other communities use ffmap.
Switching to that "standard" could help us to save work: There will be more likely readily available software for that format than for owm.
Maybe reimplementing owm-backend is more work, than pushing ffmap-data directly.

Akira25 · 2021-03-20T23:41:58Z

@stargieg There is a test-serve on http://hopglass-testenv.olsr

Could you please also consider to integrate wifi-channel and meshmode into the ffmap-string?
freifunk-berlin/hopglass.berlin.freifunk.net#4

sarumpaet · 2021-04-01T13:28:23Z

On the other hand, sending nearly every information on the router and its clients, does not match with current privacy and security expectations.

Only a few data points are needed, and there's no need to push more information than needed. In fact, we have removed some data points in the past (e.g., MAC IDs).

The data points actually used are...

ctime, mtime
id/hostname
2.4GHz/5GHz flag
hardware/system
lat/lon
email
firmware name/revision
links: to id/hostname, quality

Writing a shell script that retrieves that data on the client is quite simple. Formatting and sending it in ffmap format is just as simple as formatting and sending it in OWM format then.

Akira25 · 2021-04-02T20:43:44Z

Writing a shell script that retrieves that data on the client is quite simple. Formatting and sending it in ffmap format is just as simple as formatting and sending it in OWM format then.

I strongly agree on your points.

Though I tend to prefer ffmapdata a bit. My assumption is, that hopglass will be supported by a wide community in the future, as many communities use it. ATM it looks like that's consensus.

If we decide to preserve owm-backend we can easier support old nodes, which push data in that format. On the other hand, it feels to me unnecessarily complex as we'd need to transform between separate data representations.

Sending ffmap directly, would indeed raise complexity on the server side for a transition period as we must handle both formats at the same time. But I thinks its worth it.

With ffmap we can profit from a standardized and widely used format.

Akira25 · 2021-04-04T11:10:30Z

@sarumpaet I just found this issue in the hopglass-repo
freifunk-berlin/hopglass.berlin.freifunk.net#11

There you talk on implementing a new backend. If we gonna implement a new script anyway, we could think on adopting the hopglass backend too. That way, we could minimize our workload further.
https://github.com/hopglass/hopglass-server

Do you have feelings on that? May aim is to get the map-system into an environment, that will receive updates and features in the future.

pmelange · 2021-04-05T08:27:50Z

+1 for ffmap format

Akira25 · 2021-04-06T11:40:40Z

I just had a phone call with @sarumpaet. I think, maybe it's not the optimal solution to switch to ffmap-format and server-backend for some reasons:

Openwifimap supported geo-queries. Several services depend on that, for example the wiki: The wikipage for a site looked via geo-query for the position of the site. AFAICT ffmap-backend doesn't support this feature.
In addition I found that the hopglass-people currently invent a new format which would support live streaming of the data. sarumpaet and I agree on that this would be a nice technical challenge, but complicates things much more. Our goal is to keep it simple.

My first assumption, that owm-format is a fixed defined format is essentially not the whole truth. There are several dialects depending on the firmware version used. It's no problem to add new fields to the json-string for our needs with hopglass.

According to sarumpaet, the owm.lua script is fairly old (around 2005/2006) and was meant first for debugging. Thus there were such many privacy-releated data in there. The map came afterwards (around 2013) and started to interpret the data. For example, to figure out if a node has uplink or not. Some of that features broke, when we removed privacy-related data in 57374b5.

My main argument, that owm2ffmap.py grows wild with new versions can be solved, by computing the fields for i.e. 'routeronly' and so on on the routers itself, like we already discussed above. Then we do not need to adjust that on the serverside, but in the firmwares itself, which prevents increasing complexity in the server-software.

This is especially useful, as we probably will never get all people to use recent firmware. There will be deprecated nodes around anytime.

Essentially there are the ideas we agreed on in the call:

abiding with modified owm-format is easier for our server-software
sarumpaet will implement a new backend in python. Fairly simple with around 100 to 150 LOC
- further idea: writing unit-tests for each dialect. They serve as examples for software too.
write a proper specification/documentation for (new/modified) owm-format and stuff to clarify impacts on other systems of Freifunk Berlin
reimplement omw.lua in sh (@stargieg )

I'd love if we can agree on this solution, as sarumpaet signalised to do the implementation work and connect the new system with the wiki, util.berlin.freifunk.net and so on again.

Akira25 · 2021-04-07T19:42:55Z

On the Wednesday meeting at 07th march we agreed on implementing the variant of the post above.

stargieg · 2021-04-11T10:06:09Z

hi
Release early, release often
Hier ist eine erste version
https://github.com/stargieg/luci2-ffwizard/blob/master/luci-app-falter-owm/files/owm.sh

Was soll dieses "sysinfo": ["system is deprecated" ... ?

Akira25 · 2021-04-11T10:14:07Z

Good question. Probably that's deprecated stuff. Not sure on this. @sarumpaet do you know the purpose of that field?

PolynomialDivision · 2021-04-11T15:43:01Z

hi
Release early, release often
Hier ist eine erste version
https://github.com/stargieg/luci2-ffwizard/blob/master/luci-app-falter-owm/files/owm.sh

Was soll dieses "sysinfo": ["system is deprecated" ... ?

NICE! :)

sarumpaet · 2021-04-11T20:14:14Z

The sysinfo thing seems to be a note due to previous OpenWrt updates: freifunk-berlin/firmware-packages@222efe8

https://openwifimap.net is back, by the way, using the new Python backend. Wiki links work again, too.
Currently finishing things up.

pmelange · 2021-04-14T13:53:37Z

https://openwifimap.net is back

Is there any chance to get the green lines back too? Thanks for all the great work!

Akira25 · 2021-04-21T08:01:46Z

@stargieg @sarumpaet It would be really great if we somehow could solve that issue also freifunk-berlin/hopglass.berlin.freifunk.net#4 , which is related to this one.

SvenRoederer · 2021-05-30T15:51:31Z

Was soll dieses "sysinfo": ["system is deprecated" ... ?

As freifunk-berlin/firmware-packages@222efe8 tells - compatibility with legacy tools. In this case it was for at least the old OWM-based router-statistics. But as they are dead, this special code can be dropped.

With the regained interest in the OWM I see again some relation the issue freifunk-berlin/firmware#382 related to the undefined / open structure of the JSON-dataset. I made Issue freifunk/openwifimap-api#13 to find a common structure that frontend-clients can rely on, without having to interpret data in an uploading-script specific way. This will keep the technical implementation of collecting the nodes data separate from storing and using them.

Akira25 · 2021-06-09T22:59:41Z

@stargieg I've pulled the script from your repo into this repo by preserving the commit history: I've refactored the whole stuff and added the feature to send the json to the server. The recent version of this is here:
https://github.com/Freifunk-Spalter/packages/blob/feature/owm_sh_stargieg/luci/luci-app-falter-owm/files/owm.sh

AFAICT, only the olsr-section is missing to get the full data like in the old script.

Akira25 mentioned this issue Mar 21, 2021

hotspot as "routeronly" in map #41

Closed

Akira25 added this to the falter-1.2.0 milestone Mar 25, 2021

sarumpaet mentioned this issue Apr 12, 2021

OWM stats: not well defined JSON structure freifunk-berlin/firmware#382

Closed

Akira25 mentioned this issue Apr 21, 2021

Show Node's WiFi-Channel and Mesh-Mode freifunk-berlin/hopglass.berlin.freifunk.net#4

Open

Akira25 mentioned this issue May 16, 2021

OWM-App: verschieben ins globale Freifunk-repo freifunk-berlin/firmware#855

Closed

SvenRoederer mentioned this issue May 16, 2021

owm: replacing owm freifunk-berlin/firmware#819

Closed

SvenRoederer mentioned this issue May 30, 2021

define a common dataset for a node freifunk/openwifimap-api#13

Open

Akira25 mentioned this issue Jun 17, 2021

Feature/owm sh stargieg old api #185

Merged

pmelange closed this as completed in #185 Jul 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Replacement of owm.lua #159

RFC: Replacement of owm.lua #159

Akira25 commented Mar 10, 2021

pmelange commented Mar 11, 2021

Noki commented Mar 12, 2021 •

edited

Loading

stargieg commented Mar 12, 2021

Akira25 commented Mar 12, 2021 •

edited

Loading

sarumpaet commented Mar 15, 2021 •

edited

Loading

Akira25 commented Mar 20, 2021

Akira25 commented Mar 20, 2021

sarumpaet commented Apr 1, 2021

Akira25 commented Apr 2, 2021

Akira25 commented Apr 4, 2021

pmelange commented Apr 5, 2021

Akira25 commented Apr 6, 2021

Akira25 commented Apr 7, 2021

stargieg commented Apr 11, 2021

Akira25 commented Apr 11, 2021

PolynomialDivision commented Apr 11, 2021

sarumpaet commented Apr 11, 2021

pmelange commented Apr 14, 2021

Akira25 commented Apr 21, 2021

SvenRoederer commented May 30, 2021

Akira25 commented Jun 9, 2021

RFC: Replacement of owm.lua #159

RFC: Replacement of owm.lua #159

Comments

Akira25 commented Mar 10, 2021

Current Situtation

Proposal

Format

Notes

pmelange commented Mar 11, 2021

Noki commented Mar 12, 2021 • edited Loading

stargieg commented Mar 12, 2021

Akira25 commented Mar 12, 2021 • edited Loading

sarumpaet commented Mar 15, 2021 • edited Loading

Akira25 commented Mar 20, 2021

Akira25 commented Mar 20, 2021

sarumpaet commented Apr 1, 2021

Akira25 commented Apr 2, 2021

Akira25 commented Apr 4, 2021

pmelange commented Apr 5, 2021

Akira25 commented Apr 6, 2021

Akira25 commented Apr 7, 2021

stargieg commented Apr 11, 2021

Akira25 commented Apr 11, 2021

PolynomialDivision commented Apr 11, 2021

sarumpaet commented Apr 11, 2021

pmelange commented Apr 14, 2021

Akira25 commented Apr 21, 2021

SvenRoederer commented May 30, 2021

Akira25 commented Jun 9, 2021

Noki commented Mar 12, 2021 •

edited

Loading

Akira25 commented Mar 12, 2021 •

edited

Loading

sarumpaet commented Mar 15, 2021 •

edited

Loading