Whale sightings data in Signal K

[puzzledcherry]

I'm running into errors with having the signalK server parse my JSON file. I've attached a screenshot of the server log and the JSON file. I can't seem to find the issue as to why it's not parsing.

signalkSightings.json

[sbender9]

How are you sending this?

[puzzledcherry]

Through a file stream with the data type as SignalK

[sbender9]

It should be one delta per line (not an array), no new lines in the deltas

Also, your "context" is not right, you probably want it to be "vessels.self" ?

{     "context": "vessels.self",        "updates": [          {                "timestamp": "2024-08-28T09:15:00-07:00",                "values": [                    {                        "path": "environment.sightings.whales",                        "value": {                            "id": 0,                            "type": "Orca",                            "latitude": 47.6205,                            "longitude": -122.3493,                            "no_sighted": 2,                            "comment": "Whale breaching near ferry",                            "recent": 1                        }                    }                ]            }        ]    }
{        "context": "vessels.self",        "updates": [            {                "timestamp": "2024-08-28T09:45:00-07:00",                "values": [                    {                        "path": "environment.sightings.whales",                        "value": {                            "id": 0,                            "type": "Orca",                            "latitude": 47.6694,                            "longitude": -122.3866,                            "no_sighted": 2,                            "comment": "Spotted heading north",                            "recent": 1                        }                    }                ]            }        ]    }

[puzzledcherry]

NDJSON format right? When i try to save that to a JSON file it gives me errors.

[sbender9]

the whole file is not really valid json since it's one json per line

[puzzledcherry]

what should the file type be?

[sbender9]

It's just text. Extension does not matter.

[puzzledcherry]

Thank you for the info, the inbound datastream icon is now flashing and there's a yellow bar- is this supposed to happen? How can I see my data on a map now? I've attached an image below:

[tkurki]

I think you are now playing the same datafile over and over again, as fast as the server can process (6000+ messages per second).

What are you actually trying to achieve? Where is your data coming from?

[puzzledcherry]

I am working on a project to visualize whale sightings from a JSON data source on a map. Currently, I have a Python script that generates a static JSON file with whale sighting data. My immediate goal is to display this data on a map and ensure the visualization correctly represents the sightings.

As a next step, I plan to periodically update the JSON file at regular intervals and reflect these updates on the map automatically. Eventually, I aim to overlay this whale sighting data with an AIS viewer to analyze vessel interactions with the whales, but this is a future development phase.

At the moment, I am seeking guidance on how to get started with displaying the static JSON data on a map and understanding how I can have the map stay up-to-date when I run my python script to update the JSON file.

[panaaj]

Given your description of what you are trying to achieve it aligns with the concept of Resources.
11.3 Resource Provider Plugins in the Server Documentation will provide some insight on how this works.

You can use the built-in Resources provider plugin configuration to define a path for your resource files (in your case, Whale Sighting entries) which will then be available to client apps such as Freeboard-SK for display.

[tkurki]

Adrian has a point. Is this more static in nature, like POIs (points of interest with location) or dynamic (like AIS data: "here's a vessel at this point")?

[puzzledcherry]

The map is dynamic in that it updates to display new sightings as they are submitted by the community. However, the sightings themselves are more static compared to something like AIS data, since vessel positions are constantly updating but the sightings aren't.

In other words, the sightings are static and don't move on the map but newly submitted sightings should be displayed on the map as they roll in. So the map is dynamic but the sightings are static? If that makes sense.

[puzzledcherry]

@panaaj following your instructions, would I then save the data file into signalk/v2/api/resources/whales? Will it then automatically display the data to the freeboard-SK?

I appreciate everyone's help, I am new to all of this

[puzzledcherry]

is there a sample JSON i could use when testing the resource plug-in? im messing with it and im wondering if my file is the issue

[panaaj]

For testing the resource plugin, you can keep it simple and start with the Waypoint resource schema.

You can post this schema to the /waypoints path or any custom path you create.
e.g. /signalk/v2/api/resources/whales.

Note: If you use the /waypoints path you will see them displayed on the map in Freeboard-SK.
If you use a custom path you will need to use the resource set schema detailed here.

Example: Create a new sighting location by sending a POST request with the following payload:

{
    "name": "Sighting #1",
    "description": "Whales sighted here!",
    "type": "Whale Sighting",
    "feature": {
        "type": "Feature",
        "geometry": {
            "type": "Point",
            "coordinates": [
                5.47728,
                53.24771
            ]
        },
        "properties": {}
    }
}

Deltas are sent when resources are created, updated or deleted using the API.

[puzzledcherry]

Ok, i got that to display!
How would I pass in multiple sightings though? I tried using the signalk resource set structure which was mentioned in the documentation you linked me to, and also a standard JSON list but neither worked. I've attached my resource/json data files with mock data- if someone could tell me how to display multiple sightings that would be great.

I feel close to solving this issue.

signalkSightings_JSON.json
signalkSightings_SignalKResourceSet.json

[panaaj]

How do you want to show multiple sightings?

I will assume one location and some kind of label indicating the number of sightings.

The easiest way to do that without any code updates is to use a waypoint and set the name to be the number of sightings in that location.
The waypoint label is displayed when zoomed in which means no extra clicks.

Currently the waypoint icon does not change based on the type but that is in the pipeline.
"Whale" could be one of the defined icon types.

[panaaj]

I feel close to solving this issue.

signalkSightings_JSON.json
signalkSightings_SignalKResourceSet.json

I will have a look at these when I get back to land... Won't be for a few days though.

[puzzledcherry]

I want to show multiple sightings as multiple points on the map, such as one near Seattle, Bainbridge Islands, and one near Edmonds in Washington state. Not necessarily how many whales were spotted in a pod- in that case, I would display it in the description or title as you said.

My data pull retrieves whale sightings throughout the west coast of Washington state, and I want to display all those sightings along the coast. I'll attach an example of my previous project that did this.

Eventually, I'd like to implement transparency to show time passed and also the connecting lines between sightings, but I feel as though that's a project development further into the future so no need to worry about that right now.
Defining some icon types once we get these whale sightings on the map sounds awesome.

No rush responding! I'll be out of the office all of next week, so I won't be active again until 09/23.

[panaaj]

I feel close to solving this issue.
signalkSightings_JSON.json
signalkSightings_SignalKResourceSet.json

@puzzledcherry So close.....

• With regards the file signalkSightings_JSON.json, if you want to use waypoints to display sightings, then you will need to place each waypoint in a separate file with a filename that is a v4 UUID (e.g. d16ebf97-f860-44c8-bc9c-b2d4553af61d). These will then appear in the waypoint list with all the other waypoints and can be selected for display in the usual way.
  The easiest way to do this is via the Signal K Resources API as it will both create the waypoint resource and emit a delta to notify clients.

Example:
curl -d "@data.json" -X POST http://localhost:3000/signalk/v2/api/resources/waypoints

data.json

{
    "name": "Whale Sighting #1",
    "description": "Whale sighting from shore.",
    "type": "whale",
    "feature": {
        "type": "Feature",
        "geometry": {
            "type": "Point",
            "coordinates": [
                4.964859905436475,
                53.38410865639514
            ]
        },
        "properties": {}
    }
}

• The signalkSightings_SignalKResourceSet.json file is formatted correctly and works just fine.
  To display them in Freeboard-SK on the map you will need to select them from the resource layers menu in the following way:

1. Place your file in the correct location, this is the folder (with the same name you set up in the plugin configuration) located in the resource provider plugin's configuration i.e. ~/.signalk/plugin-config-data/resources-provider/resources/<my_res_set>
   e.g. I created a resource type called islands in the plugin config and then placed the file you provided in the folder ~/.signalk/plugin-config-data/resources-provider/resources/islands

2. In Freeboard-SK, select your resource set by:

• Clicking the resource layers button to display the menu and then select the resource type you configured (In my case..islands)
• In the displayed list of resource sets, select the desired one.

• Once selected, the features will be displayed on the map.
  

[panaaj]

FYI.... the next release of Freeboard-SK will support the "whale" waypoint type.

[puzzledcherry]

@panaaj Thank you so much! Your instructions were clear, and I was able to successfully replicate the results. I also love the addition of the "whale" waypoint type – it's exactly what I need!

Could you please guide me/point me in the direction of some resources on how to display hover information for these points? Each sighting includes some details I'd like to show when users hover over them.

[panaaj]

So "hover information" I am going to assume refers to information displayed by the client.

The waypoint resource schema contains a feature.properties attribute which could be populated with relevant data and displayed by the client.

Ultimately the behaviour will be determined by the client, so for example, currently Freeboard-SK will:

• display the resource name at zoom levels above a configurable minimum
• show a popup with summary details including name and description.

For an example of how Freeboard-SK could display feature properties for waypoints, click on an AIS target and then click Info