Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to ingest private metadata #243

Open
drnextgis opened this issue Feb 29, 2024 · 8 comments · May be fixed by #327
Open

Unable to ingest private metadata #243

drnextgis opened this issue Feb 29, 2024 · 8 comments · May be fixed by #327

Comments

@drnextgis
Copy link
Collaborator

Although the items table contains private column, the pypgstac loader is unable to ingest private metadata.

@zacdezgeo
Copy link
Collaborator

@drnextgis, could you provide some information on how to reproduce the issue?

@drnextgis
Copy link
Collaborator Author

drnextgis commented Feb 29, 2024

Just examining the loader code reveals that the private property isn't handled at all. You can replicate the issue easily:

  1. Follow the instructions here to create a collection named noaa-emergency-response.
  2. Create a items.json file and include the private object:
{"type":"Feature","private":{"a":1},"stac_version":"1.0.0","id":"20200307aC0852700w360900","properties":{"event":"Nashville Tornado","datetime":"2020-03-07T00:00:00Z"},"geometry":{"type":"Polygon","coordinates":[[[-85.4501,36.1501],[-85.4501,36.1249],[-85.4249,36.1249],[-85.4249,36.1501],[-85.4501,36.1501]]]},"links":[{"rel":"collection","href":"noaa-emergency-response","type":"application/json"}],"assets":{"cog":{"href":"s3://noaa-eri-pds/2020_Nashville_Tornado/20200307a_RGB/20200307aC0852700w360900n.tif","type":"image/tiff; application=geotiff; profile=cloud-optimized"}},"bbox":[-85.4501,36.1249,-85.4249,36.1501],"stac_extensions":[],"collection":"noaa-emergency-response"}
{"type":"Feature","private":{"b":2},"stac_version":"1.0.0","id":"20200307aC0852700w361030","properties":{"event":"Nashville Tornado","datetime":"2020-03-07T00:00:00Z"},"geometry":{"type":"Polygon","coordinates":[[[-85.4501,36.1751],[-85.4501,36.1499],[-85.4249,36.1499],[-85.4249,36.1751],[-85.4501,36.1751]]]},"links":[{"rel":"collection","href":"noaa-emergency-response","type":"application/json"}],"assets":{"cog":{"href":"s3://noaa-eri-pds/2020_Nashville_Tornado/20200307a_RGB/20200307aC0852700w361030n.tif","type":"image/tiff; application=geotiff; profile=cloud-optimized"}},"bbox":[-85.4501,36.1499,-85.4249,36.1751],"stac_extensions":[],"collection":"noaa-emergency-response"}
  1. Proceed to load it into the database:
$ pypgstac load items items.njson \
    --dsn postgresql://username:password@localhost:5439/postgis \
    --method insert
  1. Connect to the database and ensure the private column is empty:
postgis=# select private from items;
 private 
---------
 
 
(2 rows)

@vincentsarago

This comment was marked as off-topic.

@drnextgis
Copy link
Collaborator Author

But this is the case. Despite the presence of the private column in the items table, there appears to be no way to populate it using pypgstac. The only method I can see is resorting to raw SQL queries. Am I missing something?

@mmcfarland
Copy link
Collaborator

My understanding for the private column in items (and also maybe on collections) was in anticipation of future work, and the column is not currently used by pgstac sql or pypgstac (e.g., I'm not sure you could do anything with it through existing pgstac interfaces even if you were able to load it in). The rationale was there was a breaking version jump (0.7 -> 0.8) occurring and @bitner wanted to get the schema change in for it, with follow up work to be done within the 0.8 series.

@bitner
Copy link
Collaborator

bitner commented Mar 7, 2024

Correct, currently there is no plumbing to do anything with the private column or for the loader to populate it. It is there for future development to allow for ad hoc storage of anything that does not belong in the public stac metadata (ie user name who created the record).

@drnextgis
Copy link
Collaborator Author

Thank you all for the comprehensive response. We currently require a way to store private metadata alongside STAC items. Therefore, I would like to implement the ability to populate this column using the pypgstac loader. What do you think?

@bitner
Copy link
Collaborator

bitner commented Mar 11, 2024

@drnextgis if you'd like to take a stab at it, please do! If you'd like to pair on anything to get going, let me know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants