Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data-mappings architecture to be dataset-driven instead of topic-driven #650

Closed
maaikelimper opened this issue Apr 18, 2024 · 3 comments
Closed
Assignees
Milestone

Comments

@maaikelimper
Copy link
Collaborator

Right now the wis2box-architecture processes data based on the directory that is presumed to match the topic-hierarchy.

However users can have different datasets on the same topic-hierarchy (example: Chile with manual and AWS data).

The data-mappings architecture should be dataset-driven instead of topic-driven, which requires a redesign of things currently work.

Related:
#584

@maaikelimper
Copy link
Collaborator Author

proposal:
incoming bucket + dataset_id

@maaikelimper
Copy link
Collaborator Author

I have noted that our auth options are based on locking down topics, I'm updating this to have the command-lines for locking down datasets. This might require changes in wis2box-auth as well. I also note that we are missing automated testing that our command-line options for locking down topics/datasets, I should add this.

@tomkralidis tomkralidis added this to the sprint-015 milestone Apr 23, 2024
@maaikelimper
Copy link
Collaborator Author

This was implemented by #686

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants