Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sitemap check before harvest #132

Closed
valentinedwv opened this issue Aug 8, 2024 · 0 comments · Fixed by #134
Closed

sitemap check before harvest #132

valentinedwv opened this issue Aug 8, 2024 · 0 comments · Fixed by #134
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@valentinedwv
Copy link

Run a sitemap check before the run.

Gleaner returning ok, if sitemap does not exist
Dagster keeps going, because step completed fine.
So, we should also do a quick sitemap check in scheduler.

code exists in ec utilities, so let's use it.

could even do stat's, aka return some dagster operations metadata.
@valentinedwv valentinedwv added bug Something isn't working good first issue Good for newcomers labels Aug 8, 2024
@valentinedwv valentinedwv moved this to Todo in Decoder Aug 8, 2024
valentinedwv added a commit that referenced this issue Aug 8, 2024
@valentinedwv
check sitemaps #132, #133
8fde909
@valentinedwv valentinedwv linked a pull request Aug 9, 2024 that will close this issue
check sitemaps #132, #133 #134
Merged
valentinedwv added a commit that referenced this issue Aug 9, 2024
@valentinedwv
check sitemaps #132, #133 (#134)
54d4042 
* check sitemaps #132, #133

* works better if you return an error

* concurrency by tags
@github-project-automation github-project-automation bot moved this from Todo to Done in Decoder Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working good first issue Good for newcomers
Projects
Decoder
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

check sitemaps #132, #133 earthcube/scheduler
1 participant
@valentinedwv