Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Processing automation #23

Open
raphidoc opened this issue Jan 18, 2023 · 0 comments
Open

Processing automation #23

raphidoc opened this issue Jan 18, 2023 · 0 comments
Labels
enhancement New feature or request time consuming Time consuming computation

Comments

@raphidoc
Copy link
Owner

raphidoc commented Jan 18, 2023

Problem
The Algae-WISE data consists of 59 266 seconds (~16.48 hours) of acquisition (including transit). Considering the acquisition rate of the instruments, it takes a maximum of five seconds to create a discrete data point. So we have a potential database of 59 266 / 5 = 11 853 observations. It takes a minimum of 5 minutes to process a single discrete observation. So it would take (5x60)x11 853 = 3 555 900 seconds (~ 987 hours) to process it all (overestimate including transit data).

Solution
You can automate the process and leave the qualitative eye of the operator to deal with problematic observations only.

To make efficient automation, we first have to clean the data for transit recording:

  1. Speed_N <= 10
  2. Lu < threshold

Then we need to create the time interval that will be used to discretize the data:

  1. Start from a DateTime (x) take the next DateTime (y) with a minimum of three seconds difference and try to process.
  2. If there is not enough data (HOCR) take the next DateTime (y) (normally one second further) and try to process. Repeat
  3. If the next DateTime (y) has more than 10 seconds difference, start again with the next DateTime (y)

Describe alternatives
No alternative considered as of now

Additional context
Add any other context or screenshots about the feature request here.

@raphidoc raphidoc added enhancement New feature or request time consuming Time consuming computation labels Jan 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request time consuming Time consuming computation
Projects
None yet
Development

No branches or pull requests

1 participant