Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update validation.js (and comments) #726

Merged
merged 8 commits into from
Jan 9, 2024

Conversation

LDannijs
Copy link
Contributor

Summary

Picking up #633 to fix the comments in response to @KrishnaIyer

@johanstokking
Copy link
Member

What is the problem that we're trying to solve here?

@LDannijs
Copy link
Contributor Author

@KrishnaIyer asked me to take over that PR and to update the comments so.
Or did you mean the validate itself?

@LDannijs
Copy link
Contributor Author

Whoops that mention was an accident

@johanstokking
Copy link
Member

What is the problem this pull request is trying to solve?

@LDannijs
Copy link
Contributor Author

Running the validation very often results in an EAGAIN error. I do not know the details of it, only that it is a memory issue, but this fixes it by slowing the process down and other fixes (read Jaime's original PR for full details).

@johanstokking
Copy link
Member

Running the validation very often results in an EAGAIN error. I do not know the details of it, only that it is a memory issue, but this fixes it by slowing the process down and other fixes (read Jaime's original PR for full details).

Do you encounter this yourself? Are there reports from users?

@LDannijs
Copy link
Contributor Author

I encounter it everytime i run it, and users also often do.

@Jaime-Trinidad
Copy link
Contributor

Many users seem to be encountering this issue. I believe the following code could provide a temporary solution, though it may slow down the process. It still occasionally encounters an EAGAIN error, and I haven't fully grasped the underlying cause yet, especially since it doesn't appear to use all the available memory on my end. Interestingly, this issue has never occurred on my VM running Ubuntu.

Additionally, it's important to note that error and close are mutually exclusive in this code. The code is designed to promise only one outcome: if an error occurs, it rejects the promise. On a successful close, the promise is resolved. However, if it closes with issues, the promise is also rejected.

Copy link
Member

@johanstokking johanstokking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the issue is that too many child processes run at the same time, why not run them sequentially instead? What is the point of parallelism if they're scheduled 1 second apart?

validatePayloadCodecs() is awaited. So instead of pushing promises to an array and awaiting them, just await every run.

@LDannijs
Copy link
Contributor Author

LDannijs commented Jan 8, 2024

So i decided to at least give it a shot and run the feedback through ChatGPT and it seems it runs really well now. It's now running through it sequentially and it's not giving me any errors. Hope this is what we're looking for.

@Jaime-Trinidad please take a look as well.

@LDannijs LDannijs requested a review from johanstokking January 8, 2024 09:34
bin/validate.js Outdated
for (let index = 0; index < runs.length; index++) {
const r = runs[index];
await new Promise((resolve, reject) => {
setTimeout(async () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made a change that should hopefully be what your intention is.

@Jaime-Trinidad Jaime-Trinidad merged commit ed7e08b into TheThingsNetwork:master Jan 9, 2024
1 check passed
@LDannijs LDannijs deleted the validation branch January 11, 2024 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants