Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempts to make db build faster #59

Open
derekantrican opened this issue Mar 27, 2020 · 1 comment
Open

Attempts to make db build faster #59

derekantrican opened this issue Mar 27, 2020 · 1 comment

Comments

@derekantrican
Copy link
Owner

See if I can take some hints from buildxl to only rebuild things that have actually changed.

For instance: if a route has not changed, we may be able to avoid all the HTML parsing and instead skip it

@derekantrican
Copy link
Owner Author

derekantrican commented Mar 28, 2020

Ideas:

  • Hashing the HTML and if it has not changed, skipping all of the IHtmlDocument/parsing/etc lines (note that we will probably still need to get the IHtmlDocument for subareas/routes so that we can check those as well)
    • Need to check speed of hashing algorithms and see if there would actually be any time savings here to "hash HTML and check hash against previous" or "just load the IHtmlDocument and do all the parsing anyway"
  • Maybe there's a faster way to download HTML from a URL? Possibly: https://stackoverflow.com/a/54502464/2246411 (may need to investigate a couple different options)
  • See what happens if I take off the awaits before ParseRouteAsync and ParseAreaAsync (since I don't actually need to wait before those things complete. Last time I tried this I think this meant that the Task.WaitAll actually finished too quickly because all the DestAreas finished before any of the children

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant