Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full text search #318

Open
hdgarrood opened this issue Jul 12, 2017 · 5 comments
Open

Full text search #318

hdgarrood opened this issue Jul 12, 2017 · 5 comments
Labels

Comments

@hdgarrood
Copy link
Collaborator

Include comments in the search index, so that e.g. if I search for Kleisli I get Star from Data.Profunctor. Note that the documentation uses the word "Kleisli" but the word "Kleisli" doesn't appear in any of the types, which is why this query currently returns no results.

@paf31
Copy link
Contributor

paf31 commented Jul 12, 2017

I love the idea, but I don't know if we can keep an index in memory, so this might mean using something like ElasticSearch unfortunately.

@chexxor
Copy link

chexxor commented Jul 12, 2017

Wouldn't need to go all the way to ElasticSearch, do we? I think databases like sqlite and postgres have text-search. sqlite fts
Might require adding an option to the search, like "include comments", to activate this extra filter.

Would require moving from filesystem-as-database to sqlite-as-database.

@paf31
Copy link
Contributor

paf31 commented Jul 12, 2017

We could store the index on disk ourselves perhaps. I really like the way we use the filesystem right now, it makes maintenance very simple.

@felixSchl
Copy link

How about routinely merging all json files in "data/verified" into a single file containing all the comments? In terms of performance, we either search the file by scanning it or simply keeping it entirely in memory. For the former, we would just do something line-based and for the latter use json.

@felixSchl
Copy link

This just showed up in my github feed: https://github.com/well-typed/full-text-search and might be worth looking into. It comes with an example that coincidentally also searches packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants