Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(resource-full-text-search): add text vector columns and GiST indexes for notion #1159

Merged
merged 2 commits into from
Aug 31, 2023

Conversation

fontanierh
Copy link
Contributor

@fontanierh fontanierh commented Aug 28, 2023

This PR adds:

  • a sql query in connectors db.ts to enable the unaccent extension if not yet enabled (tried to switch it on in prod, worked fine)
  • a titleSearchVector TSVECTOR column on notion_pages and notion_databases
  • a GiST index on the titleSearchVector column for notion_pages and notion_databases
  • a trigger to update titleSearchVector whenever the title is updated on either a notion_pages or notion_databases row
  • a migration script to populate titleSearchVector on existing notion_pages and notion_databases rows

I went with GiST instead of GIN (means faster to insert/build, but slower to search). Rationale is that the dataset is not static, and will be more write-heavy than read-heavy. Wonder if that's the write choice ? WDYT @lasryaric @spolu ?

@fontanierh fontanierh requested review from spolu and lasryaric August 28, 2023 14:40
@fontanierh fontanierh force-pushed the feat/add-full-text-search-resource-titles branch from beb3f57 to ad7e224 Compare August 28, 2023 14:41
@fontanierh fontanierh changed the title feat(resource-full-text-search): add text vector columns and GIN indexes for notion feat(resource-full-text-search): add text vector columns and GiST indexes for notion Aug 28, 2023
@fontanierh fontanierh force-pushed the feat/add-full-text-search-resource-titles branch from ad7e224 to 3d5ca7b Compare August 28, 2023 16:17
@spolu
Copy link
Contributor

spolu commented Aug 29, 2023

I presume we can easily switch between GIST and GIN in the future anyway?

@fontanierh
Copy link
Contributor Author

I presume we can easily switch between GIST and GIN in the future anyway?

yes I believe it's a one line change + running db sync @spolu

lasryaric
lasryaric previously approved these changes Aug 29, 2023
@spolu
Copy link
Contributor

spolu commented Aug 31, 2023

What's the status here?

@fontanierh fontanierh force-pushed the feat/add-full-text-search-resource-titles branch from c71a53a to eee28d5 Compare August 31, 2023 14:53
@fontanierh
Copy link
Contributor Author

What's the status here?

I'm merging it

@fontanierh fontanierh merged commit 609fe06 into main Aug 31, 2023
@fontanierh fontanierh deleted the feat/add-full-text-search-resource-titles branch August 31, 2023 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants