Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

palomar (search) iteration #263

Merged
merged 41 commits into from
Sep 15, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
af1829a
palomar: add second README with ES ops stuff
bnewbold May 3, 2023
d01531d
palomar: proposed post and profile schema iteration
bnewbold Aug 2, 2023
ca73914
palomar: tweak proposed schemas
bnewbold Aug 2, 2023
2c782e3
search: search doc transform helpers
bnewbold Aug 14, 2023
42078d8
search: incorporate transforms
bnewbold Aug 14, 2023
4cdbdb0
palomar: update post+profile index schemas
bnewbold Aug 14, 2023
e382b9e
palomar: update README and dev setup
bnewbold Aug 14, 2023
550fc57
palomar: more progress
bnewbold Aug 14, 2023
ae34ff3
search: lint fixes
bnewbold Aug 15, 2023
adc05ba
palomar: more tweaks to schema
bnewbold Sep 1, 2023
a9e1c05
palomar: bit of progress
bnewbold Sep 1, 2023
cafef6e
palomar: basic query parsing, handle 'from:'
bnewbold Sep 1, 2023
8e78c4b
Merge branch 'main' into bnewbold/palomar-iterate
bnewbold Sep 12, 2023
10fd2dd
Makefile: build palomar (search)
bnewbold Sep 12, 2023
89c0ff4
gitignore: add more executables
bnewbold Sep 12, 2023
b5055aa
palomar: construct subscribeRepos URL using struct
bnewbold Sep 13, 2023
2cc193a
palomar: switch to slog; add prometheus and other common middleware
bnewbold Sep 13, 2023
35f09f3
palomar: don't force refresh for most indexing ops
bnewbold Sep 13, 2023
41f3ad3
palomar: progress on removing user+record database tables
bnewbold Sep 13, 2023
4489a89
identity: handle errors when doing LookupDID should not error, just i…
bnewbold Sep 13, 2023
5251558
util: allow non-fractional-second timestamps
bnewbold Sep 13, 2023
8c914b9
palomar: more cleanup
bnewbold Sep 13, 2023
f8ad174
palomar: switch HTTP API to skeleton
bnewbold Sep 14, 2023
d842aa9
palomar: fix bug in query marshal
bnewbold Sep 14, 2023
31a7212
palomar: clarify weird double-marshal
bnewbold Sep 14, 2023
cc587df
palomar: auto-create indices if needed; check existence
bnewbold Sep 14, 2023
1812e1e
palomar: logging, var names
bnewbold Sep 14, 2023
299f640
palomar: update READMEs
bnewbold Sep 14, 2023
66fef0a
palomar: hitsTotal
bnewbold Sep 14, 2023
ee3f286
identity: support skipping DNS resolution for some hosts (like bsky.s…
bnewbold Sep 14, 2023
fff34f1
palomar: skip DNS resolution on bsky.social; do try authoritative DNS
bnewbold Sep 14, 2023
25e1e95
palomar: fix unclosed HTTP connections
bnewbold Sep 14, 2023
4f7f097
palomar: default index shard sizes
bnewbold Sep 14, 2023
2ead460
palomar: tune backfill a bit
bnewbold Sep 14, 2023
c4bcc0e
palomar: clear createdAt on error, not skip record
bnewbold Sep 14, 2023
bcec416
palomar: fix go:embed schemas
bnewbold Sep 14, 2023
cddee8f
make lint
bnewbold Sep 14, 2023
8b8ab88
palomar: fix bad slog invocation
bnewbold Sep 14, 2023
8a7e7fd
palomar: feedback from review
bnewbold Sep 15, 2023
a7c32e3
palomar: better checkParams sanity check
bnewbold Sep 15, 2023
32a9856
palomar: RawQuery typo
bnewbold Sep 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
search: search doc transform helpers
  • Loading branch information
bnewbold committed Sep 1, 2023
commit 2c782e353aaa28abad9a3458de2c402f19eb2232
168 changes: 168 additions & 0 deletions search/testdata/transform-post-fixtures.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
[
{
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"rkey": "3k4duaz5vfs2b",
"cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"PostRecord": {
"$type": "app.bsky.feed.post",
"text": "post which embeds an external URL as a card",
"createdAt": "2023-08-07T05:46:14.423045Z",
"embed": {
"$type": "app.bsky.embed.external",
"external": {
"uri": "https://bsky.app",
"title": "Bluesky Social",
"description": "See what's next.",
"thumb": {
"$type": "blob",
"ref": {
"$link": "bafkreiash5eihfku2jg4skhyh5kes7j5d5fd6xxloaytdywcvb3r3zrzhu"
},
"mimeType": "image/png",
"size": 23527
}
}
}
},
"doc_id": "did:plc:u5cwb2mwiv2bfq53cjufe6yn_3k4duaz5vfs2b",
"PostDoc": {
"doc_index_ts": "2006-01-02T15:04:05.000Z",
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"record_rkey": "3k4duaz5vfs2b",
"record_cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"created_at": "2023-08-07T05:46:14.423045Z",
"text": "post which embeds an external URL as a card",
"embed_url": "https://bsky.app",
"embed_img_count": 0
}
},
{
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"rkey": "3k4duaz5vfs2b",
"cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"PostRecord": {
"$type": "app.bsky.feed.post",
"text": "longer example with #some #hashtags, emoji \u2620 \ud83d\ude42 \ud83c\udf85\ud83c\udfff, flags \ud83c\uddf8\ud83c\udde8 ",
"createdAt": "2023-08-07T05:46:14.423045Z",
"langs": ["th", "en-US"],
"facets": [
{
"index": {
"byteStart": 23,
"byteEnd": 35
},
"features": [
{
"$type": "app.bsky.richtext.facet#mention",
"did": "did:plc:ewvi7nxzyoun6zhxrhs64oiz"
}
]
},
{
"index": {
"byteStart": 74,
"byteEnd": 108
},
"features": [
{
"$type": "app.bsky.richtext.facet#link",
"uri": "https://en.wikipedia.org/wiki/CBOR"
}
]
}
],
"reply": {
"root": {
"uri": "at://did:plc:u5cwb2mwiv2bfq53cjufe6yn/app.bsky.feed.post/3k43tv4rft22g",
"cid": "bafyreig2fjxi3rptqdgylg7e5hmjl6mcke7rn2b6cugzlqq3i4zu6rq52q"
},
"parent": {
"uri": "at://did:plc:u5cwb2mwiv2bfq53cjufe6yn/app.bsky.feed.post/3k43tv4rft22g",
"cid": "bafyreig2fjxi3rptqdgylg7e5hmjl6mcke7rn2b6cugzlqq3i4zu6rq52q"
}
},
"embed": {
"$type": "app.bsky.embed.record",
"record": {
"uri": "at://did:plc:u5cwb2mwiv2bfq53cjufe6yn/app.bsky.feed.post/3k44deefqdk2g",
"cid": "bafyreiecx6dujwoeqpdzl27w67z4h46hyklk3an4i4cvvmioaqb2qbyo5u"
}
}
},
"doc_id": "did:plc:u5cwb2mwiv2bfq53cjufe6yn_3k4duaz5vfs2b",
"PostDoc": {
"doc_index_ts": "2006-01-02T15:04:05.000Z",
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"record_rkey": "3k4duaz5vfs2b",
"record_cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"created_at": "2023-08-07T05:46:14.423045Z",
"text": "longer example with #some #hashtags, emoji \u2620 \ud83d\ude42 \ud83c\udf85\ud83c\udfff, flags \ud83c\uddf8\ud83c\udde8 ",
"reply_root_aturi": "at://did:plc:u5cwb2mwiv2bfq53cjufe6yn/app.bsky.feed.post/3k43tv4rft22g",
"link_url": [ "https://en.wikipedia.org/wiki/CBOR" ],
"mention_did": [ "did:plc:ewvi7nxzyoun6zhxrhs64oiz" ],
"embed_aturi": "at://did:plc:u5cwb2mwiv2bfq53cjufe6yn/app.bsky.feed.post/3k44deefqdk2g",
"lang_code": ["th", "en-US"],
"lang_code_iso2": ["th", "en"],
"hashtag": ["some", "hashtags"],
"emoji": ["\u2620", "\ud83d\ude42", "\ud83c\udf85\ud83c\udfff", "\ud83c\uddf8\ud83c\udde8"],
"embed_img_count": 0
}
},
{
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"rkey": "3k4duaz5vfs2b",
"cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"PostRecord": {
"$type": "app.bsky.feed.post",
"text": "",
"createdAt": "2023-08-07T05:46:14.423045Z",
"embed": {
"$type": "app.bsky.embed.images",
"images": [
{
"alt": "brief alt text description of the first image",
"image": {
"$type": "blob",
"ref": {
"$link": "bafkreibabalobzn6cd366ukcsjycp4yymjymgfxcv6xczmlgpemzkz3cfa"
},
"mimeType": "image/webp",
"size": 760898
}
},
{
"alt": "brief alt text description of the second image",
"image": {
"$type": "blob",
"ref": {
"$link": "bafkreif3fouono2i3fmm5moqypwskh3yjtp7snd5hfq5pr453oggygyrte"
},
"mimeType": "image/png",
"size": 13208
}
}
]
}
},
"doc_id": "did:plc:u5cwb2mwiv2bfq53cjufe6yn_3k4duaz5vfs2b",
"PostDoc": {
"doc_index_ts": "2006-01-02T15:04:05.000Z",
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"record_rkey": "3k4duaz5vfs2b",
"record_cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"created_at": "2023-08-07T05:46:14.423045Z",
"text": "",
"embed_img_alt_text": [
"brief alt text description of the first image",
"brief alt text description of the second image"
],
"embed_img_count": 2
}
}
]
56 changes: 56 additions & 0 deletions search/testdata/transform-profile-fixtures.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
[
{
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"rkey": "self",
"cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"handle": "handle.example.com",
"ProfileRecord": {
"$type": "app.bsky.actor.profile"
},
"doc_id": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"ProfileDoc": {
"doc_index_ts": "2006-01-02T15:04:05.000Z",
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"record_cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"has_avatar": false,
"has_banner": false
}
},
{
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"rkey": "self",
"cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"handle": "handle.example.com",
"ProfileRecord": {
"$type": "app.bsky.actor.profile",
"displayName": "Big Bubba",
"description": "Big Description 🥸 #cheese",
"avatar": {
"$type": "blob",
"mimeType": "image/jpeg",
"ref": {
"$link": "bafkreiglnysron3h2je7nf6cmvtimuaxi7xe2c7rkxitmks3mzmajnc2ou"
},
"size": 106760
},
"banner": {
"cid": "bafkreih42ufe7ies4zephtkdw4tcb4hvuoc2pjzybpjxpecz3tyymnvkhm",
"mimeType": "image/jpeg"
}
},
"doc_id": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"ProfileDoc": {
"doc_index_ts": "2006-01-02T15:04:05.000Z",
"did": "did:plc:u5cwb2mwiv2bfq53cjufe6yn",
"handle": "handle.example.com",
"record_cid": "bafyreibjifzpqj6o6wcq3hejh7y4z4z2vmiklkvykc57tw3pcbx3kxifpm",
"display_name": "Big Bubba",
"description": "Big Description 🥸 #cheese",
"emoji": ["🥸"],
"hashtag": ["cheese"],
"has_avatar": true,
"has_banner": true
}
}
]
Loading