Feature - 3rd party labelers (#2032)

* adding some basic views * feed hydration, add blocks to post hydration * pass over notification hydration * tidy * merge * implement getProfile * hydrate post aggregation and viewer state * fix * fix codegen * get some tests passing! * add takedowns & some like bugfixing * all profile tests passing! * likes test * follow endpoints using data plane * reorg follow block rules * reposts * post views! * implement getList w/ dataplane caveat * adjust dataplane getListMembers to return listitem uris * implement getListMutes and -Blocks w/ dataplane * suggestions * timeline * misc view fixes * view fixes for mutes, self-mute/block * author feed * feed gen routes * tidy * misc block/mute fixes * list feed & actor likes * implement getLists, fix some empty cursors * implement getMutes, empty profile description fix * implement getBlocks, block application fix * implement getSuggestedFollowsByActor, needs some fixes * feed generation * search routes * threads * tidy * fix some snaps * fix getSuggestedFollowsByActor * implement listNotifications * implement getUnreadCount w/ dataplane * implement notifications.updateSeen w/ dataplane * 3rd party blocking tests * blocked profile viewer * add search mocks * refactor getFeed * createPipeline -> createPipelineNew * basic replygating functionality on dataplane w/o filtering violating replies * hack threadgates into dataplane, apply gates * deterministic thread orders in dataplane * misc cleanup around dataplane * upgrade typescript to v5.3 * update typescript linter deps * sync bsky proto, codegen * update dataplane, sync with bsky proto updates * remove indexer, ingester, daemon, moderation services from appview * convert more bsky internals to dataplane, remove custom feedgens, implement mute/unmuting in mock dataplane * remove bsky services. db and indexing logic into mock dataplane. * remove tests not needed by appview v2, misc reorg * add basic in-mem repo subscription to dataplane mock * fix dev-env, bsky tests, bsky build * cull bsky service entrypoint * add bsky service readme * build * tidy * tidy, fix pds proxy tests * fix * fix bsky entrypoint deps * support http2 grpc client * build * fix dataplane bad tls config/default * support multiple dataplane urls, retry when unavailable * build * tidy/fix * move dataplane mock tests into their own dir * cover label hydration through timeline test * bring back labels in appview tests * remove unused db primary/replica/coordinator from bsky dataplane * bsky proto add cids to contracts, buf codegen * sync-up bsky data-plane w/ codegen updates * start using dataplane interaction endpoints * add file * avoid overfetching from dataplane, plumb feed items and cids * pass refs through for post viewer state * Lexicons: Add labeler prefs, labeler declaration, and get labeler routes * Add labelerViewBasic and update embed views * Fix typo * switch list feeds to use feed item in dataplane * handle not found err on get-thread dataplane call * support use of search service rather than dataplane methods * mark some appview v2 todos * tidy * still use dataplane on search endpoints when search service is not configured * fix pds test * Switch to labelerViewDetailed * Move label and report values to refs * Add getActorLabelers * lint * fix up bsky tests & snaps * small diff to open pr * rm new line * codegen schemas * tidy migrations * table + indexing * protos * rename lexicons * views, hydration + rename lexicons * rest of routes * data plane routes * parse labelers from req * fix appview-v2 docker build * Support label issuer tied to appview v2 (#2033) support label issuer tied to appview * hydrate context * tidy header logic * integrating into more routes * more routes * wrap up rest * add mock labeler * rework labelerlexicons * tidy lexs * codegen new lexicons * integrate lexicon rework * add proxy logic * forward labeler headers through pds * tweak label header parsing * remove did from scheams * update indexing for lexs * tests for mod service views * label hydration test * Add 'associated' to profileViewDetailed * Rename labelers to mods in preferences * Change uri to did in mod preferences * couple more * syntax tweaks * integrate updated lexicons * update view snap * handle mod service embeds * tidy * fix mock * lint * base default labels of config var * fix label hydration * Appview v2: handle empty cursor on list notifications (#2017) handle empty cursor on appview listnotifs * Update appview v2 to use author feed enum (#2047) * update bsky protos with author feed enum, misc feed item changes * support new author feed enums in dataplane * fix build * Appview v2: utilize sorted-at field in bsky protos (#2050) utilize new sorted-at field in bsky protos * remove all dataplane usage of GetLikeCounts, switch to GetInteractionCounts * Appview v2, sync w/ changes to protos (#2071) * sync bsky protos * sync-up bsky implementation w/ proto changes * Appview v2 initial implementation for getPopularFeedGenerators (#2072) add an initial implementation for getPopularFeedGenerators on appview v2 * merge * fixes * fix feed tests * fix bsync mock * format * remove unused config * fix lockfile * another lockfile fix * fix duplicate type * fix dupplicate test * Appview v2 handling clearly bad cursors (#2092) * make mock dataplane cursors different from v1 cursors * fail open on clearly bad appview cursors * fix pds appview proxy snaps * Appview v2 no notifs seen behavior (#2096) * alter behavior for presenting notifications w/ no last-seen time * fix pds proxy tests * Appview v2 dataplane retries based on client host (#2098) choose dataplane client for retries based on host when possible/relevant * don't apply negated labels * display suspensions on actor profile in appview v2 * Appview v2 use dataplane for identity lookups (#2095) * update bsky proto w/ identity methods * setup identity endpoints on mock dataplane * move from idresolver to dataplane for identity lookups on appview * tidy * Appview v2: apply safe takedown refs to records, actors (#2107) apply safe takedown refs to records, actors * Fix timing on appview v2 repo rev header (#2113) fix timing on appview repo rev * fix post thread responses * Appview v2 don't apply 3p self blocks (#2112) do not apply 3p self-blocks * Appview v2 search for feed generators (#2118) * add protos for feedgen search * support feed search on getPopularFeedGenerators * Appview v2 config tidy (#2117) * remove mod and triage roles from appview * rename cdn and search config * remove custom feed harness from appview v2 * Appview v2: don't apply missing modlists (#2122) * dont apply missing mod lists * update mock dataplane * Update packages/bsky/src/hydration/hydrator.ts Co-authored-by: devin ivy <[email protected]> * refactor & document a bit better * fix up other routes --------- Co-authored-by: devin ivy <[email protected]> * Appview v2 enforce post thread root boundary (#2120) * enforce post thread root boundary * test thread root boundary * Appview v2 fix admin environment variable (#2137) fix admin env in appview v2 * Remove re-pagination from getSuggestions (#2145) * remove re-pagination from getSuggestions * fix test * Adjust wording for account suspension (#2153) adjust wording for account suspension * Appview v2: fix not-found and blocked uris in threads (#2201) * fix uris of not-found and blocked posts in threads * update snaps * ✨ Show author feed of takendown author to admins only (#2197) * fold in cid, auth, tracing, node version changes * remove dead config from bsky service entrypoint * build * remove ozone test codepaths for appview v2 * tidy, docs fix * fix test * add additional user counts * add associated data to profiles * update snaps * update to is_mod_service * format * tidy * 3p labeler sdk updates (#2176) * Update sdk to support 3p labeler preferences * Stick with intolerance instead of hate for the label group id * wip expand labels and label groups * Output moderationOpts (computed) and modsPref (unaltered) * Add tests for enabling/disabling mod services * Add atproto-labelers header config * Expand labels and label groups in definitions * Fix tests * Tweaks to labels * Remove label descriptions and improve output types on labels and label groups * Add mocker to exported API * Improve types of label and label group definitions * Rework moderation prefs to continue using global labelgroup settings and only disable label groups per moderator * Simplify encoding of the label preferences in definitions * Add target constraints to labels * Refactor the moderation sdk to derive more behaviors from the definition files * Small cleanup * Add hiding tool to modsdk * Track filter causes * Make mute state an alert * Fix: dont blur profileview for blocks * Prioritize causes by severity * Add moderateNotification() and drop quote post moderation code * Add mocker functions for notifications * Improve mock data * Lexicon: Add custom label definitions and remove modservice descriptions * Lexicon: Update moderation prefs * SDK updates: remove label groups, reduce builtin labels, update mod-preference apis * Lexicon: Update global labels with new reduced set * Lexicon: Remove moderation.getService and add detailed option to getServices * Lexicons: add severity=none to custom label value defs * Implement custom label-value definition tooling * All custom labels are no-self * Backend impl for labeler lexicon updates (#2258) * codegen * clean up impl * fix up tests * Lexicon: modservice -> labeler * Remove x- prefix behavior; add label value syntax rules; add custom label precedence rules * Lexicon: Remove the ability to choose a defaultSetting from custom labels * Rework test suites * Give behaviors to all labels regardless of target * sync up backend with lex changes * fix labelers in dev-env agent * lint protos * update protos & views * small bugfix & update tests * tweak protos * fix build issue from merge --------- Co-authored-by: Devin Ivy <[email protected]> Co-authored-by: Paul Frazee <[email protected]> Co-authored-by: Foysal Ahamed <[email protected]>
bluesky-social · Mar 6, 2024 · 3988543 · 3988543
1 parent bae2ce3
commit 3988543
Show file tree

Hide file tree

Showing 195 changed files with 8,534 additions and 9,819 deletions.
diff --git a/lexicons/app/bsky/actor/defs.json b/lexicons/app/bsky/actor/defs.json
@@ -67,6 +67,10 @@
         "followersCount": { "type": "integer" },
         "followsCount": { "type": "integer" },
         "postsCount": { "type": "integer" },
+        "associated": {
+          "type": "ref",
+          "ref": "#profileAssociated"
+        },
         "indexedAt": { "type": "string", "format": "datetime" },
         "viewer": { "type": "ref", "ref": "#viewerState" },
         "labels": {
@@ -75,6 +79,14 @@
         }
       }
     },
+    "profileAssociated": {
+      "type": "object",
+      "properties": {
+        "lists": { "type": "integer" },
+        "feedgens": { "type": "integer" },
+        "labeler": { "type": "boolean" }
+      }
+    },
     "viewerState": {
       "type": "object",
       "description": "Metadata about the requesting account's relationship with the subject account. Only has meaningful content for authed requests.",
@@ -122,10 +134,15 @@
       "type": "object",
       "required": ["label", "visibility"],
       "properties": {
+        "labelerDid": {
+          "type": "string",
+          "description": "Which labeler does this preference apply to? If undefined, applies globally.",
+          "format": "did"
+        },
         "label": { "type": "string" },
         "visibility": {
           "type": "string",
-          "knownValues": ["show", "warn", "hide"]
+          "knownValues": ["ignore", "show", "warn", "hide"]
         }
       }
     },
@@ -270,6 +287,29 @@
           "description": "A list of URIs of posts the account owner has hidden."
         }
       }
+    },
+    "modsPref": {
+      "type": "object",
+      "required": ["mods"],
+      "properties": {
+        "mods": {
+          "type": "array",
+          "items": {
+            "type": "ref",
+            "ref": "#modPrefItem"
+          }
+        }
+      }
+    },
+    "modPrefItem": {
+      "type": "object",
+      "required": ["did"],
+      "properties": {
+        "did": {
+          "type": "string",
+          "format": "did"
+        }
+      }
     }
   }
 }
diff --git a/lexicons/app/bsky/embed/record.json b/lexicons/app/bsky/embed/record.json
@@ -21,7 +21,8 @@
             "#viewNotFound",
             "#viewBlocked",
             "app.bsky.feed.defs#generatorView",
-            "app.bsky.graph.defs#listView"
+            "app.bsky.graph.defs#listView",
+            "app.bsky.labeler.defs#labelerView"
           ]
         }
       }

diff --git a/lexicons/app/bsky/labeler/defs.json b/lexicons/app/bsky/labeler/defs.json
@@ -0,0 +1,70 @@
+{
+  "lexicon": 1,
+  "id": "app.bsky.labeler.defs",
+  "defs": {
+    "labelerView": {
+      "type": "object",
+      "required": ["uri", "cid", "creator", "indexedAt"],
+      "properties": {
+        "uri": { "type": "string", "format": "at-uri" },
+        "cid": { "type": "string", "format": "cid" },
+        "creator": { "type": "ref", "ref": "app.bsky.actor.defs#profileView" },
+        "likeCount": { "type": "integer", "minimum": 0 },
+        "viewer": { "type": "ref", "ref": "#labelerViewerState" },
+        "indexedAt": { "type": "string", "format": "datetime" },
+        "labels": {
+          "type": "array",
+          "items": { "type": "ref", "ref": "com.atproto.label.defs#label" }
+        }
+      }
+    },
+    "labelerViewDetailed": {
+      "type": "object",
+      "required": ["uri", "cid", "creator", "policies", "indexedAt"],
+      "properties": {
+        "uri": { "type": "string", "format": "at-uri" },
+        "cid": { "type": "string", "format": "cid" },
+        "creator": { "type": "ref", "ref": "app.bsky.actor.defs#profileView" },
+        "policies": {
+          "type": "ref",
+          "ref": "app.bsky.labeler.defs#labelerPolicies"
+        },
+        "likeCount": { "type": "integer", "minimum": 0 },
+        "viewer": { "type": "ref", "ref": "#labelerViewerState" },
+        "indexedAt": { "type": "string", "format": "datetime" },
+        "labels": {
+          "type": "array",
+          "items": { "type": "ref", "ref": "com.atproto.label.defs#label" }
+        }
+      }
+    },
+    "labelerViewerState": {
+      "type": "object",
+      "properties": {
+        "like": { "type": "string", "format": "at-uri" }
+      }
+    },
+    "labelerPolicies": {
+      "type": "object",
+      "required": ["labelValues"],
+      "properties": {
+        "labelValues": {
+          "type": "array",
+          "description": "The label values which this labeler publishes. May include global or custom labels.",
+          "items": {
+            "type": "ref",
+            "ref": "com.atproto.label.defs#labelValue"
+          }
+        },
+        "labelValueDefinitions": {
+          "type": "array",
+          "description": "Label values created by this labeler and scoped exclusively to it. Labels defined here will override global label definitions for this labeler.",
+          "items": {
+            "type": "ref",
+            "ref": "com.atproto.label.defs#labelValueDefinition"
+          }
+        }
+      }
+    }
+  }
+}
diff --git a/lexicons/app/bsky/labeler/getServices.json b/lexicons/app/bsky/labeler/getServices.json
@@ -0,0 +1,43 @@
+{
+  "lexicon": 1,
+  "id": "app.bsky.labeler.getServices",
+  "defs": {
+    "main": {
+      "type": "query",
+      "description": "Get information about a list of labeler services.",
+      "parameters": {
+        "type": "params",
+        "required": ["dids"],
+        "properties": {
+          "dids": {
+            "type": "array",
+            "items": { "type": "string", "format": "did" }
+          },
+          "detailed": {
+            "type": "boolean",
+            "default": false
+          }
+        }
+      },
+      "output": {
+        "encoding": "application/json",
+        "schema": {
+          "type": "object",
+          "required": ["views"],
+          "properties": {
+            "views": {
+              "type": "array",
+              "items": {
+                "type": "union",
+                "refs": [
+                  "app.bsky.labeler.defs#labelerView",
+                  "app.bsky.labeler.defs#labelerViewDetailed"
+                ]
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
diff --git a/lexicons/app/bsky/labeler/service.json b/lexicons/app/bsky/labeler/service.json
@@ -0,0 +1,26 @@
+{
+  "lexicon": 1,
+  "id": "app.bsky.labeler.service",
+  "defs": {
+    "main": {
+      "type": "record",
+      "description": "A declaration of the existence of labeler service.",
+      "key": "literal:self",
+      "record": {
+        "type": "object",
+        "required": ["policies", "createdAt"],
+        "properties": {
+          "policies": {
+            "type": "ref",
+            "ref": "app.bsky.labeler.defs#labelerPolicies"
+          },
+          "labels": {
+            "type": "union",
+            "refs": ["com.atproto.label.defs#selfLabels"]
+          },
+          "createdAt": { "type": "string", "format": "datetime" }
+        }
+      }
+    }
+  }
+}
diff --git a/lexicons/com/atproto/label/defs.json b/lexicons/com/atproto/label/defs.json
@@ -61,6 +61,73 @@
           "description": "The short string name of the value or type of this label."
         }
       }
+    },
+    "labelValueDefinition": {
+      "type": "object",
+      "description": "Declares a label value and its expected interpertations and behaviors.",
+      "required": ["identifier", "severity", "blurs", "locales"],
+      "properties": {
+        "identifier": {
+          "type": "string",
+          "description": "The value of the label being defined. Must only include lowercase ascii and the '-' character ([a-z-]+).",
+          "maxLength": 100,
+          "maxGraphemes": 100
+        },
+        "severity": {
+          "type": "string",
+          "description": "How should a client visually convey this label? 'inform' means neutral and informational; 'alert' means negative and warning; 'none' means show nothing.",
+          "knownValues": ["inform", "alert", "none"]
+        },
+        "blurs": {
+          "type": "string",
+          "description": "What should this label hide in the UI, if applied? 'content' hides all of the target; 'media' hides the images/video/audio; 'none' hides nothing.",
+          "knownValues": ["content", "media", "none"]
+        },
+        "locales": {
+          "type": "array",
+          "items": { "type": "ref", "ref": "#labelValueDefinitionStrings" }
+        }
+      }
+    },
+    "labelValueDefinitionStrings": {
+      "type": "object",
+      "description": "Strings which describe the label in the UI, localized into a specific language.",
+      "required": ["lang", "name", "description"],
+      "properties": {
+        "lang": {
+          "type": "string",
+          "description": "The code of the language these strings are written in.",
+          "format": "language"
+        },
+        "name": {
+          "type": "string",
+          "description": "A short human-readable name for the label.",
+          "maxGraphemes": 64,
+          "maxLength": 640
+        },
+        "description": {
+          "type": "string",
+          "description": "A longer description of what the label means and why it might be applied.",
+          "maxGraphemes": 10000,
+          "maxLength": 100000
+        }
+      }
+    },
+    "labelValue": {
+      "type": "string",
+      "knownValues": [
+        "!hide",
+        "!no-promote",
+        "!warn",
+        "!no-unauthenticated",
+        "dmca-violation",
+        "doxxing",
+        "porn",
+        "sexual",
+        "nudity",
+        "nsfl",
+        "gore"
+      ]
     }
   }
 }