Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split out moderation backend #1970

Merged
merged 96 commits into from
Jan 5, 2024
Merged
Show file tree
Hide file tree
Changes from 81 commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
03a1557
mv appview
dholms Dec 14, 2023
1d80936
copy
dholms Dec 14, 2023
ef55a87
merge
dholms Dec 14, 2023
36ba182
finalize copy
dholms Dec 14, 2023
5aaf7f3
package names
dholms Dec 14, 2023
a72ba16
big WIP
dholms Dec 15, 2023
fb3161e
first pass at mod servce
dholms Dec 15, 2023
6b682ab
some tidy
dholms Dec 15, 2023
b8af648
tidy & fix compiler errors
dholms Dec 19, 2023
df4d40b
rename to ozone, db migrations, add to dev-env & pds cfg
dholms Dec 19, 2023
b121af1
getRecord & getRepo mostly working
dholms Dec 20, 2023
731473c
fix open handle
dholms Dec 20, 2023
ecb163e
get record tests all working
dholms Dec 20, 2023
6fb8d5b
moderation events working
dholms Dec 20, 2023
efa90c0
statuses working
dholms Dec 20, 2023
bb86a0f
tidy test suite
dholms Dec 20, 2023
c7d6f7a
search repos
dholms Dec 20, 2023
965669b
server & db tests
dholms Dec 20, 2023
5c4b172
moderation tests
dholms Dec 20, 2023
5ecbe43
wip daemon + push events
dholms Dec 20, 2023
4834ea9
pds fanout working
dholms Dec 20, 2023
f49ba1e
fix db test
dholms Dec 20, 2023
c133c4b
fanning takedowns out to appview
dholms Dec 20, 2023
23ea84d
rm try/catch
dholms Dec 21, 2023
c8312f5
bsky moderation test
dholms Dec 21, 2023
8016357
introduce mod subject wrappers
dholms Dec 21, 2023
c1df5af
more tidy
dholms Dec 21, 2023
c880002
refactor event reversal
dholms Dec 21, 2023
8ff0b5b
tidy some db stuff
dholms Dec 21, 2023
995f75d
tidy
dholms Dec 21, 2023
2b33089
rename service to mod-service
dholms Dec 21, 2023
4ad7e24
fix test
dholms Dec 21, 2023
fc32b36
tidy config
dholms Dec 21, 2023
f208397
refactor auth in bsky
dholms Dec 22, 2023
f4df71d
wip patching up auto-mod
dholms Dec 22, 2023
f617fe0
add label ingester in appview
dholms Dec 22, 2023
626a06a
fix a couple build issues
dholms Dec 22, 2023
dcfc21f
fix some timing bugs
dholms Dec 22, 2023
dfc2332
tidy polling logic
dholms Dec 22, 2023
20fee31
fix up tests
dholms Dec 22, 2023
a445424
merge main
dholms Dec 22, 2023
9208635
fix some pds tests
dholms Dec 22, 2023
1d07ad5
eslint ignore
dholms Dec 22, 2023
54b1710
fix ozone tests
dholms Dec 22, 2023
929bd63
move seeds to dev-env
dholms Dec 22, 2023
9b638bc
move images around
dholms Dec 22, 2023
0422450
fix db schemas
dholms Dec 22, 2023
fdb312a
use service auth admin reqs
dholms Dec 26, 2023
2ab6b24
fix remaining tests
dholms Dec 26, 2023
c1e29e3
auth tests bsky
dholms Dec 27, 2023
d2987a8
another test
dholms Dec 27, 2023
24944b9
random tidy
dholms Dec 27, 2023
4cae7ef
fix up search
dholms Dec 27, 2023
47241d5
clean up bsky mod service
dholms Dec 27, 2023
4b1b0eb
more tidy
dholms Dec 27, 2023
0880d40
default attempts to 0
dholms Dec 27, 2023
04068ee
tidy old test
dholms Dec 27, 2023
ccbcd6e
random tidy
dholms Dec 27, 2023
bd5abdb
tidy package.json
dholms Dec 27, 2023
7af7199
tidy logger
dholms Dec 27, 2023
c26b9f7
takedownId -> takedownRef
dholms Dec 28, 2023
26511a7
misc pr feedback
dholms Dec 28, 2023
617595e
split daemon out from ozone application
dholms Dec 28, 2023
00978ae
fix blob takedown mgiration
dholms Dec 29, 2023
b0c0d44
refactor ozone config
dholms Dec 29, 2023
a796638
do push event fanout on write instead of on read
dholms Dec 29, 2023
e9ff8cf
make suspend error work again
dholms Dec 29, 2023
232a4c8
add attempts check & add supporting index
dholms Dec 29, 2023
70536aa
fix takedown test ref
dholms Dec 29, 2023
a2f41ca
merge main
dholms Jan 3, 2024
f312271
get tests working
dholms Jan 3, 2024
808fa19
rm old test
dholms Jan 3, 2024
4b9c782
fix timing bug in event pusher tests
dholms Jan 3, 2024
9ed220e
attempt another fix for timing bug
dholms Jan 3, 2024
e764792
await req
dholms Jan 3, 2024
b11086e
service files
dholms Jan 4, 2024
de48f83
remove labelerDid cfg
dholms Jan 4, 2024
2660c6a
update snaps for labeler did + some cfg changes
dholms Jan 4, 2024
867310e
fix more snaps
dholms Jan 4, 2024
59cde5a
pnpm i
dholms Jan 4, 2024
9010abd
build ozone images
dholms Jan 4, 2024
58ab837
build
dholms Jan 4, 2024
83a0c23
make label provider optional
dholms Jan 5, 2024
6ef1a9b
fix build issues
dholms Jan 5, 2024
8bd7a0c
fix build
dholms Jan 5, 2024
82fe321
fix build
dholms Jan 5, 2024
2e3afd9
build pds
dholms Jan 5, 2024
de6518d
build on ghcr
dholms Jan 5, 2024
5704ea5
fix syntax in entry
dholms Jan 5, 2024
b2a1c9d
another fix
dholms Jan 5, 2024
bbffac2
use correct import
dholms Jan 5, 2024
f827502
export logger
dholms Jan 5, 2024
728a37c
remove event reverser
dholms Jan 5, 2024
d5cea8e
adjust push event fanout
dholms Jan 5, 2024
c7a7470
push out multiple
dholms Jan 5, 2024
7d63499
remove builds
dholms Jan 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
packages/api/src/client
packages/bsky/src/lexicon
packages/pds/src/lexicon
packages/ozone/src/lexicon
55 changes: 55 additions & 0 deletions .github/workflows/build-and-push-ozone-aws.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: build-and-push-ozone-aws
on:
push:
branches:
- main
- appeal-report
env:
REGISTRY: ${{ secrets.AWS_ECR_REGISTRY_USEAST2_PACKAGES_REGISTRY }}
USERNAME: ${{ secrets.AWS_ECR_REGISTRY_USEAST2_PACKAGES_USERNAME }}
PASSWORD: ${{ secrets.AWS_ECR_REGISTRY_USEAST2_PACKAGES_PASSWORD }}
IMAGE_NAME: ozone

jobs:
ozone-container-aws:
if: github.repository == 'bluesky-social/atproto'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Setup Docker buildx
uses: docker/setup-buildx-action@v2

- name: Log into registry ${{ env.REGISTRY }}
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ env.USERNAME}}
password: ${{ env.PASSWORD }}

- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v4
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=sha,enable=true,priority=100,prefix=,suffix=,format=long

- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v4
with:
context: .
push: ${{ github.event_name != 'pull_request' }}
file: ./services/ozone/Dockerfile
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
56 changes: 56 additions & 0 deletions .github/workflows/build-and-push-ozone-ghcr.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: build-and-push-ozone-ghcr
on:
push:
branches:
- main
env:
REGISTRY: ghcr.io
USERNAME: ${{ github.actor }}
PASSWORD: ${{ secrets.GITHUB_TOKEN }}

# github.repository as <account>/<repo>
IMAGE_NAME: ${{ github.repository }}

jobs:
ozone-container-ghcr:
if: github.repository == 'bluesky-social/atproto'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Setup Docker buildx
uses: docker/setup-buildx-action@v2

- name: Log into registry ${{ env.REGISTRY }}
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ env.USERNAME }}
password: ${{ env.PASSWORD }}

- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v4
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=sha,enable=true,priority=100,prefix=ozone:,suffix=,format=long

- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v4
with:
context: .
push: ${{ github.event_name != 'pull_request' }}
file: ./services/ozone/Dockerfile
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ codegen: ## Re-generate packages from lexicon/ files
cd packages/api; pnpm run codegen
cd packages/pds; pnpm run codegen
cd packages/bsky; pnpm run codegen
cd packages/ozone; pnpm run codegen
# clean up codegen output
pnpm format

Expand Down
1 change: 1 addition & 0 deletions lexicons/com/atproto/admin/defs.json
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,7 @@
"did": { "type": "string", "format": "did" },
"handle": { "type": "string", "format": "handle" },
"email": { "type": "string" },
"relatedRecords": { "type": "array", "items": { "type": "unknown" } },
"indexedAt": { "type": "string", "format": "datetime" },
"invitedBy": {
"type": "ref",
Expand Down
36 changes: 36 additions & 0 deletions lexicons/com/atproto/admin/getAccountInfos.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"lexicon": 1,
"id": "com.atproto.admin.getAccountInfos",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a batch version of getAccountInfo (implemented on PDS) - think there's a decent argument we just deprecate the non-batch version

The basic idea here is to go to all relevant services (pds & appview) for all the info they have on a user & then collate in the ozone backend

"defs": {
"main": {
"type": "query",
"description": "Get details about some accounts.",
"parameters": {
"type": "params",
"required": ["dids"],
"properties": {
"dids": {
"type": "array",
"items": { "type": "string", "format": "did" }
}
}
},
"output": {
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["infos"],
"properties": {
"infos": {
"type": "array",
"items": {
"type": "ref",
"ref": "com.atproto.admin.defs#accountView"
}
}
}
}
}
}
}
}
13 changes: 13 additions & 0 deletions packages/api/src/client/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import * as ComAtprotoAdminDisableInviteCodes from './types/com/atproto/admin/di
import * as ComAtprotoAdminEmitModerationEvent from './types/com/atproto/admin/emitModerationEvent'
import * as ComAtprotoAdminEnableAccountInvites from './types/com/atproto/admin/enableAccountInvites'
import * as ComAtprotoAdminGetAccountInfo from './types/com/atproto/admin/getAccountInfo'
import * as ComAtprotoAdminGetAccountInfos from './types/com/atproto/admin/getAccountInfos'
import * as ComAtprotoAdminGetInviteCodes from './types/com/atproto/admin/getInviteCodes'
import * as ComAtprotoAdminGetModerationEvent from './types/com/atproto/admin/getModerationEvent'
import * as ComAtprotoAdminGetRecord from './types/com/atproto/admin/getRecord'
Expand Down Expand Up @@ -153,6 +154,7 @@ export * as ComAtprotoAdminDisableInviteCodes from './types/com/atproto/admin/di
export * as ComAtprotoAdminEmitModerationEvent from './types/com/atproto/admin/emitModerationEvent'
export * as ComAtprotoAdminEnableAccountInvites from './types/com/atproto/admin/enableAccountInvites'
export * as ComAtprotoAdminGetAccountInfo from './types/com/atproto/admin/getAccountInfo'
export * as ComAtprotoAdminGetAccountInfos from './types/com/atproto/admin/getAccountInfos'
export * as ComAtprotoAdminGetInviteCodes from './types/com/atproto/admin/getInviteCodes'
export * as ComAtprotoAdminGetModerationEvent from './types/com/atproto/admin/getModerationEvent'
export * as ComAtprotoAdminGetRecord from './types/com/atproto/admin/getRecord'
Expand Down Expand Up @@ -441,6 +443,17 @@ export class AdminNS {
})
}

getAccountInfos(
params?: ComAtprotoAdminGetAccountInfos.QueryParams,
opts?: ComAtprotoAdminGetAccountInfos.CallOptions,
): Promise<ComAtprotoAdminGetAccountInfos.Response> {
return this._service.xrpc
.call('com.atproto.admin.getAccountInfos', params, undefined, opts)
.catch((e) => {
throw ComAtprotoAdminGetAccountInfos.toKnownErr(e)
})
}

getInviteCodes(
params?: ComAtprotoAdminGetInviteCodes.QueryParams,
opts?: ComAtprotoAdminGetInviteCodes.CallOptions,
Expand Down
46 changes: 46 additions & 0 deletions packages/api/src/client/lexicons.ts
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,12 @@ export const schemaDict = {
email: {
type: 'string',
},
relatedRecords: {
type: 'array',
items: {
type: 'unknown',
},
},
indexedAt: {
type: 'string',
format: 'datetime',
Expand Down Expand Up @@ -1046,6 +1052,45 @@ export const schemaDict = {
},
},
},
ComAtprotoAdminGetAccountInfos: {
lexicon: 1,
id: 'com.atproto.admin.getAccountInfos',
defs: {
main: {
type: 'query',
description: 'Get details about some accounts.',
parameters: {
type: 'params',
required: ['dids'],
properties: {
dids: {
type: 'array',
items: {
type: 'string',
format: 'did',
},
},
},
},
output: {
encoding: 'application/json',
schema: {
type: 'object',
required: ['infos'],
properties: {
infos: {
type: 'array',
items: {
type: 'ref',
ref: 'lex:com.atproto.admin.defs#accountView',
},
},
},
},
},
},
},
},
ComAtprotoAdminGetInviteCodes: {
lexicon: 1,
id: 'com.atproto.admin.getInviteCodes',
Expand Down Expand Up @@ -7875,6 +7920,7 @@ export const ids = {
ComAtprotoAdminEmitModerationEvent: 'com.atproto.admin.emitModerationEvent',
ComAtprotoAdminEnableAccountInvites: 'com.atproto.admin.enableAccountInvites',
ComAtprotoAdminGetAccountInfo: 'com.atproto.admin.getAccountInfo',
ComAtprotoAdminGetAccountInfos: 'com.atproto.admin.getAccountInfos',
ComAtprotoAdminGetInviteCodes: 'com.atproto.admin.getInviteCodes',
ComAtprotoAdminGetModerationEvent: 'com.atproto.admin.getModerationEvent',
ComAtprotoAdminGetRecord: 'com.atproto.admin.getRecord',
Expand Down
1 change: 1 addition & 0 deletions packages/api/src/client/types/com/atproto/admin/defs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,7 @@ export interface AccountView {
did: string
handle: string
email?: string
relatedRecords?: {}[]
indexedAt: string
invitedBy?: ComAtprotoServerDefs.InviteCode
invites?: ComAtprotoServerDefs.InviteCode[]
Expand Down
36 changes: 36 additions & 0 deletions packages/api/src/client/types/com/atproto/admin/getAccountInfos.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/**
* GENERATED CODE - DO NOT MODIFY
*/
import { Headers, XRPCError } from '@atproto/xrpc'
import { ValidationResult, BlobRef } from '@atproto/lexicon'
import { isObj, hasProp } from '../../../../util'
import { lexicons } from '../../../../lexicons'
import { CID } from 'multiformats/cid'
import * as ComAtprotoAdminDefs from './defs'

export interface QueryParams {
dids: string[]
}

export type InputSchema = undefined

export interface OutputSchema {
infos: ComAtprotoAdminDefs.AccountView[]
[k: string]: unknown
}

export interface CallOptions {
headers?: Headers
}

export interface Response {
success: boolean
headers: Headers
data: OutputSchema
}

export function toKnownErr(e: any) {
if (e instanceof XRPCError) {
}
return e
}
23 changes: 10 additions & 13 deletions packages/bsky/src/api/app/bsky/actor/getProfile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,16 @@ import { ModerationService } from '../../../../services/moderation'
export default function (server: Server, ctx: AppContext) {
const getProfile = createPipeline(skeleton, hydration, noRules, presentation)
server.app.bsky.actor.getProfile({
auth: ctx.authOptionalAccessOrRoleVerifier,
auth: ctx.authVerifier.optionalStandardOrRole,
handler: async ({ auth, params, res }) => {
const db = ctx.db.getReplica()
const actorService = ctx.services.actor(db)
const modService = ctx.services.moderation(ctx.db.getPrimary())
const viewer = 'did' in auth.credentials ? auth.credentials.did : null
const canViewTakendownProfile =
auth.credentials.type === 'role' && auth.credentials.triage
const { viewer, canViewTakedowns } = ctx.authVerifier.parseCreds(auth)

const [result, repoRev] = await Promise.allSettled([
getProfile(
{ ...params, viewer, canViewTakendownProfile },
{ ...params, viewer, canViewTakedowns },
{ db, actorService, modService },
),
actorService.getRepoRev(viewer),
Expand All @@ -52,15 +50,14 @@ const skeleton = async (
params: Params,
ctx: Context,
): Promise<SkeletonState> => {
const { actorService, modService } = ctx
const { canViewTakendownProfile } = params
const { actorService } = ctx
const { canViewTakedowns } = params
const actor = await actorService.getActor(params.actor, true)
if (!actor) {
throw new InvalidRequestError('Profile not found')
}
if (!canViewTakendownProfile && softDeleted(actor)) {
const isSuspended = await modService.isSubjectSuspended(actor.did)
if (isSuspended) {
if (!canViewTakedowns && softDeleted(actor)) {
if (actor.takedownRef?.includes('SUSPEND')) {
throw new InvalidRequestError(
'Account has been temporarily suspended',
'AccountTakedown',
Expand All @@ -78,10 +75,10 @@ const skeleton = async (
const hydration = async (state: SkeletonState, ctx: Context) => {
const { actorService } = ctx
const { params, actor } = state
const { viewer, canViewTakendownProfile } = params
const { viewer, canViewTakedowns } = params
const hydration = await actorService.views.profileDetailHydration(
[actor.did],
{ viewer, includeSoftDeleted: canViewTakendownProfile },
{ viewer, includeSoftDeleted: canViewTakedowns },
)
return { ...state, ...hydration }
}
Expand Down Expand Up @@ -110,7 +107,7 @@ type Context = {

type Params = QueryParams & {
viewer: string | null
canViewTakendownProfile: boolean
canViewTakedowns: boolean
}

type SkeletonState = { params: Params; actor: Actor }
Expand Down
4 changes: 2 additions & 2 deletions packages/bsky/src/api/app/bsky/actor/getProfiles.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ import { createPipeline, noRules } from '../../../../pipeline'
export default function (server: Server, ctx: AppContext) {
const getProfile = createPipeline(skeleton, hydration, noRules, presentation)
server.app.bsky.actor.getProfiles({
auth: ctx.authOptionalVerifier,
auth: ctx.authVerifier.standardOptional,
handler: async ({ auth, params, res }) => {
const db = ctx.db.getReplica()
const actorService = ctx.services.actor(db)
const viewer = auth.credentials.did
const viewer = auth.credentials.iss
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering in the back of my head if we should be looking at sub 🤔 I suppose it's known/implied that for this kind of credential iss and sub are pretty much the same, but I could imagine an implementation looking to set iss to the source pds.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any case, it's not super specific to this work. We can reconsider later, just didn't want to lose the thought.


const [result, repoRev] = await Promise.all([
getProfile({ ...params, viewer }, { db, actorService }),
Expand Down
Loading