Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add modEventDivertBlobs event to send blobs to abyss #2238

Merged
merged 32 commits into from
Mar 12, 2024
Merged

✨ Add modEventDivertBlobs event to send blobs to abyss #2238

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8c62af0
:construction: Working through an nullable review state
foysalit Feb 25, 2024
c05f2ce
Merge branch 'main' of github.com:bluesky-social/atproto into nullabl…
foysalit Feb 27, 2024
ccc8ab9
:white_check_mark: Update snapshots on some tests
foysalit Feb 27, 2024
d7f6538
:white_check_mark: Update snapshots on some tests
foysalit Feb 27, 2024
93ac0e4
:white_check_mark: Add test for reviewOptional status mutation
foysalit Feb 27, 2024
1b92941
:twisted_rightwards_arrows: Merge with upstream
foysalit Feb 27, 2024
ddd9cd5
:sparkles: Add divertBlobs event to send blobs to abyss
foysalit Feb 29, 2024
c5f2c10
:recycle: Rename reviewOptional -> reviewNone
foysalit Feb 29, 2024
4b546e9
:recycle: Rename modEventDivertBlobs -> modEventDivert
foysalit Feb 29, 2024
64dd7ac
:bug: Rename event type checker
foysalit Feb 29, 2024
9851565
Merge branch 'main' of github.com:bluesky-social/atproto into nullabl…
foysalit Feb 29, 2024
47b5d41
:twisted_rightwards_arrows: Merge with upstream
foysalit Feb 29, 2024
071712c
:sparkles: Use pds resolver to get blob straight from pds
foysalit Feb 29, 2024
5c0e20a
:white_check_mark: Use FOR UPDATE to respect db transactions
foysalit Mar 4, 2024
be01cf0
:recycle: Refactor to use event_pusher table instead of new table
foysalit Mar 6, 2024
9be9921
:twisted_rightwards_arrows: Merge with upstream
foysalit Mar 6, 2024
f04c7b6
Merge branch 'main' of github.com:bluesky-social/atproto into nullabl…
foysalit Mar 6, 2024
be498d7
Merge branch 'nullable-review-state' of github.com:bluesky-social/atp…
foysalit Mar 6, 2024
54183d8
:sparkles: Bring back missing lines in pnpm-lock
foysalit Mar 6, 2024
09f8137
:hammer: Rebuild?
foysalit Mar 6, 2024
2bd9947
:rotating_light: Formatting
foysalit Mar 6, 2024
f52cccd
Merge branch 'main' of github.com:bluesky-social/atproto into nullabl…
foysalit Mar 6, 2024
2d86f52
:twisted_rightwards_arrows: Merge with upstream
foysalit Mar 6, 2024
96e2870
:recycle: Refactor to divert blob sync
foysalit Mar 7, 2024
104ae95
Merge branch 'main' of github.com:bluesky-social/atproto into divert-…
foysalit Mar 7, 2024
93fdbc0
:broom: Cleanup
foysalit Mar 8, 2024
83ce64e
:twisted_rightwards_arrows: Merge with upstream
foysalit Mar 8, 2024
c7cff52
:white_check_mark: Use modClient seed client in blob-divert test
foysalit Mar 8, 2024
641fcd6
Merge remote-tracking branch 'origin/main' into divert-blobs
devinivy Mar 11, 2024
f2ac002
update divert blob config to use basic admin auth
devinivy Mar 11, 2024
0e4c8c8
fix
devinivy Mar 11, 2024
fefd23e
build
devinivy Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions lexicons/com/atproto/admin/defs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@
"#modEventEscalate",
"#modEventMute",
"#modEventEmail",
"#modEventResolveAppeal"
"#modEventResolveAppeal",
"#modEventDivert"
]
},
"subject": {
Expand Down Expand Up @@ -72,7 +73,8 @@
"#modEventEscalate",
"#modEventMute",
"#modEventEmail",
"#modEventResolveAppeal"
"#modEventResolveAppeal",
"#modEventDivert"
]
},
"subject": {
Expand Down Expand Up @@ -449,7 +451,12 @@
},
"subjectReviewState": {
"type": "string",
"knownValues": ["#reviewOpen", "#reviewEscalated", "#reviewClosed"]
"knownValues": [
"#reviewOpen",
"#reviewEscalated",
"#reviewClosed",
"#reviewNone"
]
},
"reviewOpen": {
"type": "token",
Expand All @@ -463,6 +470,10 @@
"type": "token",
"description": "Moderator review status of a subject: Closed. Indicates that the subject was already reviewed and resolved by a moderator"
},
"reviewNone": {
"type": "token",
"description": "Moderator review status of a subject: Unnecessary. Indicates that the subject does not need a review at the moment but there is probably some moderation related metadata available for it"
},
"modEventTakedown": {
"type": "object",
"description": "Take down a subject permanently or temporarily",
Expand Down Expand Up @@ -612,6 +623,13 @@
}
}
},
"modEventDivert": {
"type": "object",
"description": "Divert a record's blobs to a 3rd party service for further scanning/tagging",
"properties": {
"comment": { "type": "string" }
}
},
"communicationTemplateView": {
"type": "object",
"required": [
Expand Down
1 change: 1 addition & 0 deletions packages/api/src/client/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,7 @@ export const COM_ATPROTO_ADMIN = {
DefsReviewOpen: 'com.atproto.admin.defs#reviewOpen',
DefsReviewEscalated: 'com.atproto.admin.defs#reviewEscalated',
DefsReviewClosed: 'com.atproto.admin.defs#reviewClosed',
DefsReviewNone: 'com.atproto.admin.defs#reviewNone',
}
export const COM_ATPROTO_MODERATION = {
DefsReasonSpam: 'com.atproto.moderation.defs#reasonSpam',
Expand Down
18 changes: 18 additions & 0 deletions packages/api/src/client/lexicons.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ export const schemaDict = {
'lex:com.atproto.admin.defs#modEventMute',
'lex:com.atproto.admin.defs#modEventEmail',
'lex:com.atproto.admin.defs#modEventResolveAppeal',
'lex:com.atproto.admin.defs#modEventDivert',
],
},
subject: {
Expand Down Expand Up @@ -150,6 +151,7 @@ export const schemaDict = {
'lex:com.atproto.admin.defs#modEventMute',
'lex:com.atproto.admin.defs#modEventEmail',
'lex:com.atproto.admin.defs#modEventResolveAppeal',
'lex:com.atproto.admin.defs#modEventDivert',
],
},
subject: {
Expand Down Expand Up @@ -747,6 +749,7 @@ export const schemaDict = {
'lex:com.atproto.admin.defs#reviewOpen',
'lex:com.atproto.admin.defs#reviewEscalated',
'lex:com.atproto.admin.defs#reviewClosed',
'lex:com.atproto.admin.defs#reviewNone',
],
},
reviewOpen: {
Expand All @@ -764,6 +767,11 @@ export const schemaDict = {
description:
'Moderator review status of a subject: Closed. Indicates that the subject was already reviewed and resolved by a moderator',
},
reviewNone: {
type: 'token',
description:
'Moderator review status of a subject: Unnecessary. Indicates that the subject does not need a review at the moment but there is probably some moderation related metadata available for it',
},
modEventTakedown: {
type: 'object',
description: 'Take down a subject permanently or temporarily',
Expand Down Expand Up @@ -930,6 +938,16 @@ export const schemaDict = {
},
},
},
modEventDivert: {
type: 'object',
description:
"Divert a record's blobs to a 3rd party service for further scanning/tagging",
properties: {
comment: {
type: 'string',
},
},
},
communicationTemplateView: {
type: 'object',
required: [
Expand Down
23 changes: 23 additions & 0 deletions packages/api/src/client/types/com/atproto/admin/defs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ export interface ModEventView {
| ModEventMute
| ModEventEmail
| ModEventResolveAppeal
| ModEventDivert
| { $type: string; [k: string]: unknown }
subject:
| RepoRef
Expand Down Expand Up @@ -79,6 +80,7 @@ export interface ModEventViewDetail {
| ModEventMute
| ModEventEmail
| ModEventResolveAppeal
| ModEventDivert
| { $type: string; [k: string]: unknown }
subject:
| RepoView
Expand Down Expand Up @@ -497,6 +499,7 @@ export type SubjectReviewState =
| 'lex:com.atproto.admin.defs#reviewOpen'
| 'lex:com.atproto.admin.defs#reviewEscalated'
| 'lex:com.atproto.admin.defs#reviewClosed'
| 'lex:com.atproto.admin.defs#reviewNone'
| (string & {})

/** Moderator review status of a subject: Open. Indicates that the subject needs to be reviewed by a moderator */
Expand All @@ -505,6 +508,8 @@ export const REVIEWOPEN = 'com.atproto.admin.defs#reviewOpen'
export const REVIEWESCALATED = 'com.atproto.admin.defs#reviewEscalated'
/** Moderator review status of a subject: Closed. Indicates that the subject was already reviewed and resolved by a moderator */
export const REVIEWCLOSED = 'com.atproto.admin.defs#reviewClosed'
/** Moderator review status of a subject: Unnecessary. Indicates that the subject does not need a review at the moment but there is probably some moderation related metadata available for it */
export const REVIEWNONE = 'com.atproto.admin.defs#reviewNone'

/** Take down a subject permanently or temporarily */
export interface ModEventTakedown {
Expand Down Expand Up @@ -744,6 +749,24 @@ export function validateModEventTag(v: unknown): ValidationResult {
return lexicons.validate('com.atproto.admin.defs#modEventTag', v)
}

/** Divert a record's blobs to a 3rd party service for further scanning/tagging */
export interface ModEventDivert {
comment?: string
[k: string]: unknown
}

export function isModEventDivert(v: unknown): v is ModEventDivert {
return (
isObj(v) &&
hasProp(v, '$type') &&
v.$type === 'com.atproto.admin.defs#modEventDivert'
)
}

export function validateModEventDivert(v: unknown): ValidationResult {
return lexicons.validate('com.atproto.admin.defs#modEventDivert', v)
}

export interface CommunicationTemplateView {
id: string
/** Name of the template. */
Expand Down
1 change: 1 addition & 0 deletions packages/bsky/src/lexicon/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ export const COM_ATPROTO_ADMIN = {
DefsReviewOpen: 'com.atproto.admin.defs#reviewOpen',
DefsReviewEscalated: 'com.atproto.admin.defs#reviewEscalated',
DefsReviewClosed: 'com.atproto.admin.defs#reviewClosed',
DefsReviewNone: 'com.atproto.admin.defs#reviewNone',
}
export const COM_ATPROTO_MODERATION = {
DefsReasonSpam: 'com.atproto.moderation.defs#reasonSpam',
Expand Down
18 changes: 18 additions & 0 deletions packages/bsky/src/lexicon/lexicons.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ export const schemaDict = {
'lex:com.atproto.admin.defs#modEventMute',
'lex:com.atproto.admin.defs#modEventEmail',
'lex:com.atproto.admin.defs#modEventResolveAppeal',
'lex:com.atproto.admin.defs#modEventDivert',
],
},
subject: {
Expand Down Expand Up @@ -150,6 +151,7 @@ export const schemaDict = {
'lex:com.atproto.admin.defs#modEventMute',
'lex:com.atproto.admin.defs#modEventEmail',
'lex:com.atproto.admin.defs#modEventResolveAppeal',
'lex:com.atproto.admin.defs#modEventDivert',
],
},
subject: {
Expand Down Expand Up @@ -747,6 +749,7 @@ export const schemaDict = {
'lex:com.atproto.admin.defs#reviewOpen',
'lex:com.atproto.admin.defs#reviewEscalated',
'lex:com.atproto.admin.defs#reviewClosed',
'lex:com.atproto.admin.defs#reviewNone',
],
},
reviewOpen: {
Expand All @@ -764,6 +767,11 @@ export const schemaDict = {
description:
'Moderator review status of a subject: Closed. Indicates that the subject was already reviewed and resolved by a moderator',
},
reviewNone: {
type: 'token',
description:
'Moderator review status of a subject: Unnecessary. Indicates that the subject does not need a review at the moment but there is probably some moderation related metadata available for it',
},
modEventTakedown: {
type: 'object',
description: 'Take down a subject permanently or temporarily',
Expand Down Expand Up @@ -930,6 +938,16 @@ export const schemaDict = {
},
},
},
modEventDivert: {
type: 'object',
description:
"Divert a record's blobs to a 3rd party service for further scanning/tagging",
properties: {
comment: {
type: 'string',
},
},
},
communicationTemplateView: {
type: 'object',
required: [
Expand Down
23 changes: 23 additions & 0 deletions packages/bsky/src/lexicon/types/com/atproto/admin/defs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ export interface ModEventView {
| ModEventMute
| ModEventEmail
| ModEventResolveAppeal
| ModEventDivert
| { $type: string; [k: string]: unknown }
subject:
| RepoRef
Expand Down Expand Up @@ -79,6 +80,7 @@ export interface ModEventViewDetail {
| ModEventMute
| ModEventEmail
| ModEventResolveAppeal
| ModEventDivert
| { $type: string; [k: string]: unknown }
subject:
| RepoView
Expand Down Expand Up @@ -497,6 +499,7 @@ export type SubjectReviewState =
| 'lex:com.atproto.admin.defs#reviewOpen'
| 'lex:com.atproto.admin.defs#reviewEscalated'
| 'lex:com.atproto.admin.defs#reviewClosed'
| 'lex:com.atproto.admin.defs#reviewNone'
| (string & {})

/** Moderator review status of a subject: Open. Indicates that the subject needs to be reviewed by a moderator */
Expand All @@ -505,6 +508,8 @@ export const REVIEWOPEN = 'com.atproto.admin.defs#reviewOpen'
export const REVIEWESCALATED = 'com.atproto.admin.defs#reviewEscalated'
/** Moderator review status of a subject: Closed. Indicates that the subject was already reviewed and resolved by a moderator */
export const REVIEWCLOSED = 'com.atproto.admin.defs#reviewClosed'
/** Moderator review status of a subject: Unnecessary. Indicates that the subject does not need a review at the moment but there is probably some moderation related metadata available for it */
export const REVIEWNONE = 'com.atproto.admin.defs#reviewNone'

/** Take down a subject permanently or temporarily */
export interface ModEventTakedown {
Expand Down Expand Up @@ -744,6 +749,24 @@ export function validateModEventTag(v: unknown): ValidationResult {
return lexicons.validate('com.atproto.admin.defs#modEventTag', v)
}

/** Divert a record's blobs to a 3rd party service for further scanning/tagging */
export interface ModEventDivert {
comment?: string
[k: string]: unknown
}

export function isModEventDivert(v: unknown): v is ModEventDivert {
return (
isObj(v) &&
hasProp(v, '$type') &&
v.$type === 'com.atproto.admin.defs#modEventDivert'
)
}

export function validateModEventDivert(v: unknown): ValidationResult {
return lexicons.validate('com.atproto.admin.defs#modEventDivert', v)
}

export interface CommunicationTemplateView {
id: string
/** Name of the template. */
Expand Down
3 changes: 2 additions & 1 deletion packages/ozone/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
"@types/express-serve-static-core": "^4.17.36",
"@types/pg": "^8.6.6",
"@types/qs": "^6.9.7",
"axios": "^0.27.2"
"axios": "^0.27.2",
"nock": "14.0.0-beta.4"
}
}
12 changes: 12 additions & 0 deletions packages/ozone/src/config/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,18 @@ export const envToCfg = (env: OzoneEnvironment): OzoneConfig => {
plcUrl: env.didPlcUrl,
}

const blobReportServiceCfg = {
url: env.blobReportServiceUrl,
authToken: env.blobReportServiceAuthToken,
}

return {
service: serviceCfg,
db: dbCfg,
appview: appviewCfg,
pds: pdsCfg,
identity: identityCfg,
blobReportService: blobReportServiceCfg,
}
}

Expand All @@ -58,6 +64,7 @@ export type OzoneConfig = {
appview: AppviewConfig
pds: PdsConfig | null
identity: IdentityConfig
blobReportService: BlobReportServiceConfig
}

export type ServiceConfig = {
Expand All @@ -67,6 +74,11 @@ export type ServiceConfig = {
version?: string
}

export type BlobReportServiceConfig = {
url?: string
authToken?: string
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like both url and authToken are required in order to use the blob report service. Could we make these both required here, and update OzoneConfig to allow it not to be configured:

export type OzoneConfig = {
  // ...
  blobReportService: BlobReportServiceConfig | null
}


export type DatabaseConfig = {
postgresUrl: string
postgresSchema?: string
Expand Down
4 changes: 4 additions & 0 deletions packages/ozone/src/config/env.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ export const readEnv = (): OzoneEnvironment => {
moderatorPassword: envStr('OZONE_MODERATOR_PASSWORD'),
triagePassword: envStr('OZONE_TRIAGE_PASSWORD'),
signingKeyHex: envStr('OZONE_SIGNING_KEY_HEX'),
blobReportServiceUrl: envStr('OZONE_BLOB_REPORT_SERVICE_URL'),
blobReportServiceAuthToken: envStr('OZONE_BLOB_REPORT_SERVICE_AUTH_TOKEN'),
}
}

Expand All @@ -44,4 +46,6 @@ export type OzoneEnvironment = {
moderatorPassword?: string
triagePassword?: string
signingKeyHex?: string
blobReportServiceUrl?: string
blobReportServiceAuthToken?: string
}
Loading
Loading