Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evented architecture for moderation system #1617

Merged
merged 98 commits into from
Nov 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
5ea9064
:construction: WIP with proposed lexicons for event based mod archite…
foysalit Sep 16, 2023
5445a6b
:construction: Remove unnecessary moderation action lexicon
foysalit Sep 16, 2023
2b5801c
Merge branch 'main' of github.com:bluesky-social/atproto into moderat…
foysalit Sep 20, 2023
b62a397
:construction: Working on event based actions
foysalit Sep 26, 2023
8781c87
:rightwards_twisted_arrows: Merge with upstream
foysalit Sep 27, 2023
39a81fd
:sparkles: Add escalated subject status
foysalit Sep 28, 2023
1637548
Merge branch 'main' of github.com:bluesky-social/atproto into moderat…
foysalit Sep 28, 2023
e1e09b3
:bug: Alright, fixed the error in lexicon
foysalit Sep 28, 2023
619eed5
:construction: Working through reversal
foysalit Sep 28, 2023
c8c1a55
:sparkles: Cleanup build errors
foysalit Sep 29, 2023
dfc4fec
:sparkles: Add subject status endpoint
foysalit Sep 29, 2023
75625a7
:sparkles: Add handler
foysalit Oct 1, 2023
a82d3d6
:sparkles: get reports from mod actions table
foysalit Oct 3, 2023
f734703
:rightwards_twisted_arrows: Merge with upstream
foysalit Oct 3, 2023
2cc6f1d
:rightwards_twisted_arrows: Merge with upstream
foysalit Oct 3, 2023
f8b3b78
:construction: Builds but test network doesnt start
foysalit Oct 5, 2023
84d79a8
:sparkles: Tests passing on event based status change
foysalit Oct 6, 2023
6b652ef
:sparkles: Rename index
foysalit Oct 9, 2023
cc87dcb
:recycle: Rename takeModerationAction->emitModerationEvent
foysalit Oct 9, 2023
10ee0e5
:sparkles: Implement label reversal
foysalit Oct 9, 2023
dacbb3f
:white_check_mark: Auto-revert test working
foysalit Oct 10, 2023
5a9bdd9
:recycle: :white_check_mark: Refactored to event types and tests are …
foysalit Oct 11, 2023
ddce910
:sparkles: Add takedown event sequence validation
foysalit Oct 11, 2023
164f391
:sparkles: Adds support for blobCid status
foysalit Oct 13, 2023
82870ec
:broom: Cleanup unnecessary method:
foysalit Oct 13, 2023
87364a8
:sparkles: Hydrate handles with status and events
foysalit Oct 20, 2023
79f4c97
:sparkles: Re-implement auto reversal
foysalit Oct 23, 2023
a07f64f
:sparkles: Add takendown and mute filters
foysalit Oct 24, 2023
d71b910
:sparkles: Allow filtering events by type
foysalit Oct 26, 2023
aea0b55
:sparkles: Allow filtering events by creator did
foysalit Oct 26, 2023
afb9a8b
:sparkles: Add subjectStatus to record and repoview
foysalit Oct 30, 2023
cc756ad
:sparkles: Add persistent note feature
foysalit Oct 31, 2023
f5204b0
:sparkles: Log send email event
foysalit Nov 1, 2023
23cea8e
:bug: Fix logging send email event
foysalit Nov 2, 2023
85e8322
:sparkles: Better type
foysalit Nov 2, 2023
3b0e41c
:sparkles: Adjust migration to create separate moderation_event table
foysalit Nov 2, 2023
3b3a1a9
:broom: Cleanup types
foysalit Nov 2, 2023
1ca8427
:white_check_mark: Adjust tests with mod event emitter
foysalit Nov 2, 2023
c442738
:rightwards_twisted_arrows: Merge with upstream
foysalit Nov 3, 2023
9973b8e
:sparkles: Fix more tests around takedowns
foysalit Nov 3, 2023
62ea6df
:white_check_mark: Get test suite to pass
foysalit Nov 3, 2023
efc4eaa
:white_check_mark: Get test suite to pass for pds
foysalit Nov 3, 2023
eb766bd
:white_check_mark: Get test suite to pass for pds
foysalit Nov 3, 2023
bda203d
:white_check_mark: Update snapshot for feedgen
foysalit Nov 3, 2023
aa9ac28
:white_check_mark: Why are more snapshots updating?
foysalit Nov 3, 2023
226b87e
:recycle: Rename getModerationEvents -> queryModerationEvents
foysalit Nov 3, 2023
8c1f026
:recycle: Rename getModerationStatuses -> queryModerationStatuses
foysalit Nov 3, 2023
1ca527b
:recycle: Rename persistNote->sticky
foysalit Nov 3, 2023
be425b9
:bug: Rename subject
foysalit Nov 3, 2023
677e2e8
:recycle: Cleanup expiresAt for scheduled actions
foysalit Nov 3, 2023
902e6b4
:sparkles: Add more tests, allow fetching mod history for all content…
foysalit Nov 8, 2023
96395e6
:white_check_mark: Fix repo and record tests
foysalit Nov 8, 2023
49f381f
:sparkles: Migrate reports and actions to events
foysalit Nov 9, 2023
39cf166
:bug: Fix escalated status overwrite
foysalit Nov 10, 2023
1bf5acc
:sparkles: Implement direct sql query to create events from actions a…
foysalit Nov 11, 2023
143e569
:construction: Adding keyset pagination for subject statuses
foysalit Nov 11, 2023
d220257
:sparkles: Add migration for lastReportedAt
foysalit Nov 12, 2023
6e76210
:sparkles: Migrate blob cids
foysalit Nov 12, 2023
dc57dc8
:twisted_rightwards_arrows: Merge with upstream
foysalit Nov 13, 2023
006e0dc
:sparkles: Fix pagination on mod subject list endpoint
foysalit Nov 14, 2023
2792f9b
:bug: Fix blob actions
foysalit Nov 15, 2023
d44b683
:white_check_mark: All tests passing on bsky package
foysalit Nov 15, 2023
6c034bd
:white_check_mark: Bring back snapshots
foysalit Nov 15, 2023
741aef4
:white_check_mark: Skipping timeline test temporarily
foysalit Nov 15, 2023
f8a9bbe
:white_check_mark: Skipping some more tests to isolate failing ones
foysalit Nov 15, 2023
554af17
Merge branch 'main' of github.com:bluesky-social/atproto into moderat…
foysalit Nov 15, 2023
56f3a61
:white_check_mark: Bring back list-feed test
foysalit Nov 15, 2023
90c8241
:white_check_mark: Bring back timeline test
foysalit Nov 15, 2023
e4bd6d6
:white_check_mark: Fix label action in seeding
foysalit Nov 15, 2023
9f11c6c
:white_check_mark: Enable timeline proxied test
foysalit Nov 15, 2023
2617b4d
:white_check_mark: Enable search actor proxied test
foysalit Nov 15, 2023
c25168f
:white_check_mark: Enable feedgen tests
foysalit Nov 16, 2023
bcb475e
:white_check_mark: Fix test for admin/get-record
foysalit Nov 16, 2023
79b0473
:sparkles: Move note to comment for subject status
foysalit Nov 17, 2023
954b523
:sparkles: Accept comments in mute event
foysalit Nov 17, 2023
e6e8416
:sparkles: Remap flag event to ack event
foysalit Nov 20, 2023
a608eb6
:bug: Add legacyRef in report union selection
foysalit Nov 20, 2023
82fb94f
Merge remote-tracking branch 'origin/main' into moderation-api-v2
devinivy Nov 20, 2023
31d1bb2
Merge branch 'moderation-api-v2' of github.com:bluesky-social/atproto…
devinivy Nov 20, 2023
a286b84
@atproto/api 0.6.24-next.0
devinivy Nov 21, 2023
a93a162
@atproto/api 0.6.24-next.1
devinivy Nov 21, 2023
22bd5f0
:sparkles: Adjust migration export and add index for blobCids column
foysalit Nov 21, 2023
c3d0093
Merge branch 'moderation-api-v2' of github.com:bluesky-social/atproto…
foysalit Nov 21, 2023
0d1eb3f
:sparkles: Maintin action ids when migrating
foysalit Nov 21, 2023
404a42f
:sparkles: Paginate events using createdAt timestamp
foysalit Nov 21, 2023
2d1f542
:white_check_mark: Update snapshot for pds test with events cursor up…
foysalit Nov 21, 2023
91d32d4
:white_check_mark: Use only events for snapshot testing
foysalit Nov 21, 2023
006372f
:white_check_mark: Use only events for snapshot in the remaining test
foysalit Nov 21, 2023
a2e6ea6
relative paths to lexicons for build
devinivy Nov 22, 2023
d21b3d9
fix bsky periodic event reversal in service entrypoint
devinivy Nov 22, 2023
0697ffc
:sparkles: Allow comments in takedown and label
foysalit Nov 22, 2023
99aef0a
:sparkles: Only import reports on consecutive run of the migration sc…
foysalit Nov 28, 2023
655fe09
:sparkles: Adjust moderation property of blob entries
foysalit Nov 28, 2023
d69bb10
determine latest reports to migrate
devinivy Nov 29, 2023
48253a8
:sparkles: Process new reports for subject status
foysalit Nov 29, 2023
a8b95bf
:sparkles: Process unresolved reports on first migration run
foysalit Nov 29, 2023
c75131f
fix transaction error, process just unresolved reports, make reported…
devinivy Nov 30, 2023
9eebb90
tidy
devinivy Nov 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
319 changes: 234 additions & 85 deletions lexicons/com/atproto/admin/defs.json

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"lexicon": 1,
"id": "com.atproto.admin.takeModerationAction",
"id": "com.atproto.admin.emitModerationEvent",
"defs": {
"main": {
"type": "procedure",
Expand All @@ -9,14 +9,21 @@
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["action", "subject", "reason", "createdBy"],
"required": ["event", "subject", "createdBy"],
"properties": {
"action": {
"type": "string",
"knownValues": [
"com.atproto.admin.defs#takedown",
"com.atproto.admin.defs#flag",
"com.atproto.admin.defs#acknowledge"
"event": {
"type": "union",
"refs": [
"com.atproto.admin.defs#modEventTakedown",
"com.atproto.admin.defs#modEventAcknowledge",
"com.atproto.admin.defs#modEventEscalate",
"com.atproto.admin.defs#modEventComment",
"com.atproto.admin.defs#modEventLabel",
"com.atproto.admin.defs#modEventReport",
"com.atproto.admin.defs#modEventMute",
"com.atproto.admin.defs#modEventReverseTakedown",
"com.atproto.admin.defs#modEventUnmute",
"com.atproto.admin.defs#modEventEmail"
]
},
"subject": {
Copy link
Collaborator

@devinivy devinivy Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed here that blobs (subjectBlobCids) are handled separately from subject. One idea that came-up which I quite liked is to make it more uniform, i.e. making a blob within a user's repository another kind of subject. CC @dholms since it's dovetails with the work he did with e.g. com.atproto.admin.updateSubjectStatus where the subjects look like this:

  "subject": {
    "type": "union",
    "refs": [
      "com.atproto.admin.defs#repoRef",
      "com.atproto.repo.strongRef",
      "com.atproto.admin.defs#repoBlobRef"
    ]
  }

Expand All @@ -30,19 +37,6 @@
"type": "array",
"items": { "type": "string", "format": "cid" }
},
"createLabelVals": {
"type": "array",
"items": { "type": "string" }
},
"negateLabelVals": {
"type": "array",
"items": { "type": "string" }
},
"reason": { "type": "string" },
"durationInHours": {
"type": "integer",
"description": "Indicates how long this action is meant to be in effect before automatically expiring."
},
"createdBy": { "type": "string", "format": "did" }
}
}
Expand All @@ -51,7 +45,7 @@
"encoding": "application/json",
"schema": {
"type": "ref",
"ref": "com.atproto.admin.defs#actionView"
"ref": "com.atproto.admin.defs#modEventView"
}
},
"errors": [{ "name": "SubjectHasAction" }]
Expand Down
40 changes: 0 additions & 40 deletions lexicons/com/atproto/admin/getModerationActions.json

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"lexicon": 1,
"id": "com.atproto.admin.getModerationAction",
"id": "com.atproto.admin.getModerationEvent",
"defs": {
"main": {
"type": "query",
"description": "Get details about a moderation action.",
"description": "Get details about a moderation event.",
"parameters": {
"type": "params",
"required": ["id"],
Expand All @@ -16,7 +16,7 @@
"encoding": "application/json",
"schema": {
"type": "ref",
"ref": "com.atproto.admin.defs#actionViewDetail"
"ref": "com.atproto.admin.defs#modEventViewDetail"
}
}
}
Expand Down
24 changes: 0 additions & 24 deletions lexicons/com/atproto/admin/getModerationReport.json

This file was deleted.

65 changes: 0 additions & 65 deletions lexicons/com/atproto/admin/getModerationReports.json

This file was deleted.

60 changes: 60 additions & 0 deletions lexicons/com/atproto/admin/queryModerationEvents.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"lexicon": 1,
"id": "com.atproto.admin.queryModerationEvents",
"defs": {
"main": {
"type": "query",
"description": "List moderation events related to a subject.",
"parameters": {
"type": "params",
"properties": {
"types": {
"type": "array",
"items": { "type": "string" },
"description": "The types of events (fully qualified string in the format of com.atproto.admin#modEvent<name>) to filter by. If not specified, all events are returned."
},
"createdBy": {
"type": "string",
"format": "did"
},
"sortDirection": {
"type": "string",
"default": "desc",
"enum": ["asc", "desc"],
"description": "Sort direction for the events. Defaults to descending order of created at timestamp."
},
"subject": { "type": "string", "format": "uri" },
"includeAllUserRecords": {
"type": "boolean",
"default": false,
"description": "If true, events on all record types (posts, lists, profile etc.) owned by the did are returned"
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 100,
"default": 50
},
"cursor": { "type": "string" }
}
},
"output": {
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["events"],
"properties": {
"cursor": { "type": "string" },
"events": {
"type": "array",
"items": {
"type": "ref",
"ref": "com.atproto.admin.defs#modEventView"
}
}
}
}
}
}
}
}
95 changes: 95 additions & 0 deletions lexicons/com/atproto/admin/queryModerationStatuses.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
{
"lexicon": 1,
"id": "com.atproto.admin.queryModerationStatuses",
"defs": {
"main": {
"type": "query",
"description": "View moderation statuses of subjects (record or repo).",
"parameters": {
"type": "params",
"properties": {
"subject": { "type": "string", "format": "uri" },
"comment": {
"type": "string",
"description": "Search subjects by keyword from comments"
},
"reportedAfter": {
"type": "string",
"format": "datetime",
"description": "Search subjects reported after a given timestamp"
},
"reportedBefore": {
"type": "string",
"format": "datetime",
"description": "Search subjects reported before a given timestamp"
},
"reviewedAfter": {
"type": "string",
"format": "datetime",
"description": "Search subjects reviewed after a given timestamp"
},
"reviewedBefore": {
"type": "string",
"format": "datetime",
"description": "Search subjects reviewed before a given timestamp"
},
"includeMuted": {
"type": "boolean",
"description": "By default, we don't include muted subjects in the results. Set this to true to include them."
},
"reviewState": {
"type": "string",
"description": "Specify when fetching subjects in a certain state"
},
"ignoreSubjects": {
"type": "array",
"items": { "type": "string", "format": "uri" }
},
"lastReviewedBy": {
"type": "string",
"format": "did",
"description": "Get all subject statuses that were reviewed by a specific moderator"
},
"sortField": {
"type": "string",
"default": "lastReportedAt",
"enum": ["lastReviewedAt", "lastReportedAt"]
},
"sortDirection": {
"type": "string",
"default": "desc",
"enum": ["asc", "desc"]
},
"takendown": {
"type": "boolean",
"description": "Get subjects that were taken down"
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 100,
"default": 50
},
"cursor": { "type": "string" }
}
},
"output": {
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["subjectStatuses"],
"properties": {
"cursor": { "type": "string" },
"subjectStatuses": {
"type": "array",
"items": {
"type": "ref",
"ref": "com.atproto.admin.defs#subjectStatusView"
}
}
}
}
}
}
}
}
Loading