-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
spike(scoring): Investigate Post Scoring System Limitations and Gaming Vulnerabilities #36
Comments
This comment was marked as resolved.
This comment was marked as resolved.
Verification that a user is an agent on XX has a requirement that all agents on X are marked as an automated account (agent). Read X's terms of service here: https://help.x.com/en/using-x/automated-account-labels Therefore, we can use the X public API to verify that an X profile is indeed an automated account (agent). This is the truncated response body that we can GET from X for X profile: {
"errors": [...],
"data": {
"user": {
"result": {
// Highlighted section
"__typename": "User",
"id": "VXNlcjoxODYyMTAyNjUzNDY5NDMzODU2",
"rest_id": "1862102653469433856",
"affiliates_highlighted_label": {
"label": {
"badge": {
"url": "https://pbs.twimg.com/semantic_core_img/1428827730364096519/4ZXpTBhS?format=png&name=orig"
},
"description": "Automated",
"longDescription": {
"text": "Automated by @CreatorBid",
"entities": [
{
"fromIndex": 13,
"toIndex": 24,
"ref": {
"type": "TimelineRichTextMention",
"screen_name": "CreatorBid",
"mention_results": {
"result": {
"__typename": "User",
"legacy": {
"screen_name": "CreatorBid"
},
"rest_id": "1747737231916240896"
}
}
}
}
]
},
"userLabelType": "AutomatedLabel"
}
},
// Rest of response truncated
"is_blue_verified": true,
"profile_image_shape": "Circle",
"legacy": {...},
...
}
}
}
} We propose adding verification that an X account is automated as an immediate improvement to miner/agent registration and make this an immediate criteria not only for registration to the subnet but implicitly for scoring. Here is a feature ticket for this improvement: #52 |
Community feedback
|
[SPIKE] Comment: LLM-Based Tweet Quality AssessmentAfter investigating the current scoring vulnerabilities outlined in the spike ticket, we propose leveraging Subnet 19's LLM capabilities to create a more sophisticated content quality assessment system. Key Insights
Proposed FeatureSee full feature ticket: [FEATURE] Integrate Subnet 19 LLM for Enhanced Tweet Quality Assessment The feature proposes:
Next Steps
/cc @hide-on-bush-x @theMultitude @grantdfoster @mudler @Luka-Loncar Let's discuss the technical feasibility of this approach - this is low hanging fruit and we can easily spin up some notebooks to do an analysis here on the impact. |
Linking to feature ticket for Subnet 19 integration bonus: #55 This feature will add a scoring bonus for SN59 agents that use Subnet 19 for inference, requiring:
GET "https://api.nineteen.com/v1/metrics/{nineteen-userid}"
Will help drive quality content generation and create stronger subnet synergies. See full ticket for technical details and acceptance criteria. cc @namoray for input on metrics API implementation |
Quick win updates and bug fixes here: #61 |
Also, you can use post interval checking to determine manual posting, it is quite difficult to keep a strict schedule manually |
@ryssroad we will do an analysis here |
We should create follow up tasks on this Spike - syncing with the team on next grooming. |
This results into Scoring v4: #96 |
[SPIKE] Investigate Post Scoring System Limitations and Gaming Vulnerabilities
Problem Statement
The current scoring system (@post_scorer.py) has critical limitations in accurately measuring genuine engagement and content quality. The system's simplistic scoring metrics make it vulnerable to gaming and manipulation, potentially rewarding artificial rather than authentic social media engagement.
Current Issues
Engagement Metric Vulnerabilities
Content Quality Limitations
Interaction Scoring Flaws
Structural Issues
Impact
Evidence if Gaming Happening
Questions to Address
Labels:
scoring
,security
,investigation
,high-priority
The text was updated successfully, but these errors were encountered: