Developer sandbox ideas

[rishighan]

Hello,

In testing the Metron API, and referring back to the issue I opened in regards to a POST endpoint to write to Metron I thought about a couple of things:

1. Who should be allowed to write to the API?
2. If my app (ThreeTwo) allows writing to Metron in a trivial manner, are there rate-limiting and/or scalability issues that Metron might run into? To aim for ThreeTwo to be an auxiliary frontend for Metron is tall order, but it is worthwhile since the long-term goal is to get Metron on par with Comic Vine. The easier I can make it for users to write to Metron, the more approachable the goal becomes.
3. Following up on point 1, do you have ideas on how moderation is performed? I think that if we are allowing users to edit/write to Metron, it needs to get into a moderation queue, and then human intervention would be needed to get the edits published.
4. For a developer, such as myself, do you think you can give me like a sandbox where I can test the POST requests? (you can implement most endpoints in the sandbox with strict rate limits to prevent abuse)

Let's get the conversation started!

[bpepple]

1. Who should be allowed to write to the API?

I'm still looking into whether it's a good idea to implement POST support (and if you know of any sites that provide that I'd love to get in contact with them), but let's say I do implement it, the dream scenario would be any contributor with an account. The downside with that approach would be it would require a few things like:

1. Really good data validation on Metron's side, since the source could come from many different sources (ThreeTwo, Scripts, etc).
2. Some type of moderation (which brings it's own set of problems).
3. POST rate limits so inexperience or bad actors couldn't flood the site with bad data.

The more realistic approach would be have contributors vetted and then granted permission to use POST. To implement it, it would require a permissions module like this, and then modify each Viewset like this:

class CharacterViewSet(viewsets.ModelViewSet):
    queryset = Character.objects.all()
    filterset_class = NameFilter
    throttle_classes = (UserRateThrottle,)

    def get_serializer_class(self):
        if self.action == "list":
            return CharacterListSerializer
        return CharacterSerializer

    def get_permissions(self):
        permission_classes = []
        # Might be worthwhile to add 'update' or 'partial_update' actions.
        if self.action in ["create"]:
            permission_classes = [IsEditor]
        elif self.action in ["retrieve", "list"]:
            permission_classes = [IsEditorOrContributor]
        return [permission() for permission in permission_classes]

With a permission-based approach the need to moderation or rate limiting is less necessary, since it's assumed they've been vetted based on their knowledge/experience.

Anyway, that's my initial thinking on who's allowed to use the POST api.

[bpepple]

2. If my app (ThreeTwo) allows writing to Metron in a trivial manner, are there rate-limiting and/or scalability issues that Metron might run into? To aim for ThreeTwo to be an auxiliary frontend for Metron is tall order, but it is worthwhile since the long-term goal is to get Metron on par with Comic Vine. The easier I can make it for users to write to Metron, the more approachable the goal becomes.

This slightly goes back to my reply to your first question, any solution would need to take into account any method of using the POST api (be it ThreeTwo, another app, etc), and would most likely require certain level of trust in the user as I pointed out in my previous response. Currently, I throttle API request to 25/minute which seems to be more than fine for now:
https://github.com/bpepple/metron/blob/f164071805b8041650f5b6994ca13e2f68a0ef28/metron/settings.py#L158

Now if the server is getting hammered, I'd probably need to upgrade my server (which is currently only 1GB server at DigitalOcean), and use Django's caching framework, since I'm loath to throttle the API more than it currently is. Anyway, that's something that can looked at down the road.

[bpepple]

3. Following up on point 1, do you have ideas on how moderation is performed? I think that if we are allowing users to edit/write to Metron, it needs to get into a moderation queue, and then human intervention would be needed to get the edits published.

My first instinct is also it needs a moderation, but there are some definite drawbacks:

1. In my experience, moderation becomes a serious bottleneck where it becomes some folks sole job just to review items. I mean we've been battling the whole moderating/review items in the Fedora Project for almost it's whole existence (17+ years).
2. Would increase technical debt quite a bit. Using Django-Moderation might work, tho they don't support m2m relations which would be a problem since they are used quite a bit in Metron.

[bpepple]

4. For a developer, such as myself, do you think you can give me like a sandbox where I can test the POST requests? (you can implement most endpoints in the sandbox with strict rate limits to prevent abuse)

There a couple of ways to go on this:

1. Depending on your knowledge of django/python/postgresql, the easiest way would be run a local version of Metron on your development machine.
2. Run a test instance of Metron. The drawbacks would be cost, admin time for setup/maintenance.
3. Modify production Metron to use some type of sandboxing/moderation.

Obviously, I'm leaning toward option 1, since the other options are fairly time intensive. 😉

[kingfenix]

I like the idea of vetting and maybe via a self handling process? have all contributions (POSTs) require approval before being applied to the db. And once a person/account has say 100 approvals, or whatever number makes you comfortable, then that account gains a preferred status to bypass the approval?

I do agree that ThreeTwo becoming a GUI / front end for Metron will help remove the sole complaint most seem to have about using it, which is not as much as data history as CV, with a solution here then the community itself can help resolve and remove that excuse.

[bpepple]

I agree it's a nice goal, but extremely tough to do without getting tons of garbage data. I can't think of a single website (not just in the comic-book sphere) that provides a POST API, since there are a lot of drawbacks to providing one, but I am willing to look at it to see if it's possible to mitigate the risks.

[bpepple]

And hey, if you are aware of a site that provides a POST API, I'd love to get in touch with them.

[rishighan]

Thanks for the ideas @kingfenix and @bpepple

And hey, if you are aware of a site that provides a POST API, I'd love to get in touch with them.

Off the top of my head, Stack Overflow comes to mind, they do have lots of user-generated content, and I think they have a bunch of POST endpoints. The endpoint for adding an answer to a question is an example.

I agree it's a nice goal, but extremely tough to do without getting tons of garbage data.

I do get your concern about garbage data, spamming and abuse in general. I think that with Metron, at least from a third-party app's perspective, the goal is to get solid, vetted metadata in a seamless way; a goal at which ComicVine only barely passes.

That said, what, according to you @bpepple, is the vision for Metron?

1. Quality of metadata ?
2. A pool of reliable contributors ?
3. A robust API ?

From my perspective, these points aren't mutually exclusive. You need all of the above to varying degrees, and there will be tradeoffs. We need to figure out which tradeoffs to make to get to our stated goal.

---

Regarding the earlier points you made about your concerns with moderation, I think that your suggestion:

The more realistic approach would be have contributors vetted

is good. How does the vetting happen? Can this be achieved programmatically? (in line with @kingfenix's suggestion)
I am all for whatever we need to do to make sure that Metron does not become CV.

In terms of next steps, should be brainstorm the moderation process more?

[bpepple]

I have zero knowledge of python, and you may already know about this, but I found this https://github.com/dominno/django-moderation, so in terms of next steps we can start moving in some direction. (at least discuss the pros and cons)

Yeah, I mentioned it earlier. One big drawback of it is that it doesn't support many2many relationships which is a problem since they are used quite extensively in Metron.

15 years of ComicVine and GCD been around and ComicVine isn't as reliable, has its own issues with long-time contributors leaving (pikahyper), which puts a question mark on the quality and quantity of data. GCD been around forever doesn't change anything for devs such as myself or anyone with casual interest in comics. What good is data, if there is no easy way to consume it?

To be fair to GCD, they've been planning to add an API for ages, but due to the technical debt of converting to Django from their previous software in the 2000's it's been a very slow moving process.

[rishighan]

All valid points @bpepple.

I think I have a proposal for the next steps:

1. I start adding validations to my form within ThreeTwo, and test-writing it to Metron. Do you think you can add a test POST endpoint on a branch, which I can clone and run locally?

2. Moderation process: How about for the first iteration, we vet contributors offline, and only give them write access? That way you are only adding the permission hashes to their credentials, and maybe IP-restricting it.

3. Allow for ThreeTwo users to authenticate with Metron from within ThreeTwo either with OAuth or whatever mechanism you decide, as a first draft.

[bpepple]

1. I start adding validations to my form within ThreeTwo, and test-writing it to Metron. Do you think you can add a test POST endpoint on a branch, which I can clone and run locally?

Yeah, I can create a branch of Metron for work on adding POST endpoints, tho it might take me awhile to get it done since I've got quite a few other projects I'm in the midst of. I might have something ready by next week depending how much free time I have this week. I'll also most likely need to write some kind of wiki instructions for running a local version of Metron, since I'm guessing you haven't done much development with Django.

2. Moderation process: How about for the first iteration, we vet contributors offline, and only give them write access? That way you are only adding the permission hashes to their credentials, and maybe IP-restricting it.

3. Allow for ThreeTwo users to authenticate with Metron from within ThreeTwo either with OAuth or whatever mechanism you decide, as a first draft.

Provided I actually add a POST API, I'll need to make a final decision on the authentication method before rolling it out to production, since I don't want to create a bunch of extra work by having to re-implement the authentication method. For now, this can be tabled until I've got the POST methods implemented in a test branch.

[bpepple]

I started some basic work on this (adding a POST endpoint to the Publisher), and currently doing it in this branch:

https://github.com/bpepple/metron/tree/post-api-%E2%9A%A1

[rishighan]

I started some basic work on this (adding a POST endpoint to the Publisher), and currently doing it in this branch:

https://github.com/bpepple/metron/tree/post-api-%E2%9A%A1

Thank you very much @bpepple, I will get to it!